Difference between revisions of "Achievements Considered Harmful?"

m
m
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
<center>''"The intrinsic reward for knifing dudes is knifing dudes."''</center>
<blockquote>''"The intrinsic reward for knifing dudes is knifing dudes."''</blockquote>


I waded into the debate on game achievements with my lecture at the 2010 [[Game Developers Conference]] entitled '''''Achievements Considered Harmful?''''', with a strong emphasis on the "'''''?'''''".  Since the game industry seems to be careening head first into a future of larding points and medals and cute titles on players for just starting up a video game, I wanted to raise awareness of the large body of research studying the impact on motivation from various types of rewards.  Trying to be "fair and balanced", I delved into what the data show and what they don't show.
{|align=right
|__TOC__
|}
I waded into the debate on game achievements with my lecture at the 2010 [[Game Developers Conference]] titled '''''Achievements Considered Harmful?''''', with a strong emphasis on the "'''''?'''''".  Since the game industry seems to be careening head first into a future of larding points and medals and cute titles on players for just starting up a video game, I wanted to raise awareness of the large body of research studying the impact on motivation from various types of rewards.  Trying to be "fair and balanced", I delved into what the data show and what they don't show.


[[Image:schoolyardbrawl.jpg|200px|right]]
[[Image:schoolyardbrawl.jpg|200px|right]]
Sadly, there is a contentious debate amongst psychologists about how rewards affect motivation, and I spend a bunch of time in the talk discussing this debate.  Psychology is at the soft end of science, to put it mildly, and so it's easy for people&mdash;including academics&mdash;to have an agenda or opinion and ''interpret'' the data in a way that backs up that agenda or opinion.  This is human nature, of course, and [http://en.wikipedia.org/wiki/Confirmation_bias confirmation bias] is everywhere in life, but reading some of the papers reminds me more of a schoolyard yelling contest than of peer reviewed research.
Sadly, there is a contentious debate amongst psychologists about how rewards affect motivation, and I spend a bunch of time in the talk discussing this debate.  Psychology is at the soft end of science, to put it mildly, and so it's easy for people&mdash;including academics&mdash;to have an agenda or opinion and "interpret" the data in a way that backs up that agenda or opinion.  This is human nature, of course, and [http://en.wikipedia.org/wiki/Confirmation_bias confirmation bias] is everywhere in life, but reading some of the papers reminds me more of a schoolyard yelling contest than of peer reviewed research.
 
===Highlights===


To hack my way out of the thicket, I focused on the two results that both sides seem to begrudgingly agree are true.
To hack my way out of the thicket, I focused on the two results that both sides seem to begrudgingly agree are true.
Line 16: Line 21:
I define all these terms in the slides below, and I'll fill out this page more as time goes on.
I define all these terms in the slides below, and I'll fill out this page more as time goes on.


The first is the scary one, since it seems to have a lot in common with the ways games reward players with achievements and the like.  I reiterated many times that there are no studies that I'm aware of on achievements for games, but if they really could be are sapping intrinsic motivation to play games&mdash;and I can't see how you can argue with the ''possibility'' this is true after looking at the mountain of available data&mdash;then I think somebody should start funding research into this.  I think [http://research.microsoft.com/ Microsoft Research] would be the perfect people to do this work, since they have both Xbox Live (meaning a great source of data, and a vested interest in figuring out the truth for the longevity of the platform) and a bunch of smart psychologists on staff.  Hopefully they'll heed my call.
The first is the scary one, since it seems to have a lot in common with the ways games reward players with achievements and the like.  I reiterated many times that there are no studies that I'm aware of on achievements for games, but if they really could be sapping intrinsic motivation to play games&mdash;and I can't see how you can argue with the ''possibility'' this is true after looking at the mountain of available data&mdash;then I think somebody should start funding research into this.  I think [http://research.microsoft.com/ Microsoft Research] would be the perfect people to do this work, since they have both Xbox Live (meaning a great source of data, and a vested interest in figuring out the truth for the longevity of the platform) and a bunch of smart psychologists on staff.  Hopefully they'll heed my call.
 
After talking about the results for "interesting tasks", I ask the following question when I pondered covering the results for "dull tasks":
 
<center>'''Why are you making games?'''</center>
<blockquote>
''If you’re intentionally making dull games with variable ratio extrinsic motivators to separate people from their money, you have my pity.''
<br/><br/>
''If you’re making intrinsically interesting games and want to make them even better, be very careful with extrinsic motivators.''
</blockquote>


Towards the end of the talk, I outline a potential '''''Nightmare Scenario''''' based on all the implications of the research going the wrong way for games:
Towards the end of the talk, I outline a potential '''''Nightmare Scenario''''' based on all the implications of the research going the wrong way for games:
Line 23: Line 37:
# use extrinsic motivators to make your game better
# use extrinsic motivators to make your game better
# destroy intrinsic motivation to play your game
# destroy intrinsic motivation to play your game
# ''metrics fetishism'' pushes you towards designs where extrinsic motivation works
# [[Metrics Fetishism|''metrics fetishism'']] pushes you towards designs where extrinsic motivation works
# BONUS: women lose even more intrinsic motivation than men do given extrinsic motivation!
# BONUS: women lose even more intrinsic motivation than men do given extrinsic motivation!
</blockquote>
</blockquote>
Line 54: Line 68:
===Metrics Fetishism===
===Metrics Fetishism===


I'm going to put up a page about this concept at some point, but in the meantime, I want to rant briefly about what I call ''metrics fetishism'', and how's it's taking over game design and how we should be careful.
I had a section here about the trend I have started calling ''metrics fetishism'', but it was long enough that I moved it to [[Metrics Fetishism|its own page]].
 
In the old days, people designed games by their instinct and '''intuition'''.  Notice I didn't say the "good old days", it was just how things were done.  If you wanted feedback, you'd playtest the game, and watch the players struggle with the controls, or waltz through some section you thought was hard, or whatever.  Then you'd iterate.
 
Lately, as computers got networked and more powerful, we started recording '''metrics''' about how players played our games, and feeding that back into the design process.  We dump a lot of information to a database, and then we mine that information for clues about how to optimize certain aspects of our games, whether it's "fun" or "ARPU" or whatever.  This happens during development, and now that a lot of games are online, it happens after we ship, with the design being constantly iterated based on metrics gathered live after launch.
 
Now, metrics and closed feedback loops are great in the abstract, but like any tool, they can be misused.  Just as it's usually a bad idea to design a game based totally on your intuition without grounding that intuition in the cold hard reality of another human being touching the controls, it's also bad to blindly gather and follow metrics data.
 
In the old days, we erred too far on the intuition side of the equation, but now I fear we're erring too far on the metrics side of the equation.  This is what I call ''metrics fetishism''.
 
The problem is not that the data is wrong<ref>Assuming there aren't bugs in the metrics code instrumentating the game code, of course, which is probably not a great assumption, but anyway...</ref>, the problem is that we tend to gather the data that is convenient to gather, '''we worship that data because it is at least some concrete port in the storm of game design and player behavior''', and then we assume we can take a reasonable derivative from that data and hill-climb to a better place in design-space.
 
But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed.  The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space.  If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.
 
'''We really want a mix of intuition and metrics.'''  The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills.  In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space.  The metrics allow you to polish those new points to perfection.
 
<center>[[Image:foothills.jpg]]</center>


===Materials===
===Materials===
Line 82: Line 80:
</center>
</center>


I'll definitely put the raw ppt and mp3 up for download soon as well, but in the meantime I want to see how well this worksIf it works out, I might convert all my other talks to this format as well.
If you're on a phone that won't play Flash, or want to listen to my dulcet tones on your Walkman/iPod/whatever-the-kids-are-using-these-days, or you just want to study my poorly constructed PowerPoint deck, here you go:
* [http://chrishecker.com/html/gdc10-AchievementsConsideredHarmful.mp3 The mp3 audio]
* [[Media:gdc10-AchievementsConsideredHarmful.ppt|The ppt slides]]
 
===Links and Notes===
 
Here's where I put miscellaneous notes and links, both to articles talking about this lecture, and related stuff:
 
* [http://www.destructoid.com/gdc-10-are-achievements-harmful--166646.phtml Anthony Burch at Destructoid wrote about the lecture], as did [http://www.gamasutra.com/view/news/27646/GDC_Heckers_Nightmare_Scenario__A_Future_Of_Rewarding_Players_For_Dull_Tasks.php Chris Remo at Gamasutra]
* Hmm, there are a lot of links, and I haven't had time to go through them all, but you can [http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=achievements+considered+harmful see them here]
* John Teti tried to get (and succeeded in getting) 10,000 gamer points in one day, and lived to tell about it here: [http://www.eurogamer.net/articles/overachiever-for-a-day Overachiever For A Day]
* Soren Johnson wrote up a bunch of notes from the GDC on this and related topics, called [http://www.designer-notes.com/?p=195 Fear and Loathing in Farmville]
* Daniel Carvalho wrote [http://danielcarvalho.com/articles/what-have-we-achieved/ What Have We Achieved]
* Alex Meer at Rock, Paper, Shotgun on TF2 Achievement Servers, [http://www.rockpapershotgun.com/2008/06/21/achieving-nothing/ Pyro Maniacs: Achieving Nothing in TF2]
* There are more, but I haven't had time to go through them yet...
* As I say in the talk, [http://www.alfiekohn.org Alfie Kohn]'s book [http://www.alfiekohn.org/books/pbr.htm Punished By Rewards, The Trouble with Gold Stars, Incentive Plans, A's, Praise, and Other Bribes] was the introduction to this topic for me.  I read the book because I have a child&mdash;Kohn is a respected name in education circles&mdash;and most of his books are about education oriented topicsI think it's an important book, and you should read it, but it comes across as pretty biased, and like most of these popular psychology books, it could be ⅕ the length.  In a similar vein, more recently but with less detail, [http://www.danpink.com/ Daniel Pink] wrote [http://www.danpink.com/drive Drive: The Surprising Truth About What Motivates Us].  It covers some of the same research as Kohn, with a few more recent studies, but then also spends a lot of time on a kind of self-help motivation take on the results, which might annoy you if you're looking for rigor.  Pink also gave [http://www.ted.com/talks/dan_pink_on_motivation.html a talk at TED] about the book.  After reading some of the back and forth between the anti-rewards people and the behaviorists I was introduced to the term [http://dictionary.reference.com/browse/tendentious tendentious], which definitely applies here.
* As an example of the softness of all this research, you can start with the Wikipedia entry on [http://en.wikipedia.org/wiki/Pizza_Hut#Book_It.21 Pizza Hut's Book-It! program].  Book-It! is often mocked for rewarding kids with pizzas for reading books.  As Kohn joked, we should really reward kids with books for eating pizzas, since we want them to eat less pizza and read more books.  The Wikipedia entry cites a paper, [http://findarticles.com/p/articles/mi_hb3538/is_1_49/ai_n28729335/ Effects of extrinsic reinforcement for reading during childhood on reported reading habits of college students, Flora & Flora] that says there was no impact on reading.  However, looking closer, the Flora paper heavily quotes the Cameron meta-analysis, which has been discredited by Deci, which was questioned by Cameron, which was responded to by Deci, and on and on.  As I discuss in the slides, it's really hard to know who or what to believe.  This is why I use the [[#Highlights|'''two conclusions above''']], because they are the only points on which Cameron and Deci seem to agree.


<br/>
<center>[[Image:achievement-unlocked.png]]</center>
<center>[[Image:achievement-unlocked.png]]</center>
<br/>


<references/>
<references/>
Line 90: Line 106:
[[Category:Lectures]]
[[Category:Lectures]]
[[Category:Design]]
[[Category:Design]]
[[Category:Opinions]]

Latest revision as of 17:34, 6 March 2011

"The intrinsic reward for knifing dudes is knifing dudes."

I waded into the debate on game achievements with my lecture at the 2010 Game Developers Conference titled Achievements Considered Harmful?, with a strong emphasis on the "?". Since the game industry seems to be careening head first into a future of larding points and medals and cute titles on players for just starting up a video game, I wanted to raise awareness of the large body of research studying the impact on motivation from various types of rewards. Trying to be "fair and balanced", I delved into what the data show and what they don't show.

Schoolyardbrawl.jpg

Sadly, there is a contentious debate amongst psychologists about how rewards affect motivation, and I spend a bunch of time in the talk discussing this debate. Psychology is at the soft end of science, to put it mildly, and so it's easy for people—including academics—to have an agenda or opinion and "interpret" the data in a way that backs up that agenda or opinion. This is human nature, of course, and confirmation bias is everywhere in life, but reading some of the papers reminds me more of a schoolyard yelling contest than of peer reviewed research.

Highlights

To hack my way out of the thicket, I focused on the two results that both sides seem to begrudgingly agree are true.

For interesting tasks,

  1. Tangible, expected, contingent rewards reduce free-choice intrinsic motivation, and
  2. Verbal, unexpected, informational feedback, increases free-choice and self-reported intrinsic motivation.

I define all these terms in the slides below, and I'll fill out this page more as time goes on.

The first is the scary one, since it seems to have a lot in common with the ways games reward players with achievements and the like. I reiterated many times that there are no studies that I'm aware of on achievements for games, but if they really could be sapping intrinsic motivation to play games—and I can't see how you can argue with the possibility this is true after looking at the mountain of available data—then I think somebody should start funding research into this. I think Microsoft Research would be the perfect people to do this work, since they have both Xbox Live (meaning a great source of data, and a vested interest in figuring out the truth for the longevity of the platform) and a bunch of smart psychologists on staff. Hopefully they'll heed my call.

After talking about the results for "interesting tasks", I ask the following question when I pondered covering the results for "dull tasks":

Why are you making games?

If you’re intentionally making dull games with variable ratio extrinsic motivators to separate people from their money, you have my pity.

If you’re making intrinsically interesting games and want to make them even better, be very careful with extrinsic motivators.

Towards the end of the talk, I outline a potential Nightmare Scenario based on all the implications of the research going the wrong way for games:

  1. make an intrinsically interesting game, congratulations!
  2. use extrinsic motivators to make your game better
  3. destroy intrinsic motivation to play your game
  4. metrics fetishism pushes you towards designs where extrinsic motivation works
  5. BONUS: women lose even more intrinsic motivation than men do given extrinsic motivation!

Who knows whether things will actually go this way, but it seems clear to me that the potential is there, and so we should look into this more instead of blindly moving forward.

In the talk I also address a bunch of the Common Buts:

  • Players like them!
  • Our data shows they work!
  • We make lots of money!
  • Just ignore them if you don't like them!
  • They show players different ways of playing!

I go through each of these in turn, trying to address the core of the point.

Finally, I talk a bit about how to Minimize the Damage, if you're forced to have achievements and rewards in your games. As you may know, I'm working on an indie spy game called SpyParty, and since some platform holders currently require you to give away achievements to pass certification, I gathered a list of ways of implementing rewards so they do less harm:

  • Don’t make a big fuss about them.
  • Use unexpected rewards.
  • Use absolute, not relative measures.
  • Use endogenous rewards.
  • Make them informational, not controlling.

Again, the data shows even following this advice reduces intrinsic motivation, but it's at least something you can do. I talk about this in more detail in the slides below.

Metrics Fetishism

I had a section here about the trend I have started calling metrics fetishism, but it was long enough that I moved it to its own page.

Materials

I usually just dump a ppt and mp3 up here, but I decided I'd try something fancier this time. After searching far and wide for a good tool for putting presentations online, I found MyPlick, which is a silly name, and the site is unquestionably ugly, but the tools for syncing your slides to audio are really great compared to all the others[1], and their player is simple and efficient and not bogged down with menus and whatnot.

So, here you go:

Welp, the web service I was using for the slide syncing died, so I'll have to redo it on youtube I guess.

If you're on a phone that won't play Flash, or want to listen to my dulcet tones on your Walkman/iPod/whatever-the-kids-are-using-these-days, or you just want to study my poorly constructed PowerPoint deck, here you go:

Links and Notes

Here's where I put miscellaneous notes and links, both to articles talking about this lecture, and related stuff:


Achievement-unlocked.png


  1. I tried SlideShare, and their sync app is a complete joke, the person who wrote it clearly never tried to use it to mark up a large presentation, and a few others, including one that wanted you to upload a separate mp3 for each slide!
This page was last edited on 6 March 2011, at 17:34.