Level 15: Blindtesting

When your game is playtested without you (or another of the game’s designers) being personally present to observe, it is called blindtesting. This is the subject we cover today.

Readings

No readings for today. As before, if you know of any relevant readings you have encountered, post it as comments to this blog post, or on Twitter with the #GDCU tag.

The Challenges with Blindtesting

As you might imagine, blindtesting has a lot of problems compared with a playtest that you run in person:

• Without you there, any issue the players run into is a show-stopper that prevents them from finishing.
• Perhaps worse, the players may interpret the rules incorrectly and play anyway. If they are unaware they are playing “wrong” your test results may be skewed, and the testers won’t even know it.
• Feedback is highly variable; as you have no doubt seen by now, some playtest groups are better than others. This problem is made worse when you are not present and cannot ask targeted questions.
• You cannot observe in real-time. The only information you get is what the group can report back to you in retrospect. The fidelity of information is much lower than when you are present.
• Setting up a blindtest takes a bit more time. You cannot simply bring your prototype with you to a friend’s house and play right then. You have to find a way to get a copy of your game into the hands of your testers, and then you have to leave. Then you have to wait while your testers play your game, on their schedule, and then finally get back to you with some results. Even in the best case, blindtest results may take a day or two to get back to you, which is far more than an in-person playtest which you can conduct in a matter of minutes or hours.

Blindtesting is therefore limited in the type of information it can provide, and the schedule on which it can provide it.

Why Blindtesting?

If blindtesting is so limited, why bother with it at all? What use is it?

This is a technique we borrow from science. Awhile ago, some researchers noticed that the results of an experiment would be different if the researchers are in the room. Sometimes, test subjects would say what they thought the researchers wanted to hear rather than what they actually thought. Sometimes, the researchers would give subtle non-verbal cues that neither they nor the subjects would consciously notice, but that would bias the results of the test. For example, if the experiment is a taste test between two leading soft drinks, if the researchers know which drink is which and they know which one they want the test subjects to pick, they might suddenly find that all of the test subjects are choosing the “right” drink… but only because the researchers are (accidentally) telling them what to choose!

The solution to this is the so-called double-blind experiment, where neither the researchers nor the test subjects know what the “right” answer is. This eliminates many sources of accidental bias, making the results of an experiment more valid.

Playtests share a lot in common with science experiments. In both cases there is a hypothesis (“I think this game is fun”), an experiment is designed (building a prototype), the experiment is run (the playtest), results are analyzed. The purpose of both is to find out more information about the inner workings of the system that you are studying.

Blindtests, then, are the only way to get a true idea of what your game is like “in the wild” – that is, how players will react to it if they have just purchased it from a store and are playing it for the first time, without a designer present at their table to answer questions.

When to Blindtest?

Because you can get so little information from blindtesting, it is not suitable for an early test when your game is in a rough state (the testers would likely run into problems, be unable to continue, and you would have spent a lot of time waiting for something that you could have figured out much faster with an in-person playtest. Blindtesting is most suitable near the end of development, when you already have a high degree of confidence in your game.

The purpose of blindtesting is to catch the non-obvious problems that you may not be catching in your in-person playtests, because you accidentally bias the results of the test by helpfully being available to answer questions. Even if your playtesters never have to ask you anything, the very knowledge that you could can sometimes make people relaxed enough to get through the rules, where they might otherwise get flustered and give up. Or, you might say some key piece of information when introducing the players to the game (“this is an auction/bidding game”) that is not written anywhere and clarifies a lot by creating some preconceived notions in the minds of your testers. Without you present, you’ll see just how accurate your in-person playtests are, and you may catch some surprising errors that you did not notice before.

Who to Blindtest With?

Most board games require multiple players. For practical reasons, most blindtests done with professional games are done with regular game groups. Some companies keep a list of volunteer groups and put out calls to their private list for playtesters; they will then send out advance copies of their game in exchange for feedback. For our purposes, this is unhelpful, because most of you do not work at such a company and do not have a database of volunteers to call at a moment’s notice.

Ideally, your blindtest should be with people who are in the target audience for your game. If you’re making a children’s game, your blindtest should involve kids in your target age range (and perhaps their parents). If your game is made for people who are already avid Eurogame players, you’d do well to find a group that plays these kinds of games regularly. And so on.

There are a few ways to find blindtesters for your Design Project in this course:

• Your social network of friends, family, and colleagues. These people may be local, or they may live in another city or even another country from you. Since you don’t have to be present, at least this time, geography is not a barrier.
• Other people who are taking this course. If you already have a local playtest group that you’ve used before, you can put out a call for help on Twitter or the forums. Offer a blindtest exchange: you’ll blindtest their game if they will return the favor with yours.
• As a last resort, you can look outside of this course for other discussion forums or other online hangouts for board game designers and/or playtesters, and see if there is a method for recruiting blindtesters.

How to Blindtest?

With in-person tests you can afford to be a little bit sloppy. Maybe you accidentally left some of your game components at home, or some of your rules are a little unclear and need clarification. It’s not ideal, but a playtest session that runs into problems can be salvaged when you are there. With blindtesting, you do not have this luxury, so you must be extra careful to make sure that your testers have everything they need to give your game a proper playtest. This includes:

• A complete set of game components. Double and triple check to make sure that you have everything the players need, together, in one package.
• A list of everything that your package should contain, so that the testers can verify for themselves that nothing was left out (and if it was, it will at least give them a clue of what they need to supply as replacements).
• A complete set of rules describing how to play. (The components list can simply be part of the rules.)
• A separate set of instructions on how to conduct the blindtest. Is there anything in particular you are looking for the players to do? Do you just want them to see if they can play through the game? Do you want to know if they find the game enjoyable? Do you want them to try to find rules exploits and imbalances?
• A final set of instructions on what should be done on conclusion of the playtest. How should the testers contact you (phone, email, etc.)? What should they tell you when they contact you – if you don’t give a list of questions for them to answer, you will just get whatever they happen to feel like telling you, making your results a lot less focused than they could be. Give some thought to what information would be the most useful to have, what kinds of feedback are most important to you… and ask for it!

How do you get your package into the hands of the playtesters? If they are local to you, it is as simple as taking your playable prototype and handing it off. If your blindtesters are not local, this presents additional challenges. You have two options here.

First, you can mail your prototype to them through the postal service. Depending on where they are, this step alone can slow things down considerably (not to mention making it more expensive), so plan your schedule accordingly. If you want your prototype back, consider including a return package inside the main one, already addressed to you and with postage already paid.

Second, you can handle everything over email. Send documents that can be printed out and assembled to make a playable prototype, and include instructions on how to print everything. Do everything you can to reduce the amount of work that must be done at the playtesters’ end of things. After all, you are already asking for their time in the form of a playtest; asking for another hour or two to print and cut sheets of cards is adding insult to injury. Along the same lines, try to keep the cost of materials down and availability high if you’re expecting your blindtesters to provide their own. For example, do not insist on printing on heavy card stock (which may require a special trip to a print shop) when printing on plain paper will do.

Contingency Planning

In the field, game companies do not rely on a single blindtest group, but several. Aside from the obvious reason that more tests give more data, there is also the problem that blindtest groups are unreliable. Without you in the same room, they may take awhile to organize a playtest, and they may take even longer to get back to you. Some groups may lose the components, or forget about their obligation, or they may simply be so busy with other things in their life that your game takes lower priority. Or maybe you’ll send your game through the post and it will get lost. In this course, when you set up a blindtest, you may find yourself in the frustrating situation of waiting for test results that are simply not coming… or at least, they may not arrive within the schedule you set for yourself.

You have three options for dealing with this potential hazard. It is up to you which method best fits your personal situation.

1. You can set up multiple blindtest groups. As a rule of thumb, three is a fairly safe number. That way, if one or two groups don’t show, you’ll at least have some results. (Note that if you are doing a “blindtest exchange” with other participants of this course, that means you’ll be testing three other projects, so make sure you have time for this.)
2. You can choose a blindtester that you know and trust to be reliable. If you are sending your game to a friend who is highly organized and always keeps their promises, you may have confidence that they will get back to you when they say they will.
3. You can cross your fingers and hope that something bad won’t happen to you. This third method is not recommended for professional projects.

Homeplay

Your homeplay this past Thursday was to arrange for a playtest session with non-designers. You may have already performed this playtest, or you may have just scheduled it to take place over the first part of this week, but that playtest session should be concluded before this Thursday, August 20, noon GMT.

In addition, you should find a blindtest group and arrange for a blindtesting session, to take place after the non-designer playtest. If possible, you should plan to have blindtest results on or before Thursday, August 27.

Feedback

Do you know of any great articles on blindtesting? As you conduct your own session, or you’re your own personal experience if you’ve done this before, have you found any helpful tips or tricks that you’d like to share? Post in the comments on this blog, or on Twitter with the #GDCU tag.