When you use a computer enough it can start to feel like the mouse becomes a part of you. Very quickly you forget about consciously moving it right or left to control the cursor as it becomes second nature. Essentially it is an extension of your physical self, with which you manipulate the screen as naturally as you pick up an object with your hand. Combined with a keyboard it is the only form of input for most ‘serious’ video games in the home because of the level of precision required. I think there are other forms of input that can be used in conjunction with traditional mouse and keyboard for a much richer, more immersive experience.
- Eye tracking
- Voice control
- Video gestures
Eye tracking has been employed in usability and behavioral studies for over a decade, but it has so far been demonstrated only in a very limited sense for actual control input. These two videos demonstrate eye tracking as used for controlling camera movement instead of the mouse and aim instead of what would normally be a physical gun in an arcade.
In the case of a free camera the player moves the camera by looking towards the edge of the screen. When the eyes are centered the camera is still. This is the same kind of control that a joystick uses. Without having tried it myself, I feel like it could be unintuitive because when we look at something we don’t expect it to move away. With the second example you can see that with a fixed camera in place of a physical gun, eye tracking is quite effective, but I don’t see any major advantages over the traditional method.
In first person shooters I feel the best combination will be mouse for camera control and eye tracking for aim. When you look at something on the screen it doesn’t move away – you shoot precisely where you look. Gamers can use the same intuitive interface they’re already used to for moving the character around and firing the weapon. For the quickest, most precise control of aiming, the lighting fast twitch reflex of eye movement is perfect and, in fact, is something players already do.
In 2005, Erika Jonsson of the Royal Institute of Technology in Sweden conducted a study of the methods I described and arrived at similar conclusions:
The result showed that interaction with the eyes is very fast, easy to learn and perceived to be natural and relaxed. According to the usability study, eye control can provide a more fun and committing gaming experience than ordinary mouse control. Eye controlled computer games is a very new area that needs to be further developed and evaluated. The result of this study suggests that eye based interaction may be very successful in computer games.
The problems facing this method are primarily the cost of hardware. Current technologies are used in a limited fashion with academics and usability studies and are generally not available in the mass market. Accurate eye tracking is apparently not an easy thing to do. I’m guessing that, were there a serious market opportunity, some enterprising group of young researchers could simplify the hardware and prepare it for mass production, bringing costs down to reasonable levels for a luxury product.
Voice control can be used for high level commands that might otherwise be accessed from menus or other complex key commands. By using voice it saves the player from having to break from the immersive experience of controlling the character. It has the additional side effect of engaging other parts of the brain and encourages more realistic style of interaction that people encounter in daily life.
The jury is still out on whether people like talking to their computer. I think reception of it will rest, as usual, on the intuitiveness of the commands. The game should never force the player to use voice commands – there always needs to be another path of access – and the player should never have to remember what that command or sequence is. You just say what you want to happen. I think this could be especially interested in non-linear story lines where the options aren’t necessarily clear to the player. Instead of selecting from a menu of pre-defined choices the player could explore (as long as they don’t feel they’re searching in the dark).
Video gestures have been used as a fun, gimmicky activity with the PS3 eye and soon with microsoft’s project natal, but I haven’t seen a use that actually results in interesting game play with what we call “serious” games. One idea is to take hints from the camera and not direct input. When communicating with team mates in multiplayer a user might say a command in voice chat and point in a general direction relative to where he’s looking. The camera could take that hint and cause his character to also point in that direction. This is not something that can be used for precise control, but we can attempt to mimic non-essential body language as added value.
When combined with voice command video gestures could enhance the level of realism and natural interface. For instance the direction of the face could indicate whether the player is giving a command to the game or talking to another person in the room. In story telling dialog, more than one player can input and the video indicates which one of them is talking. Again, I think the best we can do with camera input at this point is imprecise input. Project natal looks like it might do a great job in the casual games space, but we’ll have to see it in the wild controlling games by 3rd party developers.