Dealing With Designer Input Latency

Input responsiveness is one of the most important aspects of any game. Often overlooked in favor of more glamorous features, a snappy input system is one of those things that the average gamer doesn’t realize they need until they pick up a game that fails to provide it.

When you bring up the subject of input lag most people think in terms of hardware and software processing latency – that is, the time it takes to translate a physical button press into something the game code can actually react to. For most of us the hardware side is out of our control, and on the software processing side the biggest factor is often pure update rate. You can get a lot more granularity out of 60hz than 30hz, which is why many games put their input handles on a separate thread. It’s important to keep both of these latency sources in mind, but what about the design side of latency? The decisions we make when creating our game mechanics can have a significant impact on the responsiveness of our controls and it’s critical that we think carefully about our decisions in this area.

For the sake of simplicity, in this article we’ll be dealing specifically with standard console game pad buttons. Motion controls, touch screens and joysticks have their own unique challenges that are outside the scope of our discussion here. Some basic, standard state terms for buttons are:

  • Button Up (state) – the button is not being held down. Typically the neutral state.
  • Button Down (state) – the state of a button when it’s being held down.
  • Button Pressed (event) – the act of pushing the button down. More specifically, the button was in the up state on the previous update and is in the down state on this update.
  • Button Released (event) – the act of releasing a button. The button was in the down state on the previous update and is in the up state on this update.

Using these 2 events and 2 states you can define most of the core button actions. We’ll leave aside things like analog triggers, as they can be hammered into this digital framework if need be.

How do we create designed latency?

Ideally, the most responsive way to hook into our game pad’s button is to utilize the button pressed event. This event represents our earliest opportunity to handle an input, and as a result we’re guaranteed that all of our latency is on the hardware and processing side of the equation. For the purposes of this article we can consider this event to have zero latency.

Of course, there are two common situations that keep us from just limiting ourselves to button pressed events and calling it a day: player state changes and button holds. Pretty much every game has the former and many games (though not all) have the second as well. The way in which we design latency into these two systems is quite different, and each one requires a different set of tools to handle smoothly.

Dealing with holds.

To begin, let’s take a look at Halo’s famous Plasma Pistol. The weapon has two methods of firing: a standard semi-automatic projectile that fires if you repeatedly pull the trigger and a charged blast that fires if you hold the trigger down. This combination of inputs means that we can’t rely on the button pressed event to make a decision on what to do – we don’t yet know if the player wants to fire a single shot or start charging for the blast attack. Instead, we need to use the button released event, something that introduces a finite amount of latency into the system. No matter what, the player can’t both press and release the button in the same update cycle. At best, we’re dealing with a minimum of one update cycle’s worth of added lag and in reality we’re almost certainly talking about at least 100 milliseconds before the average player can get his finger back off the button.

What’s more, it’s often not viable to choose the smallest possible window to detect a hold state. In the case of a simple firearm it’s probably safe to hew close to the minimum, but if the accuracy of our hold time is important (in, say, a variable length jump) we need to be a little bit more generous. If we want to be casual friendly, the situation gets worse – it’s not unreasonable to expect some inputs to need as much as 200 milliseconds for a casual gamer to be able to intentionally choose a hold instead of a tap.

Hide your shame.

Using holds to delay or charge up attacks is a core part of Soul Calibur’s fighting system.

We’ve got a few different ways to contain the damage. The first option is to try and hide it, and this is a technique that applies particularly well to fighting games. In Namco’s popular Soul Calibur series, character attacks are never instantaneous – there’s always some amount of windup, even if it’s very short, so that opposing players have a small window in which to react. As a result, it’s possible to hide the input latency of the many charge attacks inside these windup animations. If the player has released the button by the time the attack window arrives, the move completes its animation. If the player is still holding the button, the move transitions into a hold state, usually a slow-motion animation or a pretty particle effect.

This technique is especially powerful because it turns a weakness (needing to wait on additional input data) into a strength. Once the player enters the hold state, you can allow them to vary the duration of the hold or cancel the hold into a different move altogether to create interesting mixups and mind games.

Fake it ‘til you make it.

Another option is to just start doing whatever it is you’d do if the player decided to tap the button and then make adjustments later if it’s determined that, instead, he kept the button down. This is a technique that’s often applied to lofted jumping in platforming games: when the button is pressed the character immediately begins a standard jump, and then a few hundred milliseconds later that jump gets a boost if the button is still held down. The result is a jump that feels very responsive when tapping but still allows for some in-flight adjustment of the distance and height.

This works well enough, but its applicability depends somewhat on the type of game you are. In a more cartoony universe like Super Mario Bros. or Ratchet and Clank, this sort of technique blends in pretty well. In a more realistic setting, such as Red Dead Redemption or Uncharted, this sort of floaty solution sticks out a lot more. As with most solutions in game design, your mileage may vary.

Utilize your latent psychic powers.

A third approach is to try to mitigate the impact of the latency by building a more robust set of detection rules. As a simple example, if your character’s jump action fires on the <em>button released</em> event but the player doesn’t get his thumb off of the button before he runs off a ledge, you could easily have the character automatically jump instead of letting him fall to his death. How generous you decide to be is a function of both how friendly you want your controls to feel and how confident you are that your predictions are correct.

Prediction like this is not without risks. Particularly if you have overloaded inputs – such as a button that doubles as reload and pick up weapon – you run the risk of guessing wrong and doing something the player doesn’t expect. The more complex and context sensitive your input is the more risk involved in trying to guess what the player wants to do. Generally speaking you can just avoid putting hold states on buttons with widely varying functions but, in case you do, it’s something to keep in mind.

A note on double tap inputs.

I didn’t include this in the description of a “hold” state because they’re not especially common, however their impact on latency is essentially same and, as a result, you can deal with them using the same techniques. For example, in Crysis 2 you switch to grenades by tapping the weapon switch button twice in rapid succession. Since both a single tap and a double tap result in a weapon switch action (and thus it’s just a question of which weapon you end up swapping to) this problem is easily handled via the “fake it ‘til you make it” technique.

Dealing with state changes.

When it comes to input latency, the only state changes we care about are the ones that require us to respond in different ways to player input. For example, the player might be able to press the “crouch” button while running or standing idle but not while soaring through the air (due to a jump or getting thrown by an explosion). Unfortunately, if what the player actually wanted to do was start crouching right when he hit the ground, this intent is lost unless we decide to keep the input around. Ignored inputs are often very frustrating for casual players, especially if they were ignored during a finite-duration state (again, like a jump) toward the very end of that state’s window.

Although not latency in the traditional sense, ignoring inputs feels very similar. The player wanted to do something, they asked the game to do it in a way consistent with their expectations, and the game did not react in a timely manner. There are two ways of dealing with this problem: buffering and interruption.

Buffering is helpful (unless you’re RealPlayer.)

Buffering input is pretty straightforward: we record incoming inputs but don’t respond immediately. Consider a standard player jump – unless your game has a double jump action, it just doesn’t make sense to let the player jump again before he reaches the ground. Of course, if he’s trying to do something particularly tricky (like a rapid series of timed hops) ignoring a jump request could end up ruining his day. Instead, we can buffer that input and then act on it the moment the player is back on the ground, maintaining the player’s intent without changing our mechanics to allow for a double jump.

Many complex combos can be created by buffering entire input streams during moves with long animations.

Many complex combos can be created by buffering entire input streams during moves with long animations. Actually recording the input is also simple: we remember a certain number of inputs for a defined window of time. Your buffer can be as large or small as your problem requires. Fighting games like Street Fighter and Tekken have buffer windows that can hold a half dozen or more inputs, and most action games will buffer at least one input when appropriate. How long you keep those inputs around also varies. In the case of short, finite duration states it’s usually best to keep them for the entire state. If the state is particularly lengthy, however, it’s possible that the player will “forget” about that input by the time you get around to processing it. For this reason, time sensitive inputs are typically dropped after a designated duration.

Despite the fact that buffered inputs are far friendlier, by their very nature they introduce a source of latency. Buffering is better than ignoring, but we’re still acting on these inputs well after they were actually received, and if your buffer window is large the character may end up responding to inputs in a way that feels unpredictable. In addition, unless you have some sort of visual reaction to a buffered input it’s likely that more casual players won’t even be aware that it’s happening (which is why intentional buffering is an advanced technique in fighting games). Since it’s often not possible to provide this feedback in a way that doesn’t feel artificial it’s best to keep your input window small.

As was mentioned earlier, great care must be taken when applying buffering to highly time sensitive inputs. For example, if your player was firing his weapon and it ends up auto-reloading when he runs out of ammo, he might decide to hit the weapon switch button to use a different one. The player’s intent is to swap to a weapon that has ammo, but if your game buffers that input the result will be a frustrating mess: the weapon finishes reloading, which is annoying enough since no firing could occur during that time, and then the moment the weapon becomes usable again it starts playing a weapon switch animation that further delays the desired shooting!

Of course, all inputs are time sensitive to some degree, but recognizing the extent to which this is true in each case is critical for determining the correct way to handle them. When dealing with time sensitive inputs, the more appropriate response is to allow a state to be interrupted.

I’m really happy for you, and I’mma let you finish, but…

Continuing with our reloading example from before, if the weapon switch input was acted on immediately (instead of being buffered) then the reload state would end and the player would switch to his usable weapon. What happens in this case is up to your design discretion – in most FPS games, interrupting a reload means that the weapon remains empty when you switch back to it (to avoid exploits). It’s probably more newbie friendly to let the weapon reload anyway, and which one you choose depends entirely on your audience and the particular actions in question.

Guilty Gear’s cancel system adds a ton of depth.

Guilty Gear’s cancel system adds a ton of depth.State interruptions are not an all or nothing matter – there’s no particular reason why you have to allow interruptions at every point in the state, and defining interruption windows can create some very compelling mechanics. For example, in Soul Calibur many attacks can be cancelled pre-hit – that is, their animations can be interrupted at any time prior to actually doing damage. Advanced players often use this mechanic to bait their opponents into a disadvantaged, punishable situation by starting an attack and then cutting if off to create a fake. In Guilty Gear, some moves can have their recovery (post-hit) animation cancelled to allow for guaranteed combos, and doing so is so strong that it actually requires the use of a finite player resource (the Tension bar).

I could easily write an entire article just on the topic of state interruptions and buffering, but I don’t need to because Eric Williams – a designer on the first God of War title – already did! His excellent article Combat Cancelled is required reading for all game designers, and I encourage you to absorb its deep wisdom if you haven’t already.

They go together like lamb and tuna fish.

Both buffering and interruption have their uses, and they’re most effective when used for their respective strengths. Buffering works well if you have a more complicated input possibility space – like most fighting games – or if you have a state where interruption simply doesn’t make sense (such as jumping during a jump). Interruption works well for time sensitive inputs and in situations where waiting for a buffered input to fire might result in unpredictable or undesirable behavior.

Further, there’s no reason that you can’t use both techniques simultaneously. Why not have a state with a defined interruption window and allow the actual input that causes the cancel to be buffered prior to the window? You get the best of both worlds: the latency minimizing aspects of an interrupt with the granular designer control of a buffer!

Wrapping it up.

As we discovered in the introduction, having your inputs react on the button pressed event is always the most responsive option. The addition of hold inputs forces you to react on button released events instead, and as a result introduces latency. When pondering adding hold inputs to your game, be sure to ask yourself if the benefit (another input) outweighs the cost (designed lag).

Even when dealing with normal button presses, player states can muck with your ability to respond immediately – adding buffering and interruptions whenever possible goes a long way toward alleviating this pain. Buffering input should be the rule, not the exception, and if you choose to ignore player inputs it should be a careful and considered design decision.

I didn’t mention it before, but even if true buffering isn’t an option, you should always check your inputs when exiting a state; in that case, we’re looking for the button down state as opposed to the button pressed event. This should go without saying, which is why I didn’t dwell on the point previously, but I wouldn’t be doing my job if I didn’t at least point it out.

Finally, it all comes down to feedback. Even when you can’t respond immediately to an input, it’s critical that you do something to let your player know what’s happening. Hiding your latency behind windup animations, particle effects or sounds can help the player feel like something is happening even if the important part of the action hasn’t yet occurred.

 

Pick Up That Can: The Problem With Interactive Objects

I’m sure that most of you have already seen the recently released Deus Ex 3 gameplay trailer. One of the game elements highlighted in the footage is Eidos’ method of calling out interactive items in the game world: a bold, bright yellow outline and highlight over anything you can interact with that’s more or less in the direction the player is looking.

Some of the fan reactions to the trailer have been quite surprising, with a few gamers going so far as to call themselves “outraged.” They don’t like having information in their face, and what’s more they seem to find the assertion that they need to be vaguely condescending. Some folks have even brought out the dreaded “immersion breaking” phrase. All this fuss over some simple object highlighting! Clearly, most modern games need some way of pointing out what is and isn’t relevant to help streamline gameplay, so what’s the big deal with Deus Ex’s method?

Framing the issue

First, let’s note that there are actually two different problems to address when it comes to interactive objects in modern games:

  1. Showing the player objects that can be used, picked up or manipulated.
  2. Indicating to the player which objects are actually important at this moment.

The concepts are similar but, importantly, the second problem is really a subset of the first. Further, the degree to which these two things differ (that is, how much of the former problem the latter encompasses) can vary a great deal between games. We can indicate the general range of this with the Internet’s favorite tool: a two-axis graph.

I’ve taken the liberty of inserting what I feel are four representative examples, one for each quadrant. I should emphasize that this graph does not assume any sort of value judgments. No quadrant is any better or worse than the others, it’s just a simplified way of quantifying our options.

Fleshing out the axes

Valve Software’s Portal gets categorized as both austere and interactive. Excepting the latter third of the game, there isn’t a lot of stuff sitting around in the Aperture Science testing grounds. This is not a game that focuses on props, but what is included is almost all interactive: auto-turrets that will attack you and can be knocked over; cubes to pick up and move around; giant red buttons to press; bouncing energy spheres to re-direct. We have Spartan environments combined with highly interactive game objects.

Note the clean, uncluttered areas and bold primary colors.

Mirror’s Edge sits at the intersection of austere and static. Though not nearly as focused as Portal, the level design of Mirror’s Edge is still beautifully direct. There are some props around to give the world flavor – piles of boxes on pallets, wooden planks to indicate good jump points; the occasional planter box or advertisement – but very few of these objects are meant to be interacted with. You’ll sometimes find a wheel that needs to be rotated or an enemy weapon that can be picked up, but that’s pretty much the extent of what you need to manipulate in the game.

The detail work in this shot is breathtaking.

Sticking with static but moving along the horizontal axis toward cluttered we have Uncharted 2. This game has some of the most striking environments in this generation (not to mention being one of my personal favorites!) and they’re filled to the brim with stuff. In particular, the Nepal level is practically overflowing with props: burned out cars; piles of rubble and trash; orphaned furniture and appliances; plastic crates and other detritus of a city in conflict. For the most part, though, these things are there purely for purposes of immersion. It’s fairly rare that you need to actually manipulate part of the environment, and these are mostly limited to either cinematic moments (such as bits of train car that start to break as you climb on them) or navigation elements like ropes and doors.

The final quadrant of our graph is represented by Half Life 2, a game renowned for being one of the first to introduce general physics objects as a core mechanic in a mainstream shooter. There is stuff all over the place in Half Life 2, and almost all of it can be picked up and manipulated with the iconic Gravity Gun. In fact, many of the game’s puzzles involve using the various physics objects to solve simple lever or weight problems (and this has been increasingly true in the episodes.) As a result, Half Life 2 is both cluttered and interactive.

The problem of consistency

Now that we’ve broken down our examples, it’s very interesting to note that the two games at the top of our interactive axis do almost nothing to indicate to the player what can or can’t be manipulated. The Gravity Gun in Half Life 2 does have a subtle animation that activates when an object can be picked up, but this is pretty much the extent of their signaling mechanism. There’s no indication at all if you’re just picking stuff up with your hands.

Both of the games at the static end of our graph, on the other hand, do attempt to indicate when objects are interactive. Mirror’s Edge applies the same red highlight treatment that they use to suggest efficient climbing routes, while Uncharted 2 has two different approaches depending on the item in question. For things that are meant to be picked up – which are primarily guns and grenades – they play a flashing effect on the item itself. For environmental objects such as statues or carts that can be pushed the game generates a HUD prompt when the player is sufficiently close to the object to initiate interaction.

Why would games that have less interactive objects feel the need to highlight them? The answer is actually fairly obvious: because these objects are the exception rather than the rule, it’s necessary to counteract the player’s expectation that items in the environment are there primarily for aesthetic reasons. In essence, these games need to momentarily break the immersion they’ve crafted in order to make certain that the player understands what needs to be done.

The problem of fidelity

It’s worth going back to the concept of immersion as it applies to interactive objects. Games like Uncharted 2 fill their environments with interesting props precisely because it makes the world feel more alive, more lived in. This is despite the fact that these objects typically don’t do anything, but that’s because they don’t really need to. From a development standpoint, it’s much easier (not to mention more efficient) to make great looking stuff to fill out the world when you don’t have to find a use for all of them or spend valuable CPU time handling their physics.

Furthermore, the high quality of the environment and the desire for seamless immersion creates pressure to make the objects that are interactive blend in as well as possible. This is, of course, exactly the opposite of what’s easiest from a game design perspective, and it isn’t a new problem. Back in the days of yore, adventure games found themselves in a similarly difficult situation. As hardware improved, background art got increasingly lavish and detailed and, as a result, it became important for interactive items to meld well with these more immersive scenes. One of the side effects of this progression can be found in the phrase “pixel hunt,” a derogatory term that came to be associated with many later adventure games.

Because the worlds were so detailed, filtering objects that were important to the game from the ones that were important for reasons of aesthetics became a matter of hunting around the scene looking for spots that would give you a “use” cursor. This was not a particularly fun mechanic, and the problem contributed to the eventual decline of the genre. More modern takes on adventure games offer various aids to reduce the issue, with many offering player abilities that cause all interactive items to flash or highlight briefly.

Jumping back to our modern game examples, yours truly once spent several minutes trapped in a tiny room in Uncharted 2 simply because I didn’t anticipate that a statue could be moved. There are dozens, if not hundreds, of statues scattered throughout Drake’s adventure, and seldom are they interactive objects. I ended up resorting to the 3D equivalent of pixel hunting, in which I systematically walked around the room looking for a HUD prompt to appear and indicate what it was I needed to do.

Bringing it on home

Let’s get back to our original topic: Deus Ex 3’s object highlighting scheme. We know that the problem it’s trying to solve is real, and that similar techniques have been employed in other games. Given that, why are people so unhappy about this particular example?

The crux of the matter is this: no matter what, any attempt to break interactive objects out of the environment results in a momentary loss of immersion. Even when it takes on a much more subtle form – such as exploding barrels that are all red, or electrical switch boxes that all happen to look exactly the same – the presence of these cues reminds us that our character exists in a world of limited interactions. The result of Deus Ex’s extensive, always-on highlighting is to constantly remind the player that no matter how alive and immersive the world feels, most of it isn’t actually interactive.

Tactical options available.

Of course, different sorts of games require different solutions. Slower paced games (such as adventure games) have more freedom to let the player dink around and discover interactivity, whereas fast-paced games with lots of time pressure situations (such as shooters) have to be more explicit. Some settings are more restrictive than others, as well. A game set in ancient Rome doesn’t have many hooks to integrate something like Deus Ex’s system into the narrative, whereas a more futuristic, sci-fi setting like Crysis 2 is less restrictive. In fact, it’s almost certain that Eidos was hoping to leverage the cyberpunk themes in Deus Ex to make it easier for players to accept their augmented-reality approach.

The ideal solution might be creating world in which everything that should be interactive is interactive, but the reality is often isn’t practical. It’s easy enough to conceive of a game where most of the props can be picked up – just look at Half Life 2 – but what about one where all the doors open? All the vehicles can be driven? All the TVs turned on? Grand Theft Auto 4 probably comes closest, but it’s not clear to me if more linear experiences would be significantly improved by these additions.

I think the most important takeaway is this: the amount of player feedback you need to give is inversely proportional to the amount of interactive objects in your game. That is, the more interactive you are the less you need to worry about getting the player’s attention because you have already created the expectation that things are interactive. If you have less frequent (or less important) cases of this in your game, you need to do something to remind the player that items in the environment sometimes need to be manipulated to succeed. It could be that Deus Ex 3’s scheme goes a little too far given their level of interactivity, in which case their best option is simply to scale it back until the proverbial Goldilocks is just right.

First person jumping metrics.

Game development has always been about metrics. Even the simplest games needed a basic set of variables around which the mechanics could be balanced. How fast does the character move? How high can he jump? How fast do the projectiles fly? As games have become more complex and teams larger, metrics have taken on an even greater importance. The game programmers want to know the player’s stats so that AI can be properly balanced; the level designers need to know how far the player can jump so that they arrange accomplishable paths through their block outs; modelers might need to know the player’s crouching height so that they can design usable cover objects.

Although one can arrive at these values via experimentation (and play testing should always be used to double-check) it’s best to think critically about the details surrounding the metric you want to lay down.

Take the relatively common problem of horizontal jump distances in first person shooters, for example. Say you’re working in an engine where the game units are equivalent to 1 centimeter and you’d like the player to be able to comfortably jump 4 meter gaps. Your first guess would probably be to set the player’s jump distance at 400 units. If you’re forward-thinking, you might make it 412 or so to provide a little bit of forgiveness.

Your first test of this setting is likely to be quite surprising: you’ll almost certainly find yourself falling well short of clearing your 400 unit gap. Why would this be the case? Let’s take a look at some screenshots from Valve’s Half Life 2: Episode 2 and see if we can spot the problem.

Note the location of the highlighted edge…

In this screenshot you can just barely see the edge of the ledge I want to jump from (highlighted in red.) Once this edge leaves the bottom of my field of view (FOV) it is essentially out of my mind. Since I can’t see it anymore, my brain will reflexively start my jump right before or just after that moment. However, if we look at this next screenshot, you’ll see that in reality I’m quite far away from the edge!

The reason for this is straightforward: the height of the player’s camera (eye-height) and the vertical FOV angle create a predictable blind spot in front of the player character’s feet. Depending on these two values, the point at which the edge disappears from the player’s view can be a quite far. In my experience, this additional distance can easily be as much as 70% of the camera height with a wide-screen FOV of 70 degrees. The blind distance increases as the FOV angle decreases, which makes sense since narrowing the FOV is how one accomplishes zooming (such as with a sniper rifle.)

The edge has totally disappeared with the FOV reduced to 70.

 

The edge actually looks much further away with the FOV increased to 90.

Notice how this issue can also be affected by the player’s view pitch.

Looking up slightly, the edge disappears once again.

 

Looking down, the edge is both in view and appears further away again.

Some players will naturally look down as they approach a jump, which means they’ll be able to get much closer before losing sight of it. A player who leaves their view flat will have less distance, and players who happen to be looking up will find themselves with significantly less distance! Below is a quick diagram I whipped up to show the relevant values that determine this player blind spot.

The blind spot is the distance from the player to the edge.

With this information in mind, in order to create a newbie-friendly jump we need to make our jump distance quite a bit longer than the actual gap we want to clear – using our previous example, something in the neighborhood of 500 units is safer. The exact value you want will depend on your game’s FOV, your player’s eye height and your valid pitch range (i.e. how far up or down you expect the player to be looking.)

There are other factors to consider as well: the speed of your player; the minimum reaction time of a typical gamer; the size (in units) and type (axial-aligned bounding box, capsule, etc.) of the player’s collision. However, these have a much smaller overall effect on the end result.

Of course, this information leads to another metric: our un-jumpable gap size. Since advanced players will learn to wait out the blind spot (rather than jumping immediately) we’ll need to make sure those gaps are at least 512 units away. Longer is better, where possible, since it will be visually clear to the player which gaps he can jump and which ones he can’t. You can also use this data to create more difficult jumps with less padding; however, this can easily create confusion among your more casual players.