The Bug Backlog - An Evergrowing Mountain

The Bug Backlog - An Evergrowing Mountain

If you are part of a development team working on a game, and you are working in some kind of Agile way, you most likely have a bug backlog, or at least bugs as part of some kind of backlog. The bug backlog looks very different during different stages of the game development cycle - it starts out empty, and then as features and complexity is added, it grows. And in most cases it never stops growing.

The first question to answer is why this is happening in general - why are we just not fixing the bugs?

Let’s break it down a little bit:

• Value of fixing bug < Cost of fixing bug

  • If it costs more to fix a bug, than the value it adds to fix it, then it makes perfect sense not to spend time on fixing it 

  • We may or may not have some bugs in our bug backlog that just do not make sense to fix because it costs too much

• Value of fixing bug < Spending your time on something else

  • If the opportunity cost is too high, and spending your time on something else adds more value than fixing the bug, it makes sense not to spend time on fixing it

  • Maybe adding a new feature, or fixing other bugs makes more sense than fixing a specific bug you care about

So basically we are not fixing a bug, because we are adding more value doing something else.

But there is a cost of allowing a bug backlog to grow uncontrollably. Let’s break it down.

• If you have a large number of bugs in your bug backlog, it is easy to lose visibility of what is in the backlog

• It becomes much easier to lose track of important bugs, as they are hidden by piles and piles of other bugs

• It takes time to go through, refine and prioritize the backlog - larger backlog, more time

• It has a psychological cost on the team as well - they get a feeling of that their work never ends, and can cause unnecessary stress

So we don’t want to fix all bugs, because we create more value doing something else, but we don’t want the bug backlog to grow uncontrollably. And just prioritizing the bugs will not be enough.

How do we solve this conundrum? Basically we have to close bugs, even though we are not fixing them. We just have to accept that some bugs we will never fix - out of sight, out of mind.

Unfortunately calculating the actual value of a bug or a feature is never simple. The more data you have, the easier it gets, but even with all the data in the world, it is still a complex procedure. Value also changes over time, as context changes, which makes it even more complex.

Let’s cover the easy parts first.

• All bugs with an obvious low value and cost higher than that value can be closed

  • Should the value of fixing the bug change in the future for some reason, the bug can be reopened

• Bugs with an extremely high cost will most likely need to be rebranded as some type of refactoring or broader changes to the system - so we can tag them as new stories instead of bugs

• Bugs that are critical to the game’s release we will fix, so they always stay at the top of the backlog

But even with these parts done, most games will most likely have hundreds of bugs which would add value if they were fixed, but are still not prioritized because there are more important things to spend your time on. So how do we trim down the backlog to a workable state, to avoid the problems listed above?

Basically there are three factors that come into play.

• Bug Inflow

  • How many new bugs are reported every sprint?

• Development Capacity

  • What bandwidth do we have to develop features and fix bugs?

• Game Roadmap

  • How do we expect to add value to the game over the coming time period?

If we know how many bugs are usually added to the backlog each sprint, and we know how many we usually manage to fix, and the game roadmap looks similar to what it has before and no major course shifts are coming, then we have some guidance to how large we should allow the backlog to be.

Let’s look at an example.

• If we usually have 2-4 new relevant non-critical bugs each sprint, and we have a healthy feature pipeline of high value things we want to develop

• We usually have time to fix between 2-4 of these types of bugs depending on feature workload and how many critical bugs we have to fix

• We want to spend our time fixing the 2-4 most valuable non-critical bugs, so we continuously prioritize them to make sure the high value ones are at the top of the backlog

  • This of course includes all new bugs as well

  • Prioritization should be done with the entire team - the Product Owner/Producer is the final arbiter of priority

• In the best case scenario where we fix more than we introduce, we want to have a backlog of high value bugs to pick from that will last us a significant amount of time

• In the worst case scenario were we introduce more than we fix, we want to have some kind of a limit to how many bugs we have in our backlog to avoid the problems mentioned before

So let’s decide on a reasonable timeframe, say 3 months. For the bug backlog to last three months under the most favorable conditions, with sprints lasting 2 weeks, we would have to have 12 bugs in our bug backlog. 3 months = 6 sprints, and with -2 bugs in the backlog every sprint. 

So let’s say we originally had a bug backlog of 30 bugs. But we only need 12 for our healthy bug backlog. What do we do?

• The most important thing is to make sure that the bug backlog is properly prioritized, and that the top 12 bugs are indeed the most important ones

• When we start this experiment, we monitor the backlog for a few sprints to ensure that our calculations for inflow and bandwidth are correct

• Basically we can now say with relative certainty that the bottom 18 bugs in the backlog will never rise to the top and be fixed in any reasonable time frame

• Now we just close the bottom 18 bugs in the bug backlog, and begin our work with our healthy bug backlog

But we need to ensure that the bug backlog remains healthy. We keep it healthy by reviewing it at the end of each sprint. Look at the new bugs that have come in. Prioritize the backlog, and close all bugs that exceed the threshold, in this case 12. 

But what if we have 14 bugs that we know are all of high value, and something that we will fix in the coming 3 months? What if at this moment in time we have had a high concentration of high value bugs? Then you make an exception, and allow the bug backlog to grow ever so slightly. But as soon as you notice that a bug has stayed in the backlog for more than 3 months, you automatically close it, because it shows that your initial assumption that it would be fixed was wrong. 

Are there risks to closing bugs that are of moderate importance, because you think they will never rise to the top of the backlog? Of course. 

Is there a risk that a previously closed bug will become more important as time goes by and context changes? Of course. 

But it is a risk we have to take, as the gains of working with a healthy bug backlog outweighs it. 

And you can always reopen bugs if you suddenly have more capacity than expected, if context changes, and if some old bug suddenly increases in importance. 

The numbers in this example (number of bugs, inflow, bandwidth, time frame, etc.) need to be tailored to every specific project and team, and this needs to be kicked off with the entire time so that everyone understands how and why we are doing this, and how we want to work with our new healthy bug backlog.

Johan Hoberg