Issue

After two consecutive experiences that slowed development due to trying to do too much at once, the Nuve team had learned its lessons about keeping pull requests (PRs) small. We were shipping code very quickly. The team was releasing multiple times per week, and those releases were adding up to meaningful customer facing changes.

Then we started to notice some issues. We released to production and found some unexpected user facing bugs. We caught them within a few minutes and fixed them with hotfixes. So there was no major issue. But then we had a similar issue with our next release. We found user facing bugs in production and fixed them quickly, but we could have (and should have) caught them in our QA environment.

The first release with production issues could have been just an unlucky miss, but two in a row felt more like a process issue so we dug in. What we found was that while all team members were releasing smaller bits, not all team members were fully testing their code - or asking somebody else to do so.

Solution

We needed a process. We didn’t want it to be too cumbersome, but we wanted it to be effective. We settled upon asking for the following. For each PR, the person who wrote the code must:

• Include a checklist of test cases - user actions that would prove the change worked and didn’t break anything - or
• Include in the PR a reason why a list of test cases wasn’t necessary

This may sound cumbersome, but our previous two learnings were about keeping PRs small. So we were creating small PRs. Small PRs, almost by definition, meant that the number of test cases per PR should be small. And the test cases should be exactly the test cases that the developer used to test that their own code was working in their own isolated dev environment before they created the PR.

So essentially we were asking the team members to think more carefully about the test cases from the user perspective, and then use those user facing tests to test their own code. Writing the test cases in a PR when this has been done properly is easy.

The other benefits of this process are that tests:

• Can be confirmed by someone other than the person who wrote the code on the QA system, and
• Can be run again to double-check mission critical tests in production if desired

We were essentially writing a test plan in a distributed fashion.

Results

Bugs fell basically to zero over the next dozen or so releases. It was, and continues to be, very successful at preventing production bugs.

And the burden for checking it is very low. Any reviewer should ask for a test plan whenever they don’t see it, and that should prompt the developer who wrote the PR to write the tests and to run them.

The interesting irony of this exercise is that the process of requiring the test plan almost makes running the tests unnecessary. The vast majority of the quality improvements come from the developer carefully considering what could go wrong, running the tests themselves, and ensuring they pass. They will almost certainly pass in QA and on production if they passed locally because of the quality of our team’s development environments.

We still do run the tests to confirm, though. And that gives us peace of mind that our end users won’t experience any bugs.

Aside: Iterating Toward Better With Agile Cycles

This experience highlighted a repeating pattern. First, we would do something to speed up our pace of development. Then we would encounter bugs due to the increased pace. Then we would fix the issues leading to bugs, losing a bit of speed. Then we would push to increase our development pace again, starting this cycle all over again.

This wasn’t a two steps forward, two steps back, pattern. We were continuously improving in spite of small setbacks along the way. Overall the trend has been toward faster development and higher quality releases.

It can be frustrating to notice yourself or the team forgetting and relearning lessons that could have been learned before, but this is how it goes sometimes. We’re human. This type of pattern is typical of the agile development process. Improvements are not perfectly linear, but progress is organic and continuous.