This is part of our ongoing series about Agile in practice. You can get up to speed on where this post by starting where we began.

Issue

Nuve was fresh off a win fixing the vast majority of our release quality issues. Our QA environment was a great end-to-end testing ground.

But one day we had a bug slip through that end-to-end QA process. On top of that, a developer on the team had already merged a new feature int our QA environment that wasn’t fully tested. That meant we couldn’t make the fix on QA and release immediately to production - there was too much risk that the new code on QA could lead to new bugs.

We needed a way to release a fix to production bypassing our normal QA process.

Solution

Agile Brainstorming: Options for Solving

Possibility 1: Make a fix on QA, Test, and Pluck that Commit into Production

This is a promising idea. Theoretically you can make a fix and move that fix into QA and then pluck just that change from QA into production, ignoring all other changes on QA.

In practice, though, it’s kind of messy to do that. Not a lot of developers - even very good ones - know how to pluck just one or two commits from one branch into another. It’s a non-standard workflow with lots of room for error. Also, the more attempts there are at a fix (i.e. the more commits there are) the more complicated this pattern becomes.

We wanted something a little bit more foolproof when working with production so we passed on this idea.

Possibility 2: Hotfix Workflow - Fix on Production, then Move those Changes into QA

This idea is exactly what it sounds like. We create a feature branch off of main (production - like we used to when we used the GitFlow workflow). Then make the changes we needed to make to fix the bug. Then we merge the feature branch right back into our main branch AND into the develop (QA) branch when we prove the fix works.

We only do that for speed when we know that there is a bug on production and we can’t do a full release from QA to production. It’s like a safety release valve for making fast fixes.

This was the process we chose to do.

Implementation: The Workflow

Because we had separated our QA (develop) and production (main) branches, we could do the following:

• Create a branch hotfix-xyz off of main
• Make the fixes we need to make
• Merge hotfix-xyz back into main
• Deploy to production and test that it works (repeat if not)
• Merge hotfix-xyz into develop once it works as expected

At the end of that process, both develop and main (QA and Production) have the fix and we didn’t risk merging anything from QA to production that wasn’t fully tested. We are confident about this because develop never merged into main !

Further, we always were doing the same simple deploy process

• On production - pull code from main
• On QA - pull code from develop

Just like any other release. No magical git commands required.

Results

Because there was little overhead associated with fixing production and the flow was simple, everybody on the team could do a hotfix and we knew we could do it quickly. We saw this in results. Our average time from production bug discovery to fix dropped from several hours to 30 minutes after this change was implemented.

But arguably the biggest win with this process was that it inspired confidence. We knew that our QA environment (develop branch) was open for new features as soon as we completed a full release to production because we could always hotfix production if necessary.

We rarely hotfix because our QA process is strong - there have been 3 or 4 hotfixes in the last several months - but knowing that we can hotfix enables us to work quickly in QA.