This is part of our ongoing series about Agile in practice. You can get up to speed on where this post by starting where we began.
Issue
Nuve was fresh off a win fixing the vast majority of our release quality issues. Our QA environment was a great end-to-end testing ground.
But one day we had a bug slip through that end-to-end QA process. On top of that, a developer on the team had already merged a new feature int our QA environment that wasn’t fully tested. That meant we couldn’t make the fix on QA and release immediately to production - there was too much risk that the new code on QA could lead to new bugs.
We needed a way to release a fix to production bypassing our normal QA process.
Solution
Agile Brainstorming: Options for Solving
Possibility 1: Make a fix on QA, Test, and Pluck that Commit into Production
This is a promising idea. Theoretically you can make a fix and move that fix into QA and then pluck just that change from QA into production, ignoring all other changes on QA.
In practice, though, it’s kind of messy to do that. Not a lot of developers - even very good ones - know how to pluck just one or two commits from one branch into another. It’s a non-standard workflow with lots of room for error. Also, the more attempts there are at a fix (i.e. the more commits there are) the more complicated this pattern becomes.
We wanted something a little bit more foolproof when working with production so we passed on this idea.
Possibility 2: Hotfix Workflow - Fix on Production, then Move those Changes into QA
This idea is exactly what it sounds like. We create a feature branch off of main
(production - like we used to when we used the GitFlow workflow). Then make the changes we needed to make to fix the bug. Then we merge the feature branch right back into our main
branch AND into the develop
(QA) branch when we prove the fix works.
We only do that for speed when we know that there is a bug on production and we can’t do a full release from QA to production. It’s like a safety release valve for making fast fixes.
This was the process we chose to do.
Implementation: The Workflow
Because we had separated our QA (develop
) and production (main
) branches, we could do the following:
- Create a branch
hotfix-xyz
off of main
- Make the fixes we need to make
- Merge
hotfix-xyz
back into main
- Deploy to production and test that it works (repeat if not)
- Merge
hotfix-xyz
into develop
once it works as expected
At the end of that process, both develop
and main
(QA and Production) have the fix and we didn’t risk merging anything from QA to production that wasn’t fully tested. We are confident about this because develop
never merged into main
!
Further, we always were doing the same simple deploy process
- On production - pull code from
main
- On QA - pull code from
develop
Just like any other release. No magical git commands required.
Results
Because there was little overhead associated with fixing production and the flow was simple, everybody on the team could do a hotfix and we knew we could do it quickly. We saw this in results. Our average time from production bug discovery to fix dropped from several hours to 30 minutes after this change was implemented.
But arguably the biggest win with this process was that it inspired confidence. We knew that our QA environment (develop
branch) was open for new features as soon as we completed a full release to production because we could always hotfix production if necessary.
We rarely hotfix because our QA process is strong - there have been 3 or 4 hotfixes in the last several months - but knowing that we can hotfix enables us to work quickly in QA.
Want to code like the top 1% of elite SAP developers?
Elevate your skills with exclusive insights, expert tips, and cutting-edge best practices. Don't miss out – become part of the elite SAP community today!