Issue

Nuve knew customers wanted to host their systems on the cloud hosting provider of their choice within their own cloud environment. We had set clear objectives and leadership to carry the project through to completion.

We take a hands-off approach to managing implementation, embracing and enabling ownership and autonomy. The lead on this project decided to release this functionality all at once, suggesting that a big push would be faster and cleaner. We targeted six weeks, give or take a couple of weeks, to complete the project.

About a month went by and nothing had been released. Verbal updates were coming, but there wasn’t any code yet.

So we started to inquire about timelines. The response: next week code would be ready for review. Then next week came and went with no review. The timeline was updated to the following week. That deadline came and went. And that cycle continued a couple more times.

So we hit pause and pulled the project lead aside. We learned that the person was feeling stuck. They had made progress, but finishing was proving challenging. So we asked them to share the changes thus far so we could get more insight into what was necessary to finish the project.

After seeing the outstanding changes, a few tactical things became clear.

The changes touched all parts of the codebase, which was leading to code conflicts

Almost every time another team member released some code, the lead on this project would have to re-do some of their code to incorporate the conflicts from the newly added code. This was very cumbersome and tiring.

Partial progress had been made on multiple integrations, but no single integration was complete

Part of the reason that nothing had released was that nothing was complete. Releasing the changes as is would’ve broken existing functionality and would not have added anything new. If we wanted to release everything at once, it would be a while before everything would be done.

A lot of effort had been devoted to refactoring code, which increased the risk of bugs in existing functionality

On top of adding new functionality, we were re-writing old code in new ways to try and simplify the overall code environment. This was a good idea in theory, but it added a new layer of risks to an already large set of changes - we might accidentally break old functionality when releasing new functionality.

In Summary: This Change Had Become Way Too Big

The issue here was the execution strategy. We were trying to change too much all at once, and the complexity of releasing everything at once had become unmanageable. Too many things could go wrong.

We needed a different path.

Solution

We transitioned leadership of this project to a team member had experience dealing with situations like this. They outlined the following plan.

Step 1: Merge in the Outstanding Changes and Ensure Existing Functionality Works

The point of this step was to move the unmerged changes into the codebase. That would enable the team to integrate new features into the project code in a distributed fashion, rather than forcing the team managing this change to integrate everybody else’s work.

That, then, would free up the the project team to make progress on completing the work rather than fixing conflicts.

This had to be done quickly, within one release cycle. Because if it took longer, it would block others’ work.

Step 2: Release Functionality Feature-by-Feature Across Multiple Releases

After stabilization, we prioritized releasing user facing functionality frequently so we wouldn’t get stuck with large chunks of unmerged code again. This would enable us to demonstrate progress more clearly as well, and with estimated dates for each chunk of features to be released, we would know if we were ahead of or behind schedule.

We would test these new features as we went to ensure the ongoing stability of the user experience.

The principles we followed here were our critical learnings and are outlined in full below.

Results

The team completed Step 1 - merging in big changes and ensuring existing functionality works - within one week of transferring ownership of the project. That paved the way to work more quickly, releasing in chunks, as expected.

The remaining features were released in weekly or twice-per-week releases over the course of the next few weeks. Because we were exposing well-prioritized features as we went, we were able to set up our first customer on a new cloud provider one week ahead of our scheduled launch date. The functionality had not yet reached our full vision for the project, but it was sufficient to meet their needs.

The full planned implementation was completed one week behind schedule, but no customer work was directly affected by missing that internally imposed deadline. We had achieved the customer milestones ahead of schedule because we released bit-by-bit.

Actionable Take Aways: Principles for Releasing Features Chunk By Chunk

Releasing functionality chunk by chunk was the most critical learning from this experience. In fact, if we had followed this practice from the beginning, we wouldn’t have become stuck in the first place.

Now all of our development follows these principles.

Principle 1: Small PRs (Smallest Possible User-Complete Feature)

If you’re making a bug fix, just fix the bug. The change could be just one line. That’s fine because it’s easy to review, easy to test, and easy to call done.

If you’re releasing a feature, just finish the feature. Package it up, get it reviewed, and ship it out. Then do the next change.

It’s tempting to expand things, but remember you can always expand things after the first working version is shipped.

Principle 2: Separate Features and Refactors / Cleanups

One of the most common ways that small things become big is when developers decide, “oh, while I’m in here, I’ll also make this other change.” Not saying you should never do that, but 95% of the time it’s best to finish what you’re doing first. Then ship it, then go back and make the other change.

When in doubt, try a 5 minute rule. If you can do the cleanup or refactor in 5 minutes, go ahead and add it to the current feature work. Set a timer (not kidding). The change can’t be that big if it can be done in five minutes. But if it’s been 5 minutes and the change is not done, be really, really honest with yourself. The first estimate was wrong. Maybe there was something unexpected that also needed to happen that added complexity, or there was another “little fix” that popped up.

There’s always a reason to try to do more or to add something else, but seriously, just finish the original feature. After that feature is done and in review, start the bigger refactor or cleanup in a new PR.

And if you realize the current feature requires a big refactor to be finished, that’s something to flag to the team ASAP. You’ll probably want to refactor first (in its own PR) and then implement the feature (in a separate PR).

Principle 3: If You Must Launch All At Once, Still Release in Pieces, But Hide The Pieces Until Release

Let’s say the business wants to release a big feature set or even a whole product all at once, big bang style. That might sound like a justification for making a whole bunch of changes and merging them all at the same time.

Don’t do it.

It’s generally better to hide the upcoming changes, release them in little bits, and test as you go. It usually doesn’t take long to hide the features. Hack in a little change so that only the internal team can see it (this is known as a “feature flag”).

Then when you’re a week from the release date and leadership asks, “How’s this going?” you can calmly show them where what’s done explain it’s already working in production today. You’ve just hidden it and are completing final testing.

Then tell a story about a botched release you experienced when everything was merged on release day and explain how you wanted to avoid that experience. Ask for a promotion when the release happens seamlessly.

Summary Principle: Merge Changes as Quickly as Possible

If you’ve done the above, this should become easier. The point here is that you want to avoid code change conflicts between developers, if possible. If you use git, it’ll flag conflicts them, but not having them at all is smoother.

Our team moves pretty fast. Fast enough that if our code has been sitting on a branch for more than a week, odds are it’s starting to get stale and will run into someone else’s work. This “week long” rule of thumb changes under the following circumstances.

Bigger Changes Need to be Merged Faster

The more files or objects your changes touch, the more likely you will run into a conflict. Make those changes fast, test them (well but) fast, and merge them fast.

Faster Moving Teams Need to Merge Faster

This sounds like it’ll happen naturally, but think about it like driving on the highway. You need to move with the pace of traffic. If everybody else is going 60 MPH (100Km/h) and you are going 30MPH (50Km/h), you’re gonna slow everyone down because they have to weave around you.

Given a Fixed Codebase Size, Bigger Teams Need to Merge Faster

Think about fitting 5 people into a small room. You have some space between everyone and are unlikely to step on each others toes. Now put 30 people in that room and the odds of people stepping on others' toes goes way up.

Same with code. Merging your code is like stepping out of the room to free up space. So do that a lot, but also note there is a limit to how many people you can have working on a certain size codebase.