This is part of our ongoing series about Agile in Practice - Nuve Platform's own iterative journey to ship higher quality software faster.

Issues

It was essential that we:

• Work very quickly - we needed to be able to get a minor feature from idea to production in a day or less
• Fix bugs in minutes - when issues arose, we had to be able to diagnose them quickly, test them, and release a fix in minutes to hours
• Test outside of production - because testing in production is risky and costly

Many of our team members had years of experience working in shared development environments (within SAP and other enterprise systems). We would inevitably run into the following types of issues which prevented quick development and fixes, testing.

Object Locks (AKA: two people trying to change the same thing at the same time)

"Object Lock" is a fancy term for one person "locking" a part of the code (the "object") so that another person can't work on it.

The lack of object locks enables a cool trick that many young pranksters discover and it's a funny to mess with other people overwriting their work once or twice when you're a kid ("Haha look at how confused Frank is!"), but it's a huge productivity loss.

So locking is necessary in a shared environment because people could accidentally (or intentionally) work on the same thing and overwrite each others work, making progress impossible.

Break Points

When we worked in shared environments, sometimes a different developer, let's call him Joe, puts a "break point" into the code. A break point stops the program from running to allow Joe to inspect code, which is very helpful for him.

But when Joe forgets to tell you about that break point, the system stops working and you think it's, well, broken. And maybe it's your fault? So you run down a rabbit hole of trying to find the solution to some issue, completely unaware that it has nothing to do you or your work.

Later in the day Joe says, "Hey, done working with that break point, things are working again." And you curse your lost hour of investigating a phantom error. And Joe.

But if you really think about it, Joe telling you about the break point only kind of helps. If testing your code requires you to run anything that touches the break point, you still have to wait until he's done with it to test your work unless you have something else to do. And a lot of people don't.

Predictable (and High Quality) Data

We all had experiences changing systems that have little data or bad data. And it was bad for our code and our productivity. Edge cases were missed. Code was harder to test. Some Issues arose with bad data that wouldn't arise with real data that had to be handled. The final result: Less confidence when releasing and more bugs in production.

How Isolated Environments Helped

Using isolated environments solved so many speed and quality issues so quickly that it amazes us why people would ever choose to work in shared environments (yes costs, but we'll address that later).

Briefly isolated environments allowed us to:

Bypass "Object Lock" and "Break Point" Issues Entirely

The easiest way to prevent object locks and break point issues: Replicate the whole codebase into two isolated systems. You get one, Joe gets one, and Frank gets one.

When the development environment is yours alone, Joe can't set a breakpoint in your system to surprise you without you knowing it. And you can't prank Frank by locking his objects (or overwriting his work), nor can he affect your work.

Develop with High Integrity Data

We copied non-private data from production to development environments and filled in around it with anonymized or dummy data. So our data essentially was production data.

By creating predictable and reliable data in our isolated development environments, we were able to test our features in a controlled environment that approximated production nearly perfectly. It significantly reduced the risk of bugs and errors in the production environment vs. developing in environments with mediocre data.

Results

Giving isolated environments to each user mostly solved all of the above issues, and it enabled us to achieve our goals in a straightforward fashion:

• Work quickly - any developer could spin up a new system and write the small feature immediately then ship it to qa/production.
• Fix in minutes - Replicate the production conditions in your high-data-quality development environment, find the issue, fix it, then ship the fix.
• Test outside of production - because our data was high quality, we could do basically all testing outside of production prior to releasing.

This was the first pillar that allowed us to be properly agile, adapting quickly to changing needs and shipping changes iteratively.

But Costs?

Yes, giving each developer an isolated development environment sounds expensive. Well not so much if it can all fit on a laptop, but some enterprise systems require very large compute systems that cost a few dollars per hour to run.

It's true, they are expensive to run if you run them 24/7. A $3.00 / hour system costs $72 / day to run or $26,280 per year to run.[1] But if you only use the systems when the developer is actually working or testing - call that 2,000 hours per year - you can reduce the cost to $24 / 8 hour working day or $6,000 per year. That means if a $100,000 / year devleoper is spending more than 6% of their time waiting for a shared development environment, the economical decision is to give them their own.

That type of savings is now very possible, and it's what's enabled our team to develop on big machines on a shoestring budget.

[1] That's not a ton compared to a $100,000 developer salary. If the developer spends more than 26% of their time waiting for Joe's break points or fixing bugs that could've been avoided with better data in a development system - and this CAN happen - then giving them the 24/7 system makes economic sense. Add in the costs of benefits, office space, recruiter fees, and so on, and the economics move even more in favor of giving each developer a dev environment.