Why not call Database DevOps simply DevOps?

Sometimes people are annoyed by the term ‘Database DevOps’. Why not call it simply ‘DevOps’? After all, Database DevOps follows the same core principles.

The answer is simple: implementing DevOps is tricky by itself, but most teams are set up to fail when it comes to implementing DevOps for databases. This makes it worth defining as a specialization.

What is DevOps?

Donovan Brown defines DevOps as “the union of people, process, and products to enable continuous delivery of value to our end users.”

Donovan’s definition encapsulates the essence of what DevOps aims to achieve: an emphasis that people are essential, a focus on collaborative processes, and the importance of tooling and automation with the goal to continuously deliver value all the way to the customers who use it.

As Donovan writes:

It is very important to realize that DevOps is not a product. You cannot buy DevOps and install it. DevOps is not just automation or infrastructure as code. DevOps is people following a process enabled by products to deliver value to our end users.

Why is database DevOps uniquely difficult?

I generally see three major challenges with teams implementing DevOps for Database code:

1. Data is stateful. Blue/Green (sometimes called Red/Black) deployment models don’t generally work for database code, as data may be lost by returning to the state of the data before deployment began. This adds significant complexity to finding the right products and building the right processes.
2. Key stakeholders get quite nervous. Data loss or data breaches are incredibly expensive and damaging for brands. This has led to much more rigid change management practices when it comes to deploying changes to databases, especially in more regulated industries.
3. DBAs and Developers are historically misaligned. Database Administrators (DBAs) are often not deeply familiar with tooling and practices that make implementing devops safer. DBAs are often trained to act as the “protectors” of production environments, as well, and to look for everything that might possibly go wrong. This positioning of the DBA role creates significant friction between DBAs and development and DevOps teams, and makes it much harder for these teams to co-create new processes.

Not everyone has all three of these challenges. Sometimes significant cultural groundwork has been laid and DBA and Development teams are aligned on a strategy. This is somewhat rare, however.

I more often see teams who are successful with Database DevOps because there is only a development team or only a DBA team involved: this tends to make things simpler because of the lack of friction that cross-team collaboration can create. However, in these cases there is often a lack of specialized knowledge that would enrich the process because a key team is missing.

But still, the first two challenges of stateful data and wary stakeholders are nearly ever-present and take significant work to manage.

Avoid the pitfalls of Database DevOps

While these guidelines can apply to any kind of DevOps initiative, they are especially critical when it comes to databases.

1. Spend a lot of time carefully scoping your MVP

The biggest mistake that I see folks make is trying to ensure they can solve the very hardest problems with stateful data at the very beginning. This often takes the form of, “we need to make sure we can fully automate rollbacks for every problem in 30 seconds in production.” Because of the nature of stateful data, different business requirements, and implications of different data architectures, there is no One-Size Fits All approach to this. Even if you solve this for one database, that doesn’t mean you can cut and paste the same approach to others in your organization. Teams can find this hugely disheartening.

The truth is, there is huge value to simply getting your database code into version control. Automating deployments to dev environments and finding ways to reset those environments automatically also will raise the quality of your code and empower developers to work faster. There are a variety of ways to scope your MVP and plan implementations, and it’s important to carefully identify the right one for your teams.

2. Talk about guardrails early and often to stakeholders

One pitfall I see is folks focussing too much on avoiding failures as the major benefit of DevOps for databases.

Things are going to break. DevOps can help you identify failures earlier in your process and mitigate risk, but implementing the right processes and the right tooling to do this, while getting folks on board to collaboratively build those processes is an iterative effort. There will be errors, there will be failures.

A strategy that helps with this is the concept of guardrails. If you don’t like that term, you can also talk about safety. The important thing to note is that guardrails– aka safety checks– can be implemented in a variety of ways. Automation is great, but takes time to build. Start with human-powered checks first. Make these human checks stronger, more resilient, and use the information you learn to prioritize how to automate these iteratively.

This will take time. It’s important to plan this time into your effort from the beginning, and ensure that leadership is aware that regular investment in the process over time will be required.

3. Culture, culture, culture

A lot of the hardest problems with Database DevOps are not technical: they are human problems. Database administrators in particular can be highly skeptical of change. I know many DBAs and DBREs (Database Reliability Engineers) who rigidly cling to the identity of the Protector of the Data.

As these data folks see it, developers who aren’t experts on the database they are using are just going to mess things up – but it’s the DBA or DBRE who is going to get blamed and have to clean up the mess.

But you know what? This point of view didn’t come from nowhere. This perspective has been set in IT culture for a long time and is still continually reinforced by tech leadership in subtle ways. There is some truth in this perspective.

For many of these teams, many conversations and a lot of listening to concerns are needed to open up possibilities for change. Bridging the gaps between other teams may take significant work as well. Teams need to be brought together in a way that they can effectively teach and learn from one another.

Although difficult, Database DevOps is worth the investment

The payoff goes back to Donovan Brown's definition of DevOps : continuous delivery of value to your end users.

Delivering value to your customers continually can be the difference between success and failure of products and entire companies. Don’t let your databases be the gating factor in your organization’s success.