# I Got A Surprise Lesson In Database Change Control In Front Of 150 People

> Date: March 29, 2017 | Author: Yaniv Yehuda | Category: Database DevOps

For years, I’ve been preaching the gospel of rigors database change control and the importance of automating who can make what changes to the database, and when. As part of my efforts to educate the public about database change control, automation, continuous delivery and DevOps, we hold regular webinars and  demos, which are widely attended.   The Scenario A few days […]

For years, I’ve been preaching the gospel of rigors database change control and the importance of automating who can make what changes to the database, and when. As part of my efforts to educate the public about database change control, automation, continuous delivery and DevOps, we hold regular webinars and demos, which are widely attended.

 
The Scenario
A few days ago, I was presenting at a live webinar on the importance of safe release automation for financial service enterprises. Towards the end of the webinar, I simulated a case of configuration drift in production due to a manual hotfix.

 

The idea was to demonstrate how DBmaestro would prevent me from pushing through the changes and overriding the hotfix.

According to my demo scenario I was supposed to manually undo the change, and then push the automated update forward successfully.


Uh-oh
This is where I started running into problems. Because I had done this part manually, I failed to fully revert my changes. So, when I selected to run my automated pre-deployment validation, DBmaestro prevented me from going down that path. It prevented me from deploying over the drifted configuration, which in a real-world scenario, could potentially cause a costly downtime.



Because this was all happening in a live webinar with 150 people watching, I didn’t realize all this at the time. All I saw was that glaring red light and felt the urgency to keep things moving. Under this pressure, I moved on to demonstrate the next scenario and did not deploy to production.

After the webinar ended and I had some breathing room, I went back and looked at the warning messages and the audit trail we had logged. It was then that I realized the mistake was all mine.



When I tried to revert the change, I had introduced into production I had only reverted part of it - the comment, but I failed to continue and revert the missing columns. Hence, I had not reverted to the correct configuration. For that reason, DBmaestro kept failing.

 
The Lesson
To me, there are a number of lessons here. Conducting a webinar in front of a large, live audience is comparable to working under the pressure of deploying changes to a large and critical database. I had played out this scenario dozens of times, but under these circumstances, I made a mistake. We are all humans and we can all make mistakes when doing something manually.

The way it turned out, the demo gods forced me to show live and in real-time, how downtime happens, and how automation and the correct database change control tool can prevent it.

It ended being a fantastic way to demonstrate the entire point of the webinar: when working manually, mistakes are unpreventable.

So how do we prevent downtime? We automate, we validate configuration, we enforce policy, we enforce security roles, and we audit.

In the scenario played out in this webinar, damage to production was prevented.

## End-to-end database CI/CD, governed and audited.

See how DBmaestro accelerates your release pipeline while keeping every change secure.

[Book a demo](https://www.dbmaestro.com/schedule-demo/) | [Guided Tour](https://demo.dbmaestro.com/demo/cmdeiq2j900lvvm0ibx1296yt)

## Related Articles

### [Stateful Is the New DevOps Fault Line](https://www.dbmaestro.com/blog/database-devops/stateful-is-the-new-devops-fault-line/)

DevOps has transformed how enterprises build and deliver software, but the stateful database remains its hardest problem. Teams package applications into containers. They increasingly define infrastructure as code. CI/CD pipelines automate testing and deployment. When something goes wrong, teams can often replace an artifact, redeploy a previous version, or recreate an environment. But one part […]

### [From Provisioning to Control: The New Database DevOps Reality](https://www.dbmaestro.com/blog/database-devops/new-database-devops-reality/)

New Database DevOps is entering a new reality, and serverless databases are becoming the next programmable infrastructure. That was my biggest takeaway from IBM's announcement of the Db2 Serverless technical preview. At first glance, it looks like any modern cloud database: instant provisioning, automatic scaling, pay-per-use, and no infrastructure to manage. But the more I […]

### [Scaling Elite Software Delivery: A Blueprint for Modern Engineering](https://www.dbmaestro.com/blog/database-devops/scaling-elite-software-delivery-a-blueprint-for-modern-engineering/)

Elite software delivery teams share one trait: they accelerate without sacrificing quality. To achieve consistent, high-velocity delivery, organizations must move beyond manual, fragmented processes and embrace a strategy centered on simplicity, automation, and platform engineering. If you are looking to transform your software delivery lifecycle (SDLC), here is a blueprint for building a high-performance, autonomous […]

---
Canonical URL: https://www.dbmaestro.com/blog/database-devops/database-change-control/