# How a Manual Database Change Brought Salesforce down from Cloud Nine

> Date: May 20, 2019 | Author: Yaniv Yehuda | Category: Database DevOps

Over the weekend, Salesforce, a tool used by 150,000 companies and millions of people around the globe, experienced one of a global enterprise corporation's worst nightmares: a database outage. According to Salesforce, "On May 17, 2019, Salesforce blocked access to certain instances that contained customers affected by a database script deployment that inadvertently gave users broader […]

Over the weekend, Salesforce, a tool used by 150,000 companies and millions of people around the globe, experienced one of a global enterprise corporation's worst nightmares: a database outage. According to Salesforce, "On May 17, 2019, Salesforce blocked access to certain instances that contained customers affected by a database script deployment that inadvertently gave users broader data access than intended".

In other words, an error in a manual database script deployment led to, in many instances, complete revocation of authorizations and permissions in place, leaving things wide open for all users to make any and all changes they desired. Salesforce had gone well beyond the leaky sieve stage and jumped directly to "The door's wide open, come on in!"

Companies, from global industrial behemoths to small startups all protect their databases with the ferocity of a mother bear protecting its cub, and for one simple reason. Database downtime, as estimated by Gartner, has an average cost is $5,600 per minute. According to this report by Statista, nearly 60% of companies reported the average hourly downtime cost of their servers as being between over $400,000. For nearly one-third of the companies who responded, each hour of downtime cost $1,000,000 or more, and 14% estimated each hour at OVER $5,00,000. Yes, FIVE MILLION dollars for each hour of downtime.



But surely, if downtime costs so much and companies all were aware of this, they would take steps to ensure this doesn't happen, right? On the surface, you'd be forced to agree, because the people running global enterprises such as Salesforce didn't get to where they were by not being smart. If you scratch just a bit deeper, though, you'd see why this assumption turns out to be false.

The world of DevOps took the world by storm earlier this decade, with the drive towards mainstream coming with the help of Gene Kim and his seminal book "The Phoenix Project". Since then, most major companies have adopted DevOps and CI/CD as the way to move forward. This has held true for the world of software development, application development and a multitude of other fields. The only area which DevOps seems to not have penetrated is the database. In the 2018 DBmaestro DevOps Survey, two-thirds of the respondents (66%) replied that manual execution of changes was still the main method of performing changes to the database.



Salesforce seems to have fallen in the same hole, running manual database changes instead of automating them. As we just saw, this leaves themselves open to massive errors of the type which sidelined them for over half a day. Running manual changes exposes the databases to configuration drifts, errors in the code, and overall sloppy handling. Let's face it, even the best DBAs are still only human, and we all make mistakes. It is exactly for this reason why you need a database automation platform, to remove the human factor and to decrease errors and downtime.

Using a platform such as the DBmaestro DevOps Platform, which provides a full set of database DevOps tools, such a massive downtime could have been avoided. DBmaestro's visual database pipeline builder, drift detection, prerun and checks, security & governance, version control and business monitoring combine to allow DBmaestro to provide a comprehensive DevSecOps platform. Integrations with dozens of other platforms, including Jenkins, Bamboo, TeamCity, jFrog, Sonatype, IBM UrbanCode, Azure DevOps, git, Bitbucket, JIRA, Actifio, CyberArk and numerous others ensure that regardless of what your DevOps environment looks like, DBmaestro can help.

According to Salesforce, their service disruption lasted slightly over 15 hours. Even using extremely conservative estimates, that downtime could have potentially cost them over $5 million. Higher estimates place the cost at over $15 million. If you don't want a similar catastrophic downtime to take out your databases, request your demo now!

## End-to-end database CI/CD, governed and audited.

See how DBmaestro accelerates your release pipeline while keeping every change secure.

[Book a demo](https://www.dbmaestro.com/schedule-demo/) | [Guided Tour](https://demo.dbmaestro.com/demo/cmdeiq2j900lvvm0ibx1296yt)

## Related Articles

### [Db2 DevOps: Automate Releases Without Losing DBA Control](https://www.dbmaestro.com/blog/database-devops/db2-devops-automate-releases/)

Your Db2 DevOps team already runs one of the most disciplined change processes in the enterprise. This series is about making it count for more. Ask a good barista how they pull an espresso and they’ll talk about grind, dose, temperature and timing. Ask a Db2 team how they release a change and you get […]

### [Stateful Is the New DevOps Fault Line](https://www.dbmaestro.com/blog/database-devops/stateful-is-the-new-devops-fault-line/)

DevOps has transformed how enterprises build and deliver software, but the stateful database remains its hardest problem. Teams package applications into containers. They increasingly define infrastructure as code. CI/CD pipelines automate testing and deployment. When something goes wrong, teams can often replace an artifact, redeploy a previous version, or recreate an environment. But one part […]

### [From Provisioning to Control: The New Database DevOps Reality](https://www.dbmaestro.com/blog/database-devops/new-database-devops-reality/)

New Database DevOps is entering a new reality, and serverless databases are becoming the next programmable infrastructure. That was my biggest takeaway from IBM's announcement of the Db2 Serverless technical preview. At first glance, it looks like any modern cloud database: instant provisioning, automatic scaling, pay-per-use, and no infrastructure to manage. But the more I […]

---
Canonical URL: https://www.dbmaestro.com/blog/database-devops/salesforce-down/
