Several years ago, continuous delivery was a new concept being tried by a small group of early adopters. Boy have things changed! Now, continuous delivery, or CD, is a practice that companies big and small are embracing as part of a new, faster moving, on-demand business culture.

We’re now hearing the term described often as being a “table stakes” obligation that companies need to pursue to compete. Companies that ante up have an advantage. Companies that don’t, fall behind. Continuous delivery is now seen as the first stepping-stone of a DevOps transformation.

The Need

Before the move to a more continuous, more agile model, Yahoo!’s Ad Exchange department was not only slow but also fragile. They launched builds to production only three times per year. Those builds frequently broke and would take a week or so on average to rebuild.
Coding would take place in intense six-week sessions, followed by eight weeks of testing. QA would sign off with a long list of exceptions and it would usually take another four weeks to launch into production.
The system had a long windup and was not very good at course-correcting once parts were in motion. This made for a fairly non-responsive and cumbersome process that to a long time to deliver middling updates.

The Challenge

You’re not going to find many joint advertising and data platforms that are busier or more complex than Yahoo! Ad Exchange. The media titan’s platforms are backed by a massively distributed system that processes over 100 billion events each day.

The system consists of hundreds of unique software components and thousands of servers. You have hundreds of programmers working in more than a dozen languages, on different teams with different priorities and schedules twenty-four hours a day.

Bringing this type of complex corporate structure into the DevOps Age is by no means impossible, but it’s most assuredly the stuff of logistical nightmares.

The Process

The first thing that needed to be tackled was the culture. To successfully implement continuous delivery, Yahoo! would need to build a culture of both inter- and intra-team collaboration, procedural controls, peer review, task chunking, smart change management, and automation.

Screwdriver-architecture-and-developer-flow.png

These sorts of wholesale changes are sure to cause a little bit of turbulence and cannot be realized overnight. Credit therefore must be given to the decision makers at Yahoo! who had clarity of vision and fortitude of conviction to stay the course and see the process through.

With culture a work in progress, attention was then given to the build process and the tools that facilitate it. Yahoo! developed Screwdriver.cd to serve as their dynamic infrastructure build system. The system was specially designed for the needs of Yahoo! to smoothly and conveniently handle deployment pipelines, to make trunk development easier and better, and to remove the hassle and delays from roll backs.

From there, it was mostly just a matter of bringing everyone up to speed, continuing to push for the required culture, securing employee buy-in, sharing wins to develop a sense of collective purpose,  and building out core competencies from within this new working model. With each passing month, the system ran better and Ad Exchange moved closer to a model for truly continuous delivery.

The Results

Today, the system hums – thanks to a process overhaul that started with the decision to implement continuous delivery. The Ad Exchange team generates more than 8,000 builds a day, committing code to production without human intervention.

challenges-and-best-practices-to-implement-continuous-delivery.png

Before the CD initiative, it was a very different story. “The same team today launches regularly,” says Yahoo! senior product architect Stas Zvinyatsokovsky.

The team goes from commit to certification in about six hours and they queue up to launch every day. If there’s a break or a security issue, it’s fixed the same day.

Zvinyatsokovsky says the Ad Exchange team has gone “from continuous debacle to continuous delivery” in the space of two years.

“Continuous delivery has become part of the culture,” he said. “Nowadays, when a new product is delivered we expect then to launch to production multiple times a day.”

Respect the Process: Implement Continuous Delivery

A decade ago, continuous delivery was seen as a new, confusing, and potentially risky proposition. Nowadays, many companies are still figuring out the best ways to integrate CD into their operations. But there can be no doubt that continuous delivery has today gone mainstream. That fact is thanks in large part to the well-publicized cases of CD success among industry giants like Yahoo!.

Companies small and large – from Choose Digital all the way up to Amazon – across industries as varied as retail, travel and digital music delivery are generating real, tangible results. They’re deploying more often, with less friction. They’re creating whole new business models and they’re beginning their DevOps transformation. Don’t you think it’s time you join the fray?

Making the Database Part of Your Continuous Delivery Pipeline? Read more to find out why is it important.

Read more about:

Continuous delivery best practices.

Continuous delivery principles

——

This article was based on a lecture given by Stas Zvinyatsokovsky.