Spotify started using Google Cloud Platform (GCP) in 2016, with investments of $450 million over three years. For Google, the music streaming platform became a reference customer, not only for its brand and scale, but also for its reputation as an engineering-centric and data-driven company.
Since then, Spotify has closed both of its US-based data centers and will be free of on-premise infrastructure by the end of the year, following a complex migration.
Why migrate?
Ramon van Alteren, director of engineering at Spotify, comments that if you think about the amount of effort required to maintain the compute capacity, storage, and network of a global company serving over 170 million users, it is a considerable amount of work.
In addition to preventing developers from worrying about infrastructure provisioning and maintenance, the company also wanted to leverage some of Google Cloud's innovations, specifically the BigQuery cloud data warehouse, Pub/Sub for messaging, and the DataFlow tool for batch and stream processing.
Service migration: step-by-step
The current migration plan was formulated in 2015 and split into two parts: services and data. The service migration focused on moving nearly 1,200 microservices from data centers to Google Cloud Platform.
The three main goals during the migration, according to van Alteren, were to minimize disruption to product development, finish as quickly as possible to avoid the cost and complexity of running a hybrid environment, and ensure that Spotify had no services running in its data centers.
One of the first actions the companies took was to build a small migration team of engineers and create a real-time visualization of the entire migration state so that engineers could watch and see the project's progress.
This visualization looks like a set of red (data center) and green (Google Cloud) bubbles, with each bubble representing a system and the size of the bubble representing the number of machines involved.
The service migration began with mapping dependencies, as the architecture at Spotify means that each microservice relies on 10 to 15 people to serve a customer request. This meant that a "big bang" migration, where everything stops, was not an option, as customers expect constant service uptime.
Instead, Spotify's engineering teams were tasked with moving their services to the cloud in a two-week sprint, during which they effectively halted all product development. This also allowed these teams to start assessing their architecture and decommission anything unnecessary.
One thing Google Cloud did specifically for Spotify during the migration was the Virtual Private Cloud (VPC) option. "This allows you to build something similar to an internal network that connects multiple projects, and they can cross-communicate," said van Alteren.
This allows teams to have good control over their demands, enabling them to do what they need to do, and if something goes wrong, it only affects that sector, not the entire company.
Once the migration was in full swing, the central migration team began secretly inducing failures in these cloud systems, recording how the teams reacted to the new architecture.
"This helped ensure that monitoring systems were properly extended to the new cloud deployment; if a team didn't notice, we ultimately had this playbook where they could start using cloud failure modes they might not have had in the past," explains Peter Mark Verwoerd, solutions architect at Google.
By May 2017, each migration sprint was completed, and traffic was being routed to Google Cloud. Then, in December 2017, Spotify reached 100% of users and had already closed the first of its four data centers. Since then, the second data center has been closed, and the last two, both in Europe, will be shut down by the end of this year.
Data migration
Due to a highly complex dependency graph, it was a challenge to move 20,000 daily data jobs to GCP without causing pipeline failures, according to Josh Baer, senior product manager for Spotify's machine learning infrastructure.
Spotify began by evaluating the possibility of a 'big bang' migration. However, even with a 160-gigabit-per-second network link, it would take two months to copy the data from the Hadoop cluster to Google's infrastructure. "We wouldn't be much of a business if we went dark for two months," he added.
The best strategy, then, was to copy the data. "As you move your job to GCP, you copy your dependencies, and then you can run your job," he explained. "Then, if you have downstream consumers, you might have to copy the output of your job back to our on-premise cluster so they don't break. Since most of the data migration lasted six to 12 months, we were running many of these jobs to fill gaps in our dependency tree."
Lessons learned
Max Charas, cloud engineer at Google, warns: "this migration strategy is highly customized for Spotify, so for anyone else wanting to do something like this, it might look very different."
The company learned some key lessons from the migration. The first was preparation. "We prepared probably two years before the migration, and each migration took about a year. We tried to create a minimal use case to show the benefits of moving to GCP, but it couldn't be a small thing to show the true value," says Charas.
Second was focus. For Van Alteren, it is truly amazing what can be done with a team of engineers focused on a single thing. This also helps business partners, who are happier with a short period without product development rather than a long, drawn-out period.
The third was building a dedicated migration team to act as a buffer to help them know what they need, pass on past experiences and learnings, and provide just the resources they need.
The last was "getting out of hybrid as quickly as possible—all of those copy jobs are expensive and complex," said Baer.
Results
With the migration, developers have more freedom and greater scale, without sacrificing quality of service. "Quality of service is something we measure diligently, and there was no degradation," said Van Alteren. "The benefits include our event delivery pipeline, which loads royalty payments for rights holders. When we moved to the cloud, the pipeline carried a peak of 800,000 events per second, and now it loads three million per second," he concludes.
"And what companies can learn from it; The migration was complex, but now developers have more freedom and greater scale."