Spotify introduces new product to help software development teams with with A/B testing

Spotify is introducing a new commercial product for software development teams called “Confidence,” which is based on its homegrown experimentation platform. Confidence is designed to make it easy for teams to set up, run, coordinate, and analyze their own user tests so they can quickly optimize their ideas. Confidence is currently only available as a private beta.

“Spotify’s data scientists and engineers have been developing and honing our product testing methods for years.” the company wrote in a blog post. “Whether it’s automatically coordinating simultaneous A/B tests or orchestrating the rollout of an AI recommendation system across mobile, desktop, and web, the platform we built scales experimentation best practices and capabilities to all our teams. Soon this experimentation platform will be available to any company that wants to build, test, and iterate ideas the way we do at Spotify: quickly, reliably, and with confidence.”

Spotify says Confidence is designed to help software development teams whether they have outgrown their current testing platform or are looking for a quick and easy way to get started with A/B testing.

The music streaming service says its experimentation efforts in the early 2010s when a few data scientists and engineers started conducting small A/B tests internally. Although these tests were manual and error-prone, Spotify understood the importance of experimenting and wanted to get better at it. The company then decided to build its own basic A/B testing platform, which it called ABBA.

The platform did feature flagging and analysis for a set of standardized metrics. Spotify says this unlocked a wave of experimentation across the company, and that it grew from running fewer than 20 priority experiments per year to running hundreds of experiments per year across multiple squads.

Confidence will be available to customers in three ways. First, it can be run as a managed service, which means you can access the experimentation platform as a standalone web service managed Spotify. Second, you can run it as a Backstage plugin. Last, you can integrate the Confidence platform into your own infrastructure via APIs.

Spotify has previously detailed its experimentation platform, and is now making it public, which is an interesting move for a company that it is mostly seen as a consumer-facing service.

Software development teams can now sign up for the waitlist to be eligible for an invite to Confidence. Spotify did not say when it plans to make Confidence more widely available.