Abstract: Controlled experiments, or A/B tests, are the gold standard for optimizing websites. From Amazon's checkout flow to Google's search results, A/B tests can help companies improve workflows to download software, sign-up for subscriptions, click on content, or make a purchase. But, controlled experiments have roots far beyond optimizing websites for conversion. In its simplest form, controlled experiments help establish a causal relationship of a change for a target audience.

 

Microsoft takes the basics of controlled experiments, and applies the concept in unsuspecting places to help software development teams restructure the way they think about shipping even the most mundane changes. This talk will give you a flavor of how Microsoft products like Bing, Office, and Xbox to weave controlled experiments into the day-to-day software engineering processes. From trusting the data to using it to make backlog decisions, you will learn how controlled experiments are an essential tool to making informed decisions at Microsoft.

Short Bio: Jennifer Beckmann is a Principal Engineering Manager for the Microsoft Office Experimentation team. Her team is responsible for the programming interface and development lifecycle surround the rich set of feature flags within Office's client and online versions. Prior to joining Office she was a Principal Program Manager on the Microsoft Experimentation team (ExP) where she helped guide and develop the Experimentation services that runs hundreds of experiments every day across a variety of products. Jennifer earned her PhD in database systems from the University of Wisconsin-Madison and early in her career she was a developer in the SQL Server team.