SPEAKER: Julie Novak; Netflix
TITLE: “Statistical Methodologies in Streaming Experimentation at Netflix”
ABSTRACT: This talk will begin with an overview of how controlled randomized tests assist in informed data-driven decision making at Netflix. I will then go over streaming Quality of Experience and the components necessary to improve it. I plan to dive into two specific challenges associated with experiments in this space and describe the statistical solutions that we use to address them. First, the metrics we are comparing often have very ‘non-normal’ distributions and extreme outliers, so t-tests and standard non-parametric tests have undesirable properties. We develop a suite of bootstrapping tools which provide practical and statistical significance, as well as easy to interpret visualizations. Second, we need a fast, automated failure detection system to ensure that tests do not break any devices. To solve this we built a Bayesian regression model which updates its decision as data streams come in.
DATE: Thursday, October 24th, 4:10pm
LOCATION: MSB 1147, Colloquium Room
REFRESHMENTS: 3:30pm MSB 4110 (4th floor lounge)