Название: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing Автор: Ron Kohavi, Diane Tang
Издательство: Cambridge University Press
Год: 2020
Страниц: 200
Язык: английский
Формат: pdf (conv), mobi, epub
Размер: 10.2 MB
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.
Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to:
Use the scientific method to evaluate hypotheses using controlled experiments.
Define key metrics and ideally an Overall Evaluation Criterion.
Test for trustworthiness of the results and alert experimentersto violated assumptions.
Build a scalable platform that lowers the marginal cost of experiments close to zero.
Avoid pitfalls like carryover effects and Twyman's law
Understand how statistical issues play out in practice.
Our goal in writing this book is to share practical lessons from decades of experience running online controlled experiments at scale at Amazon and Microsoft (Ron), Google (Diane), and Microsoft and LinkedIn (Ya). While we are writing this book in our capacity as individuals and not as representatives of Google, LinkedIn, or Microsoft, we have distilled key lessons and pitfalls encountered over the years and provide guidance for both software platforms and the corporate cultural aspects of using online controlled experiments to establish a data-driven culture that informs rather than relies on the HiPPO (Highest Paid Person’s Opinion). We believe many of these lessons apply in the online setting, to large or small companies, or even teams and organizations within a company. A concern we share is the need to evaluate the trustworthiness of experiment results. We believe in the skepticism implied by Twyman’s Law: Any figure that looks interesting or different is usually wrong; we encourage readers to double-check results and run validity tests, especially for breakthrough positive results. Getting numbers is easy; getting numbers you can trust is hard!
Online controlled experiments are used heavily at companies like Airbnb, Amazon, Booking.com, eBay, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yahoo!/Oath, and Yandex. These companies run thousands to tens of thousands of experiments every year, sometimes involving millions of users and testing everything, including changes to the user interface (UI), relevance algorithms (search, ads, personalization, recommendations, and so on), latency/performance, content management systems, customer support systems, and more. Experiments are run on multiple channels: websites, desktop applications, mobile applications, and e-mail.
"This is the new bible of how to get from data to decisions in the digital age." -- Scott Cook, Intuit Co-founder & Chairman of the executive committee
Скачать Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing