Christina
Katsimerou

Roundtables –  Experimentation without control group 

Booking.com

Christina Katsimerou

Christina
Katsimerou

Roundtables –  Experimentation without control group 

Booking.com

Christina Katsimerou

Bio

Christina works as Senior Data Scientist in Booking.com, the largest travel e-commerce company worldwide. She studied Electrical and Computer Engineering at Aristotle University of Thessaloniki, and later moved to the Netherlands to pursue her PhD in Computer Science at Delft University of Technology. For the last three years, she has been applying Machine Learning in Booking.com to problems such as dynamic pricing and fraud detection. Last year, she formed a team that is currently driving the development of an experiment tool for conducting non standard A/B experiments.

Bio

Christina works as Senior Data Scientist in Booking.com, the largest travel e-commerce company worldwide. She studied Electrical and Computer Engineering at Aristotle University of Thessaloniki, and later moved to the Netherlands to pursue her PhD in Computer Science at Delft University of Technology. For the last three years, she has been applying Machine Learning in Booking.com to problems such as dynamic pricing and fraud detection. Last year, she formed a team that is currently driving the development of an experiment tool for conducting non standard A/B experiments.

Abstract

 

Before deploying new features or products, it is a common practice to run randomised experiments or A/B tests to estimate the effects of the proposed change. Randomised experiments are considered the gold standard in experimentation because they isolate the effect of the manipulated variable from other potential causal influences. Unfortunately, such randomisation is not always possible, because data may not be available at the individual customer level. A typical example is mass marketing campaigns, where the experimenter cannot control who gets exposed to an ad. Without a control group, it is unclear how to measure the incremental value of a campaign.

 

In this round table we will discuss the limitations of A/B experiments, with a focus on experiments that lack a control group. I will present how to construct “synthetic” controls using time series, to discuss further the assumptions behind this method, what can go wrong and how we can safeguard good experimentation practices.

 

 

Abstract

 

Before deploying new features or products, it is a common practice to run randomised experiments or A/B tests to estimate the effects of the proposed change. Randomised experiments are considered the gold standard in experimentation because they isolate the effect of the manipulated variable from other potential causal influences. Unfortunately, such randomisation is not always possible, because data may not be available at the individual customer level. A typical example is mass marketing campaigns, where the experimenter cannot control who gets exposed to an ad. Without a control group, it is unclear how to measure the incremental value of a campaign.

 

In this round table we will discuss the limitations of A/B experiments, with a focus on experiments that lack a control group. I will present how to construct “synthetic” controls using time series, to discuss further the assumptions behind this method, what can go wrong and how we can safeguard good experimentation practices.

 

 

Discussion Points

  • When is it not valid to run a randomised (A/B) experiment?
  • How often is randomisation not possible in practice?
  • How can we measure the causal effect when there is no control group?
  • I will briefly describe the method of time series “synthetic” controls  (which received attention with Google’s Causal Impact Package) for estimating effects for experiments that lack control group. Together we will discuss Under which assumptions is this method applicable?
  • What data can we use to validate such methods?
  • What can go wrong?
  • How can we safeguard good experimentation practices?

Discussion Points

  • When is it not valid to run a randomised (A/B) experiment?
  • How often is randomisation not possible in practice?
  • How can we measure the causal effect when there is no control group?
  • I will briefly describe the method of time series “synthetic” controls  (which received attention with Google’s Causal Impact Package) for estimating effects for experiments that lack control group. Together we will discuss Under which assumptions is this method applicable?
  • What data can we use to validate such methods?
  • What can go wrong?
  • How can we safeguard good experimentation practices? 

Planned Agenda

Planned Agenda