All work

Conversion-rate prediction for ad bidding

What
The models that predicted how likely an ad click was to become a sale, for each product, keyword and hour, on Amazon, Walmart and Tmall.
Result
Up to about 20 million predictions a day at peak on the largest pipeline, feeding live ad bids. Data checks that stop a bad run before it reaches the bids.
My role
Owner from February 2023 through two role changes. I later trained the colleague who runs it now.
  1. Data
  2. Checks
  3. Train
  4. Compare with current model
  5. Score 36 hours ahead
  6. Checks
  7. Bids

The daily run. Either set of checks could stop it.

Read the full story

The models were gradient-boosted trees: CatBoost, after a migration from XGBoost. A daily batch run forecast conversion rates 36 hours ahead at hourly granularity. I moved training from CPU to GPU and built multi-GPU training for all the pipelines. One of them runs on it in production.

The part I would point to is the safety net. After a data-quality incident that I traced to a change upstream of the models, I built checks that ran on the training data and on the predictions themselves. Small deviations sent an alert. Large ones failed the pipeline closed, so a bad day of data stopped the run.

Before a new model replaced the current one, its predictions were compared with the current model's, and any divergence had to be justified on validation data.

CatBoost, XGBoost, SageMaker training jobs, Airflow, S3, Streamlit for the monitoring dashboard.