Builders' Echo — Where work wins the job

Overview

Prize pool: $4500

Deadline: Jul 10, 2026, 8:57 PM

Blind judging on

Entries: 0 submissions

Entry codes are blind labels. Names appear on the leaderboard after the company turns on identity reveal for this contest.

Scoring

Calibration quality

25%

Predictive performance on hidden holdout

25%

Feature engineering depth and justification

20%

Code quality and reproducibility

20%

Engineering judgment and production readiness write-up

10%

See full rubric →

Brief

Overview

StackAdapt's bidder must estimate click-through rates (CTR) in milliseconds to decide whether to bid on an ad impression and at what price. Accurate, well-calibrated CTR models directly drive advertiser ROI and platform revenue across billions of daily auctions.

What you'll build

A production-ready CTR prediction pipeline trained on the Criteo Display Advertising dataset. You will deliver a scikit-learn–compatible model (or PyTorch/TF equivalent with a sklearn-style predict_proba interface), a reproducible training script, and a written analysis — not a notebook prototype, but code structured for hand-off to an ML Engineering team.

Requirements

Dataset: Use the Criteo Display Advertising Challenge dataset (Kaggle, train.txt, ~45.8 M rows, 13 integer + 26 categorical features, binary label click). Download URL: https://www.kaggle.com/c/criteo-display-ad-challenge/data. You must use the first 5 million rows (rows 0–4,999,999 of train.txt) as your training set. The platform will evaluate your model on a hidden holdout drawn from rows 5,000,001–6,000,000 (not available to you). Do not train on rows outside 0–4,999,999.
Your training script must be runnable end-to-end with a single command (e.g., python train.py) and produce a serialized model artifact (pickle or equivalent).
Handle missing values explicitly — document your imputation strategy in the README.
Apply at least one feature engineering step beyond raw one-hot encoding for the categorical columns (e.g., target encoding, frequency encoding, feature hashing, embedding, or cross features). Justify your choice.
The model must output calibrated probabilities (verify with a reliability/calibration curve on your local validation split — include the plot in the repo).
Report on your local validation split: Log-Loss (primary metric), AUC-ROC, and a calibration plot. The platform's hidden holdout is scored by Log-Loss only.
Restrict training wall-clock time to ≤ 30 minutes on a single modern CPU core (or ≤ 10 minutes with GPU). State the measured runtime in your README.
All random seeds must be set explicitly and documented so results are reproducible byte-for-byte.
Do not use AutoML libraries (AutoSklearn, FLAML, H2O AutoML, etc.). You must select and tune the model architecture manually.

Deliverables

A public GitHub repo containing: train.py (or equivalent entry point), predict.py (loads artifact, scores new rows), requirements.txt or environment.yml, and any helper modules.
A README.md covering: setup instructions, how to reproduce training, feature engineering decisions and tradeoffs, calibration analysis, runtime measurement, and a section on what you would do differently given more time or compute.
calibration_curve.png committed to the repo.
The serialized model artifact (pickle, ONNX, or joblib) committed or made available via a public release/link.

How we'll evaluate

Calibration quality: Reliability diagram shows predicted probabilities match observed click rates across deciles; ECE (Expected Calibration Error) is low.
Predictive performance: Log-Loss and AUC on your reported local validation split; these are cross-referenced against the hidden holdout Log-Loss to check for overfitting or data leakage.
Feature engineering depth: The chosen technique is appropriate, correctly implemented, and its impact is measured and discussed.
Code quality and reproducibility: train.py runs cleanly from scratch with a single command; seeds are fixed; no data leakage between train and validation.
Engineering judgment and write-up: README clearly explains tradeoffs, failure modes, and what a production version would need (online updates, latency budget, monitoring, etc.).

Out of scope

Building a full bidding simulation or auction system.
Real-time inference API or serving infrastructure.
Using the full 45 M rows — stick to the prescribed 5 M training slice.
Hyperparameter search exceeding the stated runtime budget.
Ensembles of more than 3 base models.

Loading content…

#	Entry	Review	Reveal
No submissions yet.

Entry

Review

Reveal

No submissions yet.

What you'll build

Requirements

Dataset: Use the Criteo Display Advertising Challenge dataset (Kaggle, train.txt, ~45.8 M rows, 13 integer + 26 categorical features, binary label click). Download URL: https://www.kaggle.com/c/criteo-display-ad-challenge/data. You must use the first 5 million rows (rows 0–4,999,999 of train.txt) as your training set. The platform will evaluate your model on a hidden holdout drawn from rows 5,000,001–6,000,000 (not available to you). Do not train on rows outside 0–4,999,999.

Your training script must be runnable end-to-end with a single command (e.g., python train.py) and produce a serialized model artifact (pickle or equivalent).

Handle missing values explicitly — document your imputation strategy in the README.

Apply at least one feature engineering step beyond raw one-hot encoding for the categorical columns (e.g., target encoding, frequency encoding, feature hashing, embedding, or cross features). Justify your choice.

The model must output calibrated probabilities (verify with a reliability/calibration curve on your local validation split — include the plot in the repo).

Report on your local validation split: Log-Loss (primary metric), AUC-ROC, and a calibration plot. The platform's hidden holdout is scored by Log-Loss only.

Restrict training wall-clock time to ≤ 30 minutes on a single modern CPU core (or ≤ 10 minutes with GPU). State the measured runtime in your README.

All random seeds must be set explicitly and documented so results are reproducible byte-for-byte.

Do not use AutoML libraries (AutoSklearn, FLAML, H2O AutoML, etc.). You must select and tune the model architecture manually.

Deliverables

A public GitHub repo containing: train.py (or equivalent entry point), predict.py (loads artifact, scores new rows), requirements.txt or environment.yml, and any helper modules.

A README.md covering: setup instructions, how to reproduce training, feature engineering decisions and tradeoffs, calibration analysis, runtime measurement, and a section on what you would do differently given more time or compute.

calibration_curve.png committed to the repo.

The serialized model artifact (pickle, ONNX, or joblib) committed or made available via a public release/link.

How we'll evaluate

Calibration quality: Reliability diagram shows predicted probabilities match observed click rates across deciles; ECE (Expected Calibration Error) is low.

Predictive performance: Log-Loss and AUC on your reported local validation split; these are cross-referenced against the hidden holdout Log-Loss to check for overfitting or data leakage.

Feature engineering depth: The chosen technique is appropriate, correctly implemented, and its impact is measured and discussed.

Code quality and reproducibility: train.py runs cleanly from scratch with a single command; seeds are fixed; no data leakage between train and validation.

Engineering judgment and write-up: README clearly explains tradeoffs, failure modes, and what a production version would need (online updates, latency budget, monitoring, etc.).

Click-Through Rate Prediction for Programmatic Ad Auctions

Overview

Brief

Overview

What you'll build

Requirements

Deliverables

How we'll evaluate

Out of scope

Leaderboard

Overview

What you'll build

Requirements

Deliverables

How we'll evaluate

Out of scope