Curated library of real product experiments
Browse a structured repository of experiments run by real teams, with goals, designs, outcomes, and methodological commentary. Use it as case material for your product team or research.
Free Experiment Design
Experiment Library
Unlock the full dataset
One‑time purchase for the complete library, delivered as a clean, structured CSV.
- 69 experiments across SaaS, consumer, marketplace, and media products — including standout headline tests from the Upworthy Research Archive.
- CSV columns: experiment_id, title, product_name, sector, goal, experiment_type, primary_metric, design_summary, sample_size, outcome_summary, p_value, decision, methodology_notes, source, year, tags.
- Methodological commentary: threats to validity, design notes, and analysis hints.
- Plus a companion Upworthy headline-test subset (18 standout tests, with competing headlines and click-through rates).
- Ready for use in R, Python, SQL, or BI tools.
Want early access or a custom export? Email: jared@productscience.consulting
Learnings from 27,000 news headline tests at Upworthy
Between 2013 and 2015 the media company Upworthy ran a randomized A/B test on almost every story it published, rotating competing headlines and images past visitors and keeping the one that earned the most clicks. They later released the raw logs as the Upworthy Research Archive.
Six things the data says
The headline decides the outcome
Within a single test, the best headline beat the worst by a median 1.7× in click-through — and by 2× or more in over a third of tests. Same story, same audience; only the words changed.
Experts can't call the winner
Lining up the archive's measured effects across 50+ language features against what marketing professionals, content writers, and the academic literature predicted, agreement with the real direction was barely better than a coin flip.
The "curiosity gap" is oversold
Curiosity is the single most-recommended tactic in industry guidance — yet headlines coded as curious showed no significant lift, and asking a question in the headline actually lowered clicks.
Visceral emotion travels
Emotional intensity and negative emotions — anger, fear, even disgust — were among the features that reliably raised click-through. Calm, positive-tone framings tended to underperform.
Specific beats clever
Everyday words, real numbers, and named, visible people lifted clicks. Abstract appeals to authority, goals, and comparison framing pushed them down.
Small effects, big samples
At a ~1.5% average click rate, telling a real winner from noise takes tens of thousands of impressions per variant — which is exactly why Upworthy tested everything before publishing.
Two real tests from the archive
Built from the Upworthy Research Archive (Matias, J.N., Munger, K., Le Quere, M.A., & Ebersole, C., 2021), released for research use. The aggregate statistics and the comparison of measured effects to expert, practitioner, and literature predictions are computed from the archive's confirmatory dataset and its companion language analysis; the full archive catalogs 32,487 experiments.
Research Papers on Online Experimentation Methods
Curated bibliography of academic work on online experimentation, A/B testing, and related causal methods (2000–2025). It comes from a PRISMA-style systematic literature review and is separate from the product case-study library above. A sample of rows is embedded so the table works even when you open this file locally. On a live site the full list loads from the CSV, or you can download the file below.
If you use this dataset, please cite:
Return-Aware Platform Experimentation: New Directions for Research, 2025.
Authors: Jacqueline Doremus, Joel Persson, Brian St. Thomas, Carlos A. Flores, Sebastian Ankargren, Mårten Schultzberg, Kyle Kretschman.
Dataset available at: https://github.com/jopersson/online-experimentation-literature-review/
| Title | Year | Venue | Field | Micro | Macro | Database |
|---|
Why Experiment?
A randomized experiment gives you a more accurate estimate of the effect of a change. That is true whether the change is a website redesign, a new school policy, or a tweak to your own health and lifestyle. You make one deliberate change, measure what happens, and learn how much of the result the change actually caused.
The hard part is comparison. To know the effect of a change you need a believable picture of what would have happened without it. When people are not randomly assigned to groups, the groups differ in hidden ways. The people who user your website, the classrooms who adopt the policy, or the people who start the new diet are often more motivated to begin with. These hidden differences are called confounders, and they make non-randomized studies untrustworthy because you cannot tell whether the change or the different type of people caused the result.
For years, observational studies suggested that hormone replacement therapy protected women against heart disease. But the women who took it were healthier and wealthier to start with. When the randomized Women's Health Initiative trial (JAMA, 2002) finally tested it head to head, the supposed benefit vanished and some risks rose. Randomization revealed an effect that confounding had completely hidden. That is exactly why I built Health Evidence, a companion tool that sorts health studies by how much you can trust them — strong evidence from large randomized experiments versus weak evidence from studies that only observed patterns — so you can make informed decisions about your health.
Randomization vs Self-selection
Each shape is a person. Switch between random assignment and letting people choose to see how confounders sneak into the groups. Watch the balance line and caption below.
Frequently asked questions
How teams use this library
Product managers, data scientists, and researchers use this as a reference when planning new experiments, teaching methodology, or quantifying patterns across many tests.
Where do these experiments come from?
The library curates experiments from public case studies, talks, and research reports, then standardizes them into a consistent schema with added methodological notes.
Can I use this in teaching or training?
Yes. The commentary is designed to be used as case material for internal training, workshops, or university courses. You can slice the dataset by sector, goal, or method.
Will you add my experiments?
If you run a substantial experiment program and want anonymized entries included, reach out. Contributions can be credited and included in future dataset releases.