Writing
Infographic: Tips On Digital Experiments
A visual, practical guide to designing digital experiments, connected to Paulo's Experiment-Oriented Computing field, with a full accessible-text transcript.
- Reflection
- Inquiry
My second infographic, related to my work on Experiment-Oriented Computing. Please share if you find it valuable and would like to see more content like this.
Full text of the infographic
Tips on Digital Experiments
Define your purpose first. Before running any digital experiment, be explicit about what you want to learn and how you will decide whether it worked.
Example use cases: A/B testing a website change; comparing two machine-learning models; evaluating a new feature with a subset of users; testing a pricing strategy; validating a product hypothesis; comparing manual and automated workflows.
An iterative process: Explore the problem and possibilities → Learn by running a disciplined experiment → Store the results, code, and configuration for reuse → Repeat, refining the question each time.
Explore. Map out the space of plausible options before committing to one experimental design.
Learn. Run experiments that can actually surprise you, not only confirm what you already believe.
Store. Keep the artifacts of an experiment (data, code, configuration, results) so it can be inspected, reproduced, or extended later.
Finally, also consider:
- Reproducibility — could someone else (or you, later) rerun this experiment and get the same result?
- Traceability — can you trace a result back to the exact data, code, and parameters that produced it?
- Durability — will the experiment's artifacts still be usable and meaningful months or years from now?
Sources: Based on Paulo Salem's work on Experiment-Oriented Computing, including The Case for Experiment-Oriented Computing (ESEC/FSE 2018).
Paulo Salem, Ph.D., Data Scientist. Version 1, June/2019. Licensed under CC BY-SA 4.0 International. Powered by Piktochart.