Experimentation

Experimentation is a core phase in the LLM development workflow in which practitioners use collected datasets to run systematic tests and comparisons on model behavior, prompts, or pipeline configurations.

Role in the LLM Development Workflow

In the context of LLM Observability, Dat Ngo of Arize describes experimentation as a downstream activity that follows data collection: once a representative dataset has been assembled — typically sourced from production traces or curated examples — engineers can begin running experiments against it. 20:53

This positions experimentation as tightly coupled to evals: the dataset provides the ground truth or reference inputs, and experiments operationalize evaluation by varying a controlled factor (e.g., model version, prompt template, retrieval strategy) to measure the effect on output quality. The workflow Ngo outlines — observe → collect data → experiment — reflects a standard iterative loop common to LLM engineering platforms, where observability infrastructure feeds directly into the experimentation surface.