Evaluation with executor
Evaluate a Laminar pipeline by running it during the evaluation
Key concepts
- Executor pipeline – the pipeline being evaluated
- Evaluator pipeline - the pipeline evaluating the results
- Dataset format – two fixed JSON objects:
target
anddata
each with any keys.target
– data sent to the evaluator pipelinedata
– data sent to the executor pipeline
Flow overview
![High-level overview of the evaluation flow with the executor pipeline](https://mintlify.s3-us-west-1.amazonaws.com/laminarai/images/evaluations/old-evaluation-overview-with-executor.png)
High-level overview of the evaluation flow
What happens if executor pipeline has an output node named same as one of the keys in target?
Executor outputs have a higher priority over and overwrite the values in
target
. Detailed flow
For every datapoint in the dataset, evaluation does the following:
- Set all the values in
data
of the datapoint as values to the input nodes of the executor pipeline. This is done by matching* the keys in thedata
object to the input node names of the executor pipeline. - Run the evaluation pipeline. All required env variables must be set for this to succeed.
- All outputs of the executor pipeline are extracted and set as inputs to the evaluator pipeline. This is done by matching the output node names of the executor pipeline to the input node names of the evaluator pipeline.
- Set all the values in
target
of the datapoint as values to the input nodes of the evaluator pipeline. This is done by matching* the keys in thetarget
object to the input node names of the evaluator pipeline. - Evaluator pipeline is run. All required env variables must be set for this to succeed.
- Evaluator pipeline produces a single numeric output. This is stored in the results of the evaluation.
Requirements
- Executor pipeline has at least one commit.
data
at least contains keys matching* the names of the input nodes on the executor pipeline.- All required environment variables for the executor run are set in the Env vars page.
- Evaluator pipeline has at least one commit.
- Names of the output nodes of the executor pipeline match* the names of the corresponding input nodes of the evaluator pipeline.
- If you want to use some target data from the dataset,
target
must at least contain keys matching* the names of the input nodes on the evaluator pipeline. - All required environment variables for the evaluator run are set in the Env vars page.
- Evaluator pipeline must produce one output. This output must be parsable into a number (8-bit float).
* Exact case-sensitive match