​
Key concepts

  • Executor pipeline – the pipeline being evaluated
  • Evaluator pipeline - the pipeline evaluating the results
  • Dataset format – two fixed JSON objects: target and data each with any keys.
    • target – data sent to the evaluator pipeline
    • data – data sent to the executor pipeline

​
Flow overview

High-level overview of the evaluation flow with the executor pipeline

High-level overview of the evaluation flow

What happens if executor pipeline has an output node named same as one of the keys in target? Executor outputs have a higher priority over and overwrite the values in target.

​
Detailed flow

For every datapoint in the dataset, evaluation does the following:

  1. Set all the values in data of the datapoint as values to the input nodes of the executor pipeline. This is done by matching* the keys in the data object to the input node names of the executor pipeline.
  2. Run the evaluation pipeline. All required env variables must be set for this to succeed.
  3. All outputs of the executor pipeline are extracted and set as inputs to the evaluator pipeline. This is done by matching the output node names of the executor pipeline to the input node names of the evaluator pipeline.
  4. Set all the values in target of the datapoint as values to the input nodes of the evaluator pipeline. This is done by matching* the keys in the target object to the input node names of the evaluator pipeline.
  5. Evaluator pipeline is run. All required env variables must be set for this to succeed.
  6. Evaluator pipeline produces a single numeric output. This is stored in the results of the evaluation.

​
Requirements

  • Executor pipeline has at least one commit.
  • data at least contains keys matching* the names of the input nodes on the executor pipeline.
  • All required environment variables for the executor run are set in the Env vars page.
  • Evaluator pipeline has at least one commit.
  • Names of the output nodes of the executor pipeline match* the names of the corresponding input nodes of the evaluator pipeline.
  • If you want to use some target data from the dataset, target must at least contain keys matching* the names of the input nodes on the evaluator pipeline.
  • All required environment variables for the evaluator run are set in the Env vars page.
  • Evaluator pipeline must produce one output. This output must be parsable into a number (8-bit float).

* Exact case-sensitive match