Online evaluators provide a powerful way to automatically assess and score your LLM calls as they come. Once registered, evaluators run automatically whenever a span matches their registered path, immediately attaching evaluation scores based on how you define to score them.

​
How Online Evaluators Work

The core concept:

  1. Custom Evaluators: You define your own evaluator logic by writing Python functions that analyze the outputs of your LLM calls. These can range from simple checks to sophisticated analysis. It should always return number.

  2. Span Path Registration: Each evaluator is registered to a specific span path - a unique identifier that corresponds to a particular LLM function or call location in your code.

  3. Automatic Execution: Once registered, evaluators run automatically whenever a span matches their registered path. The evaluator score is immediately attached to the span.

​
Getting Started

  1. Go to the Evaluators page in your Laminar dashboard
  2. Click “New Evaluator” to start creating your custom evaluator
  3. Create your evaluator by writing the Python function that will assess your LLM outputs
  4. Go to a span in your traces and register the evaluator to that specific span path