How Online Evaluators Work
The core concept:- Custom Evaluators: You define your own evaluator logic by writing Python functions that analyze the outputs of your LLM calls. These can range from simple checks to sophisticated analysis. It should always return number.
- Span Path Registration: Each evaluator is registered to a specific span path - a unique identifier that corresponds to a particular LLM function or call location in your code.
- Automatic Execution: Once registered, evaluators run automatically whenever a span matches their registered path. The evaluator score is immediately attached to the span.
Two Ways to Produce Evaluator Scores
There are two main approaches to generating evaluator scores in Laminar:1. Hosted Evaluators (Automated)
Create custom Python functions that run automatically on our platform. These evaluators:- Execute automatically when spans match their registered path
- Require no additional code in your application
- Are managed and hosted by Laminar
2. SDK Scoring (Programmatic)
Create scores programmatically using our SDK or REST API. This approach:- Gives you full control over when and how scores are created
- Allows integration with your existing evaluation pipeline
- Supports custom scoring logic that runs in your environment
Getting Started
- Go to the Evaluators page in your Laminar dashboard
- Click “New Evaluator” to start creating your custom evaluator
- Create your evaluator by writing the Python function that will assess your LLM outputs
- Go to a span in your traces and register the evaluator to that specific span path