Configuration

​

Registering human evaluators

You can register human evaluators right from your code. To do this, you will need to first create a labeling queue, and then pass the queue name to the evaluate function.

In this example, let’s assume you have created labeling queues with names my_queue and my_other_queue.

• JavaScript/TypeScript
• Python

import { evaluate, HumanEvaluator } from '@lmnr-ai/lmnr';

evaluate({
    data: evaluationData,
    executor: async (data) => await getCapital(data),
    evaluators: { checkCapitalCorrectness: evaluator },
    projectApiKey: process.env.LMNR_PROJECT_API_KEY,
    // note, this is new from `@lmnr-ai/lmnr==0.4.18`
    humanEvaluators: [
        HumanEvaluator("my_queue"),
        HumanEvaluator("my_other_queue"),
    ],
})

This will run your programmatic evaluator (“check capital correctness”) and then send the target and executor_output to the queues my_queue and my_other_queue.

When a label is added to an item in the queue, it will be added back to the evaluation alongside the programmatic evaluator scores.

You can then visualize the human labeler scores in the UI, and compare them to the programmatic evaluator scores.

​

Configuring evaluations to report results to locally self-hosted Laminar

In this example, we configure the evaluation to report results to a locally self-hosted Laminar instance.

Evaluations send data to Laminar over both HTTP and gRPC. HTTP is used to create an evaluation and report the datapoints, stats, and trace ids. OpenTelemetry traces themselves are sent over gRPC.

Assuming you have configured Laminar to run on ports 8000 and 8001 on your localhost, you will need to pass these values to the evaluate function.

• JavaScript/TypeScript
• Python

import { evaluate } from '@lmnr-ai/lmnr';
evaluate({
    data: evaluationData,
    executor: async (data) => await getCapital(data),
    evaluators: [evaluator],
    config: {
        projectApiKey: process.env.LMNR_PROJECT_API_KEY,
        baseUrl: 'http://localhost',
        httpPort: 8000,
        grpcPort: 8001,
    }
})

​

Using a Laminar dataset for evaluations

​

Prerequisites

Have a dataset uploaded to Laminar, or collected from traces. See datasets for more information.

​

Defining data

To run an evaluation with a Laminar dataset, you pass the dataset object as data instead of a list of dictionaries.

Use LaminarDataset to create a dataset object. The dataset name should match the name of the dataset in Laminar. The constructor also takes an optional fetch_size/fetchSize parameter, which specifies the number of datapoints to fetch at once. The default value is 25. We strongly recommend setting this value to a number that is a multiple of the evaluation batch size for best performance.

• JavaScript/TypeScript
• Python

import { evaluate, LaminarDataset } from '@lmnr-ai/lmnr';
const data = new LaminarDataset("name_of_your_dataset");
evaluate({
    data,
    executor: yourExecutorFunction,
    evaluators: yourEvaluators,
    config: {
        projectApiKey: process.env.LMNR_PROJECT_API_KEY,
        // ... other optional parameters
    }
})

​

Technical details and extension

LaminarDataset is an implementation of an abstract class EvaluationDataset which defines 2 methods besides initialization:

• __len__ (size in JS): Returns the number of datapoints in the dataset.
• __getitem__ (get in JS): Returns a single datapoint by index.

We also implement a concrete slice method to make slicing easier than using __getitem__ directly.

This is inspired by the PyTorch Dataset class, and is designed to be used in a similar way.

You can re-use the EvaluationDataset class to create your own dataset classes, for example, to fetch data from a database or an API.

• JavaScript/TypeScript
• Python

import { EvaluationDataset } from '@lmnr-ai/lmnr';

class MyCustomDataset extends EvaluationDataset {
    constructor(customProperty) {
        super();
        // Your custom initialization code here
    }

    public async size() {
        // Your custom implementation here
        return 0;
    }

    public async get(index: number) {
        // Your custom implementation here
        return { data: {}, target: {} };
    }

    // Optionally, you can implement other custom methods here
}

​

Configuring evaluations

Evaluations in Laminar are configured using the evaluate function. The function takes the following arguments:

• data: Either (1) A list of dictionaries, where each dictionary contains the data and target for a single evaluation; or (2) An instance of LaminarDataset – read more in the dedicated section above.
• executor: An optionally async function that takes a single argument, the evaluation data, and returns the output.
• evaluators: A dictionary of async functions that take the output and target as arguments and return a score. Keys in the dictionary are the names of the evaluators.
• humanEvaluators/human_evaluators : A list of HumanEvaluator objects, which register human evaluators for the evaluation. Read more in the dedicated section above.
• name (optional): Evaluation name, so it is easier to identify the evaluation in the UI. If not provided, a random name is assigned.
• groupId/group_id (optional): An optional string that groups evaluations together. Only evaluations with the same group_id can be visually compared.

Additional optional configuration parameters are passed as a config object in JavaScript/TypeScript and directly to evaluate in Python.

• JavaScript/TypeScript
• Python