Skip to main content

​
Prerequisites

Have a dataset uploaded to Laminar, or collected from traces. See datasets for more information.

​
Defining data

To run an evaluation with a Laminar dataset, you pass the dataset object as data instead of a list of dictionaries. Use LaminarDataset to create a dataset object. The dataset name should match the name of the dataset in Laminar. The constructor also takes an optional fetch_size/fetchSize parameter, which specifies the number of datapoints to fetch at once. The default value is 25. We strongly recommend setting this value to a number that is a multiple of the evaluation batch size for best performance.
  • JavaScript/TypeScript
  • Python
import { evaluate, LaminarDataset } from '@lmnr-ai/lmnr';
const data = new LaminarDataset("name_of_your_dataset");
evaluate({
    data,
    executor: yourExecutorFunction,
    evaluators: yourEvaluators,
    config: {
        projectApiKey: process.env.LMNR_PROJECT_API_KEY,
        // ... other optional parameters
    }
})

​
Technical details and extension

LaminarDataset is an implementation of an abstract class EvaluationDataset which defines 2 methods besides initialization:
  • __len__ (size in JS): Returns the number of datapoints in the dataset.
  • __getitem__ (get in JS): Returns a single datapoint by index.
We also implement a concrete slice method to make slicing easier than using __getitem__ directly. This is inspired by the PyTorch Dataset class, and is designed to be used in a similar way. You can re-use the EvaluationDataset class to create your own dataset classes, for example, to fetch data from a database or an API.
  • JavaScript/TypeScript
  • Python
import { EvaluationDataset } from '@lmnr-ai/lmnr';

class MyCustomDataset extends EvaluationDataset {
    constructor(customProperty) {
        super();
        // Your custom initialization code here
    }

    public async size() {
        // Your custom implementation here
        return 0;
    }

    public async get(index: number) {
        // Your custom implementation here
        return { data: {}, target: {} };
    }

    // Optionally, you can implement other custom methods here
}