Configuring Laminar evaluations

​

Configuring evaluations to report results to locally self-hosted Laminar

In this example, we configure the evaluation to report results to a locally self-hosted Laminar instance.

Evaluations send data to Laminar over both HTTP and gRPC. HTTP is used to create an evaluation and report the datapoints, stats, and trace ids. OpenTelemetry traces themselves are sent over gRPC.

Assuming you have configured Laminar to run on ports 8000 and 8001 on your localhost, you will need to pass these values to the evaluate function.

import { evaluate } from '@lmnr-ai/lmnr';
evaluate({
    data: evaluationData,
    executor: async (data) => await getCapital(data),
    evaluators: [evaluator],
    config: {
        projectApiKey: process.env.LMNR_PROJECT_API_KEY,
        baseUrl: 'http://localhost',
        httpPort: 8000,
        grpcPort: 8001,
    }
})

import { evaluate } from '@lmnr-ai/lmnr';
evaluate({
    data: evaluationData,
    executor: async (data) => await getCapital(data),
    evaluators: [evaluator],
    config: {
        projectApiKey: process.env.LMNR_PROJECT_API_KEY,
        baseUrl: 'http://localhost',
        httpPort: 8000,
        grpcPort: 8001,
    }
})

from lmnr import evaluate
evaluate(
    data=data,
    executor=get_capital,
    evaluators={'check_capital_correctness': evaluator},
    project_api_key=os.environ["LMNR_PROJECT_API_KEY"],
    base_url="http://localhost",
    http_port=8000,
    grpc_port=8001,
)

​

Configuring evaluations

​

evaluate reference

Evaluations in Laminar are configured using the evaluate function. The function takes the following arguments:

• data: Either (1) A list of dictionaries, where each dictionary contains the data, target, and metadata for a single evaluation; or (2) An instance of LaminarDataset – read more in the dedicated page.
• executor: An optionally async function that takes a single argument, the evaluation data, and returns the output.
• evaluators: A dictionary of async functions that take the output and target as arguments and return a score.
• name (optional): Evaluation name, so it is easier to identify the evaluation in the UI. If not provided, a random name is assigned in the backend.
• groupName/group_name (optional): An optional string that groups evaluations together. Only evaluations with the same group name can be visually compared.

Additional optional configuration parameters are passed as a config object in JavaScript/TypeScript and directly to evaluate in Python.

​

eval CLI reference

lmnr eval subcommand is used to run evaluations.

lmnr eval [options]
# or in most Node settings
npx lmnr eval [options]

​

Formatting evaluation files

When you run an evaluation file from the CLI, the evaluate functions must be called at the top level of the file. You can have one or more evaluate calls in the file.

​

Options

First positional argument is the path to the evaluation file. E.g.

lmnr eval ./evals/my_evaluation.eval.ts

If file is not provided, lmnr eval will run all files in the evals directory that match the naming pattern. For TypeScript/JavaScript, the pattern is *.eval.{ts,js}. For Python, the pattern is eval_*.py or *_eval.py.

​

Params

--fail-on-error – if set, the CLI will fail on non-critical errors, for example, if evaluate is not called in the file. Default is false.