​
evaluate reference

Evaluations in Laminar are configured using the evaluate function. The function takes the following arguments:

  • data: Either (1) A list of dictionaries, where each dictionary contains the data and target for a single evaluation; or (2) An instance of LaminarDataset – read more in the dedicated section.
  • executor: An optionally async function that takes a single argument, the evaluation data, and returns the output.
  • evaluators: A dictionary of async functions that take the output and target as arguments and return a score. Keys in the dictionary are the names of the evaluators.
  • name (optional): Evaluation name, so it is easier to identify the evaluation in the UI. If not provided, a random name is assigned.
  • groupName/group_name (optional): An optional string that groups evaluations together. Only evaluations with the same group name can be visually compared.

Additional optional configuration parameters are passed as a config object in JavaScript/TypeScript and directly to evaluate in Python.

  • projectApiKey: The API key of the project where the evaluation results will be stored. Required, unless you set the LMNR_PROJECT_API_KEY environment variable.
  • concurrencyLimit: The number of evaluations to run in parallel. Default is 5.
  • baseUrl: The base URL of the Laminar instance. Do NOT include port here. Default is https://api.lmnr.ai.
  • httpPort: The port of the Laminar instance for HTTP. Used to send evaluation results and metadata. Default is 443. For local self-hosted Laminar, use 8000.
  • grpcPort: The port of the Laminar instance for gRPC. Used to send traces via OTel gRPC exporter. Default is 8443. For local self-hosted Laminar, use 8001.
  • instrumentModules: A map from module names to packages to be instrumented. Read more in the instrumentation guide.

​
eval CLI reference

lmnr eval subcommand is used to run evaluations.

lmnr eval [options]
# or in most Node settings
npx lmnr eval [options]

​
Options

First positional argument is the path to the evaluation file. E.g.

lmnr eval ./evals/my_evaluation.eval.ts

If file is not provided, lmnr eval will run all files in the evals directory that match the naming pattern. For TypeScript/JavaScript, the pattern is *.eval.{ts,js}. For Python, the pattern is eval_*.py or *_eval.py.

​
Params

--fail-on-error – if set, the CLI will fail on non-critical errors, for example, if evaluate is not called in the file. Default is false.