SDK Reference
Reference documentation for evaluations in Laminar, including the evaluate function and CLI options.
evaluate
reference
Evaluations in Laminar are configured using the evaluate
function. The function takes the following arguments:
data
: Either (1) A list of dictionaries, where each dictionary contains the data and target for a single evaluation; or (2) An instance ofLaminarDataset
– read more in the dedicated section.executor
: An optionally async function that takes a single argument, the evaluation data, and returns the output.evaluators
: A dictionary of async functions that take the output and target as arguments and return a score. Keys in the dictionary are the names of the evaluators.humanEvaluators
/human_evaluators
: A list ofHumanEvaluator
objects, which register human evaluators for the evaluation. Read more in the dedicated section.name
(optional): Evaluation name, so it is easier to identify the evaluation in the UI. If not provided, a random name is assigned.groupName
/group_name
(optional): An optional string that groups evaluations together. Only evaluations with the same group name can be visually compared.
Additional optional configuration parameters are passed as a config
object in JavaScript/TypeScript and directly to evaluate
in Python.
projectApiKey
: The API key of the project where the evaluation results will be stored. Required, unless you set theLMNR_PROJECT_API_KEY
environment variable.batchSize
: The number of evaluations to run in parallel. Default is5
.baseUrl
: The base URL of the Laminar instance. Do NOT include port here. Default ishttps://api.lmnr.ai
.httpPort
: The port of the Laminar instance for HTTP. Used to send evaluation results and metadata. Default is 443. For local self-hosted Laminar, use 8000.grpcPort
: The port of the Laminar instance for gRPC. Used to send traces via OTel gRPC exporter. Default is 8443. For local self-hosted Laminar, use 8001.instrumentModules
: A map from module names to packages to be instrumented. Read more in the instrumentation guide.
projectApiKey
: The API key of the project where the evaluation results will be stored. Required, unless you set theLMNR_PROJECT_API_KEY
environment variable.batchSize
: The number of evaluations to run in parallel. Default is5
.baseUrl
: The base URL of the Laminar instance. Do NOT include port here. Default ishttps://api.lmnr.ai
.httpPort
: The port of the Laminar instance for HTTP. Used to send evaluation results and metadata. Default is 443. For local self-hosted Laminar, use 8000.grpcPort
: The port of the Laminar instance for gRPC. Used to send traces via OTel gRPC exporter. Default is 8443. For local self-hosted Laminar, use 8001.instrumentModules
: A map from module names to packages to be instrumented. Read more in the instrumentation guide.
project_api_key
: The API key of the project where the evaluation results will be stored. Required, unless you set theLMNR_PROJECT_API_KEY
environment variable.batch_size
: The number of evaluations to run in parallel. Default is5
.base_url
: The base URL of the Laminar instance. Do NOT include port here. Default ishttps://api.lmnr.ai
.http_port
: The port of the Laminar instance for HTTP. Used to send evaluation results and metadata. Default is 443. For local self-hosted Laminar, use 8000.grpc_port
: The port of the Laminar instance for gRPC. Used to send traces via OTel gRPC exporter. Default is 8443. For local self-hosted Laminar, use 8001.instrument_modules
: A set of modules to instrument. Read more in the instrumentation guide.
eval CLI reference
lmnr eval
subcommand is used to run evaluations.
Options
First positional argument is the path to the evaluation file. E.g.
If file is not provided, lmnr eval
will run all files in the evals
directory that match the naming pattern.
For TypeScript/JavaScript, the pattern is *.eval.{ts,js}
. For Python, the pattern is eval_*.py
or *_eval.py
.
Params
--fail-on-error
– if set, the CLI will fail on non-critical errors, for example, if evaluate
is not called in the file. Default is false
.