Troubleshooting

My JS auto-instrumentation is not working.

Problem

I have instrumented my JavaScript code with Laminar, and I can see the manual spans in the Laminar UI, but I don’t see the auto-instrumented spans.

Cause

Laminar auto-instrumentations are based on OpenTelemetry’s auto-instrumentations, and they are optimized for commonjs modules.

Solution

Make sure that you pass `instrumentModules`

The instrumentModules option is required to instrument the modules you need.

import { Laminar } from '@lmnr-ai/lmnr';
import { OpenAI } from 'openai';
import anthropic from 'anthropic';

Laminar.initialize({
  projectApiKey: process.env.LMNR_PROJECT_API_KEY,
  instrumentModules: {
    OpenAI: OpenAI,
    anthropic: anthropic,
    // ... other modules you want to instrument
  }
});

If this didn’t solve your issue, continue with the next steps.

Check if you are using ESM modules

Due to the differences between Node versions, bundlers, and TypeScript compilers, it is not easy to definitely determine if you are using ESM modules.

Some strong indicators:

You have a package.json file with "type": "module" – this is a 100% indicator.
Your files have .mjs extension – this is a strong indicator.
You are using import statements and this is not TypeScript – this is a strong indicator.

See more in Node.js documentation and Node.js documentation on determining module system.

If you are not sure, you can try to run node --input-type=module yourfile.js and see if it works.

Try to migrate to CommonJS

For modern Node versions, migrating to CommonJS is straightforward and most code actually is written in ESM syntax.

It might be as simple as changing the file extension from .mjs to .js and adding "type": "commonjs" to your package.json file.

[Optional] Further reading

Refer to the CJS vs ESM guide in OpenTelemetry documentation for more information.

My JS traces are not showing up in the Laminar UI.

Problem

I have instrumented my JavaScript code with Laminar, but I don’t see any traces in the Laminar UI.

Cause

This often happens in Edge runtimes, Lambda functions, or in one-off scripts that are not running in a long-lived process.

JavaScript’s OpenTelemetry batch span processor runs in a background async function, and, if the process exits before the function has a chance to send the traces, they will be lost. In theory, there is a way to force flush the pending spans onShutdown, but it is not implemented in the OpenTelemetry JS SDK. Apparently, doing that may cause the consequent incoming spans to block the process.

See originating commit where onShutdown was only implemented for the browser contexts, but not for Node.js.

Solution

Use Laminar’s flush() function to ensure that the traces are sent before the process exits.

import { Laminar } from '@lmnr-ai/lmnr';

Laminar.initialize({
  projectApiKey: process.env.LMNR_PROJECT_API_KEY,
  instrumentModules: {
    // your instrumented modules
  }
});

// your code here

// at the end of your script, when you are done tracing
await Laminar.flush(); // NOTE: don't forget to await

My Python auto-instrumentation is not working.

Problem

I have instrumented my Python code with Laminar, and I can see the manual spans in the Laminar UI, but I don’t see the auto-instrumented spans.

Cause

Most likely, you have not installed the right extras that enable the auto-instrumentations.

Solution

Install the extras for LLM providers you are using and wish to instrument. For instance,

pip install lmnr[openai]

Read the installation docs for more information.

I can see traces and spans in the UI, but they are not showing inputs and/or outputs.

Problem

I see traces and spans in the Laminar UI, but they are not showing the inputs and/or outputs. They have attributes and duration, but no inputs or outputs.

Cause

One possible reason is that we send span inputs and outputs as span attributes, and OpenTelemetry has a limited set of possible attribute types, namely string, number, boolean, and array of each. If the input or output is not one of these types, it will not be sent as an attribute.

To work around this, we serialize the input and output to JSON and send it as a string. We do our best to serialize the inputs and outputs, but it may be that the serialization fails.

Solution

Make sure inputs and outputs to your functions or spans are serializable to JSON. If you are using custom objects, make sure their fields are serializible. In Python, that may mean having to implement default() method on some of them, or on the parent object.

Many of my JS route requests are instrumented in the Laminar UI, even though I don’t want that.

Problem

I see many traces and spans in the UI, but most of them are just noise, they merely are route requests.

Some examples of noise spans:

GET /my/route
POST /my/other/route
dns.lookup
middleware - query
tcp - connect
fs statSync

Cause

Even though Laminar only enables OpenLLMetry-specific instrumentations, i.e. just model calls, some other dependency in your code may have enabled other instrumentations for Node libraries. In particular, we have seen this issue when such initialization is done before Laminar’s initialization. For example:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require(
  '@opentelemetry/auto-instrumentations-node'
);
const { BatchSpanProcessor, SamplingDecision } = require(
  '@opentelemetry/sdk-trace-base'
);

// this can also be not raw OpenTelemetry, but some other observability library
const sdk = new NodeSDK({
  instrumentations: [getNodeAutoInstrumentations()],
  spanProcessors: new BatchSpanProcessor(exporter),
});

const { Laminar } = require('@lmnr-ai/lmnr');
Laminar.initialize({ projectApiKey: "...", });

Read more on OpenTelemetry’s Node.js auto-instrumentations in OpenTelemetry.

Solution

The most effective solution is to update lmnr-ai/lmnr to the latest version. Since version 0.6.0 Laminar has its own Tracer provider, and it’s not affected by other auto-instrumentations.

SSL Certificate and GRPC Connection Issues for Laminar

Problem

Experiencing errors related to SSL certificate verification and GRPC connections in Laminar. Error messages include CERTIFICATE_VERIFY_FAILED and StatusCode.UNAVAILABLE when attempting to export traces to a GRPC server.

Cause

SSL Certificate Verification Error:
- The certificate chain may be incomplete or incorrect.
- The client is not configured to accept the server’s certificate.
GRPC Connection Error:
- SSL configuration issues prevent establishing a secure connection.
- The server may actually be unavailable.

Solution

To resolve these errors, follow these steps:

Import the Certifi Library:
- Use the certifi library to obtain an updated set of trusted CA certificates.
- Install using pip: pip install certifi
Verify the Certifi Path:
- After installation, you can verify the path to the certifi CA bundle by running:
  python -m certifi
- This command will print the path to the cacert.pem file, which you can use for further configurations.
Configure Environment Variables:
- SSL_CERT_FILE: Set the path to the trusted CA certificate file.
  export SSL_CERT_FILE=$(python -m certifi)
- GRPC_DEFAULT_SSL_ROOTS_FILE_PATH: Configure this variable so that GRPC uses the same CA certificate file.
  export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=$(python -m certifi)

Observe decorator is not working in Python with async generators, i.e. model stream responses

Problem

@observe decorator in Python is not correctly capturing the spans when I use async generators, e.g. when I am streaming responses from a model.

Cause

Async generators only end when they are exhausted, i.e. when the async for loop is done. Given the nature of Python’s async, the chance of the interruptions within stream, and the way OpenTelemetry’s contexts work, there is no way to ensure that @observe works in 100% of the cases.

Solution

One possible workaround is to observe only the outer functions that are calling the async generator, and handle the final result, not the generator itself.

However, if you want to track partial stream results as well, you can create the spans manually. Refer to manually creating spans for more information.

ERR_BUFFER_OUT_OF_BOUNDS in Node.js

Problem

I see the following error in my Node.js application:

node:internal/buffer:1066
      throw new ERR_BUFFER_OUT_OF_BOUNDS('length');
      ^

RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: "length" is outside of buffer bounds
    at proto.utf8Write (node:internal/buffer:1066:13)
    at Op.writeStringBuffer [as fn] (/Users/sebastianscheibe/Code/XXX/node_modules/protobufjs/src/writer_buffer.js:61:13)
    at BufferWriter.finish (/Users/sebastianscheibe/Code/XXX/node_modules/protobufjs/src/writer.js:453:14)
    at /Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:177:109
    at Array.map (<anonymous>)
    at createPackageDefinition (/Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:177:39)
    at Object.fromJSON (/Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:230:12)
    at GrpcClient.loadProtoJSON (/Users/sebastianscheibe/Code/XXX/node_modules/google-gax/build/src/grpc.js:228:51)
    at new ImageAnnotatorClient (/Users/sebastianscheibe/Code/XXX/node_modules/@google-cloud/vision/build/src/v1/image_annotator_client.js:148:38)
    at Object.<anonymous> (/Users/sebastianscheibe/Code/XXX/test.js:5:16) {
  code: 'ERR_BUFFER_OUT_OF_BOUNDS'
}

Node.js v22.7.0

Cause

This is most likely due to a bug in NodeJS versions 22.6 and 22.7 with handling utf-8 buffers. See more in the Node.js issue.

Solution

Upgrade to Node.js version 22.8.0 or higher. You can also downgrade to 22.5.

Overview

Tracing

Evaluations

Labels

Datasets

My JS auto-instrumentation is not working.

Problem

Cause

Solution

My JS traces are not showing up in the Laminar UI.

Problem

Cause

Solution

My Python auto-instrumentation is not working.

Problem

Cause

Solution

I can see traces and spans in the UI, but they are not showing inputs and/or outputs.

Problem

Cause

Solution

Many of my JS route requests are instrumented in the Laminar UI, even though I don’t want that.

Problem

Cause

Solution

SSL Certificate and GRPC Connection Issues for Laminar

Problem

Cause

Solution

Observe decorator is not working in Python with async generators, i.e. model stream responses

Problem

Cause

Solution

ERR_BUFFER_OUT_OF_BOUNDS in Node.js

Problem

Cause

Solution

Overview

Tracing

Evaluations

Labels

Datasets

​My JS auto-instrumentation is not working.

​Problem

​Cause

​Solution

​My JS traces are not showing up in the Laminar UI.

​Problem

​Cause

​Solution

​My Python auto-instrumentation is not working.

​Problem

​Cause

​Solution

​I can see traces and spans in the UI, but they are not showing inputs and/or outputs.

​Problem

​Cause

​Solution

​Many of my JS route requests are instrumented in the Laminar UI, even though I don’t want that.

​Problem

​Cause

​Solution

​SSL Certificate and GRPC Connection Issues for Laminar

​Problem

​Cause

​Solution

​Observe decorator is not working in Python with async generators, i.e. model stream responses

​Problem

​Cause

​Solution

​ERR_BUFFER_OUT_OF_BOUNDS in Node.js

​Problem

​Cause

​Solution

My JS auto-instrumentation is not working.

Problem

Cause

Solution

My JS traces are not showing up in the Laminar UI.

Problem

Cause

Solution

My Python auto-instrumentation is not working.

Problem

Cause

Solution

I can see traces and spans in the UI, but they are not showing inputs and/or outputs.

Problem

Cause

Solution

Many of my JS route requests are instrumented in the Laminar UI, even though I don’t want that.

Problem

Cause

Solution

SSL Certificate and GRPC Connection Issues for Laminar

Problem

Cause

Solution

Observe decorator is not working in Python with async generators, i.e. model stream responses

Problem

Cause

Solution

ERR_BUFFER_OUT_OF_BOUNDS in Node.js

Problem

Cause

Solution