Laminar uses OpenTelemetry for tracing. That is, Laminar’s backend is compatible with OpenTelemetry. This page details the methods to send your OpenTelemetry traces to Laminar, even if you are not using Laminar’s SDK. If you are using Laminar’s SDK, refer to the tracing page to get started. Still feel free to read this page to get basic understanding of the internals of OpenTelemetry and how it works with Laminar.

​
Introduction

OpenTelemetry is a framework for tracing and monitoring distributed systems. OpenLLMetry is an extension of OpenTelemetry that is specifically designed for LLM tracing. The standards introduced in OpenLLMetry are gradually being adopted by OpenTelemetry. OpenTelemetry covers tracing, metrics, and logs, but Laminar only uses tracing. For more details about tracing in Laminar, and a quick intro to the core concepts, see the Tracing page. Processing of OpenTelemetry traces has many concepts and components, namely, exporters, processors, propagators, samplers, and collectors. For the purposes of this page, we will focus on the exporters.

​
Exporters

Exporter is the client-side component that sends spans (and thus traces) to an OpenTelemetry-compatible backend. It is responsible for the payload format and the transport protocol. The default protocol is OTLP, which underneath uses protobuf in one of the following formats:
  • Protobuf in gRPC (commonly known as OTLP/gRPC)
  • Protobuf over HTTP with binary encoding (commonly known as OTLP/HTTP/proto)
  • Protobuf over HTTP with JSON encoding (commonly known as OTLP/HTTP/json, is used rarely).
Laminar’s backend (both cloud and self-hosted) supports OTLP/gRPC and OTLP/HTTP/proto formats. OTLP/gRPC is recommended for performance and reliability reasons.
Laminar’s backend does not support OTLP/HTTP/json.
Here’s a quick comparison of formats that Laminar supports:
OTLP/gRPCOTLP/HTTP/proto
Supported✅✅
Recommended✅❌
Underlying protocolgRPC over HTTP/2HTTP/1.1
Encoding formatprotobufprotobuf
Base URL for Laminar cloudhttps://api.lmnr.aihttps://api.lmnr.ai
Port at Laminar cloud8443443
Default port for self-hosted backend80018000
Path/v1/traces [1]/v1/traces [1]
[1] In OpenTelemetry Node and Python SDKs, this path is appended (unless specified explicitly) in gRPC exporter, but not in the HTTP exporter.

​
Getting started

Usually, OpenTelemetry tracing in other libraries is initialized once at the start of the application by calling a function like initTracer(). Functions like this usually accept a configuration object or a set of parameters, including the exporter configuration. To send traces to Laminar, you need to configure the endpoint and the authorization.

​
Example

import { Metadata } from '@grpc/grpc-js';
// it is important to import the OTLP exporter from the
// `@opentelemetry/exporter-trace-otlp-grpc` package
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';

const metadata = new Metadata();
metadata.set('authorization', `Bearer ${process.env.LMNR_PROJECT_API_KEY}`);
const exporter = new OTLPTraceExporter({
  url: "https://api.lmnr.ai:8443/v1/traces",
  metadata,
});

functionThatInitiatesTracer({
    exporter
});

​
Endpoint

The default base url for the endpoint is https://api.lmnr.ai listening for gRPC traffic on port 8443. For the self-hosted backend, the base url is http://<your-self-hosted-backend-url> and the default port is 8001, unless you have changed the configuration. The /v1/traces path is the default OpenTelemetry trace endpoint, and Laminar listens at this path. In both JavaScript (OpenTelemetry Node SDK) and Python OpenTelemetry SDKs, the gRPC implementation appends /v1/traces to the base url, if you don’t specify it. Be careful though if you are using the HTTP exporter, as the HTTP implementation does not append it.

​
Authorization

The authorization header is required to send traces to Laminar. We use bearer authentication, so the header should start with Bearer and the token is your project API key. The right way to set the headers for gRPC requests is to use the metadata object. Even though gRPC is sent over HTTP/2, and metadata is sent as HTTP headers, it is different from raw HTTP headers. Learn more about metadata in the gRPC documentation.
If you specify the raw HTTP headers for a gRPC request, Laminar will not be able to process them. This is especially relevant for the exporter in Node.js, which accepts both metadata and headers parameters. The latter is effectively ignored by Laminar backend.
gRPC metadata has a restricted set of keys, and so many client implementations, including the OpenTelemetry Python SDK, check the keys for validity, effectively imposing case-sensitivity. That is, authorization has to start with lowercase a.
At the time of writing, Python SDK throws an error if the key is Authorization with capital A. If you are specifying the key as Authorization and not seeing an error saying TypeError: not all arguments converted during string formatting, you are probably using the HTTP exporter, which is not recommended.

​
Troubleshooting

If you are facing issues, please refer to the Troubleshooting OpenTelemetry page.

​
Span reference

This section is solely for information purposes. As a user of Laminar, you don’t have to deal with the internals of OpenTelemetry.

​
Span object

AttributeDescriptionTypeLaminar representation (if different)Example
trace_idUnique identifier for the trace16-bytesUUID [1]01234567-89ab-4def-0123-456789abcdef
span_idUnique identifier for the span8-bytesUUID [1]00000000-0000-0000-0123-456789abcdef
parent_span_idUnique identifier for the parent span8-bytesUUID [1]00000000-0000-0000-0123-456789abcdef
nameName of the spanstringmy_function
eventsEvents associated with the spanArray<Event>
attributesAttributes associated with the span. Fully compatible with the gen_ai semantic conventionsKey-value pair. Key must comply to semantic conventions, value must be of AttributeType [2]{"gen_ai.usage.output_tokens": 369}
start_time_unix_nanoStart time of the span in nanoseconds [3]numbertimestamp with UTC timezone1630000000000000000
end_time_unix_nanoEnd time of the span in nanoseconds [3]numbertimestamp with UTC timezone1630000000000000000
[1] 13th digit in hex UUID depends on UUID version, so technically converting OTel/WC3’s 16-byte and, especially, 8-byte IDs to UUIDs is not exactly UUID. [2] AttributeType is a union of string, number, boolean, Array<string>, Array<number>, Array<boolean> [3] In most OpenTelemetry client implementations, you don’t have to convert the timestamp to nanoseconds manually, you can simply pass the Date / datetime object and the client will convert it to nanoseconds.

​
Span attributes

Span attribute values that Laminar bases some of its functionality on:
AttributeDescriptionTypeExample
lmnr.span.typeType of the spanstring (LLM or DEFAULT)LLM
lmnr.span.pathPath of the spanstringagent.generate.openai.chat
gen_ai.systemModel providerstringopenai
gen_ai.usage.output_tokensNumber of tokens the LLM producednumber369
gen_ai.usage.input_tokensNumber of tokens in the LLM inputnumber42
llm.usage.total_tokensTotal number of tokens in the LLM input and output. If not specified, sum of input_tokens and output_tokensnumber411
gen_ai.usage.costCost of the LLM call. If not specified, system, input_tokens, and output_tokens are used to estimate the cost Laminar-side in the backgroundnumber0.012
gen_ai.usage.input_costCost of the inputs to the LLM call. If not specified, system and input_tokens are used to estimate the cost Laminar-side in the backgroundnumber0.003
gen_ai.usage.output_costCost of the outputs of the LLM call. If not specified, system, and output_tokens are used to estimate the cost Laminar-side in the backgroundnumber0.009
gen_ai.usage.request_modelModel name specified in the requeststringgpt-4o
gen_ai.usage.response_modelModel name returned in the responsestringgpt-4o-2024-08-06
gen_ai.prompt.{i}.contentContent of the input message number i (starting from 0)stringwrite a poem about laminar flow
gen_ai.prompt.{i}.roleRole of the input message number i (starting from 0)stringuser
These and all other attributes are stored as key-value pairs in the attributes field of the span.