OpenTelemetry

Laminar uses OpenTelemetry for tracing. That is, Laminar’s backend is compatible with OpenTelemetry. This page details the methods to send your OpenTelemetry traces to Laminar, even if you are not using Laminar’s SDK.

If you are using Laminar’s SDK, refer to the tracing page to get started. Still feel free to read this page to get basic understanding of the internals of OpenTelemetry and how it works with Laminar.

​

Introduction

OpenTelemetry is a framework for tracing and monitoring distributed systems. OpenLLMetry is an extension of OpenTelemetry that is specifically designed for LLM tracing. The standards introduced in OpenLLMetry are gradually being adopted by OpenTelemetry.

OpenTelemetry covers tracing, metrics, and logs, but Laminar only uses tracing. For more details about tracing in Laminar, and a quick intro to the core concepts, see the Tracing page.

Processing of OpenTelemetry traces has many concepts and components, namely, exporters, processors, propagators, samplers, and collectors.

For the purposes of this page, we will focus on the exporters.

​

Exporters

Exporter is the client-side component that sends spans (and thus traces) to an OpenTelemetry-compatible backend.

It is responsible for the payload format and the transport protocol. The default protocol is OTLP, which underneath uses protobuf in one of the following formats:

• Protobuf in gRPC (commonly known as OTLP/gRPC)
• Protobuf over HTTP with binary encoding (commonly known as OTLP/HTTP/proto)
• Protobuf over HTTP with JSON encoding (commonly known as OTLP/HTTP/json, is used rarely).

Laminar’s backend (both cloud and self-hosted) supports OTLP/gRPC and OTLP/HTTP/proto formats. OTLP/gRPC is recommended for performance and reliability reasons.

Here’s a quick comparison of formats that Laminar supports:

[1] In OpenTelemetry Node and Python SDKs, this path is appended (unless specified explicitly) in gRPC exporter, but not in the HTTP exporter.

​

Getting started

Usually, OpenTelemetry tracing in other libraries is initialized once at the start of the application by calling a function like initTracer(). Functions like this usually accept a configuration object or a set of parameters, including the exporter configuration.

To send traces to Laminar, you need to configure the endpoint and the authorization.

​

Example

import { Metadata } from '@grpc/grpc-js';
// it is important to import the OTLP exporter from the
// `@opentelemetry/exporter-trace-otlp-grpc` package
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';

const metadata = new Metadata();
metadata.set('authorization', `Bearer ${process.env.LMNR_PROJECT_API_KEY}`);
const exporter = new OTLPTraceExporter({
  url: "https://api.lmnr.ai:8443/v1/traces",
  metadata,
});

functionThatInitiatesTracer({
    exporter
});

import { Metadata } from '@grpc/grpc-js';
// it is important to import the OTLP exporter from the
// `@opentelemetry/exporter-trace-otlp-grpc` package
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';

const metadata = new Metadata();
metadata.set('authorization', `Bearer ${process.env.LMNR_PROJECT_API_KEY}`);
const exporter = new OTLPTraceExporter({
  url: "https://api.lmnr.ai:8443/v1/traces",
  metadata,
});

functionThatInitiatesTracer({
    exporter
});

# it is important to import the OTLP exporter from the
# `opentelemetry.exporter.otlp.proto.grpc.trace_exporter` package
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter
)
import os

exporter = OTLPSpanExporter(
    endpoint="https://api.lmnr.ai:8443/v1/traces",
    # important: `authorization` starts with a lowercase letter
    headers={"authorization": f"Bearer {os.getenv('LMNR_PROJECT_API_KEY')}"},
)

function_that_initiates_tracer(
    exporter=exporter,
    # other parameters
)

​

Endpoint

The default base url for the endpoint is https://api.lmnr.ai listening for gRPC traffic on port 8443.

For the self-hosted backend, the base url is http://<your-self-hosted-backend-url> and the default port is 8001, unless you have changed the configuration.

The /v1/traces path is the default OpenTelemetry trace endpoint, and Laminar listens at this path. In both JavaScript (OpenTelemetry Node SDK) and Python OpenTelemetry SDKs, the gRPC implementation appends /v1/traces to the base url, if you don’t specify it. Be careful though if you are using the HTTP exporter, as the HTTP implementation does not append it.

​

Authorization

The authorization header is required to send traces to Laminar. We use bearer authentication, so the header should start with Bearer and the token is your project API key.

The right way to set the headers for gRPC requests is to use the metadata object. Even though gRPC is sent over HTTP/2, and metadata is sent as HTTP headers, it is different from raw HTTP headers. Learn more about metadata in the gRPC documentation.

gRPC metadata has a restricted set of keys, and so many client implementations, including the OpenTelemetry Python SDK, check the keys for validity, effectively imposing case-sensitivity. That is, authorization has to start with lowercase a.

​

Troubleshooting

If you are facing issues, please refer to the Troubleshooting OpenTelemetry page.

​

Span reference

This section is solely for information purposes. As a user of Laminar, you don’t have to deal with the internals of OpenTelemetry.

​

Span object

[1] 13th digit in hex UUID depends on UUID version, so technically converting OTel/WC3’s 16-byte and, especially, 8-byte IDs to UUIDs is not exactly UUID.

[2] AttributeType is a union of string, number, boolean, Array<string>, Array<number>, Array<boolean>

[3] In most OpenTelemetry client implementations, you don’t have to convert the timestamp to nanoseconds manually, you can simply pass the Date / datetime object and the client will convert it to nanoseconds.

​

Span attributes

Span attribute values that Laminar bases some of its functionality on:

These and all other attributes are stored as key-value pairs in the attributes field of the span.