Troubleshooting for using Laminar as the backend for OpenTelemetry.
This page is relevant for the users not using Laminar’s SDK for tracing, but sending
their OpenTelemetry traces to Laminar. For a getting started guide, please refer to
the OpenTelemetry page.
I am using the OpenTelemetry Node.js SDK and I see the following error:
Copy
Error: 16 UNAUTHENTICATED: Failed to authenticate request
Copy
Error: 16 UNAUTHENTICATED: Failed to authenticate request at callErrorFromStatus (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/call.ts:82:17) at Object.onReceiveStatus (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/client.ts:360:55) at Object.onReceiveStatus (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34) at Object.onReceiveStatus (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48) at /workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24 at processTicksAndRejections (node:internal/process/task_queues:85:11)for call at at ServiceClientImpl.makeUnaryRequest (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/client.ts:325:42) at ServiceClientImpl.export (/workspace/node_modules/.pnpm/@grpc+grpc-js@1.12.5/node_modules/@grpc/grpc-js/src/make-client.ts:189:15) at /workspace/node_modules/.pnpm/@opentelemetry+otlp-grpc-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-grpc-exporter-base/src/grpc-exporter-transport.ts:159:26 at new Promise (<anonymous>) at GrpcExporterTransport.send (/workspace/node_modules/.pnpm/@opentelemetry+otlp-grpc-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-grpc-exporter-base/src/grpc-exporter-transport.ts:146:12) at OTLPExportDelegate.export (/workspace/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/otlp-export-delegate.ts:82:23) at OTLPTraceExporter.export (/workspace/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/OTLPExporterBase.ts:32:26) at doExport (/workspace/node_modules/.pnpm/@opentelemetry+sdk-trace-base@1.30.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/sdk-trace-base/src/export/BatchSpanProcessorBase.ts:193:32) at /workspace/node_modules/.pnpm/@opentelemetry+sdk-trace-base@1.30.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/sdk-trace-base/src/export/BatchSpanProcessorBase.ts:219:11 at AsyncLocalStorage.run (node:internal/async_local_storage/async_hooks:91:14) { code: 16, details: 'Failed to authenticate request', metadata: Metadata { internalRepr: Map(3) { 'content-type' => [Array], 'content-length' => [Array], 'date' => [Array] }, options: {} }}
I have checked that my LMNR_PROJECT_API_KEY is correct and I set the headers when initializing the exporter.
One common cause is that the authorization header on the gRPC exporter is set
as a raw HTTP header, instead of a gRPC metadata header. Read the
authorization section for more information.
This error occurs when the exporter is configured to use the HTTP endpoint, but
sends traces to the gRPC endpoint. The error message indicates the mismatch
between the HTTP versions. gRPC uses HTTP/2.0, while the HTTP exporter uses
HTTP/1.1.
Make sure that you are importing the gRPC exporter from the
@opentelemetry/exporter-trace-otlp-grpc package.
Copy
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';// not from `@opentelemetry/exporter-trace-otlp-proto`// not from `@opentelemetry/exporter-trace-otlp-http`
I am using the OpenTelemetry Node.js SDK and I see the following error:
Copy
OTLPExporterError: Not Found
Copy
OTLPExporterError: Not Found at IncomingMessage.<anonymous> (/workspace/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/transport/http-transport-utils.ts:75:23) at IncomingMessage.emit (node:events:525:35) at IncomingMessage.emit (node:domain:489:12) at endReadableNT (node:internal/streams/readable:1696:12) at processTicksAndRejections (node:internal/process/task_queues:90:21) { data: '', code: 404}
We recommend using the gRPC exporter,
as it is more reliable and faster. Make sure you are importing
OTLPTraceExporter from @opentelemetry/exporter-trace-otlp-grpc.
If you have to use the HTTP exporter, make sure that you are using the correct endpoint.
The endpoint for HTTP is https://api.lmnr.ai:443/v1/traces (port 443).
Copy
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';const exporter = new OTLPSpanExporter({ url: "https://api.lmnr.ai:443/v1/traces", // note the path `/v1/traces` headers: { Authorization: `Bearer ${process.env.LMNR_PROJECT_API_KEY}`, },});
I am using the OpenTelemetry Node.js SDK and I see the following error:
Copy
OTLPExporterError: Internal Server Error
Copy
OTLPExporterError: Internal Server Error at IncomingMessage.<anonymous> (/workspace/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/transport/http-transport-utils.ts:75:23) at IncomingMessage.emit (node:events:525:35) at IncomingMessage.emit (node:domain:489:12) at endReadableNT (node:internal/streams/readable:1696:12) at processTicksAndRejections (node:internal/process/task_queues:90:21) { data: '', code: 500}
While this may indicate any issue with the Laminar server, one of the most common cases
is attempting to send HTTP/json traces to HTTP/proto endpoint. Laminar does NOT support
HTTP/json traces.
We recommend using the gRPC exporter,
as it is more reliable and faster. Make sure you are importing
OTLPTraceExporter from @opentelemetry/exporter-trace-otlp-grpc.
If you have to use the HTTP exporter, make sure that you are using are importing
OTLPTraceExporter from @opentelemetry/exporter-trace-otlp-proto, NOT from
@opentelemetry/exporter-trace-otlp-http, as the latter is for HTTP/json traces.
I am using the OpenTelemetry Python SDK and I see the following error:
Copy
TypeError: not all arguments converted during string formatting
Copy
Traceback (most recent call last): File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 360, in _export_batch self.span_exporter.export(self.spans_list[:idx]) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/grpc/trace_exporter/__init__.py", line 143, in export return self._export(spans) ^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 299, in _export self._client.Export( File "/workspace/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1178, in __call__ ) = self._blocking( ^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1146, in _blocking call = self._channel.segregated_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 547, in grpc._cython.cygrpc.Channel.segregated_call File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 416, in grpc._cython.cygrpc._segregated_call File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 410, in grpc._cython.cygrpc._segregated_call File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 262, in grpc._cython.cygrpc._call File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 305, in grpc._cython.cygrpc._call File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 62, in grpc._cython.cygrpc._raise_call_error File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 32, in grpc._cython.cygrpc._call_error File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 23, in grpc._cython.cygrpc._call_error_metadataTypeError: not all arguments converted during string formatting
This error indicates that some of the headers that you passed to the gRPC trace exporter
are not within the allowed set of keys. For Laminar uses, this is mosth likely the
authorization header.
I am using the OpenTelemetry Python SDK and I see the following error:
Copy
ConnectionResetError: [Errno 54] Connection reset by peer
Copy
Traceback (most recent call last): File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 292, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/socket.py", line 707, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ConnectionResetError: [Errno 54] Connection reset by peerDuring handling of the above exception, another exception occurred:Traceback (most recent call last): File "/workspace/.venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 292, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/socket.py", line 707, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 360, in _export_batch self.span_exporter.export(self.spans_list[:idx]) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 189, in export return self._export_serialized_spans(serialized_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 159, in _export_serialized_spans resp = self._export(serialized_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 133, in _export return self._session.post( ^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/.venv/lib/python3.12/site-packages/requests/adapters.py", line 682, in send raise ConnectionError(err, request=request)requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
This error indicates that the connection was reset at Laminar’s backend. Most likely,
this error indicates an HTTP version mismatch. One common cause for this is using
the HTTP exporter against the gRPC endpoint.
Laminar accepts traces via both gRPC and HTTP. We recommend using the gRPC exporter,
as it is more reliable and faster. Make sure you are importing
OTLPTraceExporter from opentelemetry.exporter.otlp.proto.grpc.trace_exporter and not
from opentelemetry.exporter.otlp.proto.http.trace_exporter.
If you have to use the HTTP exporter, make sure that you are using the correct endpoint.
The endpoint for HTTP is https://api.lmnr.ai:443/v1/traces (port 443).
Copy
import osfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporterexporter = OTLPSpanExporter( endpoint="https://api.lmnr.ai:443/v1/traces", # note the port 443 headers={"authorization": f"Bearer {os.getenv('LMNR_PROJECT_API_KEY')}"},)
We recommend using the gRPC exporter,
as it is more reliable and faster. Make sure you are importing
OTLPTraceExporter from opentelemetry.exporter.otlp.proto.grpc.trace_exporter and not
from opentelemetry.exporter.otlp.proto.http.trace_exporter.
If you have to use the HTTP exporter, make sure that you are using the correct endpoint.
The endpoint for HTTP is https://api.lmnr.ai:443/v1/traces (port 443).