Observability tools

Every call to the inference API is on Models > Observability for 30 days. If your team already watches its model calls somewhere else, Runcrate sends each call there too: its model, tokens, cost, how long it took, its TTFT, its status, and the end user, session and tags you sent with it.

How it works

Connect the tool

In Settings > Integrations, select Connect beside the tool and paste its key. Runcrate sends one sample call first and saves the connection only if the tool accepts it. Keys are encrypted at rest and never shown again. Owners and managers can connect tools.

Pick the environments

On Models > Observability, the Send to line names the connected tools. Select Choose per environment and tick the tools each environment sends to. An environment sends nowhere until you pick one, and a tool gets calls made from the moment it was picked, not the days before.

Watch the calls arrive

New calls are sent in batches every minute. The tool’s row in Integrations shows the last batch it took, the last error in plain words and how many calls are still to send. Test sends one more sample call and shows the tool’s answer.

Prompts and answers

A call is sent without its prompt and answer unless both of these are on:

the environment saves prompts and answers (Settings > Environments), and
the tool’s Send prompts and answers switch (off by default).

Prompts and answers saved as cut short are sent as saved; images, audio and video are never sent.

When a tool fails

If a tool refuses a batch or does not answer within 15 seconds, Runcrate keeps the batch and tries again after 1, 2, 4, 8 and 16 minutes, then every 30 minutes, for 24 hours. Calls are always sent in order and none is skipped: a tool that is down for an hour gets that hour’s calls once it is back. After 24 hours of failures the tool is Paused. Fix the cause (a new key, a new address) and select Resume: the calls it missed are sent while they are still kept (30 days). A batch can arrive twice (after an answer that came too late). Each call keeps the same ids every time it is sent: its trace and span ids, its PostHog event uuid, its data[].id on the webhook, so a repeat can be told apart. What is never sent: which supplier served a call, or what it cost us.

Langfuse

Calls go to Langfuse’s OpenTelemetry endpoint (/api/public/otel/v1/traces, on its v4 data model, so they show at once). Each call is one generation (an embedding for /v1/embeddings) in a trace of its own:

Langfuse	From the call
Trace name	The endpoint and model, such as `chat meta-llama/Llama-3.3-70B-Instruct`
`metadata.request_id`	The `X-Request-Id` the API returned
`model`	The model id, such as `meta-llama/Llama-3.3-70B-Instruct`
`usageDetails`	`input`, `output`, `input_cached_tokens`, `input_cache_write_tokens`, `output_reasoning_tokens`, `total`
`costDetails.total`	What the call was billed, in USD
`startTime`, `completionStartTime`, `endTime`	When it arrived, its first token, its last byte
`userId`, `sessionId`	The end user and session you sent
`tags`	Your tags, as `key:value`
`environment`	The Runcrate environment’s name, in lowercase
`level`, `statusMessage`	`ERROR` and the error, for a failed call
`input`, `output`	The prompt and answer, when both switches are on

Feedback you send for a call (thumbs or a rating, from the API, MCP, the playground or the dashboard) becomes a NUMERIC score on its generation, from -1 to 1, named by its label or feedback, with your comment. A changed rating replaces its score. To connect: pick the region (EU cloud, US cloud or your own host, Langfuse v3.22 or later) and paste a public key (pk-lf-…) and secret key (sk-lf-…) from your project’s Settings > API keys.

Datadog

Choose how each call arrives:

A log (every Datadog plan): source:runcrate, your service name, status:error for a failed call, tags env, model, endpoint, source, session_id and your own, the end user as usr.id, and the call under the runcrate attribute (tokens, cost, timings, status).
An LLM Observability span (where your account has LLM Observability): one llm span per call under your ML app name, with model_name, input and output tokens, TTFT, cost and status, and the prompt and answer as messages when both switches are on.

To connect: pick your site (the region in your Datadog address: US1, US3, US5, EU, AP1, AP2 or US1-FED) and paste an API key from Organization settings > API keys. An application key will not work.

OpenTelemetry

Any backend that takes OTLP over HTTP. Each call is one CLIENT span, named like chat meta-llama/Llama-3.3-70B-Instruct, with the GenAI semantic conventions:

Attribute	From the call
`gen_ai.operation.name`	`chat`, `text_completion`, `embeddings` or `generate_content`
`gen_ai.provider.name`	`runcrate`
`gen_ai.request.model`, `gen_ai.response.model`	The model id
`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`	Tokens in and out
`gen_ai.usage.cache_read.input_tokens`	Cached input tokens
`gen_ai.response.finish_reasons`	Why it stopped
`gen_ai.conversation.id`	Your session
`gen_ai.input.messages`, `gen_ai.output.messages`	The prompt and answer, when both switches are on
`user.id`	Your end user
`error.type`, span status	The error, for a failed call
`runcrate.*`	Request id, environment, cost (USD), TTFT, and `runcrate.tag.<key>` per tag

To connect: paste the OTLP/HTTP endpoint (Runcrate adds /v1/traces unless it is there) and the headers your backend wants. Start from one of these:

Backend	Endpoint	Header
Honeycomb	`https://api.honeycomb.io`	`x-honeycomb-team: <ingest key>`
Grafana Cloud	`https://otlp-gateway-prod-<region>.grafana.net/otlp`	`Authorization: Basic <base64 of instance ID:token>`
SigNoz Cloud	`https://ingest.<region>.signoz.cloud:443`	`signoz-ingestion-key: <key>`
Elastic	Your deployment’s APM endpoint	`Authorization: ApiKey <key>`

The address must be public and use https.

PostHog

Each call is a $ai_generation event in PostHog’s LLM analytics: $ai_model, $ai_provider (runcrate), $ai_input_tokens, $ai_output_tokens, $ai_total_cost_usd, $ai_latency and $ai_time_to_first_token in seconds, $ai_http_status, $ai_is_error and $ai_error, $ai_trace_id (the call’s id), $ai_session_id, and $ai_input and $ai_output_choices when both switches are on. Your tags arrive as tag_<key> properties. The event’s person is your end user. A call without one is sent under runcrate:<API key id> without a person profile, so anonymous traffic does not fill your persons list. Feedback is a $ai_metric event on the call’s trace (and a $ai_feedback event when it has a comment). To connect: pick US cloud, EU cloud or your own host and paste the project API key (phc_…) from Project settings. It can only send events, never read them.

Webhook

Your own HTTPS endpoint gets each batch as one JSON POST, signed like every Runcrate webhook:

POST /your/endpoint
Content-Type: application/json
Runcrate-Event: inference.requests
Runcrate-Delivery: 6f1d1a0e-8a5b-4d3c-9a43-0d6c9f0f2a11
Runcrate-Signature: t=1792756800,v1=5d2f8c0b9e...e91a

{
  "type": "inference.requests",
  "api_version": "2026-10-23",
  "id": "6f1d1a0e-8a5b-4d3c-9a43-0d6c9f0f2a11",
  "created_at": "2026-10-23T12:05:00.000Z",
  "workspace_id": "1c8c7fa0-0497-4600-b284-16609d52d991",
  "data": [
    {
      "id": "a0000000-0000-4000-8000-000000000001",
      "request_id": "req_8f2c",
      "created_at": "2026-10-23T12:00:00.000Z",
      "environment": { "id": "e0000000-0000-4000-8000-000000000001", "name": "Production" },
      "api_key_id": "k0000000-0000-4000-8000-000000000001",
      "endpoint": "chat",
      "model": "meta-llama/Llama-3.3-70B-Instruct",
      "status": 200,
      "outcome": "ok",
      "error": null,
      "latency_ms": 1500,
      "ttft_ms": 200,
      "tokens": { "input": 100, "cached_input": 20, "cache_write": 0, "output": 50, "reasoning": 0 },
      "cost_usd": 0.00012,
      "stream": true,
      "source": "api",
      "free": false,
      "end_user": "user-42",
      "session_id": "sess-7",
      "tags": { "team": "search" },
      "finish_reason": "stop",
      "output_tokens_per_sec": 41.5,
      "tool_calls": 1
    }
  ]
}

With both switches on, each call also carries prompt, answer and body_truncated. Feedback arrives as its own batch, "type": "inference.feedback", with request_id, call_id (the call’s id), score (-1 to 1), rating, label, comment and source. Runcrate-Signature is the hex HMAC-SHA256 of t, a full stop and the raw body, keyed with the signing secret Runcrate shows once when you connect the webhook. Verify it with the code in verifying signatures. Answer with a 2xx within 15 seconds; anything else is retried as above.

Verifying signatures

Check the Runcrate-Signature header on your webhook.

Webhooks

Events about instances, jobs and billing, posted as they happen.

Welcome

SDKs

CLI

Inference

Compute

Jobs

Schedules

Secrets

Storage

Dedicated Clusters

Billing

Account

Webhooks

Integrations

FAQ

Observability tools

How it works

Prompts and answers

When a tool fails

Langfuse

Datadog

OpenTelemetry

PostHog

Webhook

Verifying signatures

Webhooks

​How it works

​Prompts and answers

​When a tool fails

​ Langfuse

​ Datadog

​ OpenTelemetry

​ PostHog

​ Webhook

​Related

Verifying signatures

Webhooks

How it works

Prompts and answers

When a tool fails

Langfuse

Datadog

OpenTelemetry

PostHog

Webhook

Related