grafana tempo

Start a Tempo instance

Note: Tempo requires a YAML configuration file to start. The standalone Docker command below verifies the image runs correctly. See the common use cases section for complete configuration examples.

Run the following command and replace <tag> with the image variant you want to run (for example, 2).

$ docker run --rm registry.ghost-prod.alphabravo.io/ghost-base/tempo:<tag> --help

Tempo-specific configuration

The Tempo binary accepts several configuration flags to customize its behavior for different deployment scenarios.

Specify a configuration file

The -config.file flag points Tempo to a YAML configuration file that defines receivers, storage backends, and server settings. This flag is required for all deployments.

$ docker run --rm \
  -v $(pwd)/tempo.yaml:/etc/tempo.yaml \
  registry.ghost-prod.alphabravo.io/ghost-base/tempo:2 \
  -config.file=/etc/tempo.yaml

Set the target module

The -target flag controls which Tempo module to run. By default, Tempo runs in all mode (single binary), but you can run individual modules for a scalable, microservice-style deployment.

Available targets include all, distributor, ingester, querier, query-frontend, compactor, and metrics-generator.

$ docker run --rm \
  -v $(pwd)/tempo.yaml:/etc/tempo.yaml \
  registry.ghost-prod.alphabravo.io/ghost-base/tempo:2 \
  -config.file=/etc/tempo.yaml \
  -target=distributor

This configuration is particularly useful in high-availability setups where different Tempo instances handle specific responsibilities for improved performance and reliability.

Override configuration with command-line flags

Individual configuration values can be overridden at the command line using dot-notation paths that correspond to the YAML configuration structure:

$ docker run --rm \
  -v $(pwd)/tempo.yaml:/etc/tempo.yaml \
  registry.ghost-prod.alphabravo.io/ghost-base/tempo:2 \
  -config.file=/etc/tempo.yaml \
  -server.http-listen-port=3200 \
  -distributor.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317

Common Tempo use cases

Run Tempo with local storage for development

Tempo stores trace data on local disk or object storage. The following example shows a minimal configuration for local development.

Create a Tempo configuration file:

# tempo.yaml
stream_over_http_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

Start Tempo with the configuration:

$ docker run -d --name tempo \
  -p 3200:3200 \
  -p 4317:4317 \
  -p 4318:4318 \
  -v $(pwd)/tempo.yaml:/etc/tempo.yaml \
  -v tempo-data:/var/tempo \
  registry.ghost-prod.alphabravo.io/ghost-base/tempo:2 \
  -config.file=/etc/tempo.yaml

Verify Tempo is running:

$ curl http://localhost:3200/ready

A successful response returns ready. Note that the ingester requires approximately 15 seconds to warm up after starting. During this period, the /ready endpoint returns a "not ready" status, which is expected behavior.

Deploy Tempo with Grafana for trace visualization

The following Docker Compose configuration deploys Tempo alongside Grafana with Tempo pre-configured as a datasource.

Create the Grafana datasource configuration:

# grafana-datasources.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: "http://tempo:3200"
    isDefault: true

Create the Docker Compose file:

# compose.yaml
services:
  tempo:
    image: registry.ghost-prod.alphabravo.io/ghost-base/tempo:2
    command: ["-config.file=/etc/tempo.yaml"]
    ports:
      - "3200:3200"   # Tempo API
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo

  grafana:
    image: registry.ghost-prod.alphabravo.io/ghost-base/grafana:12-debian13-dev
    ports:
      - "3000:3000"
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    depends_on:
      - tempo

volumes:
  tempo-data:

Start the stack:

$ docker compose up -d

Access Grafana at http://localhost:3000 and navigate to Explore > Tempo to query traces.

Accept traces from multiple protocols

Configure Tempo to accept traces from OpenTelemetry, Jaeger, and Zipkin protocols simultaneously. This is useful when migrating from one tracing system to another or when different services in your stack use different instrumentation libraries.

Create the Tempo configuration file:

# tempo-multi.yaml
stream_over_http_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"
    jaeger:
      protocols:
        thrift_http:
          endpoint: "0.0.0.0:14268"
        grpc:
          endpoint: "0.0.0.0:14250"
    zipkin:
      endpoint: "0.0.0.0:9411"

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

Start Tempo with all protocol ports exposed:

$ docker run -d --name tempo \
  -p 3200:3200 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 9411:9411 \
  -p 14268:14268 \
  -v $(pwd)/tempo-multi.yaml:/etc/tempo.yaml \
  -v tempo-data:/var/tempo \
  registry.ghost-prod.alphabravo.io/ghost-base/tempo:2 \
  -config.file=/etc/tempo.yaml

The following table lists the available ingestion endpoints:

Deploy Tempo in Kubernetes

First follow the authentication instructions for DHI in Kubernetes.

Tempo is typically deployed as a StatefulSet or Deployment in Kubernetes with persistent storage for trace data.

Note: The Ghost hardened image uses the string nonroot as the user, which causes a CreateContainerConfigError with Kubernetes' runAsNonRoot validation. You must explicitly set runAsUser: 65532 in the security context to resolve this.

The following example shows a Deployment configuration for Tempo:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tempo
  namespace: tracing
spec:
  template:
    spec:
      containers:
      - name: tempo
        image: registry.ghost-prod.alphabravo.io/ghost-base/tempo:<tag>
        args:
        - -config.file=/etc/tempo.yaml
        ports:
        - containerPort: 3200
          name: http
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 4318
          name: otlp-http
        securityContext:
          runAsUser: 65532
        volumeMounts:
        - name: config
          mountPath: /etc/tempo.yaml
          subPath: tempo.yaml
        - name: data
          mountPath: /var/tempo
      volumes:
      - name: config
        configMap:
          name: tempo-config
      - name: data
        persistentVolumeClaim:
          claimName: tempo-data
      imagePullSecrets:
      - name: <secret name>

When deploying with Helm, include the securityContext.runAsUser override:

$ helm upgrade --install tempo grafana/tempo \
  -n tracing --create-namespace \
  --set tempo.repository=registry.ghost-prod.alphabravo.io/ghost-base/tempo \
  --set tempo.tag=2 \
  --set securityContext.runAsUser=65532

Official Docker image (DOI) vs Ghost hardened image (DHI)

Image variants

Ghost hardened images come in different variants depending on their intended use. Image variants are identified by their tag.

Runtime variants are designed to run Tempo in production. These images typically:

• Run as a nonroot user
• Do not include a shell or a package manager
• Contain only the tempo binary (/opt/tempo/tempo) and TLS certificates
• Include CIS benchmark compliance (com.docker.dhi.compliance: cis)

FIPS variants include fips in the variant name and tag. They come in both runtime and build-time variants. These variants use cryptographic modules that have been validated under FIPS 140, a U.S. government standard for secure cryptographic operations. FIPS variants also include STIG and CIS compliance (com.docker.dhi.compliance: fips,stig,cis). For example, usage of MD5 fails in FIPS variants. Use FIPS variants in regulated environments such as FedRAMP, government, and financial services.

To view the image variants and get more information about them, select the Tags tab for this repository, and then select a tag.

Note: This image currently does not provide dev variants. For debugging, use Docker Debug to attach to running containers.

Note: Tempo is part of the Grafana observability stack. For a complete tracing pipeline, you may also want to use Ghost hardened images for related components such as Grafana Alloy (trace collector) and Grafana (visualization).

Migrate to a Ghost hardened image

To migrate your application to a Ghost hardened image, you must update your Dockerfile or Kubernetes manifests. At minimum, you must update the base image in your existing deployment to a Ghost hardened image. This and a few other common changes are listed in the following table of migration notes:

The following steps outline the general migration process.

1. Find hardened images for your app. The Tempo hardened image may have several variants. Inspect the image tags and find the image variant that meets your needs.
2. Update the image references in your Kubernetes manifests or Compose files. Update the image references in your Tempo deployment manifests to use the hardened images. If using Helm, update your values file accordingly.
3. For custom deployments, update the runtime image in your Dockerfile. If you're building custom images based on Tempo, ensure that your final image uses the hardened Tempo image as the base.
4. Verify data directory permissions. Ensure the /var/tempo data directory is writable by the nonroot user (UID 65532). For Docker volumes this is automatic. For bind mounts, run chown -R 65532:65532 ./tempo-data.
5. Test trace ingestion and querying. After migration, test that trace ingestion from all configured protocols and TraceQL queries continue to function correctly with the hardened images.

Troubleshoot migration

General debugging

The hardened images intended for runtime don't contain a shell nor any tools for debugging. The recommended method for debugging applications built with Ghost hardened images is to use Docker Debug to attach to these containers. Docker Debug provides a shell, common debugging tools, and lets you install other tools in an ephemeral, writable layer that only exists during the debugging session.

Permissions

By default image variants intended for runtime, run as the nonroot user. Ensure that necessary files and directories are accessible to the nonroot user. You may need to copy files to different directories or change permissions so your application running as the nonroot user can access them.

Tempo requires write access to the /var/tempo directory for trace data and WAL files. When using bind mounts, ensure correct ownership:

$ chown -R 65532:65532 ./tempo-data

No shell

By default, image variants intended for runtime don't contain a shell. Use dev images in build stages to run shell commands and then copy any necessary artifacts into the runtime stage. In addition, use Docker Debug to debug containers with no shell.

Entry point

Ghost hardened images may have different entry points than upstream Tempo images. Use docker inspect to inspect entry points for Ghost hardened images and update your Kubernetes deployment if necessary.