zero gravitySOLUTIONS
← All insightsCLOUD & OPERATIONS / OPERATING GUIDE

Tracing one request across API queue and worker.

QUICK ANSWER

Carry trace context with queue messages and create a processing span for each delivery attempt. Keep the durable business operation ID separate so retries and completion remain explainable even when traces are sampled or expired.

Inside this article 5 sections

To trace a request across an API, queue, and worker, propagate trace context with the message and create a separate processing span for each delivery attempt. Keep a durable business operation ID alongside the trace. Traces explain execution; the operation record establishes whether the customer's work was accepted and completed.

On AWS, an API running on ECS can enqueue SQS work that another ECS or Fargate task processes later. HTTP instrumentation alone cannot explain the gap between acceptance and completion. The asynchronous handoff needs its own context, timing, and retry model.

Give each identifier one job

Consider an illustrative account export. The application assigns operation export-204, stores the export request, and sends a message. The trace ID connects spans from the original interaction. The SQS message ID identifies a transport message. A worker attempt ID identifies one execution. None of these identifiers should replace the account-scoped idempotency key that prevents duplicate exports.

A customer may query the operation hours later, after the original trace is no longer retained. An operator may replay a failed message under a new trace. The durable operation record must still connect those attempts to the same requested result. Store that relationship in logs or application state instead of relying on the tracing backend as the system of record.

Original trace handoff · API to SQS to worker
API requestAccept export-204Request span and durable operation IDSQS messageCarry contextTrace carrier plus business envelopeWorker deliveryProcess attempt 2New span; same operation identity
Queue wait, worker execution, and customer-visible completion are separate durations.
The second processing attempt is observable without pretending that the first attempt never happened or that a new business operation was requested.

Choose the propagation path explicitly

For new instrumentation, use OpenTelemetry or AWS Distro for OpenTelemetry rather than starting a new dependency on the legacy X-Ray SDK. AWS placed the X-Ray SDKs and daemon in maintenance mode in February 2026 and recommends migration. X-Ray remains a possible trace destination; instrumentation and destination are separate choices.

Agree on a propagator across producer and consumer. A W3C carrier commonly contains traceparent and optionally tracestate. An X-Ray propagation path uses its own header format. SQS reserves the AWSTraceHeader system attribute for a correctly formatted X-Ray header; placing a W3C traceparent string there is not a valid substitution.

If the application carries W3C context in custom SQS message attributes, explicitly request those attributes when receiving messages and extract them before starting the processing span. If AWS instrumentation handles the X-Ray system attribute, verify the actual producer and consumer behavior rather than adding a second unrelated trace chain.

Make the carrier transformation testable

This Python example demonstrates only the application-controlled W3C attribute mapping. The carrier supplied to it should come from the configured OpenTelemetry propagator; extraction should pass the returned dictionary back to that propagator. It is not a complete AWS client or tracing setup.

TRACE_FIELDS = ("traceparent", "tracestate")

def to_sqs_attributes(carrier):
    return {
        key: {"DataType": "String", "StringValue": carrier[key]}
        for key in TRACE_FIELDS
        if carrier.get(key)
    }

def from_sqs_attributes(attributes):
    return {
        key: attributes[key]["StringValue"]
        for key in TRACE_FIELDS
        if key in attributes
        and attributes[key].get("DataType") == "String"
        and "StringValue" in attributes[key]
    }

Allowlist propagation fields. Do not forward arbitrary request headers or personal data as baggage. Treat externally supplied trace context as untrusted metadata, enforce size limits at the boundary, and never use a trace identifier as authorization. The configured propagator should handle malformed or absent trace context without rejecting otherwise valid business work.

Model batches and retries honestly

A single-message consumer can use the producer's creation context as a parent when that matches the tracing design. For a batch containing several independently created messages, one processing span cannot have several parents. OpenTelemetry messaging conventions use span links to represent those relationships. Per-message spans can provide more detail when the operational value justifies their volume.

Each retry should create a fresh processing span with its attempt number and outcome. Record whether the worker failed before committing the export, after committing but before acknowledgment, or during notification. Reusing one span identifier across attempts obscures those boundaries and may produce invalid telemetry.

Queue wait should be derived from a defined timestamp and observation point. Compare the business enqueue time with processing start, while accounting for clock skew if they come from different hosts. SQS metadata can help diagnose transport age, but business latency also includes time before the message was sent and after processing finished.

Keep context local to each message handler. A worker may process several accounts concurrently in one process; a global variable holding the current trace can attach one customer’s spans to another customer’s job. Use the language runtime’s supported asynchronous context mechanism and test concurrent handlers with different carriers.

An outbox adds another handoff. Persist the chosen creation context with the outbox record if it is part of the trace design, then have the dispatcher preserve it when publishing. Give dispatcher retries their own execution evidence. A database transaction that commits before the dispatcher runs should not keep an HTTP span artificially open until the queue eventually accepts the message.

Verify the exporter and the business path separately

With ADOT, configure the SDK, resource identity, sampling, and export path deliberately. A collector can receive application telemetry and export to X-Ray with the required permissions. Collector health is another dependency: a broken exporter should be visible without blocking every customer export indefinitely.

Test a normal delivery, a missing context attribute, a malformed carrier, and a worker crash before SQS deletion. Confirm the trace relationships, attempt logs, and durable operation result. Then test a batch with two different originating traces and inspect the links. A trace diagram that looks connected is insufficient if operation IDs or tenant context are wrong.

Use metrics for complete service-level counts because traces may be sampled. The background job reconciliation guide supplies completion evidence, and customer-impact alerting explains which failures deserve action. Architecture design can align trace context, operation state, and worker recovery into one observable execution path.

FROM INSIGHT TO ACTIONSoftware architecture consulting

Make asynchronous execution traceable end to end.

We can align OpenTelemetry propagation, AWS export configuration, operation records, and recovery evidence across your API and workers.

Explore architecture consultingDiscuss your architecture

Start with your goals.
We’ll work through the next step together.