OpenTelemetry

Hatchet supports exporting traces from your workflows to an OpenTelemetry Collector to improve visibility into your Hatchet tasks.

Usage

Setup

Hatchet’s SDK provides an instrumentor that auto-instruments Hatchet code, if you opt in. Setup is straightforward:

First, install the otel extra with (e.g.) pip install hatchet-sdk[otel]. Then, import the instrumentor:

from path.to.your.trace.provider import trace_provider
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
 
HatchetInstrumentor(tracer_provider=trace_provider).instrument()

You bring your own trace provider and plug it into the HatchetInstrumentor, call instrument, and that’s it!

Providing a traceparent

In some cases, you might also want to provide a traceparent so any spans created in Hatchet are children of a parent that was created elsewhere in your application. You can do that by providing a traceparent key in the additional_metadata field of the following methods:

• hatchet.event.push
• hatchet.event.bulk_push
• hatchet.admin.run_workflow
• hatchet.admin.run_workflows
• hatchet.admin.run_workflow_async
• hatchet.admin.run_workflows_async

For example:

hatchet.event.push(
    "user:create",
    {'userId': '1234'},
    options={
    "additional_metadata": {
        "traceparent":"00-f1aff5c5ea45185eff2a06fd5c0ed6c5-6f4116aff54d54d1-01" ## example traceparent
    }
})

The HatchetInstrumentor also has some methods for generating traceparents that might be helpful:

1. create_traceparent creates a traceparent
2. inject_traceparent_into_metadata injects a traceparent into the additional_metadata field

Spans

By default, Hatchet creates spans at the following points in the lifecycle of a workflow run:

1. When a trigger is run on the client side, e.g. run_workflow() or push() is called.
2. When a worker handles a task event, like starting running the task or cancelling the task

In addition, you’ll get a handful of attributes set (prefixed by hatchet.) on the step run events, such as the workflow name and the worker id, as well as success / failure states, and so on.