Observability

Syntropic137 captures every agent action in real time. Events flow from workspace containers through the collector service into TimescaleDB, where projections build the read models that power the dashboard.

Two-Channel Pipeline

Workspace containers emit observability data through two complementary channels:

Both channels flow through syn-collector (/events for plugin hooks, /v1/metrics + /v1/logs for OTel) and are stored in the same agent_events table in TimescaleDB.

Why two channels?

OTel can't capture everything: subagent lifecycle, context compaction, and granular git operations (branch, merge, push) are not exported by Claude Code's OTel implementation. Conversely, plugin hooks can't give you per-API-call cost breakdowns or retry counts. The channels are complementary:

Event Flow

Workspace Container
  ├── Plugin hooks → stderr → orchestrator captures JSONL → POST /events
  └── Claude Code OTel → OTLP JSON push → POST /v1/metrics, /v1/logs
                                                    ↓
                                              syn-collector
                                          (dedup, store, batch)
                                                    ↓
                                              TimescaleDB
                                          (agent_events hypertable)
                                                    ↓
                                            Projections
                                     (session_cost, tool_timeline,
                                      execution_cost, realtime SSE)
                                                    ↓
                                              Dashboard + API

Event Types

Plugin Hook Events (JSONL channel)

OTel Events (OTLP channel)

Collector Configuration

syn-collector receives events from workspace containers. It must be reachable on the Docker network that workspace containers use (agent-net).

The collector endpoint is auto-injected into workspace containers by the orchestrator when COLLECTOR_URL is set in syn-api's environment:

# In your .env / environment config
COLLECTOR_URL=http://collector:8080

When COLLECTOR_URL is set, every workspace container gets:

• CLAUDE_CODE_ENABLE_TELEMETRY=1: activates Claude Code's OTel export
• OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:8080: directs OTel push to the collector

If COLLECTOR_URL is not set, OTel export is a graceful no-op: containers still emit plugin hook events via the JSONL channel.

Verifying Events Are Flowing

Health check

curl http://localhost:8080/health
# → {"status": "healthy"}

Check recent events

curl -s http://localhost:8080/events/recent | \
  jq 'group_by(.event_type) | map({type: .[0].event_type, count: length})'

After a workflow execution you should see both channels represented:

[
  {"type": "api_request", "count": 12},
  {"type": "cost_recorded", "count": 3},
  {"type": "git_commit", "count": 2},
  {"type": "session_started", "count": 1},
  {"type": "subagent_started", "count": 2},
  {"type": "token_usage", "count": 18},
  {"type": "tool_execution_completed", "count": 24}
]

Verify collector is reachable from agent-net

docker run --rm --network syn-dev_agent-net curlimages/curl \
  curl -s http://collector:8080/health
# → {"status": "healthy"}

Token Cost Optimization

Workspace images include built-in optimizations that reduce token consumption, and the observability pipeline captures the evidence.

RTK Bash Compression

RTK is pre-installed in workspace images and intercepts Bash tool calls. It compresses output from commands like ls, find, git log, and tree into token-efficient formatting. Claude Code uses these aliases transparently, no configuration needed.

Measured impact from A/B testing on an "explore repository" task:

RTK primarily reduces input tokens, the largest cost driver. This shows up in the OTel channel's api_request events: compare input_tokens vs cache_read_tokens to see how much context the model is consuming per call.

Observing the Impact

With both channels active, you can track token efficiency per session:

• api_request events (OTel): input_tokens, output_tokens, cache_read_tokens per API call. Lower input_tokens relative to task complexity indicates RTK compression is working.
• token_usage events (both channels): cumulative input/output/cache totals per turn.
• cost_recorded events (OTel): running USD cost. Compare sessions with similar tasks to measure savings.

The dashboard's session cost view aggregates these into per-session and per-execution cost breakdowns.

Deduplication

Events are identified by a deterministic SHA256 event_id derived from content (session ID, metric name, timestamp, index). The collector deduplicates within a sliding window, safe to retry any event submission without double-counting.