# Observability
> "How Syntropic137 collects agent events: two-channel pipeline, event types, and how to verify everything is flowing."

Syntropic137 captures every agent action in real time. Events flow from workspace containers through the collector service into TimescaleDB, where projections build the read models that power the dashboard.

## Two-Channel Pipeline

Workspace containers emit observability data through two complementary channels:

| Channel | Transport | What It Captures |
|---------|-----------|-----------------|
| **Plugin hooks** | JSONL to stderr, captured by the orchestrator | Session lifecycle, subagent tracking, tool start/end, git operations, context compaction, permissions |
| **Native OTel** | OTLP HTTP push to syn-collector | Per-API-call cost and model, cache token breakdown, API errors, active time, lines of code |

Both channels flow through **syn-collector** (`/events` for plugin hooks, `/v1/metrics` + `/v1/logs` for OTel) and are stored in the same `agent_events` table in TimescaleDB.

### Why two channels?

OTel can't capture everything: subagent lifecycle, context compaction, and granular git operations (branch, merge, push) are not exported by Claude Code's OTel implementation. Conversely, plugin hooks can't give you per-API-call cost breakdowns or retry counts. The channels are complementary:

| Event | Source |
|-------|--------|
| Subagent started / stopped | Plugin hooks only |
| Context compaction (before/after tokens) | Plugin hooks only |
| Git commit, push, merge details | Plugin hooks only |
| Per-API-call cost + model | OTel only |
| API errors (status code, retries) | OTel only |
| Cache token breakdown | OTel only |
| Token usage totals | Both (deduplicated) |
| Tool execution completed | Both (deduplicated) |

## Event Flow

```
Workspace Container
  ├── Plugin hooks → stderr → orchestrator captures JSONL → POST /events
  └── Claude Code OTel → OTLP JSON push → POST /v1/metrics, /v1/logs
                                                    ↓
                                              syn-collector
                                          (dedup, store, batch)
                                                    ↓
                                              TimescaleDB
                                          (agent_events hypertable)
                                                    ↓
                                            Projections
                                     (session_cost, tool_timeline,
                                      execution_cost, realtime SSE)
                                                    ↓
                                              Dashboard + API
```

## Event Types

### Plugin Hook Events (JSONL channel)

| Event Type | Trigger | Key Data |
|------------|---------|----------|
| `session_started` | Agent session begins | source, cwd, permission_mode |
| `session_ended` | Agent session ends | reason, duration_ms |
| `agent_stopped` | Agent stopped | reason |
| `subagent_started` | Task tool invoked | subagent_id |
| `subagent_stopped` | Task tool completed | subagent_id, reason |
| `tool_execution_started` | PreToolUse hook | tool_name, tool_use_id, input_preview |
| `tool_execution_completed` | PostToolUse hook | tool_name, duration_ms, success |
| `pre_compact` | Context compaction | before_tokens, after_tokens, reduction_percent |
| `git_commit` | post-commit hook | sha, branch, files_changed, insertions, deletions |
| `git_push_started` | pre-push hook | remote, branch, commits_count |
| `git_merge_completed` | post-merge hook | branch, merge_sha |
| `user_prompt_submitted` | UserPromptSubmit hook | prompt_preview |

### OTel Events (OTLP channel)

| Event Type | OTel Source | Key Data |
|------------|------------|----------|
| `token_usage` | `claude_code.token.usage` metric | value, type (input/output/cache) |
| `cost_recorded` | `claude_code.cost.usage` metric | value (USD) |
| `api_request` | `claude_code.api_request` log | model, cost_usd, duration_ms, input_tokens, output_tokens, cache_read_tokens, speed |
| `api_error` | `claude_code.api_error` log | model, error, status_code, duration_ms, attempt |
| `tool_execution_completed` | `claude_code.tool_result` log | tool_name, success, duration_ms |
| `otlp_session_count` | `claude_code.session.count` metric | (counter increment) |
| `otlp_commit_count` | `claude_code.commit.count` metric | (counter increment) |

## Collector Configuration

syn-collector receives events from workspace containers. It must be reachable on the Docker network that workspace containers use (`agent-net`).

The collector endpoint is auto-injected into workspace containers by the orchestrator when `COLLECTOR_URL` is set in syn-api's environment:

```bash
# In your .env / environment config
COLLECTOR_URL=http://collector:8080
```

When `COLLECTOR_URL` is set, every workspace container gets:
- `CLAUDE_CODE_ENABLE_TELEMETRY=1`: activates Claude Code's OTel export
- `OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:8080`: directs OTel push to the collector

If `COLLECTOR_URL` is not set, OTel export is a graceful no-op: containers still emit plugin hook events via the JSONL channel.

## Verifying Events Are Flowing

### Health check

```bash
curl http://localhost:8080/health
# → {"status": "healthy"}
```

### Check recent events

```bash
curl -s http://localhost:8080/events/recent | \
  jq 'group_by(.event_type) | map({type: .[0].event_type, count: length})'
```

After a workflow execution you should see both channels represented:

```json
[
  {"type": "api_request", "count": 12},
  {"type": "cost_recorded", "count": 3},
  {"type": "git_commit", "count": 2},
  {"type": "session_started", "count": 1},
  {"type": "subagent_started", "count": 2},
  {"type": "token_usage", "count": 18},
  {"type": "tool_execution_completed", "count": 24}
]
```

### Verify collector is reachable from agent-net

```bash
docker run --rm --network syn-dev_agent-net curlimages/curl \
  curl -s http://collector:8080/health
# → {"status": "healthy"}
```

## Token Cost Optimization

Workspace images include built-in optimizations that reduce token consumption, and the observability pipeline captures the evidence.

### RTK Bash Compression

[RTK](https://github.com/rtk-ai/rtk) is pre-installed in workspace images and intercepts Bash tool calls. It compresses output from commands like `ls`, `find`, `git log`, and `tree` into token-efficient formatting. Claude Code uses these aliases transparently, no configuration needed.

Measured impact from A/B testing on an "explore repository" task:

| Metric | Without RTK | With RTK | Savings |
|--------|-------------|----------|---------|
| Context tokens | 61,117 | 28,663 | **53%** |
| Cost (USD) | $0.13 | $0.09 | **29%** |
| Conversation turns | 12 | 6 | **50%** |

RTK primarily reduces **input tokens**, the largest cost driver. This shows up in the OTel channel's `api_request` events: compare `input_tokens` vs `cache_read_tokens` to see how much context the model is consuming per call.

### Observing the Impact

With both channels active, you can track token efficiency per session:

- **`api_request` events** (OTel): `input_tokens`, `output_tokens`, `cache_read_tokens` per API call. Lower `input_tokens` relative to task complexity indicates RTK compression is working.
- **`token_usage` events** (both channels): cumulative input/output/cache totals per turn.
- **`cost_recorded` events** (OTel): running USD cost. Compare sessions with similar tasks to measure savings.

The dashboard's session cost view aggregates these into per-session and per-execution cost breakdowns.

## Deduplication

Events are identified by a deterministic SHA256 `event_id` derived from content (session ID, metric name, timestamp, index). The collector deduplicates within a sliding window, safe to retry any event submission without double-counting.