LLM / Platform

Tardis
Autonomous Incident Investigation Agent

Shortens the 3 AM page → dashboard-chase → hypothesis loop. Tardis investigates first, so on-call decides with evidence in hand.
Infra Team

The Problem & What Tardis Is

Page fires. Where do you look first?

How It Works — One Diagram

Architecture

Tardis architecture solid = implemented · dashed = in progress · green arrow = the compounding self-improvement loop INGESTION PagerDuty webhook /webhook/pagerduty Slack mention @tardis investigate ... Manual API POST /investigate Event router classify service · classify severity dedup (5-min window) · classify request_type AlertEvent PER-INVESTIGATION RESOLVE severity ∈ {critical · high · warning · info} request_type ∈ {investigation · data_task · quick_check · conversational} → resolves: timeout · task_budget · effort · soft_limit · cost_cap two-axis lookup: investigation→severity map; else→request_type map Main agent Tardis investigation loop · Opus 4.7 · adaptive thinking while not done: build context → call API → tool_use? → execute → checkpoint → repeat stop on: end_turn (trusted) · timeout · cost · turn cap SELF-REGULATION GUARDRAILS ① task budget (advisory · model self-regulates) ② wall-clock timeout (real-world incident urgency) ③ cost ceiling (financial circuit breaker) ④ emergency turn cap (pure runaway protection) TardisAnthropic WRAPPER (agent/api.py) single point for all 10 messages.create() sites routes by purpose · attaches cache_control · selects effort · enforces max_tokens Trust stop_reason="end_turn" no soft-limit nudges for critical/high — model decides when investigation is complete SYSTEM PROMPT (tuple) stable_prefix (cache_control: ephemeral, ttl=1h) + dynamic_suffix 90% cost reduction on stable prefix · accuracy unchanged CONTEXT INJECTION Stable knowledge prefix (cached) 15 always-loaded markdown files — system_prompt · playbooks · environments · domain_reference · grafana_dashboards · sumo_alerts · model_oncall · error_triage · health_check · tundra · ice ... Tier 2 — investigation memory SQLite-vec · per-investigation summaries · embedding similarity recall: vector + FTS5 + service match + verified boost + temporal decay Tier 3 — learned patterns effective queries · anti-patterns · failure signatures (per service) extracted post-investigation by Opus 4.7 (was Haiku — accuracy upgrade) Team knowledge @tardis remember JSON on PVC Incident recall S3-backed COC bot embedding similarity Refuted-claims tracker IN PROGRESS — only added if 4.7 + thinking doesn't fix it organically per-thread list of claims user pushed back on; never re-stated as findings assemble TOOL EXECUTION Tool registry sumo · prometheus · kubectl · database redshift · kong · github · slack_reader code_search · ask_human · submit_finding parallel: asyncio.gather (semaphore=10) serial: ask_human · submit_finding · alerter triage gate (turns 1-3) always serial PII redaction in-process · HIPAA-critical · never skipped runs BEFORE compaction or context entry 12 regex: MRN · DOB · SSN · phone · keys · JWT Compactor (display only) DISABLED for investigation context post-upgrade Haiku 4.5 · only fires on Slack RCA formatting raw tool output → Opus (preserves error codes, traces) tool_use raw result redacted PERSISTENCE / RECOVERY Checkpoint store SQLite · saves full conversation state every turn crash recovery: pod restart resumes mid-investigation staleness limit: 30 min (older = fresh investigation) Ledger per-investigation cost · turns · tools · cache hit rate tracks: input · output · cache_creation · cache_read tokens save / restore Delivery · cross-judge feedback before posting if confidence is low → loop back to main agent to gather missing info → re-synthesize → re-judge (max 2 retries) → post Synthesis RCA draft from main agent Opus 4.7 + xhigh thinking (critical) timeline · root cause · evidence · ruled-out Cross-judge gate non-Anthropic LLM (e.g. GPT-4o) scores RCA against rubric high conf → post low conf → gather more info retry cap: 2 cycles, then post w/ low-conf label Slack post · fires after gate passes verified RCA goes to Slack thread ▸ headline + thread RCA · 👍 / 👎 / 🔁 buttons ▸ confidence label baked in (✓ verified / ⚠️ retried-out) ▸ progress updates during investigation (live status) Slack thread is then read-only input human corrections + reactions accumulate naturally during the incident post-investigation NEVER updates the post afterwards draft approved low confidence → "gather missing info: ..." → re-run main agent end_turn → synthesize if posted with low-conf label → skip Tier 2 store Post-investigation · self-improvement loop reads the completed thread (RCA + corrections + reactions) → writes to Tardis's own knowledge → NEVER updates the Slack post 1 · Thread analyzer runs after thread quiescence (5 min idle) reads the full Slack thread: ▸ the posted RCA ▸ human corrections in-thread ▸ 👍 / 👎 / button reactions read-only — Slack post is final 2 · Failure-mode identifier where did Tardis go wrong? compares evidence vs corrections: ▸ refuted claims ▸ tools skipped or misused ▸ knowledge gaps that caused it e.g. NLB IPs ≠ pod IPs (sm4, Apr 28) 3 · Improvement generator produces fix candidates candidate fixes: ▸ knowledge file additions (.md) ▸ Tier 3 anti-pattern entries ▸ new eval case (regression test) ▸ (optional) code-patch suggestion Opus 4.7 generates · human reviews 4 · Open PR (human-gated) no auto-apply — owners review every change creates GitHub PR proposing: ▸ knowledge/*.md additions ▸ Tier 3 anti-pattern entries ▸ evals/cases/ regression test posts link to #infra-owners merges only after human approval 5 min after thread goes quiet on merge → improves future investigations ▶ The compounding loop. Each investigation makes Tardis smarter. Slack thread is read-only input · Tardis's own knowledge files are the output ROADMAP: started with manual @tardis remember + 👍/👎; automating the analyzer + generator is the next compound OBSERVABILITY Prometheus metrics · JSONL event log · cache hit rate · turn distribution · cross-judge agreement rate · cost per investigation

~1K lines in the loop · ~3.6K lines across 13 tool plugins · 15+ knowledge files · checkpointed to SQLite so pod restarts don't lose work.

Same loop, four shapes

How Tardis behaves by request_type

Tardis call graph — by request_type Common path on top · per-type lanes below · convergence at cleanup COMMON ENTRY (all request types) PagerDuty webhook /webhook/pagerduty (server.py) Slack mention app_mention → handle_mention Manual API POST /investigate Slack-only short-circuits stop · pause · resume · delete · rerun → no agent loop handled in listener, return immediately Slack mid-investigation pending question → answer · running → inject context routed via session, no new investigation Event router (events/router.py) classify_service · classify_severity · dedup (5-min window) classify request_type → spawn asyncio task → call AgentLoop.run() ↓ branch by request_type ↓ investigation data_task quick_check conversational ▼ STEP A · BUILD SYSTEM PROMPT (loop.py:_build_system_prompt) Full context 15 knowledge files (stable, cached) + Tier 3 patterns (proven queries / anti-patterns) + Tier 2 memory (5 past investigations) + Team knowledge facts + Incident recall (S3) Lean context 15 knowledge files (stable, cached) + Team knowledge (relevant only) SKIP: Tier 2 memory · Tier 3 patterns · Incident recall data tasks don't need historical reasoning Lean context Same as data_task SKIP: Tier 2 / Tier 3 / Incident recall code: is_data_task = type in (data_task, quick_check, conversational, ...) Lean context + thread Knowledge + team + thread context SKIP: Tier 2 / Tier 3 / Incident recall match the conversation already in progress, don't restart from scratch ▼ STEP B · BUILD INITIAL USER MESSAGE (loop.py:_build_initial_prompt) + TRIAGE GATE Step 1: database_query (call.calls) Step 2: database_query (7-day baseline) Step 3: ask_human (decide / proceed) prom/sumo banned for first 3 turns No triage Just: request + format hint "Run queries, return clean tables. No RCA. 5–8 turns." advisory only — no enforced limit No triage Just: request + format hint "Answer directly. 2–3 turns max." expects a one-shot lookup No triage · thread anchored Includes thread context Match question depth. Conversational tone. "thanks", "what about X", "explain" ▼ STEP C · RESOLVE LIMITS (config.py two-axis lookup) By severity (4 tiers) timeout: 1200/600/300/180s budget: 950K/700K/400K/200K tokens effort: xhigh/high/high/adaptive soft_limit: None/None/30/20 critical/high never get a turn nudge Flat (no severity) timeout: 180s budget: 40K tokens effort: adaptive soft_limit: None Flat (smaller) timeout: 60s budget: 15K tokens effort: low soft_limit: None no thinking — direct lookup Flat (medium) timeout: 90s budget: 25K tokens effort: adaptive soft_limit: None ▼ STEP D · ask_human TOOL WIRING Wired (if Slack session) tool can pause loop, post Slack msg, await reply Cleared model can call but tool returns error Cleared no human-in-loop for quick checks Cleared conversation already has the human ▼ STEP E · AGENT LOOP (loop.py for-loop) — same code, different shape per type Multi-turn deep dive Turns 1–3: triage gate enforced Turns 4+: prom · sumo · kubectl · kong Cross-verify (Track 1 + Track 2) Hypothesis testing, ruled-out tracking Typical: 8–30 turns, 2–15 min checkpoint each turn (resume on crash) progress broadcast to Slack between turns Query execution Run: sumo / prom / db / kubectl Format result as a clean table No RCA, no hypothesis chain Typical: 3–8 turns, 30s–2 min "how many calls failed today?" "top 5 error reasons last hour?" Direct lookup Often 1 tool call Single-source answer No methodology, no thinking Typical: 1–3 turns, <30s "is l405s healthy right now?" "what's snow-call's p99 latency?" Follow-up reply Often 0–2 tool calls Match thread tone & depth Reference earlier RCA / data Typical: 1–4 turns, <1 min "explain the timeline more" "what about the previous 30 min?" ▼ STEP F · LOOP EXIT (stop_reason="end_turn") · OUTPUT SHAPE Structured RCA (6 sections) Summary · Timeline · Root Cause Evidence · Recommendations · Ruled Out MECHANISM / DOSE-RESPONSE / REVERSAL retry once if < 500 chars on forced synthesis Tables & numbers Markdown tables only No "summary" or "recommendations" Direct: rows + counts + timestamps One-line answer "sm4 healthy — 0 fails, p99 240ms" No structure, no headers Direct status / fact Conversational reply Match question shape Reference prior RCA / data Hedge: "looks like" / "probably" ▼ STEP G · MEMORY & LEDGER WRITES Tier 2 + Tier 3 + Ledger Store full RCA as Tier 2 (embedding) Extract Tier 3 patterns (Opus 4.7) Append ledger row (cost, turns, tools) future investigations on this service recall this RCA + learn its patterns Tier 2 + Ledger Store as Tier 2 (with embedding) Append ledger row no Tier 3 extraction (no methodology to learn from a query result) Tier 2 + Ledger Stored same as data_task code-wise: only conversational is ephemeral — quick_check still stores. Arguably should also be ephemeral — low-value Tier 2 noise. (open Q) SKIP all storage No Tier 2 · No Tier 3 · No ledger code: is_ephemeral = type in ("conversational",) → skip both follow-ups don't deserve a row ▼ STEP H · SLACK OUTPUT (slack/listener.py + formatter.py) Multi-block thread reply Investigation Report (header) + Timeline + Evidence + Recommendations + Buttons: 👍 / 👎 / 🔁 dig deeper PLANNED: headline-first compression, full RCA collapsed behind 📄 button Tables in thread Just the result data No preamble, no commentary ("Here's the answer:" → drop) One-line in thread No header, no structure Just the fact "sm4 healthy — 0 fails, 240ms p99" Conversational reply Plain text in thread Match prior message tone human-like, not report-like ▼ STEP I · CLEANUP & STATE TRANSITION (common to all) Cleanup (loop.py end of run) checkpoint_store.delete(event.id) — investigation done, no resume needed ask_human_tool.clear_session(event.id) — release the per-investigation Slack wiring state_store: status "running" → "completed" · session_manager.mark_completed Legend & key differences investigation full memory · triage · RCA · stored data_task no triage · tables only · stored quick_check low effort · 1-line · stored (?) conversational ephemeral · NOT stored · NOT in ledger Critical splits: • Triage gate: investigation only • Tier 2/3 memory injection: investigation only (rest skip) • Tier 3 pattern extraction: investigation only • Memory + ledger storage: all except conversational • ask_human wiring: investigation only • Severity-driven limits: investigation only (others use flat per-type)

Key Features

What makes Tardis tick

1 · PII redaction before context

  • Tool results pass through 12 regex rules — MRN · DOB · SSN · phone · AWS keys · JWTs — before entering Claude's window. HIPAA-critical, runs on every tool call, never skipped.

2 · Three-tier memory + semantic recall

  • Tier 1: 15+ knowledge files (always). Tier 2: past investigations with embeddings (BAAI/bge-small-en). Tier 3: learned patterns — proven queries, anti-patterns. Hybrid recall: vector + FTS5 + service match + temporal decay.

3 · Comprehensive context

  • 13 read-only tools spanning the full stack — infra (Kong, kubectl, Prometheus model+app, Sumo) · app (Postgres, Redshift, GitHub code search) · team (Slack reader, team_knowledge). Plus knowledge files for runbooks, on-call playbooks, domain refs. No artificial silos.

4 · Self-improvement loop (PR-gated)

  • Post-investigation analyzer reads the thread (RCA + corrections + 👍/👎), identifies refuted claims and knowledge gaps, and opens a GitHub PR proposing fixes — knowledge-file additions, anti-patterns, eval cases. Posted to #infra-owners; merges only after human review. Each incident makes Tardis smarter.

5 · Crash recovery + audit trail

  • SQLite checkpoint after every turn — pod restart resumes mid-investigation (within 30 min). Per-investigation ledger captures cost, turns, tool calls, cache hit rate. JSONL event log for full replay.

6 · Triage + verification

  • Triage gate: first-3-turns active-call check filters noise before deep investigation. Cross-judge gate (planned): non-Anthropic LLM scores every critical RCA before posting; low confidence → loops back for more info. Causal claims require MECHANISM / DOSE-RESPONSE / REVERSAL.

+ Eval framework — 20 cases replay historical investigations and score tool selection, root-cause keywords, wrong-hypothesis avoidance. CI blocks regressions on every PR.

Case Study · how Tardis learned

From a wrong answer (Mar 30) → working triage today

  • Mar 30 P0: WellSpan incoming_calls_accepting returned 503 for 2 patient calls.
  • Tardis v1 said wrong "367 patient calls never connected" — counted DB ERRORED aggregate against a Monday baseline of 273–340.
  • Reality: 2 WellSpan 503s, both callbacks completed within 2 min. Real cause: GIL contention from find_all_utterances on a long-running VNS call — not CPU saturation.
  • What we shipped: mandatory_investigation_rules.md (8 rules), call_failure_taxonomy.md, baseline_rates.md, Rule -1 for incident-time scoping, plus a regression eval case (PR c4efb25).
  • Bonus: the investigation surfaced two state-leak bugs in Snow (PR #36254).
Tardis triage of kd7 503 alert in Slack — Track 1 + Track 2 methodology, already-recovered finding, low patient impact
Tardis today (Apr 28, kd7 503): two-track triage in 1 turn — 0.05% error rate, already recovered, ~6 calls impacted. Clear infra/app split. Posted before Sam finished acknowledging.

Teaching & Contributing to Tardis

Four ways to make Tardis smarter

1 · Slack — no clone needed

  • @tardis remember <topic>: <fact> — teach a team fact
  • @tardis forget <topic> — remove
  • 👍 Accurate / 👎 Wrong on any RCA — boosts or deboosts that memory for future recall
  • gated SLACK_APPROVED_USERS — ask to be added

2 · Add knowledge for a platform

  • Example: hai-agents#89 — ICE reference (19 patterns + infra + triage)
  • Touch 3 files: knowledge/<platform>_reference.md, events/router.py, agent/loop.py
  • Per pattern: alert · root cause · error signature · impact · fix
  • Required: ≥1 eval case in evals/cases/

3 · Add a service / tool / repo

  • Example: hai-agents#84 — add tundra to code_search (3-line PR)
  • New tool = one file in tools/ (auto-discovered)
  • New repo for code search = edit REPO_MAP + tool enum

4 · Propose a new feature — TIP

  • Template: agents/tardis/tip/TARDIS_AUTOIMPROVE_TIP-V1.md
  • Write a Tardis Improvement Proposal: goal, design, risks, rollout
  • Discussion happens in the PR; implementation follows

Local dev: clone hai-agents · cd agents/tardis · uv sync --extra dev · uv run uvicorn server:app · uv run python -m evals.runner --mode offline --case <name> before pushing.

Impact, Roadmap, Ask

Where it's going

Today (just shipped)

  • 13 tools · 23 knowledge artifacts · 27 eval cases
  • Opus 4.7 reasoning + synthesis · Haiku 4.5 for compaction
  • Two-axis limits (severity × request_type) + cache-aware ledger

Next

  • Flip ENABLE_PROMPT_CACHE + 1M_CONTEXT + EXTENDED_THINKING in eval → prod
  • Cross-judge verification (non-Anthropic LLM) for critical RCAs
  • Wire past_rcas into semantic recall · supersession semantics
  • Expand to all other production services
  • Create local development friendly environment - skill tardis-publish - WIP

Guardrails

  • Read-only, for the foreseeable future
  • Approval gate for new alert classes
  • Human-in-the-loop via ask_human

Ask

  • Teams with alerts → let's map your triage signal
  • Feedback on Slack RCA format
  • Candidate eval cases: past incidents with a known RCA

One more thing

PS — the team has feelings

Joe Redmond in Slack: 'It feels like the Inside Tardis presentation should be eng exchange! great app btw, seems like the most useful agent at the company IMO'
Slack thread: Tardis investigation timed out after 771s and 4 turns with partial findings. Dishank Jhaveri replies 'i love tardis @on-call-platform'. Joe Redmond replies 'We all love tardis @Revanth Reddy Airre'.
Team loves Tardis :D