crosstalktalks, distilled into a living wiki

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

AI Engineer · watch the talk ↗ · updated Jul 25, 2026

Jason Lopatecki, founder and builder at Arize, walks through Signal, Arize's autonomous debugging agent, arguing that observability is undergoing a fundamental shift: from dashboards humans click through to telemetry that agents consume. The talk traces the architecture from production event detection through trace retrieval, code repository integration, and automated PR generation, using Arize's own internal agent Alyx as the running example. Lopatecki is candid that large fixes still require human steering, but predicts a future where orders-of-magnitude more logging enables fully autonomous self-improving systems.

Key Takeaways

  • Observability is shifting from human-facing UIs to "smoke thrown off by systems for agents to read" 1:58
  • The key engineering unlock is pulling traces onto the filesystem as files, because coding harnesses like Claude Code are "magical with files and hopeless with a dashboard" 7:14
  • You should log and trace ten times more than today—not less—because agents can process what was previously just noise for humans 9:39
  • The developer role moves from responder to reviewer: the agent does the cold-start investigation, humans drive remaining steps 9:30
  • Enterprises like Uber and Booking.com will not point production systems at external APIs; VPC deployment is a hard customer requirement 13:29
  • One-to-two-line fixes are the ideal case; bigger fixes still need human co-piloting 9:08

Why Observability 2.0 Is a Different Paradigm

Lopatecki argues that traditional observability—Grafana dashboards, clicking through graphs—is being superseded 2:55. Telemetry now serves a dual purpose: human debugging and autonomous agent context. Skills built for platforms like Pyroscope or Google Cloud Logging let agents gather exactly the context they need rather than guessing across millions possible code paths 2:38. Arize's own Alyx agent, shipping ~two years ago as one of the first in-product AI agents in the observability space, was by Lopatecki's own assessment a failure in its first version, and Signal represents the second-generation approach 1:01.

The bottleneck has moved: with coding agents, writing the actual fix is no longer the hard part. The hard part is confidence—knowing the fix is correct and having sufficient evidence before pushing 3:47.

Traces on the Filesystem: The Core Architecture

The central architectural insight is deceptively simple 6:08: skills pull relevant production traces and logs down as files into the repo, directly alongside the source code. Files up to 10 MB are acceptable 17:21. This matters because coding harnesses like Claude Code operate best on the filesystem, not on API dashboards or structured queries.

The full evidence bundle for an automated fix contains:

  • Production traces (pulled by skill into the repo as temp files) 6:56
  • Logs from external systems (Google Cloud, etc.) 12:36
  • The code itself, giving the agent the exact execution path taken 2:40

Lopatecki warns against simply "pointing Claude at the data"—the skill surface area must be carefully designed so the LLM can composably find issues 17:45. Examples: a Pyroscope skill that finds memory issues; cohort-by-customer faceting to isolate whether a single customer is causing a failure 17:35.

Signal in Practice: The Alyx Stream-Canceled Bug

Signal runs continuously as a background process, triggered either periodically or on specific error events 12:10. When Alyx (Arize's own in-product assistant) produced a stream-canceled error, Signal automatically:

  1. Detected the event 8:28
  2. Pulled traces and logs as files into the repo
  3. Identified the failure (repeated to-do-update calls erroring out) 8:50
  4. Proposed a one-to-two-line fix and opened a PR 9:08

Lopatecki presents this as the ideal case. For larger fixes, a human takes the pre-populated investigation as a cold-start and drives it the rest of the way 9:22. Signal integrates with GitHub: it can open issues, create evals for newly discovered failure modes to catch them in future, and add failing cases to datasets 12:16.

Sandboxes, VPC, and the Enterprise Deployment Model

Getting this to work beyond a developer's laptop requires a sandbox environment that mirrors local debugging 7:41. Lopatecki advocates for a workflow of: get it working locally first, then move to event-driven sandbox execution 7:54.

Signal supports pluggable harnesses (Claude Code, Cursor) and pluggable sandboxes (Daytona, Arize-managed, or customer-provided) 13:16. The hard constraint for enterprise: customers like Uber and Booking.com refuse to connect production systems to Anthropic or other external endpoints directly 13:29. Arize's answer is VPC-installed deployment—many large companies are comfortable installing the platform inside their own infrastructure 14:01.

Evals as Pre-Processed Signal Layers

Responding to a Q&A question 18:04, Lopatecki explains that evals are not separate from traces—they are data layered onto traces via what Arize calls online evals. An LLM-as-judge runs periodically across production trace data, adding pre-processed assessments that Signal then consumes alongside raw trace data 18:22.

His characterization of current LLM-as-judge evals: they tend to be preset detectors for known failure modes—prompt injection catches, response quality failures, safety issues—rather than general-purpose evaluators 19:29. When Signal surfaces a new recurring failure, a practitioner can immediately create a new eval to catch it next time, closing the improvement loop 19:52.