Dat Ngo
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Dat Ngo is a practitioner at Arize AI, a company specializing in LLM observability, evaluation, and experimentation infrastructure. He is a speaker and advocate for OpenTelemetry-based approaches to understanding and improving AI agent systems.
Core Philosophy: Observability as Foundation
Ngo frames the central challenge of AI engineering as understanding what is actually happening inside complex, non-deterministic systems. He defines Observability as answering "the question of like what's happening in the thing that I've built" — positioning it as the necessary precondition before any improvement is possible. 3:27
A key provocation he offers is that code cannot audit agents — telemetry can: "Code does not audit agents or harnesses — telemetry does." 5:31 This reframes instrumentation not as a nice-to-have but as the primary lens through which agent behavior becomes legible. Everything Arize builds flows through OpenTelemetry, which he advocates as the standard substrate for this instrumentation: "everything we do is through open telemetry." 4:07
On Non-Determinism and Regression Risk
Ngo highlights a distinctive hazard of AI systems compared to traditional software: "When fixing a perceived issue in a non-deterministic AI system, you may inadvertently introduce two or three new regressions." 3:51 This motivates the need for structured evals and experimentation infrastructure — changes cannot be validated by intuition alone.
Evaluation Framework
Ngo articulates a taxonomy of five distinct evaluation signal types: LLM-as-a-judge, human feedback, golden datasets, deterministic checks, and business metrics. 11:46 He also advocates for trajectory evals as a mechanism specific to agents — these surface out-of-order tool calls that degrade performance in ways that code inspection cannot reveal. 10:00
On eval strategy, he argues for restraint: evals should be run at "the minimal sufficient set rather than exhaustively, because there is a real cost associated with running evals." 19:14 This positions cost-awareness as a first-class concern in eval design.
The Automation Vision
Ngo's most forward-looking claims concern the eventual automation of the entire observability-evals-experimentation loop. He argues the flywheel "is automatable and should eventually require no manual human intervention" 23:54, and specifically that AI systems should "automatically create evals on the fly based on trace context rather than requiring humans to choose evals manually." 25:00 This positions current human-in-the-loop eval workflows as a transitional state.
Tooling
Ngo works with two complementary products in the Arize ecosystem: Arize Phoenix, the open-source offering ("we have Arize Phoenix, which is open source" 25:37), and Arize AI's enterprise platform, Arize AX, used by the company's largest customers. 25:50