Computer Use
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Computer Use is a capability that enables AI agents to control applications behind the scenes, operating on any application rather than being constrained to specific interfaces or structured data sources.
Overview
Jason Liu describes Computer Use as a paradigm-shifting capability, stating it was "the first time in a long time I really sort of felt the AGI of being at work," suggesting it represents a qualitative leap in perceived AI autonomy and general applicability. →
How It Works
Computer Use operates by allowing AI to interact with and control applications directly, working "behind the scenes to control any application." → This approach relies on visual perception of the screen (screenshots) to understand application state and take actions.
Contrast with Appshots
Liu positions Computer Use in contrast to appshots — a technique that captures the full accessibility tree of an application, including element IDs, rather than raw pixel screenshots. According to Liu, appshots are superior for giving AI context because they enable single-hop tool calls, whereas Computer Use depends on screenshot-based perception. The key distinguishing feature of Computer Use, however, is its generality: it can work behind the scenes to control any application, not just those that expose accessibility trees. → This represents a tradeoff between precision/efficiency (appshots) and universality (Computer Use).