Claude 3.5 Sonnet Grabs Your Mouse. It Fills Your Forms. The Autonomy Is The Point, Obviously.
Anthropic's Claude 3.5 Sonnet now controls mouse and keyboard input directly, performing tasks like form completion and spreadsheet editing inside existing software. The model operates your applications as a human would, clicking and typing in real time. This is not a plugin or an API integration. It is direct GUI manipulation.
The mechanism here is agentic GUI control, which means the AI interacts with software through the same visual interface you use. This eliminates the need for APIs or developer integrations. The mental model to understand: the model sees your screen, decides where to click, and executes. That is a fundamentally different paradigm than chat-based assistance.
Anthropic released this capability with Claude 3.5 Sonnet, targeting regular users who want an AI assistant working inside their existing software rather than in a separate window.
- Visit anthropic.com and access Claude 3.5 Sonnet through their interface.
- Open a simple spreadsheet or form on your computer alongside the Claude window.
- Ask Claude to perform a concrete task like filling in a form field or editing a specific cell, then observe how it interprets and acts on your on-screen content. Expected outcome: you will see Claude attempt to interact with your software interface directly.