Claude 3.5 Sonnet Seizes Your Mouse. Literally. The Demo Filled A Form Without You.
Anthropic released Claude 3.5 Sonnet with a capability they call computer use. The model can move your cursor, click interface elements, and type text directly into your applications. It automates tasks like form completion and file organization without requiring you to write a single line of code. This is, to put it plainly, the first widely available AI that operates software the way a human operator would.
The underlying principle here is agentic interaction through GUI manipulation. Most AI assistants work through APIs, meaning they need developers to build custom integrations. Claude sidesteps that entirely by interacting with the graphical interface itself. The mental model: instead of teaching the AI a new language for every app, you let it watch the screen and use the same buttons you use. The mechanism is called visual grounding, and it collapses the integration problem into a single general solution. Your computer is now the API.
Anthropic built and released this. They demonstrated Claude 3.5 Sonnet performing real desktop tasks like filling out forms and organizing files through direct mouse and keyboard control.
- Visit anthropic.com and access Claude 3.5 Sonnet through their console or chat interface. Ask it to describe what steps it would take to organize a folder of files by type. You will see it reason through the task in plain language.
- Open a simple spreadsheet or form on your computer. Describe the form fields to Claude and ask it to generate the exact values you would type into each one. This approximates the computer use capability without granting direct control.
- If you have API access through Anthropic's console, enable the computer use beta feature and instruct Claude to interact with a basic application window. Observe how it interprets the screen and executes clicks. Expected outcome: you watch an AI reason about your desktop interface as if it were sitting in your chair.