Claude 3.5 Sonnet Seizes Your Mouse. It Clicks. It Types. You Are Now Spectator.
Anthropic's Claude 3.5 Sonnet can now control your cursor, click interface elements, and complete web forms directly inside a browser. The model operates your laptop visually rather than through APIs, meaning it can book flights, post to social media, or manage spreadsheets by interacting with software the way a human would.
This demonstrates what I would call embodied interface manipulation, if you will forgive the jargon. The principle is straightforward: instead of requiring developers to build custom integrations for every application, the model simply sees the screen and acts upon it. This collapses the integration problem entirely. The mechanism is visual grounding, where the model maps its understanding of a task to pixel coordinates and UI elements. You should understand that this shifts AI from a tool you consult to a tool that operates your tools.
Anthropic released this capability with Claude 3.5 Sonnet, enabling the model to perform real-time computer use including mouse movement, clicking, and form completion.
- Visit anthropic.com and sign up for a Claude account, then access Claude 3.5 Sonnet. Expected outcome: You gain access to the latest model with computer use capabilities.
- Open a browser task you would normally do manually, such as filling out a web form or searching for information across multiple sites. Ask Claude to perform the task by describing what you want accomplished. Expected outcome: Claude will attempt to navigate the interface visually and complete the action.
- Watch the screen as Claude moves your cursor and clicks. Intervene if needed by correcting its approach in plain language. Expected outcome: You observe the model handling a multi-step web task that previously required manual work.