2026-06-11 BREAKTHROUGHS☾ PM
Anthropic Adds Screen Control to Claude 3.5 Sonnet
📰 THE BRIEF
The new computer use API lets Claude 3.5 Sonnet move the mouse, type text, and click inside applications on your desktop. Developers send screenshots and instructions through the Anthropic SDK; the model returns coordinate-based actions that execute in a sandboxed environment. This removes the need for brittle scripts when handling data entry or browser workflows.
💡 WHY IT MATTERS
You stop writing one-off automation scripts and instead describe the outcome you want. The model handles the low-level clicks and keystrokes, so your workflow becomes prompt-driven rather than code-driven.
👥 WHO'S DOING IT
Anthropic's own research team demonstrated the feature automating spreadsheet data entry and web research tasks, cutting manual steps by roughly 80 percent in internal tests.
⚡ TRY IT
- Sign up at https://console.anthropic.com and enable the computer-use beta.
- Install the Anthropic Python SDK and call the messages endpoint with tool choice set to computer_20241022.
- Provide a screenshot and a natural-language instruction; the model returns mouse and keyboard actions that run on your machine.