Overview
Google announced a “Computer Use” capability for the macOS version of Gemini, letting users describe tasks in natural language while the assistant operates local apps, files and websites in the background — clicking, typing, scrolling and handling cross-application information. The process runs in a separate virtual window that users can pause or terminate, and the feature is rolling out to US English users.
At roughly the same time, OpenAI disclosed an internal agent system called Performance Factory: when anomalies are detected, Codex agents automatically read metrics, logs, traces and code, investigate root causes and propose fixes. Over three months it has tracked more than 1,700 performance investigations and identified over 600 root causes, averaging about 17 automated investigations a day. Signal, meanwhile, announced on November 20 that it would restrict screen access for AI agents on mobile operating systems to prevent private messages from being read across apps.
Background & Interpretation
1. Technical context and evolution
The agent capability frontier is expanding from generating content to executing actions. Where assistants once required humans to copy, paste and click across software, Computer Use stitches those actions into complete workflows. OpenAI’s production practice shows agents entering the R&D and operations loop — receiving tasks, calling tools, retaining state, handling exceptions and delivering results under human supervision. Together they point to a trend: AI shifting from a user of software to part of the software production system.
2. Core drivers and mechanisms
The leap rests on a mature perceive-decide-act-feedback loop. OpenAI’s disclosure also reveals the limits: of 100 root-cause recommendations reviewed by humans, about 70% were rated accurate, with the rest needing refinement or lacking clear causation, and final code merges still require engineer approval. Agents are not yet reliable enough in open environments to be left alone, keeping humans in the loop essential. Permission management, state management and tool-call cost optimization are displacing raw parameter scale as the key platform battleground.
3. Market structure and value-chain impact
Security and privacy are parallel threads. Signal’s move followed findings that some agents could read private messages via system notifications, accessibility features and screenshot permissions. Though imperfect, it reflects the need for operating systems to redefine app permissions: what data AI may read, which actions need confirmation, and which apps can refuse agent access. Enterprise competition is shifting from model strength to safely embedding models into workflows, making model adaptation, data security, permission governance and inference cost the new infrastructure entry point.
Implications & Outlook
In the near term, desktop agents will first land in standardized scenarios such as expense reporting, file organization and multi-step office tasks, while new attack surfaces — such as malicious screen content inducing cross-app actions — emerge in parallel. Permission design will determine user trust. Over the medium term, scaling agents from demos to production requires verifiability, failure rollback and accountability. The gap cited by Gartner — 71% of organizations planning deployment versus just 21% with governance capability — captures the core tension of this phase.
Takeaways for Industry Participants
Enterprises should make least-privilege access, auditable operations and human confirmation at critical steps the default design rather than an afterthought. Developers can adopt engineering techniques such as deferred tool schema loading to control context costs and improve tool selection accuracy. For everyday users, granting screen and app permissions judiciously and watching execution previews is a practical way to enjoy agent convenience while protecting privacy.
This column compiles industry information and shares technical perspectives; it does not constitute investment advice.