GitHub has opened a public preview that lets the Copilot CLI and its desktop client control macOS and Windows applications through simulated mouse and keyboard actions – a capability the company calls Copilot computer use. The change matters because it introduces a new way for AI‑driven agents to interact with legacy GUIs that lack APIs, while also adding a permission and policy surface that engineers must manage.
New computer‑use capability in Copilot
The preview adds a bundled plugin with its own MCP server to the Copilot CLI. When enabled, the agent reads the operating system’s accessibility tree, captures screenshots for visual context, and can click, type, scroll, or drag inside any window. The feature is accessible via the /computer on command in the terminal or through the Copilot app’s settings. Demonstrations included filling an expense report in Safari and tasks such as summarising data from a legacy app, updating a presentation, or moving information between programs.
When to prefer direct integrations
GitHub’s guidance advises using structured interfaces—APIs, MCP servers, terminal commands, filesystem tools, or dedicated browser utilities—whenever they exist. Those mechanisms provide predictable data and reduce the risk of mis‑clicks or timing issues that can arise when an agent manipulates a graphical interface. The recommendation effectively defines the scope where GitHub expects Copilot computer use to be a fallback rather than the default approach.
Permission model and enterprise controls
On macOS the agent requires Accessibility permission and, when visual context is needed, Screen Recording permission. The CLI session’s permission mode determines whether Copilot prompts before accessing an app; developers can view the current mode with /permissions show. Prompt responses include allowing the current session, always allowing, or declining. A “deny” decision overrides any saved approvals. Approvals granted in the CLI are shared with the desktop app on the same machine, and removing an app from the always‑allowed list clears future approvals but does not affect an already‑running session. Work can be stopped by pressing Esc twice in the CLI or using the Stop button or Esc in the desktop client.
Enterprise administrators can enforce policy via managed-settings.json. If a managed setting blocks computer use, the CLI reports the feature as unavailable. The same file can prevent developers from bypassing approval prompts, and the policy applies uniformly across the Copilot app, CLI, and VS Code. The default‑enablement policy for Business and Enterprise accounts does not alter the preview’s opt‑in status, and the policy will start applying to unconfigured features on October 22, though preview features remain excluded.
Reliability and security considerations
Because the agent relies on visual cues and window state, changes in timing or UI layout can cause repeated actions or stalls. The system may select the wrong control, type into an unintended field, or fail with complex, dynamic interfaces. Any sensitive data displayed in a window becomes part of the agent’s context, raising the possibility that the model could inadvertently expose that information elsewhere. Ambiguous instructions or unexpected on‑screen content can also trigger unintended device, data, or account actions.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Teams should treat Copilot computer use as a supplemental automation path for legacy GUIs, not a replacement for API‑first designs. Evaluate the necessity of enabling the feature against the added permission surface and potential reliability hiccups. Where possible, implement direct integrations and reserve computer‑use for edge cases. Review enterprise policy settings to ensure they align with your organization’s risk tolerance, and monitor session permissions to avoid accidental exposure of sensitive UI data. Finally, incorporate explicit stop mechanisms and clear fallback procedures in any workflow that depends on the agent’s UI interactions.
