FAQ for the computer use tool

What is the computer use tool

Computer use is a tool you can add to your Copilot Studio agents designed to automate tasks on websites and applications. Users provide task descriptions in natural language, and the tool automates mouse clicks and keyboard actions accordingly. For instance, you could instruct the tool to open a PDF and use its contents to fill out a web form. The tool visually interprets the screen and executes the necessary clicks and keystrokes, emulating human interactions.

What can the computer use tool do?

The computer use tool processes screen pixel data to understand on-screen content, and it performs tasks using virtual mouse clicks and keyboard inputs. The tool can manage multi-step workflows, handle unexpected scenarios, and adapt dynamically to screen changes, enabling it to perform various digital tasks such as navigating websites and filling forms without needing specialized APIs.

The computer use tool follows an iterative loop based on three steps:

• Perception: It captures screenshots to visually understand the current state of the computer screen.

• Reasoning: It evaluates the current state, previous actions, and screenshots through a chain-of-thought approach, which allows it to adapt its actions effectively.

• Action: It performs actions like clicking, typing, or scrolling until the task is complete or requires further user input.

What are the computer use tool’s intended uses?

The computer use tool is intended to automate user interactions with websites and applications through a virtual mouse and keyboard, especially useful in scenarios where APIs aren't available.

How was the computer use tool evaluated? What metrics are used to measure performance?

The computer use tool was rigorously evaluated using real-world scenarios throughout its design, development, and deployment phases. Evaluation methods included research, impact studies, and testing across different scenarios to measure accuracy, user trust, and usefulness. Both qualitative and quantitative metrics are regularly monitored to maintain high performance and a positive user experience. The tool’s availability is also ensured by adhering to established SLAs.

What are the limitations of the computer use tool? How can users minimize the impact of the computer use tool’s limitations when using the system?

Computer use is a novel technology with known security risks and functional limitations. We're empowering you to experiment with it. This early-stage capability enables automation of certain computer-based tasks, though its performance varies widely depending on the use case. To use it effectively and responsibly, understand its current limitations:

Known limitations

• Success rate varies by task: The tool performs best on web-based tasks (about 80% success) but drops significantly on desktop apps (about 35% success).
• Inconsistent performance: The same task might yield different outcomes depending on visual or timing changes.
• Difficulties with certain UI controls: Computer use can have difficulties interacting with non-standard or dynamic interface elements, such as dropdowns, date pickers, or custom widgets.
• Loops and stuck states: In some cases, the agent enters a loop or gets stuck when the screen doesn't match expectations or changes unpredictably.
• Challenges with complex tasks: While effective for simple actions, the tool might struggle with detailed graphical interfaces or tasks involving intricate text manipulation.
• Not for sensitive or high-risk use cases: The tool isn't intended for:
  • Recommending or scoring in sensitive domains, such as hiring, healthcare, or finance
  • Making financial transactions
  • Harmful actions, such as spam or misinformation
  • Sharing data outside your organization without proper authorization

How to minimize limitations

To improve reliability, users should provide clear, detailed instructions when configuring tasks. This helps reduce errors and increases task success rates.

What operational factors and settings allow for effective and responsible use of the computer use tool?

The computer use tool performs best when clear and detailed user instructions are provided. Users should give precise hints about website or application interactions to enhance reliability and accuracy during task execution.

How do I provide feedback on the computer use tool?

Have feedback on the computer use tool? Email computeruse-feedback@microsoft.com.