Share via

Use Agent Bricks: Multi-Agent Supervisor to create a coordinated multi-agent system

Important

This feature is in Beta.

This page describes how to use Agent Bricks: Multi-Agent Supervisor to create a multi-agent supervisor system that orchestrates AI agents and tools to work together on complex tasks. You can improve their coordination based on natural language feedback from your subject matter experts.

Agent Bricks provides a simple approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases.

What is Agent Bricks: Multi-Agent Supervisor?

Use Agent Bricks: Multi-Agent Supervisor to create a supervisor system that coordinates Genie Spaces, agent endpoints, and tools to work together to complete complex tasks across different, specialized domains. Multi-Agent Supervisor uses advanced AI orchestration patterns to manage agent interactions, task delegation, and result synthesis to deliver comprehensive solutions.

Agent Bricks: Multi-Agent Supervisor builds the system for you and lets you improve it over time with human feedback. It is ideal for supporting the following use cases:

  • Provide market analysis and insights by searching across research reports and usage data.
  • Answer questions about internal processes and automate a ticket backlog for it.
  • Speed up customer service by answering policy, FAQ, account, and other questions.

Multi-Agent Supervisor enables you to improve the supervisor's coordination quality and adjust agent behavior based on natural language feedback from your subject matter experts. Provide task scenarios for a labeling session and send it to experts to review in the Review App. Their responses provide labeled data that helps optimize the system's performance.

Multi-Agent Supervisor creates a comprehensive endpoint you can use downstream for your applications. For example, you can interact with the endpoint by submitting prompts in Playground or build a chat application using Databricks Apps. The supervisor has built-in access controls, so that its end users only access the subagents and data they have access to.

Requirements

Create a multi-agent supervisor system

Go to Agents icon. Agents in the left navigation pane of your workspace and click Multi-Agent Supervisor.

re[ABMAS]

Step 1: Create subagents and grant permissions

Since Agent Bricks: Multi-Agent Supervisor creates a supervisor system that coordinates subagents to work together to complete complex tasks, you need to first provide subagents for it to coordinate. These subagents can be Genie Spaces or Knowledge Assistant agent endpoints. You also need to grant end users explicit access to each subagent in order for the supervisor to return helpful responses from that subagent.

Genie space

  1. To create a Genie space, follow the steps in Set up and manage an AI/BI Genie space.
  2. Grant end users access to both the Genie space and its underlying Unity Catalog objects. Follow the steps in Share a Genie space.

Agent endpoint

  1. To create a Knowledge Assistant agent, follow the steps in Use Agent Bricks: Knowledge Assistant to create a high-quality chatbot over your documents.
  2. Grant end users the CAN QUERY permission on the Knowledge Assistant agent endpoint.

Step 2: Configure your supervisor

On the Configure tab, configure your supervisor and add the agents it will coordinate.

Note

The supervisor has built-in access controls, so that its end users only access the subagents and data they have access to. For agent endpoints, end users require the CAN QUERY permission on the endpoint. For Genie spaces, end users require access to both the Genie space and data access to its underlying Unity Catalog objects. See Share a Genie space. If the end user does not have access to any subagents, the supervisor will end the conversation. If the end user has access to some but not all subagents, the supervisor will redirect the conversation away from subagents the user cannot access.

  1. In the Name field, enter a name for your supervisor agent.

  2. In the Description field, describe what your supervisor system can do.

  3. Under Configure Agents, select up to 10 agent endpoints and/or Genie spaces.

    Genie Space

    To provide a Genie space:

    1. In the Type field, select Genie Space.

    2. Select your Genie space from the Genie space drop-down menu.

    3. The Agent name and Describe the content fields are automatically populated when possible. You can edit the name and description if desired.

      The supervisor uses the information in the description to help it coordinate agents. Provide as much detail as possible to help improve its task delegation.

    To learn more about Genie spaces, see What is an AI/BI Genie space. To set up a Genie space, see Set up and manage an AI/BI Genie space

    Agent Endpoint

    To provide an agent endpoint:

    1. In the Type field, select Agent Endpoint.
    2. Select the endpoint from the Agent Endpoint drop-down menu. Only agent endpoints created through Agent Bricks: Knowledge Assistant are supported.
    3. The Agent Name field is automatically populated. You can edit this if desired.
    4. Under Describe the content, describe what this agent can do to help the supervisor understand when to delegate tasks to this agent.

  4. (Optional) To add more agents, click + Add. You can provide up to 10 agents.

  5. (Optional) In the Instructions field, specify guidelines for how the multi-agent supervisor should respond.

  6. Click Create Agent.

You'll be redirected to the Configure tab. It can take a few minutes to a few hours to create your multi-agent system and supervisor agent.

Step 3: Test your supervisor agent

After your supervisor has finished building, you can test it by trying it out in AI Playground. The supervisor will coordinate multiple agents to handle complex tasks. In Test your Agent in the right side panel, you can chat with the agent to evaluate its responses.

  1. (Optional) You can also test the agent in AI Playground. Click Open in Playground. This opens up AI Playground with your supervisor endpoint connected. If you have AI assistive features enabled, you can enable AI Judge and Synthetic task generation to help you evaluate your supervisor.
  2. Under Test your Agent or in AI Playground, enter a complex task for your supervisor.
  3. Evaluate its response. Ensure that the supervisor successfully delegates tasks to the right agents.
  4. Based on your agent's responses, adjust the Description and Instructions fiels on the left side panel to improve its configuration.
  5. Click Update Agent.

If you're satisfied with your supervisor's performance, continue using the supervisor as-is. By default, Agent Bricks endpoints scale to zero after 30 minutes of inactivity, so you'll only be billed for the uptime.

Step 4: Improve the supervisor

Agent Bricks: Multi-Agent Supervisor can adjust the supervisor's behavior based on natural language feedback. Gather human feedback through a labeling session to improve your supervisor's coordination quality. Collecting labeled data for your supervisor can improve its performance. Agent Bricks will retrain and optimize the supervisor from the new data.

In the Improve tab, add task scenarios and start a labeling session.

  1. Add task scenarios to include in your labeling session:

    1. Click + Add to add a task scenario.
    2. In the Add a question modal, enter a question or task for the agent.
    3. Click Add. The task will appear in the UI.
    4. Repeat until you've added all questions you want to evaluate.
    5. To delete a question, click the kebab menu, then Delete.

    Databricks recommends adding at least 20 questions for a labeling session to ensure enough labeled data is collected.

  2. After you've finished adding your task scenarios, send the scenarios to experts for review to help you build a high-quality labeled dataset. On the right, click Start labeling session.

    When your labeling session is ready, the UI will update as shown below.

  3. Share the review app with experts to gather feedback. Click Grant SME permissions and add the experts to grant them the correct permission to access the labelling session.

    To learn more about labeling sessions and the review app, see Review App.

  4. Ensure the SME has access to the appropriate subagents:

    • For each agent endpoint, grant the SME the CAN QUERY permission.
    • For each Genie space, grant the SME all appropriate permissions to interact with the space. See Share a Genie space.

    If the SME does not have access to any subagents, the supervisor will end the conversation. If the end user has access to some but not all subagents, the supervisor will redirect the conversation away from subagents the user cannot access.

  5. To label the data yourself, click Open labeling session.

    This opens the review app in a new tab. As a reviewer:

    1. Click Start review.
    2. On the left, review the question and the supervisor's response
    3. On the right side, under Expectations, review any existing guidelines and add more as you see fit.
      1. To add a guideline, click + Add input.
      2. Enter the guideline in the text box that appears.
      3. Click Save.
    4. Under Feedback, enter your feedback, then click Save.
    5. When you're done reviewing a task scenario, click Next unreviewed > in the top right to move onto the next one.
    6. When you're done reviewing all task scenarios, exit the review app.

  6. When your reviewers are done with their labeling sessions, return to your supervisor's Improve coordination tab.

  7. Click Merge to merge feedback from the experts to your labeled dataset. The table of task scenarios on the right side will update with the merged feedback.

  8. Review the feedback records.

  9. Test the supervisor again in AI Playground to see its improved coordination performance. If needed, start another labeling session to gather more labeled data.

Limitations