Configure Copilot Tuning for document generation

Initial drafts of new documents can be generated using a custom document generation model trained with Copilot Tuning. Copilot Tuning document generation models use an original input document and specified changes to create a draft of output file.

Document generation works best when the drafting process follows recognizable patterns and requires incorporating consistent modifications across similar types of documents. This approach streamlines the creation of first drafts, reducing manual editing time and promoting consistency across your documents.

Some example use cases include:

• Human resources – Generate new job listings matching your organization's tone and structure based on existing job listing templates and information about new jobs.
• Legal – Draft recurring contracts quickly and consistently based on previous contracts and new or updated contract terms.
• Compliance – Create new compliance forms based on an approved form template and information tailored to new clients, deals, or jurisdictions.
• Procurement – Create draft purchase orders using previous purchase orders and information about new purchases.
• Documentation – Draft new product or feature docs based on existing documentation templates and information about new products or features.

Prerequisites

• You must have permission to use Copilot Tuning in Copilot Studio.
• A collection of original documents and corresponding final draft documents that are stored in SharePoint.
• A collection of changelogs or specifications stored in SharePoint.
• A structured version of required changes to provide in the supplementary field within Copilot Tuning.
• More than 20 well-aligned pairs of reference documents to target pairs that reflect a representative range of changes you expect the system to handle.

Create a document generation model

The following are the high-level steps to configure a custom document generation model using Copilot Tuning.

Prepare a mapping file

Your knowledge source should have more than 20 example pairs of original files and corresponding final (draft) files. In this step, you prepare a CSV file that provides more than 20 examples of original files to final (draft) documents. Copilot Tuning uses these examples to fine-tune the generation logic, helping the model learn how your organization typically edits or adapts documents.

Create a file named mapping.csv and store it in the root directory of your knowledge source. This file should have two columns:

• The first column is named precedent and contains the path to an original file in the data source.
• The second column is named target and contains the path to the final draft file in the data source that was created using the original file as a basis.

precedent,target
"https://contoso.sharepoint.com/sites/ProductSpecs/Shared%20Documents/Mark-8-FAQ.docx", "https://contoso.sharepoint.com/sites/ProductSpecs/Shared%20Documents/mark-8-faq.md"

Customize the model

To customize the model:

1. Open Copilot Studio in your browser. In the left-hand navigation bar, select the ellipses (...), and then select Copilot Tuning.
2. Select Create new.
3. Enter a name for your model and provide a description of what the model does. Ensure that the model name and description are user-friendly and clearly describe the purpose of the model.
4. Select Add knowledge to select the SharePoint location that contains your training data.
5. Under Permissions you can enter security groups or email addresses of the individuals who should be able to use this model. Copilot Tuning ensures that any knowledge that isn't accessible by these selected security groups is filtered out. If you don't explicitly give permissions, the model is only accessible to you.
6. Set the task type to Document generation.
7. Provide Model Instructions about your scenario. Instructions ensure Copilot Tuning is using the terminology your organization uses and evaluates your models output using the criteria one of your users would use to evaluate the document. It's important to accurately answer these questions because this information cues the system how to interpret the underlying data.
8. Select Prepare labeling data.

Copilot Tuning begins analyzing the ACLs of the training documents. Based on the size of your dataset, this step might take up to 24 hours. Copilot Tuning sends you an email when it's ready for you to go to the next step in the model creation process. In the meanwhile, you can check the status of the model from the Copilot Tuning landing page.

Review Copilot Tuning ACL analysis

Copilot Tuning analyzes the document access control lists (ACLs) of the training documents and then filters out any knowledge that isn't accessible to the security groups selected in the previous step. Copilot Tuning might suggest more security groups based on its analysis to ensure access to the training data.

Based on the size of your dataset, this step might take up to 24 hours. Copilot Tuning sends you an email when it's ready for you to go to the next step in the model creation process. In the meanwhile, you can check the status of the model from the Copilot Tuning landing page.

Label training data

Preparing high-quality datasets for fine-tuning can be time-consuming and often requires multiple iterations and input from data scientists. Copilot Tuning simplifies this process by automatically identifying which of your provided document pairs are most effective for fine-tuning. This learning process is iterative, and at times, Copilot Tuning might send an email asking you to label files to improve its understanding of what constitutes a high-quality target. When you receive an email indicating that data is ready for labeling, follow these steps to complete the process.

1. Open Copilot Studio in your browser. In the left-hand navigation bar, select the ellipses (...), and then select Copilot Tuning.
2. Select the model that is ready for labeling to open the labeling form.
3. The labeling form provides sample documents that Copilot Tuning identified as example final draft documents. Examine each document and label whether or not you think the document is a good example draft document. Once you label all of the documents, submit the labeling form.

Copilot Tuning continues preparing the training data based on your input. Based on the size of your dataset, this step might take up to 24 hours. You can return to the Copilot Tuning landing page and see the status of your model configuration. Your model configuration might ask you to label more data, might be processing the data in your knowledge source, or it might be ready to fine-tune the model. Copilot Tuning sends you an email when it's ready for you to go to the next step in the model creation process.

Train and evaluate results

When sufficient data is prepared by Copilot Tuning, you can proceed with training the model. Fine-tuning of your model occurs in Azure AI Foundry. After the model is fine-tuned, Copilot Tuning generates results that you can review to evaluate if the model is of sufficient quality to deploy.

1. Open Copilot Studio in your browser. In the left-hand navigation bar, select the ellipses (...), and then select Copilot Tuning.
2. Select the model that is ready for training to open the model configuration dialog form.
3. Select Start model training to begin fine-tuning of your custom model.

Based on the size of your dataset, the fine-tuning step might take up to 24 hours. You can return to the Copilot Tuning landing page and see the status of your model.

If the results don't meet your expectations, you can go to the previous steps in the flow to add more data sources, adjust the model instructions, or provide more samples in your mapping file. Retrain the model until document generation results meet expectations.

Publish the model

When the model generates results that meet your quality standards, you can publish the model through the Copilot Tuning interface. Publish the model by selecting the Publish model button. The model is deployed to an isolated and secured environment. It's only accessible to the security groups or users that were selected during model customization.

Related content

• Microsoft 365 Copilot Tuning overview (preview)
• Microsoft 365 Copilot Tuning FAQ
• Create declarative agents with Copilot Studio agent builder and Microsoft 365 Copilot Tuning models.
• Troubleshooting document generation models.