AML skill in an Azure AI Search enrichment pipeline

Use the AML skill to extend AI enrichment with a deployed base embedding model from the Azure AI Foundry model catalog or a custom Azure Machine Learning (AML) model. After an AML model is trained and deployed, the AML skill integrates the model into a skillset.

AML skill usage

Like other built-in skills, a custom AML skill has inputs and outputs. The inputs are sent to a deployed AML online endpoint as a JSON object. The output of the endpoint must be a JSON payload in the response, along with a success status code. Your data is processed in the Geo where your model is deployed. The response should provide the outputs specified by your AML skill definition. Any other response is considered an error, and no enrichments are performed.

You can call the AML skill with the 2024-07-01 stable API version or an equivalent Azure SDK. For connections to the model catalog in the Azure AI Foundry portal, use the 2024-05-01-preview API version or later.

AML skill for models in Azure AI Foundry

Starting in the 2024-05-01-preview REST API and the Azure portal, which also targets the 2024-05-01-preview, Azure AI Search provides the Azure AI Foundry model catalog vectorizer for query-time connections to the model catalog in the Azure AI Foundry portal. If you want to use that vectorizer for queries, the AML skill is the indexing counterpart for generating embeddings using a model from the model catalog.

During indexing, the AML skill can connect to the model catalog to generate vectors for the index. At query time, queries can use a vectorizer to connect to the same model to vectorize text strings for a vector query. In this workflow, you should use the AML skill and the model catalog vectorizer together so that the same embedding model is used for indexing and queries. For more information, including a list of supported embedding models, see Use embedding models from Azure AI Foundry model catalog.

We recommend using the Import and vectorize data wizard to generate a skillset that includes an AML skill for deployed embedding models in Azure AI Foundry. The wizard generates the AML skill definition for inputs, outputs, and mappings, providing an easy way to test a model before writing any code.

Prerequisites

• An Azure AI Foundry project for an embedding model deployed from the catalog, or an AML workspace for a custom model that you create.
• The model endpoint for an embedding model deployed from the catalog, or an online endpoint (real-time) of your AML workspace for a custom model.

@odata.type

Microsoft.Skills.Custom.AmlSkill

Skill parameters

Parameters are case sensitive. The parameters you use depend on what authentication your AML online endpoint requires, if any.

Authentication

AML online endpoints provide two authentication options:

• Key-based authentication. A static key is provided to authenticate scoring requests from AML skills. Set the uri and key parameters for this connection.

• Token-based authentication, where the AML online endpoint is deployed using token-based authentication. The Azure AI Search service must have a managed identity and a role assignment on the AML workspace. The AML skill then uses the service's managed identity to authenticate against the AML online endpoint, with no static keys required. The search service identity must have the Owner or Contributor role. Set the resourceId parameter, and if the search service is in a different region from the AML workspace, set the region parameter.

Skill inputs

Skill inputs are a node of the enriched document created during document cracking. For example, it might be the root document, a normalized image, or the content of a blob. There are no predefined inputs for this skill. For inputs, you should specify one or more nodes that are populated at the time of the AML skill's execution.

Skill outputs

Skill outputs are new nodes of an enriched document created by the skill. There are no predefined outputs for this skill. For outputs, you should provide nodes that can be populated from the JSON response of your AML skill.

Sample definition

  {
    "@odata.type": "#Microsoft.Skills.Custom.AmlSkill",
    "description": "A custom model that detects the language in a document.",
    "uri": "https://language-model.models.contoso.com/score",
    "context": "/document",
    "inputs": [
      {
        "name": "text",
        "source": "/document/content"
      }
    ],
    "outputs": [
      {
        "name": "detected_language_code"
      }
    ]
  }

Sample input JSON structure

This JSON structure represents the payload sent to your AML online endpoint. The top-level fields of the structure correspond to the "names" specified in the inputs section of the skill definition. The values of those fields are from the "sources" of those fields, which could be from a field in the document or another skill.

{
  "text": "Este es un contrato en Inglés"
}

Sample output JSON structure

The output corresponds to the response from your AML online endpoint. The AML online endpoint should only return a JSON payload (verified by looking at the Content-Type response header) and should be an object where the fields are enrichments matching the "names" in the output and whose value is considered the enrichment.

{
    "detected_language_code": "es"
}

Inline shaping sample definition

  {
    "@odata.type": "#Microsoft.Skills.Custom.AmlSkill",
    "description": "A sample model that detects the language of sentence",
    "uri": "https://language-model.models.contoso.com/score",
    "context": "/document",
    "inputs": [
      {
        "name": "shapedText",
        "sourceContext": "/document",
        "inputs": [
            {
              "name": "content",
              "source": "/document/content"
            }
        ]
      }
    ],
    "outputs": [
      {
        "name": "detected_language_code"
      }
    ]
  }

Inline shaping input JSON structure

{
  "shapedText": { "content": "Este es un contrato en Inglés" }
}

Inline shaping sample output JSON structure

{
    "detected_language_code": "es"
}

Error cases

In addition to your AML being unavailable or sending nonsuccessful status codes, the following cases are considered errors:

• The AML online endpoint returns a success status code, but the response indicates that it isn't application/json. The response is thus invalid, and no enrichments are performed.

• The AML online endpoint returns invalid JSON.

If the AML online endpoint is unavailable or returns an HTTP error, a friendly error with any available details about the HTTP error is added to the indexer execution history.

See also

• Create a skillset in Azure AI Search
• Use embedding models from Azure AI Foundry model catalog
• Troubleshoot Azure Machine Learning online endpoint deployment and scoring