Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Bicep resource definition
The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:
- Resource groups - See resource group deployment commands
For a list of changed properties in each API version, see change log.
Resource format
To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following Bicep to your template.
resource symbolicname 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-07-01-preview' = {
parent: resourceSymbolicName
extendedLocation: {
name: 'string'
type: 'string'
}
name: 'string'
properties: {
mode: 'string'
operations: [
{
builtInTransformationSettings: {
datasets: [
{
description: 'string'
expression: 'string'
inputs: [
'string'
]
key: 'string'
schemaRef: 'string'
}
]
filter: [
{
description: 'string'
expression: 'string'
inputs: [
'string'
]
type: 'string'
}
]
map: [
{
description: 'string'
expression: 'string'
inputs: [
'string'
]
output: 'string'
type: 'string'
}
]
schemaRef: 'string'
serializationFormat: 'string'
}
destinationSettings: {
dataDestination: 'string'
endpointRef: 'string'
}
name: 'string'
operationType: 'string'
sourceSettings: {
assetRef: 'string'
dataSources: [
'string'
]
endpointRef: 'string'
schemaRef: 'string'
serializationFormat: 'string'
}
}
]
profileRef: 'string'
}
}
Property Values
Microsoft.IoTOperations/instances/dataflowProfiles/dataflows
| Name | Description | Value |
|---|---|---|
| extendedLocation | Edge location of the resource. | ExtendedLocation (required) |
| name | The resource name | string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required) |
| parent | In Bicep, you can specify the parent resource for a child resource. You only need to add this property when the child resource is declared outside of the parent resource. For more information, see Child resource outside parent resource. |
Symbolic name for resource of type: instances/dataflowProfiles |
| properties | The resource-specific properties for this resource. | DataFlowProperties |
DataFlowBuiltInTransformation
| Name | Description | Value |
|---|---|---|
| datasets | Enrich data from Broker State Store. Dataset references a key in Broker State Store. | DataFlowBuiltInTransformationDataset[] |
| filter | Filters input record or datapoints based on condition. | DataFlowBuiltInTransformationFilter[] |
| map | Maps input to output message. | DataFlowBuiltInTransformationMap[] |
| schemaRef | Reference to the schema that describes the output of the transformation. | string |
| serializationFormat | Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json | 'Delta' 'Json' 'Parquet' |
DataFlowBuiltInTransformationDataset
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the dataset. | string Constraints: Max length = 256 |
| expression | Condition to enrich data from Broker State Store. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string |
| inputs | List of fields for enriching from the Broker State Store. | string[] (required) |
| key | The key of the dataset. | string Constraints: Max length = 64 (required) |
| schemaRef | The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7. | string |
DataFlowBuiltInTransformationFilter
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the filter. | string Constraints: Max length = 256 |
| expression | Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string (required) |
| inputs | List of fields for filtering in JSON path expression. | string[] (required) |
| type | The type of dataflow operation. | 'Filter' |
DataFlowBuiltInTransformationMap
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the mapping function. | string Constraints: Max length = 256 |
| expression | Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided) | string |
| inputs | List of fields for mapping in JSON path expression. | string[] (required) |
| output | Where and how the input fields to be organized in the output record. | string (required) |
| type | Type of transformation. | 'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename' |
DataFlowDestinationOperation
| Name | Description | Value |
|---|---|---|
| dataDestination | Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription. | string (required) |
| endpointRef | Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type. | string (required) |
DataFlowOperation
| Name | Description | Value |
|---|---|---|
| builtInTransformationSettings | Transformation configuration. | DataFlowBuiltInTransformation |
| destinationSettings | Destination configuration. | DataFlowDestinationOperation (required) |
| name | Optional user provided name of the transformation. | string Constraints: Max length = 64 |
| operationType | Type of operation. | 'BuiltInTransformation' 'Destination' 'Source' (required) |
| sourceSettings | Source configuration. | DataFlowSourceOperation (required) |
DataFlowProperties
| Name | Description | Value |
|---|---|---|
| mode | Mode for DataFlow. Optional; defaults to Enabled. | 'Disabled' 'Enabled' |
| operations | List of operations including source and destination references as well as transformation. | DataFlowOperation[] (required) |
| profileRef | Reference to the DataflowProfile CR. | string (required) |
DataFlowSourceOperation
| Name | Description | Value |
|---|---|---|
| assetRef | Reference to the resource in Azure Device Registry where the data in the endpoint originates from. | string |
| dataSources | List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +. | string[] (required) |
| endpointRef | Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type. | string (required) |
| schemaRef | Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match. | string |
| serializationFormat | Content is a JSON Schema. Allowed: JSON Schema/draft-7. | 'Json' |
ExtendedLocation
| Name | Description | Value |
|---|---|---|
| name | The name of the extended location. | string (required) |
| type | Type of ExtendedLocation. | 'CustomLocation' (required) |
ARM template resource definition
The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:
- Resource groups - See resource group deployment commands
For a list of changed properties in each API version, see change log.
Resource format
To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following JSON to your template.
{
"type": "Microsoft.IoTOperations/instances/dataflowProfiles/dataflows",
"apiVersion": "2024-07-01-preview",
"name": "string",
"extendedLocation": {
"name": "string",
"type": "string"
},
"properties": {
"mode": "string",
"operations": [
{
"builtInTransformationSettings": {
"datasets": [
{
"description": "string",
"expression": "string",
"inputs": [ "string" ],
"key": "string",
"schemaRef": "string"
}
],
"filter": [
{
"description": "string",
"expression": "string",
"inputs": [ "string" ],
"type": "string"
}
],
"map": [
{
"description": "string",
"expression": "string",
"inputs": [ "string" ],
"output": "string",
"type": "string"
}
],
"schemaRef": "string",
"serializationFormat": "string"
},
"destinationSettings": {
"dataDestination": "string",
"endpointRef": "string"
},
"name": "string",
"operationType": "string",
"sourceSettings": {
"assetRef": "string",
"dataSources": [ "string" ],
"endpointRef": "string",
"schemaRef": "string",
"serializationFormat": "string"
}
}
],
"profileRef": "string"
}
}
Property Values
Microsoft.IoTOperations/instances/dataflowProfiles/dataflows
| Name | Description | Value |
|---|---|---|
| apiVersion | The api version | '2024-07-01-preview' |
| extendedLocation | Edge location of the resource. | ExtendedLocation (required) |
| name | The resource name | string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required) |
| properties | The resource-specific properties for this resource. | DataFlowProperties |
| type | The resource type | 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows' |
DataFlowBuiltInTransformation
| Name | Description | Value |
|---|---|---|
| datasets | Enrich data from Broker State Store. Dataset references a key in Broker State Store. | DataFlowBuiltInTransformationDataset[] |
| filter | Filters input record or datapoints based on condition. | DataFlowBuiltInTransformationFilter[] |
| map | Maps input to output message. | DataFlowBuiltInTransformationMap[] |
| schemaRef | Reference to the schema that describes the output of the transformation. | string |
| serializationFormat | Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json | 'Delta' 'Json' 'Parquet' |
DataFlowBuiltInTransformationDataset
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the dataset. | string Constraints: Max length = 256 |
| expression | Condition to enrich data from Broker State Store. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string |
| inputs | List of fields for enriching from the Broker State Store. | string[] (required) |
| key | The key of the dataset. | string Constraints: Max length = 64 (required) |
| schemaRef | The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7. | string |
DataFlowBuiltInTransformationFilter
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the filter. | string Constraints: Max length = 256 |
| expression | Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string (required) |
| inputs | List of fields for filtering in JSON path expression. | string[] (required) |
| type | The type of dataflow operation. | 'Filter' |
DataFlowBuiltInTransformationMap
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the mapping function. | string Constraints: Max length = 256 |
| expression | Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided) | string |
| inputs | List of fields for mapping in JSON path expression. | string[] (required) |
| output | Where and how the input fields to be organized in the output record. | string (required) |
| type | Type of transformation. | 'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename' |
DataFlowDestinationOperation
| Name | Description | Value |
|---|---|---|
| dataDestination | Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription. | string (required) |
| endpointRef | Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type. | string (required) |
DataFlowOperation
| Name | Description | Value |
|---|---|---|
| builtInTransformationSettings | Transformation configuration. | DataFlowBuiltInTransformation |
| destinationSettings | Destination configuration. | DataFlowDestinationOperation (required) |
| name | Optional user provided name of the transformation. | string Constraints: Max length = 64 |
| operationType | Type of operation. | 'BuiltInTransformation' 'Destination' 'Source' (required) |
| sourceSettings | Source configuration. | DataFlowSourceOperation (required) |
DataFlowProperties
| Name | Description | Value |
|---|---|---|
| mode | Mode for DataFlow. Optional; defaults to Enabled. | 'Disabled' 'Enabled' |
| operations | List of operations including source and destination references as well as transformation. | DataFlowOperation[] (required) |
| profileRef | Reference to the DataflowProfile CR. | string (required) |
DataFlowSourceOperation
| Name | Description | Value |
|---|---|---|
| assetRef | Reference to the resource in Azure Device Registry where the data in the endpoint originates from. | string |
| dataSources | List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +. | string[] (required) |
| endpointRef | Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type. | string (required) |
| schemaRef | Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match. | string |
| serializationFormat | Content is a JSON Schema. Allowed: JSON Schema/draft-7. | 'Json' |
ExtendedLocation
| Name | Description | Value |
|---|---|---|
| name | The name of the extended location. | string (required) |
| type | Type of ExtendedLocation. | 'CustomLocation' (required) |
Usage Examples
Terraform (AzAPI provider) resource definition
The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:
- Resource groups
For a list of changed properties in each API version, see change log.
Resource format
To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following Terraform to your template.
resource "azapi_resource" "symbolicname" {
type = "Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-07-01-preview"
name = "string"
parent_id = "string"
body = {
extendedLocation = {
name = "string"
type = "string"
}
properties = {
mode = "string"
operations = [
{
builtInTransformationSettings = {
datasets = [
{
description = "string"
expression = "string"
inputs = [
"string"
]
key = "string"
schemaRef = "string"
}
]
filter = [
{
description = "string"
expression = "string"
inputs = [
"string"
]
type = "string"
}
]
map = [
{
description = "string"
expression = "string"
inputs = [
"string"
]
output = "string"
type = "string"
}
]
schemaRef = "string"
serializationFormat = "string"
}
destinationSettings = {
dataDestination = "string"
endpointRef = "string"
}
name = "string"
operationType = "string"
sourceSettings = {
assetRef = "string"
dataSources = [
"string"
]
endpointRef = "string"
schemaRef = "string"
serializationFormat = "string"
}
}
]
profileRef = "string"
}
}
}
Property Values
Microsoft.IoTOperations/instances/dataflowProfiles/dataflows
| Name | Description | Value |
|---|---|---|
| extendedLocation | Edge location of the resource. | ExtendedLocation (required) |
| name | The resource name | string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required) |
| parent_id | The ID of the resource that is the parent for this resource. | ID for resource of type: instances/dataflowProfiles |
| properties | The resource-specific properties for this resource. | DataFlowProperties |
| type | The resource type | "Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-07-01-preview" |
DataFlowBuiltInTransformation
| Name | Description | Value |
|---|---|---|
| datasets | Enrich data from Broker State Store. Dataset references a key in Broker State Store. | DataFlowBuiltInTransformationDataset[] |
| filter | Filters input record or datapoints based on condition. | DataFlowBuiltInTransformationFilter[] |
| map | Maps input to output message. | DataFlowBuiltInTransformationMap[] |
| schemaRef | Reference to the schema that describes the output of the transformation. | string |
| serializationFormat | Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json | 'Delta' 'Json' 'Parquet' |
DataFlowBuiltInTransformationDataset
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the dataset. | string Constraints: Max length = 256 |
| expression | Condition to enrich data from Broker State Store. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string |
| inputs | List of fields for enriching from the Broker State Store. | string[] (required) |
| key | The key of the dataset. | string Constraints: Max length = 64 (required) |
| schemaRef | The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7. | string |
DataFlowBuiltInTransformationFilter
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the filter. | string Constraints: Max length = 256 |
| expression | Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0 || $1 > $2 (Assuming inputs section $1 and $2 are provided) | string (required) |
| inputs | List of fields for filtering in JSON path expression. | string[] (required) |
| type | The type of dataflow operation. | 'Filter' |
DataFlowBuiltInTransformationMap
| Name | Description | Value |
|---|---|---|
| description | A user provided optional description of the mapping function. | string Constraints: Max length = 256 |
| expression | Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided) | string |
| inputs | List of fields for mapping in JSON path expression. | string[] (required) |
| output | Where and how the input fields to be organized in the output record. | string (required) |
| type | Type of transformation. | 'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename' |
DataFlowDestinationOperation
| Name | Description | Value |
|---|---|---|
| dataDestination | Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription. | string (required) |
| endpointRef | Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type. | string (required) |
DataFlowOperation
| Name | Description | Value |
|---|---|---|
| builtInTransformationSettings | Transformation configuration. | DataFlowBuiltInTransformation |
| destinationSettings | Destination configuration. | DataFlowDestinationOperation (required) |
| name | Optional user provided name of the transformation. | string Constraints: Max length = 64 |
| operationType | Type of operation. | 'BuiltInTransformation' 'Destination' 'Source' (required) |
| sourceSettings | Source configuration. | DataFlowSourceOperation (required) |
DataFlowProperties
| Name | Description | Value |
|---|---|---|
| mode | Mode for DataFlow. Optional; defaults to Enabled. | 'Disabled' 'Enabled' |
| operations | List of operations including source and destination references as well as transformation. | DataFlowOperation[] (required) |
| profileRef | Reference to the DataflowProfile CR. | string (required) |
DataFlowSourceOperation
| Name | Description | Value |
|---|---|---|
| assetRef | Reference to the resource in Azure Device Registry where the data in the endpoint originates from. | string |
| dataSources | List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +. | string[] (required) |
| endpointRef | Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type. | string (required) |
| schemaRef | Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match. | string |
| serializationFormat | Content is a JSON Schema. Allowed: JSON Schema/draft-7. | 'Json' |
ExtendedLocation
| Name | Description | Value |
|---|---|---|
| name | The name of the extended location. | string (required) |
| type | Type of ExtendedLocation. | 'CustomLocation' (required) |