What is the list of possible language codes returned by the "Languages" add-on in Document Intelligence?

Hi CoreyK-2554

The Document Intelligence prebuilt-layout model incorporates a sophisticated language detection system that adheres to internationally recognized standards for identifying languages in documents. Specifically, it follows the BCP-47 (IETF Best Current Practice 47) convention for structuring language tags and uses ISO-639-1 identifiers for base language codes. This dual-standard approach ensures both broad compatibility and precise language recognition across various regions and dialects. The model supports a wide range of language codes, including multiple regional variants of major languages. For instance, English is represented by codes such as en-US, en-GB, en-AU, en-CA, and en-IN, while European languages include fr-FR (French), de-DE (German), it-IT (Italian), es-ES (Spanish), pt-PT (Portuguese), and nl-NL (Dutch). Asian languages are also well-supported, with codes like ja-JP (Japanese), zh-Hans (Simplified Chinese), ko-KR (Korean), and vi-VN (Vietnamese). The model uses deep learning to automatically detect languages within documents, even when multiple languages are present, without requiring explicit input parameters. If the system cannot determine the language, it returns "unknown" as a standardized response.

While specifying language codes is optional, understanding their structure—where the base code follows ISO-639-1 and the region tag aligns with BCP-47—is essential for optimal implementation. This ensures consistent behavior across different environments and enhances the model’s ability to process multilingual content effectively. Incorrect or unsupported language codes may result in incomplete text extraction, so leveraging the model’s auto-detection capabilities is generally recommended. This implementation makes Document Intelligence particularly powerful for handling diverse and multilingual document sets while maintaining compliance with global standards.

Reference links : Language support: prebuilt models

Hope it helps

Thank you

Thank you Ravada, this is helpful. However, I'm seeing what appears to be a mix of full "BCP-47" codes as well as some of the base "ISO-639-1" codes returned in the results from Doc Intelligence language detection model.

For example, when the model encounters English text I get back "en" for the locale which appears to be an ISO-639-1 base code.

However, when I run it against Chinese text I get back "zh-Hans" which is a full BCP-47 code w/region tag.

Is there some fallback mechanism where the model first tries to identify the full BCP-47, and if it can't identify the region it simply uses the ISO-639-1 base code? Or is it the case that for certain languages it will always use BCP-47 but others it will use ISO-639-1? If the latter, is there a list of all possible language codes that can be returned by the languages feature?

Hi CoreyK-2554

Sorry for delayed response

The Document Intelligence language detection model identifies languages using either simple two-letter codes (like en for English) or more detailed codes (like zh-Hans for Simplified Chinese) depending on what's needed for accurate identification. When detecting English text, it uses the simple format because English doesn't require additional context. However, for Chinese text, it uses the longer format to specify whether it's Simplified or Traditional script, as this distinction is crucial for proper processing.

The model supports many languages, including English variations (en-US, en-GB), European languages (fr-FR, de-DE, it-IT), and Asian languages (ja-JP, ko-KR, vi-VN). It automatically detects languages in documents without requiring manual input, and returns "unknown" when it cannot determine the language. This approach ensures reliable document processing while maintaining accuracy across different languages and writing systems.

Hope it helps!

Thank you