MedImageInsight Embedding Usage + Report Generation Workflow (with Accuracy Clarification)

Question

MedImageInsight Embedding Usage + Report Generation Workflow (with Accuracy Clarification)

Ishant Singh 0

Hello Team,

We’re currently evaluating MedImageInsight as part of a medical imaging workflow aimed at automated report generation to assist radiologists. We understand that the model generates embeddings from medical images, and we’re exploring the best way to utilize them effectively.

Below are our questions and goals:

🔹 Primary Objective:

To generate preliminary medical reports from medical images (or DICOMs) using MedImageInsight or any other AI model that can support this task.

🔹 Context:

We are not aiming for 100% diagnostic accuracy.
Our goal is to reduce reporting time and assist certified radiologists by providing a draft report that they will always review and finalize.
For our use case, 60–80% accuracy is acceptable, as it serves to accelerate workflow, not replace medical judgment.

🔹 Questions & Clarifications:

Embedding Decoding
- We understand MedImageInsight generates embeddings as output.
- Some sources suggest these embeddings could be passed into another AI model to decode them into structured labels or full reports.
- However, newer insights suggest decoding embeddings directly may not be possible.
- Is there any model or recommended method to decode or interpret MedImageInsight embeddings into report-like outputs or labels?
Label Dataset for Embedding Matching
- If there’s a reference dataset of labeled embeddings (image-label or image-report pairs used during MedImageInsight’s training), we would like to access or replicate it.
- We plan to perform similarity search between new image embeddings and this labeled set, and generate preliminary reports based on the closest matches.
DICOM Support
- Is it possible to provide DICOM files directly to MedImageInsight?
- Converting DICOMs to PNG/JPEG seems to reduce volumetric (3D) detail, which may impact model effectiveness.
- This is not a blocking requirement — we’re willing to proceed with 2D inputs, but would appreciate any guidance.
Alternative Model Recommendation
- If there’s any alternative AI model that directly generates radiology reports from medical images or DICOMs with 60–80% reliability, we are open to adopting that instead of or alongside MedImageInsight.

We’re building a clinical decision support tool that works alongside certified radiologists, not as a replacement. Any support, guidance, model links, or reference materials would be highly appreciated.

Thank you in advance for your assistance.

Best regards, Ishant Singh Founder – WhatsDiscuss | SmartAcademy | SmartCarePlus

Alex Burlachenko 13,330 Reputation points Volunteer Moderator

2025-08-06T10:06:08.52+00:00
hi Ishant Singh,

thanks for posting this super detailed question ))

so u wanna speed up radiologists' workflow by generating draft reports from medical images, right? about medimageinsight embeddings... yes, the model creates embeddings, but decoding them directly into full reports isn't really its thing. think of these embeddings more like a 'fingerprint' of the image - great for comparing similarities, but not a direct translation. u could try pairing them with a language model (like gpt4 tuned for medical text) to generate report-like outputs.

no, there's no public dataset of labeled medimageinsight embeddings, but u could build ur own! train a secondary model to match new embeddings with known cases. it's extra work, but would give u that similarity search u mentioned.

dicom files? medimageinsight prefers standard image formats (png/jpeg), so u'd need to convert. yeah, u lose some 3D detail, but for 2D analysis it's still decent. if volumetric data is critical, check out azure health ai's dicom tools https://learn.microsoft.com/en-us/azure/healthcare-apis/dicom/

other models to try... look into nuance powerscribe or ibm watson health imaging - both play nice with clinical workflows. also, radiology-specific models like chexnet might help for certain scan types. worth testing a few to see what fits ur accuracy sweet spot ))

always validate any ai output against current medical guidelines. even at 80% accuracy, u wanna catch those edge cases.

hope this gets u started! if u hit snags with the embedding-to-report pipeline, come back with specifics - we'll brainstorm more hacks ))

Alex

and "yes" if you would follow me at Q&A - personaly thx. P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/
Pavankumar Purilla 10,425 Reputation points Microsoft External Staff Moderator

2025-08-07T07:09:13.2366667+00:00

Hi Ishant Singh,
Did you get any chance to check the response. Thank you!
Pavankumar Purilla 10,425 Reputation points Microsoft External Staff Moderator

2025-08-08T05:35:52.5666667+00:00

Hi Ishant Singh,
Just following up to see if you had a chance to review the above response. Thank you!

1 answer

Your answer

Alex Burlachenko 13,330 Reputation points Volunteer Moderator

2025-08-06T10:06:08.52+00:00

hi Ishant Singh,

thanks for posting this super detailed question ))

so u wanna speed up radiologists' workflow by generating draft reports from medical images, right? about medimageinsight embeddings... yes, the model creates embeddings, but decoding them directly into full reports isn't really its thing. think of these embeddings more like a 'fingerprint' of the image - great for comparing similarities, but not a direct translation. u could try pairing them with a language model (like gpt4 tuned for medical text) to generate report-like outputs.

no, there's no public dataset of labeled medimageinsight embeddings, but u could build ur own! train a secondary model to match new embeddings with known cases. it's extra work, but would give u that similarity search u mentioned.

dicom files? medimageinsight prefers standard image formats (png/jpeg), so u'd need to convert. yeah, u lose some 3D detail, but for 2D analysis it's still decent. if volumetric data is critical, check out azure health ai's dicom tools https://learn.microsoft.com/en-us/azure/healthcare-apis/dicom/

other models to try... look into nuance powerscribe or ibm watson health imaging - both play nice with clinical workflows. also, radiology-specific models like chexnet might help for certain scan types. worth testing a few to see what fits ur accuracy sweet spot ))

always validate any ai output against current medical guidelines. even at 80% accuracy, u wanna catch those edge cases.

hope this gets u started! if u hit snags with the embedding-to-report pipeline, come back with specifics - we'll brainstorm more hacks ))

Alex

and "yes" if you would follow me at Q&A - personaly thx. P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/
Pavankumar Purilla 10,425 Reputation points Microsoft External Staff Moderator

2025-08-07T07:09:13.2366667+00:00

Hi Ishant Singh,
Did you get any chance to check the response. Thank you!
Pavankumar Purilla 10,425 Reputation points Microsoft External Staff Moderator

2025-08-08T05:35:52.5666667+00:00

Hi Ishant Singh,
Just following up to see if you had a chance to review the above response. Thank you!

Answer 1

Hi Ishant Singh,

MedImageInsight is an embedding model designed to generate vector representations from medical images, which can be used for downstream tasks such as similarity search, classification, or report generation. The embeddings themselves cannot be directly decoded into structured labels or full reports. To generate draft reports, we recommend either fine-tuning a text decoder (e.g., a transformer) on top of the embeddings using image–report pairs or implementing a similarity search approach, where embeddings from new images are matched against a labeled embedding library to retrieve the closest reports for radiologist review.

Currently, no public labeled embedding dataset is provided out of the box. However, you can create one by generating embeddings from your historical image–report archives or from publicly available datasets (e.g., MIMIC-CXR) and associating them with their corresponding labels or reports.

Regarding DICOM support, MedImageInsight does not natively accept DICOM files. The images must be converted into 2D image formats such as PNG or JPEG for embedding generation. This conversion enables processing of individual slices or projections, though some volumetric and 3D spatial details may be lost. Azure supports this workflow through healthcare AI pipelines that include DICOM ingestion, conversion, and embedding generation, and Azure Health Data Services tools such as CloudSync and DICOMcast can be used to manage, synchronize, and query DICOM metadata and images in the cloud.

If your goal is to generate preliminary reports more directly, you may also consider alternative AI models such as LLaVA‑Rad, a visual–language model trained on radiology data that can produce draft reports from medical images. This can be used alongside MedImageInsight for embedding-based similarity search to accelerate radiologist workflows while maintaining human oversight.

For production scenarios, we recommend using the Azure orchestrated multimodal AI insights pipeline for MedImageInsight, which streamlines DICOM handling, embedding generation, and integration with downstream AI workflows.

For your reference: Use orchestrate multimodal AI insights (preview) in healthcare data solutions
Cloud migration for medical imaging data using Azure Health Data Services and IMS
Discovering the Power of Finetuning MedImageInsight on Your Data

How to use MedImageInsight healthcare AI model for medical image embedding generation
Healthcare AI Models

Share via

MedImageInsight Embedding Usage + Report Generation Workflow (with Accuracy Clarification)

🔹 Primary Objective:

🔹 Context:

🔹 Questions & Clarifications:

1 answer

Your answer