chatbot responds with different answers for same question

Question

chatbot responds with different answers for same question

Admin 50

Our chatbot responds different answers to the same question.

we are using our own data (2000 files) uploaded to Blob storage and Created index and Built in RAG functionality from Import and Vectorize data which has created skillset and using cosmodb for storing chat history, enabled schematic ranker too.

we use a GPT 4.1 deployment model, with text-embedding003 model.

AZURE_OPENAI_TEMPERATURE = 1

AZURE_OPENAI_TOP_P = 1

AZURE_SEARCH_STRICTNESS = 3

AZURE_SEARCH_TOP_K = 5

Could you help

Manas Mohanty 8,325 Reputation points Microsoft External Staff Moderator

2025-08-04T01:33:34.63+00:00

Hi Admin

Good day.

Agree with Divyesh Govaerdhanan's suggestions.

Azure OpenAI side

Wanted to emphasize Temperature = 1 will make model hallucinate to come with unique answer, you can keep it with 0.1 to 0.3 to come with predictable outcome.

Yes, it is recommended to adjust Top_K and Top_P too

Use chain of thoughts to pass multiple conditions and ask model not to get confused and give irrelevant answer if it does not find a match in knowledge base

Azure AI search side

You can test with Hybrid indexing (Both Semantic and Vector)

https://techcommunity.microsoft.com/blog/azure-ai-services-blog/automate-rag-indexing-azure-logic-apps--ai-search-for-source-document-processing/4266083

Ask

Could you share the indexing documentation referred for indexing the files

Thank you

Accepted answer

1 additional answer

Your answer

Manas Mohanty 8,325 Reputation points Microsoft External Staff Moderator

2025-08-04T01:33:34.63+00:00

Hi Admin

Good day.

Agree with Divyesh Govaerdhanan's suggestions.

Azure OpenAI side

Wanted to emphasize Temperature = 1 will make model hallucinate to come with unique answer, you can keep it with 0.1 to 0.3 to come with predictable outcome.

Yes, it is recommended to adjust Top_K and Top_P too

Use chain of thoughts to pass multiple conditions and ask model not to get confused and give irrelevant answer if it does not find a match in knowledge base

Azure AI search side

You can test with Hybrid indexing (Both Semantic and Vector)

https://techcommunity.microsoft.com/blog/azure-ai-services-blog/automate-rag-indexing-azure-logic-apps--ai-search-for-source-document-processing/4266083

Ask

Could you share the indexing documentation referred for indexing the files

Thank you

Answer 1

Hi Admin

Thank you for sharing the configurations.

Here are key findings

Authentications

"name": "AUTH_ENABLED",

"value": False

will be accessible to public without authentication (recommend you to add Google, Entra or any other authentication)

Environment variables

Environment variables for Embedding endpoint and keys are missing which is crucial

AZURE_OPENAI_EMBEDDING_ENDPOINT

AZURE_OPENAI_EMBEDDING_KEY

Top_P - "value": "0.1", - Can you set it to 0.2 or 0.3

Index Optimization

Weaknesses:

No Field for Business Entity: There's no business_unit or organization field to match “Sunrun business.”
No Project-Specific Field: No project_name, project_owner, or project_type field to match “projects.”
No Facetable Fields: Limits filtering by business unit or project category.
No Suggesters: Reduces autocomplete and guided search capabilities.

Please add few more fields

{
  "name": "business_unit",
  "type": "Edm.String",
  "searchable": true,
  "filterable": true,
  "retrievable": true,
  "stored": true,
  "sortable": false,
  "facetable": true,
  "key": false
},
{
  "name": "project_name",
  "type": "Edm.String",
  "searchable": true,
  "filterable": true,
  "retrievable": true,
  "stored": true,
  "sortable": true,
  "facetable": true,
  "key": false
},
{
  "name": "project_owner",
  "type": "Edm.String",
  "searchable": true,
  "filterable": true,
  "retrievable": true,
  "stored": true,
  "sortable": false,
  "facetable": false,
  "key": false
}

Update on Semantic ranking



"prioritizedFields": {
  "titleField": {
    "fieldName": "title"
  },
  "prioritizedContentFields": [
    { "fieldName": "chunk" },
    { "fieldName": "project_name" }
  ],
  "prioritizedKeywordsFields": [
    { "fieldName": "business_unit" },
    { "fieldName": "project_owner" }
  ]
}

Parent Id , title, storage path url can be set as searchable

Sample queries should be used

list all the projects or initiative being run by "sunrun" business
Find all "Sunrun business" unit projects

Hope it helps address the issue.

Please let us know if the issue still persists after implementing all above index and query optimization.

Thank you

Answer 2

Divyesh Govaerdhanan 8,345

Hello,

Welcome to Microsoft Q&A,

Even with identical queries and retrieved documents, chatbot responses may vary due to the following reasons:

High Model Temperature (temperature = 1)
1. This controls randomness. A value of 1 allows the model to generate more diverse (and sometimes inconsistent) outputs. Suggestion: Set AZURE_OPENAI_TEMPERATURE = 0.2–0.4 to improve consistency.
TOP_P = 1 (Unfiltered Sampling)
1. When combined with high temperature, this encourages creativity but reduces determinism. Suggestion: Try reducing AZURE_OPENAI_TOP_P to 0.8 or lower for more focused sampling.
Variability in Retrieved Chunks (TOP_K = 5)
1. Each call may retrieve slightly different documents or orders depending on semantic similarity and ranking noise. Suggestion: Use semantic_captions = true, and consider setting AZURE_SEARCH_STRICTNESS = 4–5 to tighten relevance filtering.

Please Upvote and accept the answer if it helps!!

Admin 50

Thank you @Divyesh Govaerdhanan i did follow your suggestion and also a blogs and changed the settings, but still i get different answers,

Question i asked my bot

1.provide me details for business sunrun

Response was ----- Here is a recent project associated with the business Sunrun:

Project Address: 11015 Santa Gertrudes Ave

Permit Type: Residential Solar

System Details: 11.5 kW AC, 17 modules, 1 inverter, 1 battery

Status Date: 2025-07-31

2.Late my follow up question was : list all the projects done by sunrun business

Response was ----- The requested information is not available in the retrieved data. Please try another query or topic.

The provided reference(s) can offer more information on this question.

Repeated the question again 1. "provide me details for business sunrun"

now the response was ---------The requested information is not available in the retrieved data. Please try another query or topic.

The provided reference(s) can offer more information on this question.

below is my Configuration in webapp and index configuration.

[

{

"name": "AUTH_CLIENT_SECRET",

"value": "xxxxxxxxxxxxxx",

"slotSetting": false

},

{

"name": "AUTH_ENABLED",

"value": "false",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_ACCOUNT",

"value": "db-asktest",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_CONVERSATIONS_CONTAINER",

"value": "conversations",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_DATABASE",

"value": "db_conversation_history",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_CONNECTION_STRING",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_CONTAINER",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_CONTENT_COLUMNS",

"value": "chunk",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_DATABASE",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_FILENAME_COLUMN",

"value": "name",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_INDEX",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_TITLE_COLUMN",

"value": "title",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_URL_COLUMN",

"value": "url",

"slotSetting": false

},

{

"name": "AZURE_COSMOSDB_MONGO_VCORE_VECTOR_COLUMNS",

"value": "text_vector",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_EMBEDDING_ENDPOINT",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_EMBEDDING_KEY",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_EMBEDDING_NAME",

"value": "text-embedding-3-large",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_ENDPOINT",

"value": "https://xxxxx.openai.azure.com/",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_KEY",

"value": "xxxxxxxxx",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_MAX_TOKENS",

"value": "800",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_MODEL",

"value": "gpt-4.1",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_MODEL_NAME",

"value": "gpt-4.1",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_RESOURCE",

"value": "whitopenai",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_STOP_SEQUENCE",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_SYSTEM_MESSAGE",

"value": "You are an AI assistant that helps people find information.",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_TEMPERATURE",

"value": "0.2",

"slotSetting": false

},

{

"name": "AZURE_OPENAI_TOP_P",

"value": "0.1",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_CONTENT_COLUMNS",

"value": "chunk",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_ENABLE_IN_DOMAIN",

"value": "true",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_FILENAME_COLUMN",

"value": "name",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_INDEX",

"value": "rag-1750751552545-test",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_KEY",

"value": "xxxxxx",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_PERMITTED_GROUPS_COLUMN",

"value": "",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_QUERY_TYPE",

"value": "vector_semantic_hybrid",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG",

"value": "rag-1750751552545-test-semantic-configuration",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_SERVICE",

"value": "whittiersearch",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_STRICTNESS",

"value": "3",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_TITLE_COLUMN",

"value": "title",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_TOP_K",

"value": "5",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_URL_COLUMN",

"value": "url",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_USE_SEMANTIC_SEARCH",

"value": "true",

"slotSetting": false

},

{

"name": "AZURE_SEARCH_VECTOR_COLUMNS",

"value": "text_vector",

"slotSetting": false

},

{

"name": "DATASOURCE_TYPE",

"value": "AzureCognitiveSearch",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_CONTENT_COLUMNS",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_EMBEDDING_MODEL_ID",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_ENABLE_IN_DOMAIN",

"value": "true",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_ENCODED_API_KEY",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_ENDPOINT",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_FILENAME_COLUMN",

"value": "name",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_INDEX",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_QUERY_TYPE",

"value": "",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_STRICTNESS",

"value": "3",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_TITLE_COLUMN",

"value": "title",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_TOP_K",

"value": "5",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_URL_COLUMN",

"value": "url",

"slotSetting": false

},

{

"name": "ELASTICSEARCH_VECTOR_COLUMNS",

"value": "text_vector",

"slotSetting": false

},

{

"name": "MONGODB_APP_NAME",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_COLLECTION_NAME",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_CONTENT_COLUMNS",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_DATABASE_NAME",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_ENABLE_IN_DOMAIN",

"value": "true",

"slotSetting": false

},

{

"name": "MONGODB_ENDPOINT",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_FILENAME_COLUMN",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_INDEX_NAME",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_PASSWORD",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_STRICTNESS",

"value": "3",

"slotSetting": false

},

{

"name": "MONGODB_TITLE_COLUMN",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_TOP_K",

"value": "5",

"slotSetting": false

},

{

"name": "MONGODB_URL_COLUMN",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_USERNAME",

"value": "",

"slotSetting": false

},

{

"name": "MONGODB_VECTOR_COLUMNS",

"value": "",

"slotSetting": false

},

{

"name": "SCM_DO_BUILD_DURING_DEPLOYMENT",

"value": "true",

"slotSetting": false

}

]

Indexes configuration.

{

"@odata.etag": ""0x8DDBE74B2AE36FB"",

"name": "rag-1750751552545-test",

"fields": [

{

  "name": "chunk_id",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": true,

  "facetable": false,

  "key": true,

  "analyzer": "keyword",

  "synonymMaps": []

},

{

  "name": "parent_id",

  "type": "Edm.String",

  "searchable": false,

  "filterable": true,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

},

{

  "name": "chunk",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

},

{

  "name": "title",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

},

{

  "name": "text_vector",

  "type": "Collection(Edm.Single)",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "dimensions": 3072,

  "vectorSearchProfile": "rag-1750751552545-test-azureOpenAi-text-profile",

  "synonymMaps": []

},

{

  "name": "name",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

},

{

  "name": "metadata_storage_path",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

},

{

  "name": "url",

  "type": "Edm.String",

  "searchable": true,

  "filterable": false,

  "retrievable": true,

  "stored": true,

  "sortable": false,

  "facetable": false,

  "key": false,

  "synonymMaps": []

}

],

"scoringProfiles": [],

"suggesters": [],

"analyzers": [],

"normalizers": [],

"tokenizers": [],

"tokenFilters": [],

"charFilters": [],

"similarity": {

"@odata.type": "#Microsoft.Azure.Search.BM25Similarity"

},

"semantic": {

"defaultConfiguration": "rag-1750751552545-test-semantic-configuration",

"configurations": [

  {

    "name": "rag-1750751552545-test-semantic-configuration",

    "flightingOptIn": false,

    "rankingOrder": "BoostedRerankerScore",

    "prioritizedFields": {

      "titleField": {

        "fieldName": "title"

      },

      "prioritizedContentFields": [

        {

          "fieldName": "chunk"

        }

      ],

      "prioritizedKeywordsFields": []

    }

  }

]

},

"vectorSearch": {

"algorithms": [

  {

    "name": "rag-1750751552545-test-algorithm",

    "kind": "hnsw",

    "hnswParameters": {

      "metric": "cosine",

      "m": 4,

      "efConstruction": 400,

      "efSearch": 500

    }

  }

],

"profiles": [

  {

    "name": "rag-1750751552545-test-azureOpenAi-text-profile",

    "algorithm": "rag-1750751552545-test-algorithm",

    "vectorizer": "rag-1750751552545-test-azureOpenAi-text-vectorizer"

  }

],

"vectorizers": [

  {

    "name": "rag-1750751552545-test-azureOpenAi-text-vectorizer",

    "kind": "azureOpenAI",

    "azureOpenAIParameters": {

      "resourceUri": "https://xxxxx.openai.azure.com",

      "deploymentId": "text-embedding-3-large",

      "apiKey": "<redacted>",

      "modelName": "text-embedding-3-large"

    }

  }

],

"compressions": []

}

Admin 50 Reputation points

2025-08-05T12:57:15.26+00:00
Thank you for the suggestion @Manas Mohanty

i did incorporate your changes except suggestor and Auth_enabled, but still not getting same answers.

list all the projects or initiative being run by "sunrun" business

no answer

projects by sunrun

gave 1 result

how many projects did sunrun work on

Based on the provided documents, there is no direct mention of Sunrun as the contractor or business

added fields in index as people, organizations, business, project (Collection(Edm.String)

"prioritizedFields": {

"titleField": { "fieldName": "title" }, "prioritizedContentFields": [ { "fieldName": "chunk" }, { "fieldName": "people" }, { "fieldName": "organizations" } ], "prioritizedKeywordsFields": [ { "fieldName": "business" }, { "fieldName": "project" } ] }

i have another question when we ask a question "latest permit or project by david" it gives response related to old data, not picking up the new ones.

when we specify july 2025 or in 2025 then it responds to the recent ones. how to train bot to pick up latest
Manas Mohanty 8,325 Reputation points Microsoft External Staff Moderator

2025-08-06T02:08:26.1633333+00:00

Hi Admin

We can modify the prompt" permit or project by david after 2025"

Can you test the same index again GPT 4-o or 4-o mini or 4.1 min models and let us know.

Thank you
Admin 50 Reputation points

2025-08-11T06:37:42.7966667+00:00

Hi @Manas Mohanty when this site is public we cannot restrict user, they ask questions however they want, is there something in prompting where our bot understands latest and check current year and months results first and then later the old records ? i did modify the prompt to provide 2025 yr recods but still it didn't. unless while asking question if mentioned year it provides.

inspite of using gpt 4.1. its the same. but people and organization level question issue has been addressed.

thank you.
Manas Mohanty 8,325 Reputation points Microsoft External Staff Moderator

2025-08-11T06:52:22.0133333+00:00
Hi Admin

Thank you for confirming that at least we were able to address the project/org issue.

From OpenAI perspective, we can enforce in system instruction too like "Go through attached data and Provide latest events when asked"

For AI search perspective, we can add a date field and enabled filtering and searching,

You can use a custom skill to enrich your data with meta data and timestamps.

Opt for Hybrid search

"queryType": "full", "scoringProfile": "hybridRRF"

Opt for recentboost scoring

"scoringProfile": "recentBoost"

Please create a new thread in AI search side to address the date wise filtering issue if the issue still persists and mark below answer as resolved based on reported scope

Thank you

Share via

chatbot responds with different answers for same question

1 additional answer

Your answer