Get notified for missing questions from QnA Maker

QnAMaker.ai is a great way to provide answers to end-users through a chat interface. And while it has been around for a while there are always new things to explore. We recently set it up internally to provide a FAQ around the whole COVID situation. To keep learning we challenged ourselves to keep our knowledge base as up to date as possible. At first, we fed it daily with the new information, but soon we wanted to have insights into what questions we where missing.

Set-up application insight

Setting up application insights is es easy as it gets. Just make sure you have your QnA maker deployed to Azure. Then navigate to the app service and select Application Insights. It is turned off by default, so just turn it on. You can create either a new application insight instance or reuse an existing one. Once this is set up you can write queries to see what is gathered. You can even use those queries to form a nice looking dashboard as described by Stephan Bisser.

Set-up alert rules

Navigate to the Application Insights. Select the ‘log’ option from the menu, and you can execute queries against the gathered data. By using a sample query, you can easily identify all missing answers.

requests
| where url endswith "generateAnswer"
| project timestamp, id, url, resultCode, duration, performanceBucket
| parse kind = regex url with *"(?i)knowledgebases/"KbId"/generateAnswer"
| join kind= inner
(
    traces
    | extend id = operation_ParentId
)
on id
| extend question = tostring(customDimensions['Question'])
| extend answer = tostring(customDimensions['Answer'])
| extend score = tostring(customDimensions['Score'])
| project timestamp, resultCode, duration, id, question, answer, score, performanceBucket,KbId

Based on that query, you can easily create a new ‘alert rule’ based on your query. This alert rule can then be used to either send an e-mail notification, or a team’s message.

Loop in your subject matter experts to answer questions that are missing and let them republish the knowledge base. That way you have a nice loop of updating your knowledge base. And an automated process to let people know what answers are missing. You could even go as far as to capture user ID’s so that you can let users know that their question has been answered.

There is one thing to keep in mind: If you start your query with a comment the button in your e-mail to go to the report will fail. It will store the query as a single line (all breaks will be removed). This results in the complete query being a comment. So store your query without a comment to use the links from e-mails to live results!

For instance, when you send an e-mail, you will get a daily overview of missing answers:

All in all it is a great way to capture what is happening, and provide feedback to the people who are managing your content!