What are evaluations?

Evaluations are a way for you to track how well Cal is responding to users. The information gathered through evaluations will enable you to improve Cal’s performance over time.

How can I evaluate Cal?

Evaluations are completed in the "Quality" tab.

1. Every interaction with Cal will be tracked from this page. 
2. Within each Cal conversation, you’ll have the ability to mark Cal’s response to a question as
   • Correct answer
   • Incorrect resource
   • Incorrect response
   • Missing knowledge
   • Impossible to answer
   • Excluded
     
3. You’ll be able to view the results of these evaluations in the Assist scorecard report:

Evaluation definitions

• Correct answer: The knowledge used to generate the answer is correct and the answer itself is correct.
• Incorrect resource: The knowledge used to generate the answer is incorrect. Meaning, Cal failed to find/use the right resource.
• Incorrect response: The knowledge used to provide this answer was correct, however, the response contained incorrect information. “Incorrect response” could also indicate that Cal’s response is missing pertinent steps or information.
• Missing knowledge: Cal does not have the knowledge required to answer the question. This could mean:
  • There is no knowledge base article with this content.
  • There is knowledge with this information, but Cal does not have access to the knowledge.
  • There is incorrect or outdated knowledge driving Cal’s response.
• Impossible to answer: The user did not provide enough information for Cal to answer the question.
  • The question from the user is missing details
• Excluded: The interaction is a test or a duplicate.

Metric definitions

In your metrics reporting you’ll see a “correct %” and an “incorrect %”. Below is how each of these metrics is calculated.

• correct % = (Correct answer) / (Correct answer + Incorrect resource + Incorrect response + Missing Knowledge)
• incorrect % = (Incorrect resource + Incorrect response + Missing Knowledge) / (Correct answer + Incorrect resource + Incorrect response + Missing Knowledge)

NOTE: Impossible to answer and Exclude are excluded from “correct %” and “incorrect %”.