Reviewing Quality evaluations

What are evaluations?

Evaluations help you track how well Cal is responding to users. The information gathered through evaluations allows you to spot trends over time and determine whether any configuration or knowledge-based updates are needed.

How can I evaluate Cal?

Accessing the Quality Review page

1. In the left sidebar, click on AI quality and select QA review
2. Every interaction with Cal will be tracked from this page. Use the tabs at the top (Email agent, Chat agent, Voice agent and Copilot) and the date range, Sort and Add filter controls to view tickets by AI agent, date, evaluation status, channel, category, and more.

Evaluating interactions

To start evaluating cases:

1. Click on any case in the list to open the detailed evaluation view
2. Review the full conversation history and Cal's response
3. Open the QA scoring tab on the right and score Cal's performance
4. Add any additional comments in the Comments (optional) fields
5. Click "Submit evaluation" when finished

Evaluation criteria

The default evaluation criteria vary by interaction type. For custom evaluation criteria, please see below.

AI Agent Reply Accuracy (All interaction types)

• Correct: The answer provided is accurate
• Somewhat incorrect: The answer is partially accurate but contains some inaccuracies
• Incorrect: The answer provided is inaccurate

Knowledge Retrieval (Co-pilot only)

• Correct source: Cal referenced the correct knowledge base articles
• Incorrect source: Cal did not use the appropriate knowledge sources

AI Agent Reply Style (Co-Pilot, Email)

• Excellent: The reply follows your Style Guide well
• Fair: The reply partially follows your Style Guide
• Poor: The reply does not follow your Style Guide

Workflow selection (Email only)

• Correct: The relevant workflow was accurately chosen
• Incorrect: The wrong workflow was selected

Custom Evaluation Criteria

Custom evaluation criteria can be added through the QA scoring page, opened with the Manage QA scoring button at the top right of AI quality > QA review.

Click 'Add score' to begin adding a custom score:

• Channel: these are the types of conversations/interactions that this score will be available for on the evaluations page

• Ratings: add the options you would like to show up as rating buttons under the score on the evaluations page, e.g. 'Correct', 'Somewhat incorrect', 'Incorrect' in this screenshot:

  

• Reasoning Tags (optional): add tags that you'd like to be able to select between for a particular score

Once saved, you will be able to see the custom score on an applicable conversation type on the case's QA scoring tab in Quality review.

Quality reporting

Once evaluations are completed, you can view performance data in the AI agent QA scorecard report (formerly the Quality report).

Note: The Quality report is currently available for Copilot only, with reporting for Email, Voice, and Chat agents coming soon.

Accessing the Quality report

1. In the left sidebar, click on Report and select AI agent QA scorecard
2. The report has two main tabs:
   • Drafted replies: Includes performance data for all Co-pilot interaction types
   • Agent guidance: Focuses on ‘Question answer’ and ‘Summarize’ interaction types
3. Use the date range, Types and Add filter filters to view data by interaction type, date, or source

Report metrics

The Quality report includes metrics based on two inputs:

• Evaluations completed by your QA team
• Agent feedback submitted via the thumbs-up (helpful) and thumbs-down (not helpful) buttons in the Copilot interface

Tips for effective quality reporting

The Quality report only shows data for cases that have been evaluated by your QA team or received agent feedback. To obtain meaningful insights:

1. Have your QA team regularly evaluate interactions in the QA review page
2. Encourage agents to provide feedback directly in the Cal interface
3. Ensure evaluations cover all relevant Copilot support types
4. Maintain consistent evaluation criteria