What is case categorization?

Assembled’s AI-driven case categorization helps you automatically classify tickets by type (i.e., intent or contact reason).

• Case categorization cuts down on handle time by allowing agents to just focus on the response to the customers, instead of also manually categorizing the ticket.
• Case categorization also provides you with your Voice of the customer to be able to drive key product insights back to your team.

Setup

You should work with your Assembled AI point of contact hand-in-hand for each step of this process. But the below, will give you an overview of the setup process.

1. First, you’ll need to ensure Case Tagging is turned on within your Assembled AI instance. Again, please work with your point of contact on the Assembled AI team to make sure this is turned on!
2. Next, you’ll need to pull in your existing case categorization hierarchy. We recommend doing so by Importing from your Zendesk custom field. This will automatically pull in the existing hierarchy of categories you have today. Once selecting the appropriate field, you can Select Continue. Assembled will then import all existing field values for the relevant tag and assume the hierarchy based on the :: between text within the value. This may take a few minutes to complete.
3. Alternatively, you can also upload your category list via CSV. Please reference the CSV Template provided. Please note that by manually uploading your categories you will need to provide an initial description and a list of True Positives for each category (ticket IDs of tickets that were correctly assigned the relevant case). For True Positives, please be sure to separate ticket IDs by a comma (e.g., 12345, 67890).
   
4. Once the field values are imported, please review the list of categories and the assumed hierarchy to make sure it aligns with your categorization hierarchy.
   
5. Upon clicking into categories, you’ll also notice that each category at the lowest hierarchy level has a list of Example cases for this category. These are historical tickets that have been assigned to this category previously. This tickets will be used to help generate descriptions based on their content and test those descriptions.
   
6. If you’d like to make any changes, you can select the three dots next to the Category name. This will allow you to:
   • Edit - change the category name
   • Move - add hierarchy level mapping (i.e., assign a parent category)
   • Delete - remove the category entirely
7. Once you’ve finalized your list of categories, select “Confirm Categories” and “Autogenerate descriptions”. Assembled will use the categories along with historical tickets assigned to those categories, to generate mutually exclusive and collectively exhaustive descriptions. This exercise does take some time, I’d recommend checking back 30 or mins or so after kicking off the auto-generation.
8. Once the description has been generated, you’ll want to get an initial baseline of accuracy by selecting “Run evaluation”. Assembled will perform a ‘mock’ category assignment test. By taking the same historical tickets that were used to generate the category description, Assembled will use these descriptions to test it’s category assignment. If the historical category matches Assembled’s test-generated category, it will be considered correct. This evaluation will not have any impact to the ticket & it’s existing field / status values. This evaluation can take some time, varying from 20 mins to overnight depending on the number of your categories.
   

Improving Accuracy

So, you have a baseline accuracy metric? What does it mean and how can you improve it?

The two most important metrics when understand how well your categorization is performing is the overall accuracy and the True Positives.

• Overall Accuracy: The % of true positives achieved relative to the total number of cases evaluated.
• True Positives: The number tickets assigned a contact reasons correctly (based on the historical comparison).
• False Negatives and False Positives: The number of tickets assigned an incorrect contact reason (based on the historical comparison.

There’s a few things you can do if you identify that your evaluation is not performing well off the bat.

1. First, depending on how accurate agents are at manually tagging categories today, you may want to change out the Example cases listed for all categories or categories that are performing particularly poorly. The better the source of truth of example tickets, the more specific our generated descriptions can be & the better they perform.
2. Second, for certain categories you may have tags that are already set on the creation of the ticket. You can also use these tags to guide when certain categories should be set. It is important to note whichever Zendesk Ticket Property that is used as a rule MUST be set at the creation of the ticket.

After making any updates to certain categories, you will want to re-optimize the description based on the new example ticket IDs or rules.

Once re-optimizing your lowest performing descriptions, we’d recommend running another evaluation to confirm the percent of correct cases has increased! Keep tweaking until you reach at least 50-60%.

At this stage, we typically recommend launching case categorization side by side with your existing categorization process, but please work directly with your point of contanct. This means you can activate Assembled’s auto-generated case categorization, but instead of overwriting your existing categorization you can write to a new field, which allows you to test new ticket accuracy relative to agents.

Activating Case Categorization

To activate case categorization, you can create a new customer field in Zendesk to automate the tagging of the category (i.e., for side by side comparison). We recommend starting with this first!

You can also use the existing field you have today, in which case Assembled will update directly within that field.

FAQ

• How will Cal handle updates to the field?
  • Assembled AI does NOT continuously sync from Zendesk so changes made in Zendesk after importing, must also be made in Assembled. However, we do write in the other direction, for example, adding categories in Assembled adds them in Zendesk. When we change a category in Assembled, we delete and replace (technically an update). We do not remove on delete.