Plans and permissions:
- Add-on: AI agents, billed separately
- Run a bulk test: Admin, Manager, or Team Lead role, or a custom role with Workflows set to Can create, edit, and delete workflows or Can test and view workflows
- Add or change test cases: Admin, Manager, or Team Lead role, or a custom role with Workflows set to Can create, edit, and delete workflows
Use bulk testing to check how accurately a workflow selects cases, by running it against a library of past cases you’ve labeled. Each test shows how many cases the workflow selected correctly, how many it missed, and how many it picked up by mistake, so you can find problems before customers do and confirm that a change made things better.
To open it, go to AI support > Workflows, open the workflow, and select the Test tab.
How bulk testing works
- Test cases are past cases, identified by case ID, that you’ve assigned to the workflow that should handle them, or to No assignment for cases that no workflow should handle. Your company has one library of test cases, shared by all workflows.
- A bulk test runs each case through workflow selection and compares the workflow that was selected with the one you expected.
- It measures selection, not the quality of replies. To check replies, test single cases as described in Building workflows.
- A test can include up to 500 cases. If your selections add up to more, Assembled picks 500 of them at random.
- Each test is listed on the Test tab with its results. While a test is running, you can’t start another one.
- A test of a few dozen cases usually takes about 10 minutes. Large samples can take 15–20 minutes.
When to run bulk tests
- Before you publish a change to a workflow’s trigger, so the change doesn’t break cases that worked before.
- Before you move from Agent review to Automated. Bulk test at least 20 varied cases before you automate a share of cases, and 40 or more before you automate all of them. Include cases the workflow shouldn’t select. See Workflow activation.
- Regularly for important workflows, such as once a month.
- After a big change to your knowledge base, products, or processes.
Build your test case library
Add cases one at a time while you review them, or many at once by case ID.
To add a case while reviewing it:
- Open the case from AI quality > QA review, or from the workflow’s View cases tab.
- Select Add test case. This is available for Zendesk cases.
- Choose the workflow that should handle the case, or choose not to assign it to any workflow.
To add several cases at once:
- Select Manage test cases, on the Workflows page or on the workflow’s Test tab.
- Select Add cases.
- In Case IDs, enter the case IDs, separated by commas. You can add up to 500 cases at a time.
- Under Workflow assignment, choose Assign to workflow and the workflow that should handle these cases, or No assignment for cases that shouldn’t select any workflow.
- Select Add.
The cases appear in the Test cases list, which you can filter by workflow.
Tips for a useful library:
- Start small. Five to 10 cases for each workflow is enough for a first test. Add more as you find them in your regular reviews.
- Cover the range. Include typical cases, edge cases, different ways customers phrase the same request, and regional variations if they apply.
- Include near misses. Cases that look similar but belong to another workflow, or to none, are the most useful negatives.
- Keep notes. Record why each case should or shouldn’t select the workflow. It helps when you review results.
Run a bulk test
- Go to AI support > Workflows, open the workflow, and select the Test tab.
- Select Test workflow. It’s unavailable until you have at least one test case.
- Under True positives (cases this workflow should select) and True negatives (cases this workflow should not select), choose which cases to test (see the options below). For a first test, 10–20 cases in each group is enough.
- Under Set test parameters, choose the other workflows to test against, where to start each case from, and which version of the workflow to test.
- Select Test workflow.
The test appears at the top of the list on the Test tab. When it finishes, its results fill in. To see each case, open the test’s menu and select View results.
Keep in mind: If you test the Draft version, the workflow must have a draft with unpublished changes. Without one, the test fails right away.
Bulk test options
| Option | What it does |
|---|---|
| Random sample from assigned test cases | True positives: picks the Sample size you enter from the cases assigned to this workflow. |
| Random sample from test cases not assigned to this workflow | True negatives: picks from every test case that isn’t assigned to this workflow, including cases with no assignment. |
| Random sample of test cases assigned to other workflows | True negatives: picks from the cases assigned to the Workflows to sample from you choose, with a Sample size (per workflow). Use it to check that this workflow doesn’t take cases from a similar one. |
| Specific case IDs | Tests the case IDs you enter, such as known edge cases. Available for both groups. |
| None | Leaves out that group. |
| Other workflows to test against | The workflows that compete with this one during the test. Include any that handle similar requests. |
| Preview each case from | Where each case starts: Initial interaction event, Last follow-up interaction event, or Current state of case. The options depend on the workflow’s case type. Most tests use the initial interaction. |
| Version of this workflow to test | Published tests the live workflow. Draft tests your unpublished changes. |
If you see Upload plain text cases (optional), you can upload a CSV of written examples instead of choosing case IDs.
Read the results
| Column | What it means |
|---|---|
| Accuracy for this workflow | The share of cases the workflow got right. See How accuracy is calculated. |
| Correctly selected (TP) | True positives: the workflow was selected, as expected. Higher is better. |
| Correctly missed (TN) | True negatives: the workflow wasn’t selected, as expected. Higher is better. |
| Incorrectly selected (FP) | False positives: the workflow was selected for a case it shouldn’t handle. These show the workflow selecting too broadly. |
| Incorrectly missed (FN) | False negatives: the workflow wasn’t selected for a case it should handle. These are missed opportunities. |
| Errors | Cases that couldn’t be tested. |
| Test group size | The number of cases tested. |
| Preview cases from, Other workflows included in test | The settings the test used. |
In View results, each case shows its Expected outcome, its Actual outcome (the workflow that was selected, or No workflow was selected), and an Outcome: Correct, Incorrectly selected, Correctly missed, Incorrectly missed, or Error. Filter by Outcome to review the cases the workflow got wrong.
Improve the workflow from the results
| Result | What it suggests | What to do |
|---|---|---|
| Many incorrectly selected | The trigger is too broad. | Add specific exclusions to the selection criteria, using the wrongly selected cases as examples. |
| Many incorrectly missed | The trigger is too narrow. | Broaden the Trigger conditions, with the missed cases as examples. |
| Low accuracy overall | The workflow’s design may not fit the requests. | Rethink the trigger, or split the workflow into two. Check that test cases are assigned correctly. |
| Errors | Some cases couldn’t run. | Open them in View results to see what happened. |
If you see Get AI suggestions in the trigger’s Selection criteria, select it and describe the problem, such as “This workflow is selecting cases about shipping delays, but should only handle order cancellations.” Review the suggested additions and deletions, apply the ones you agree with, and run the test again.
To improve a workflow step by step:
- Start with a small, high-confidence set of 10–15 cases and get it to good accuracy.
- Look for patterns in the cases the workflow got wrong.
- Change the selection criteria in a draft.
- Test the Draft version against the same cases, and compare the results with the Published version.
- Publish when the draft does better, then grow the test set.
How accuracy is calculated
Accuracy = (correctly selected + correctly missed) ÷ (correctly selected + correctly missed + incorrectly selected + incorrectly missed)
Cases with errors aren’t counted. Accuracy shows how often the workflow made the right call. It doesn’t say whether the replies were good.
For example: a test of 50 cases has 18 correctly selected, 25 correctly missed, 3 incorrectly selected, and 4 incorrectly missed. Accuracy = (18 + 25) ÷ 50 = 86%.
Precision and recall
Two more measures help you tell over-selecting from under-selecting. They aren’t shown on the page, but you can work them out from the counts:
- Precision = correctly selected ÷ (correctly selected + incorrectly selected): when the workflow is selected, how often it’s right. In the example, 18 ÷ 21 = 86%.
- Recall = correctly selected ÷ (correctly selected + incorrectly missed): of the cases the workflow should handle, how many it catches. In the example, 18 ÷ 22 = 82%.
The right target depends on the workflow and what a wrong selection costs. Many teams aim for more than 80% before they increase automation.