Auto-test results panel
Test results panel
The results from the Auto-test are displayed in the Auto-test page. If no test has been run earlier, the test results will be populated into the window once the user has started an Auto-test. If a test has been performed earlier, when opening the Auto-test page, the results from the last Auto-test are displayed.
For information on how to run an Auto-test, please look here.
The top area of the test results view includes a visual summary of the currently displayed test results (below Solution). The summary includes details about the number of tested Intent Triggers, Transitions and Links as well as a percentage for the pass rate.
Selecting any of the examples in the results view will populate the right-side panel with information related to the selected example and its Intent Trigger.
In case of a failed test, the Intent Trigger which stole the example (blocking the expected result) will have a red line on its left side and is displayed in the top under Triggered. The test example will be displayed next in the section Expected Result - Should trigger.
For both failed tests and tests with warning, an All Matches section is available displaying information about all the Intent Triggers which matched the input.
In the Action panel, the user can double-click any of the Intent Triggers to open the Flow. Furthermore, three icons are available above the All Matches section which allows the user to open the Flow, Trigger Ordering or obtain further information by clicking the Info button.
The filters are available below the test result summary.
Test results can be filtered by using the following options:
- Text filter (applies to Flow names, examples, and the test result message)
- Filter buttons to show/hide the following test items:
- Succeeded (green checkmark)
- Passed with warning (yellow triangle)
- Failed (red warning)
- Non-testable items (clipboard)
When many test results are available, the list of successful results will automatically be hidden (the Succeeded button will automatically be disabled). This happens in the following two scenarios:
- There are more than 100 results and at least one is a failed result. In this case only failed results are displayed.
- There are more than 500 results, but none of them are a failed result.
In both of the above cases, the succeeded results can always be displayed by clicking the respective button.
Test passed (with warning)
When Positive examples pass the Auto-test with warning, it means that although the example matched the correct Intent Trigger and triggered it, there are one or more Intent Triggers in the same Order group which also match the example and potentially could trigger it.
In these cases, it is recommended to create order relations between the Intent Triggers for which the warning is displayed, because even though these currently do not cause the Auto-test to fail, an order relation will help avoid conflicts or Auto-test failures in the future.
The most common reasons for a failed test are:
- Match Requirements are not covering the positive examples
- Ordering problem (the example input is stolen by another Intent Trigger with a higher rank in Ordering).
Recognize issues related to Match Requirements
When the text The example was not matched is displayed next to the failed example, it indicates that the example is not matched by the Match Requirement to which it belongs and therefore was not triggered.
Another Intent Trigger, ranked below the current one, could still trigger the example (if covered by its Match Requirements), but as this was most likely not the intention, the user is encouraged to open the failed Intent Trigger and either review the example or update the Match Requirements.
Recognize issues related to Ordering
When one or more examples are Triggered by one or more different Flows, in most cases this is due to Ordering and can be solved by either changing the Order group of one of the Intent Triggers or by creating a relation between the two Intent Triggers.