Review Inputs

You can improve the accuracy of your intent classifier model by reviewing inputs for which one or more possible classes have been identified. In Teneo, you can screen through these inputs one by one, and decide whether you want to confirm one of the classes that have been identified for this user input or whether you want to re-assign the input to a different class instead.
The results of this manual reviewing process will take effect immediately after the model has been automatically re-trained and increase the classes accuracies.

In the following, we will have a closer look at two cases:

1. A single class was indentified for a given input.
2. Two or more classes were identified for a given input.

Prerequisites

Before you can start reviewing, make sure you have:

• Created a solution with several different classes and flows
• Published this solution
• Created log data via this published solution.

Note that chats with your bot in Tryout is not stored in the Log Data Source. Hence you have to chat via the published solution.

Detect and Fix

All user inputs for which Teneo identified only one single possible class are collected in the Detect and Fix section of the Optimisation area. The focus of this section is on confirming correct classifications and correcting miss-classifications.

A total of four sub-categories may appear in this section:

• Fix - input is a negative example
• Fix - input is training data of a different class
• Review - system is not very confident
• Review - system is very confident

The two 'Fix' sections show critical examples where the system assumes a high error probability due to not sufficiently structured training data. The 'Review' sections simply display all user inputs that have been assigned to classes with different levels of confidence. By confirming or re-assigning user inputs to classes, their confidence levels are enhanced after the next model training.

Consider the following example of how to review user inputs for which the system was not very confident. Here, we are taking a look at cases where the system is not confident about classification.

In this view we can see the relevant classes and the inputs that have matched to triggers on those classes. We can also see the low confidences that have led to these inputs ending up here in the Review section.

For the TELL_ME_ABOUT_YOUR_COFFEE class, we can see several inputs that were correctly classified. To assign them to this class, we can simply click on the Assign button:

For the ANOTHER_ONE_PLEASE class, we have one incorrectly classified input. To fix this, we can click on the Fix button, and then click and drag the input to whatever class we want.

Assign and Review

Sometimes, the classifier identifies more than one possible class for a user input and might not be able to definitely assign the user input, due to similar confidence levels for the class candidates. These are the kinds of inputs of which we take care in the 'Assign when uncertain' tab of the classifier optimization. By manually reviewing these user inputs and assigning them to the most appropriate class, you help enhance the performance of Teneo's intent classifier. It has an immediate effect on the next model training where the confidences will be adjusted taking the new user inputs into account.

Up to four sub-categories may appear in this section:

• Fix - input is a negative example of at least one of the class that have been identified for this user input.
• Review - input is training data of an annotated class means that the user input is among the positive learning examples of one class, but the intent classifier was more confident assigning it to a different class.
• Choose - input is training data of a different class means that the class in whose training data the user input occurred is not among the classes that the intent classifier has identified for this user input.
• Choose - input is not training data of any class in this solution.

Consider the following example on how to choose the correct class for inputs that have not occurred in any other trigger's training data. Under Assign when uncertain, we can find inputs to review:

For the the first input, we can see the system has found two possible classes. In this case, we can see that the input "can you show me a list of coffees" belongs to the class TELL_ME_ABOUT_YOUR_COFFEE. All we have to do to assign it is click on the Assign button.

Further Reviewing options

The full list of review options, including the Assign and Fix options we discussed in the examples above, can be seen in the right-click menu for each input:

Using this menu, you can perform the following actions:

• Send the input to Tryout, where it can be verified which class it currently triggers.
• Mark the input as reviewed (makes it disappear from the review list but no further action is taken).
• Mark a whole section as reviewed.
• Confirm the input and assign it to the current class (can also be achieved by clicking the "Assign" button).
• Assign the input to a different class (can also be achieved by clicking the "Fix" button).
• Clear the action for the current input.
• Clear the action for all inputs of a section.

The reviewing options shown in this section are available for reviewing both user inputs for which one single class has been identified and user inputs for which several classes have been identified.