Teneo Developers

Review Inputs

You can improve the accuracy of your intent classifier model by reviewing inputs for which one or more possible classes have been identified. In Teneo, you can screen through these inputs one by one, and decide whether you want to confirm one of the classes that have been identified for this user input or whether you want to re-assign the input to a different class instead.
The results of this manual reviewing process will take effect immediately after the model has been automatically re-trained and increase the classes accuracies.

In the following, we will have a closer look at two cases:

  1. A single class was indentified for a given input.
  2. Two or more classes were identified for a given input.

Prerequisites

Before you can start reviewing, make sure you have:

  • Created a solution with several different classes and flows
  • Published this solution
  • Created log data via this published solution.

Note that chatting with your bot in Try out is not stored in the log data source. Hence you have to chat via the published solution.

Case 1: A single class was identified

All user inputs for which Teneo identified only one single possible class are collected in the Detect and Fix section of the Optimisation area. The focus of this section is on confirming correct classifications and correcting miss-classifications.

Optimize - Review inputs: ClassifierDetectAndFixResized

A total of four sub-categories may appear in this section:

  • Fix - input is a negative example
  • Fix - input is training data of a different class
  • Review - system is not very confident
  • Review - system is very confident

The two 'Fix' sections show critical examples where the system assumes a high error probability due to not sufficiently structured training data. The 'Review' sections simply display all user inputs that have been assigned to classes with different levels of confidence. By confirming or re-assigning user inputs to classes, their confidence levels are enhanced after the next model training. In the following, we will take a closer look at both of these sections.

This video shows an example of how to review user inputs for which the system was not very confident. Please make sure to adjust your loudspeakers accordingly before starting the video.

Case 2: Two or more classes were identified

Sometimes, the classifier identifies more than one possible class for a user input and might not be able to definitely assign the user input, due to similar confidence levels for the class candidates. These are the kinds of inputs of which we take care in the 'Assign when uncertain' tab of the classifier optimization. By manually reviewing these user inputs and assigning them to the most appropriate class, you help enhance the performance of Teneo's intent classifier. It has an immediate effect on the next model training where the confidences will be adjusted taking the new user inputs into account.

Optimize - Review inputs: ClassifierAssignWhenUncertainResized

Up to four sub-categories may appear in this section:

  • Fix - input is a negative example of at least one of the class that have been identified for this user input.
  • Review - input is training data of an annotated class means that the user input is among the positive learning examples of one class, but the intent classifier was more confident assigning it to a different class.
  • Choose - input is training data of a different class means that the class in whose training data the user input occurred is not among the classes that the intent classifier has identified for this user input.
  • Choose - input is not training data of any class in this solution.

The following video shows how to choose the correct class for inputs that have not occurred in any other trigger's training data. Please make sure to adjust your loudspeakers accordingly before playing the video.

Further Reviewing options

In the videos of the two previous sections, we have shown you how you can review your inputs. However, there are several different ways how you can perform the reviewing and we will tell you more about them here.

For each input you can perform one of the following actions:

  1. Assign to current class → adds the user input to the learning examples of that trigger in future model trainings.
  2. Assign to different class → the user input will be used as an additional learning example of the new class in future model trainings.
  3. Acknowledge → the input will dissapear from the review list but not further action is taken.

The first two actions are illustrated in the screenshot below. The first input, "I want a coffee mug", would make a good additional example input for the "Can I have a coffee mug" class. We should thus assign it using the "Assign" button. The second input, "A cup of coffee please" should be assigned to a different class instead, namely "Can I order a coffee?". This can be done by clicking the "Fix" button and then dragging the input to a more appropriate class from the list of all classes of our solution, which are shown in the window on the right-hand side.

Optimize - Review inputs: reviewScreenResized

Both of these actions and some more can also be accomplished using the menu that appears via right-click on the input:

Optimize - Review inputs: RightMouseMenu

Using this menu, you may perform the following actions:

  • Send the input to Tryout, where it can be verified which class it currently triggers.
  • Mark the input as reviewed (makes it dissapear from the review list but no further action is taken).
  • Mark a whole section as reviewed.
  • Confirm the input and assign it to the current class (can also be achieved by clicking the "Assign" button).
  • Assign the input to a different class (can also be achieved by clicking the "Fix" button).
  • Clear the action for the current input.
  • Clear the action for all inputs of a section.

The reviewing options shown in this section are available for reviewing both, user inputs for which one single class has been identified and user inputs for which several classes have been identified. We will have a closer look at both of these cases in the coming subsections.