Review Inputs

You can improve the accuracy of your intent classifier model by reviewing inputs for which one or more possible class triggers have been identified. In Teneo, you can screen through these inputs one by one, and decide whether you want to confirm one of the class triggers that have been identified for this user input or whether you want to re-assign the input to a different class trigger instead. The results of this manual reviewing process will take effect immediately after the model has been automatically re-trained and increase the class trigger's accurracies.

In the following, we will have a closer look at two cases:

  1. A single class trigger was indentified for a given input.
  2. Two or more class triggers were identified for a given input.

Prerequisites

Before you can start reviewing, make sure you have:

  • created a solution with several different class triggers and flows
  • published this solution
  • and created log data via this published solution. Note that chatting with your bot in Try out is not stored in the log data source. Hence you have to chat via the published solution.

Case 1: A single class trigger was identified

All user inputs for which Teneo identified only one single possible class trigger are collected in the Detect and Fix section of the Optimisation area. The focus of this section is on confirming correct classifications and correcting miss-classifications.

Detect and Fix panel

A total of four sub-categories may appear in this section:

  • Fix - input is a negative example
  • Fix - input is training data of a different class
  • Review - system is not very confident
  • Review - system is very confident

The two 'Fix' sections show critical examples where the system assumes a high error probability due to not sufficiently structured training data. The 'Review' sections simply display all user inputs that have been assigned to class triggers with different levels of confidence. By confirming or re-assigning user inputs to class triggers, their confidence levels are enhanced after the next model training. In the following, we will take a closer look at both of these sections.

This video shows an example of how to review user inputs for which the system was not very confident. Please make sure to adjust your loudspeakers accordingly before starting the video.

Case 2: Two or more class triggers were identified

Sometimes, the classifier identifies more than one possible class trigger for a user input and might not be able to definitely assign the user input, due to similar confidence levels for the class trigger candidates. These are the kinds of inputs of which we take care in the 'Assign when uncertain' tab of the classifier optimization. By manually reviewing these user inputs and assigning them to the most appropriate class trigger, you help enhance the performance of Teneo's intent classifier. It has an immediate effect on the next model training where the confidences will be adjusted taking the new user inputs into account.

Assign when uncertain panel

Up to four sub-categories may appear in this section:

  • Fix - input is a negative example of at least one of the class triggers that have been identified for this user input.
  • Review - input is training data of an annotated class means that the user input is among the positive learning examples of one class trigger, but the intent classifier was more confident assigning it to a different class trigger.
  • Choose - input is training data of a different class means that the class trigger in whose training data the user input occurred is not among the class triggers that the intent classifier has identified for this user input.
  • Choose - input is not training data of any class trigger in this solution.

The following video shows how to choose the correct class trigger for inputs that have not occurred in any other trigger's training data. Please make sure to adjust your loudspeakers accordingly before playing the video.

Further Reviewing options

In the videos of the two previous sections, we have shown you how you can review your inputs. However, there are several different ways how you can perform the reviewing and we will tell you more about them here.

For each input you can perform one of the following actions:

  1. Assign to current class trigger → adds the user input to the learning examples of that trigger in future model trainings.
  2. Assign to different class trigger → the user input will be used as an additional learning example of the new class trigger in future model trainings.
  3. Acknowledge → the input will dissapear from the review list but not further action is taken.

The first two actions are illustrated in the screeshot below. The first input, "I want a coffee mug", would make a good additional example input for the "Can I have a coffee mug" class trigger. We should thus assign it using the "Assign" button. The second input, "A cup of coffee please" should be assigned to a different class trigger instead, namely "Can I order a coffee?". This can be done by clicking the "Fix" button and then dragging the input to a more appropriate class trigger from the list of all class triggers of our solution, which are shown in the window on the right hand side.

Reviewing inputs menu

Both of these actions and some more can also be accomplished using the menu that appears via right-click on the input:

Right mouse click review menu

Using this menu, you may perform the following actions:

  • send the input to Tryout, where it can be verified which class trigger it currently triggers.
  • mark the input as reviewed (makes it dissapear from the review list but no further action is taken).
  • mark a whole section as reviewed.
  • confirm the input and assign it to the current class trigger (can also be achieved by clicking the "Assign" button).
  • assign the input to a different class trigger (can also be achieved by clicking the "Fix" button).
  • clear the action for the current input.
  • clear the action for all inputs of a section.

The reviewing options shown in this section are available for reviewing both, user inputs for which one single class trigger has been identified and user inputs for which several class triggers have been identified. We will have a closer look at both of these cases in the coming subsections.

Was this page helpful?