Standard Input Processors Chain

Introduction

An Input Processor (IP) pre-processes inputs for the Teneo Engine to be able to perform different processes on them, such as normalization and tokenization of inputs or spelling correction. Each language supported by the Teneo Platform has a chain of Input Processors which know how to process that particular language. The Standard Input Processor chain offers support to a a large number of the supported Teneo languages.

Supported Languages

Currently, the below listed languages are supported by the Standard Input Processor chain.

Supported languages
Afrikaans	Czech	Georgian	Kinyarwanda	Nepali	Sango	Tigrinya
Albanian	Danish	German	Kirundi (Rundi)	Norwegian (Nynorsk/Bokmål)	Scottish Gaelic	Tsonga
Amharic	Dutch	Greek	Kyrgyz	Odia	Serbian	Tswana (Setswana)
Armenian	English	Gujarati	Latvian	Oromo	Shona	Turkmen
Azerbaijani	Esperanto	Hindi	Lithuanian	Papiamento	Sinhala	Ukrainian
Basque	Estonian	Hungarian	Luxembourgish	Polish	Slovak	Uzbek
Belarusian	Ewe	Icelandic	Macedonian	Portuguese	Slovene	Vietnamese
Bengali/Bangla	Faroese	Igbo	Malagasy	Quechuan (Quechua)	Somali	Welsh
Bosnian	Finnish*	Indonesian	Malay	Romanian	Spanish	Yoruba
Bulgarian	French	Irish	Maltese	Romansh	Swahili (Kiswahili)	Zulu (isiZulu)
Catalan	Frisian	Italian	Marathi	Russian	Swazi
Croatian	Galician	Kazakh	Mongolian	Sámi	Swedish

* The Input Processor chain for Finnish language also contains the Finnish Splitting Input Processor on top of the IPs in the Standard Input Processor chain.

IP Chain Setup

The following graph displays the setup of the Standard Input Processors chain; each Input Processor is described further in the following sections.

graph TD subgraph ips [ ] split[Standard Splitting] autocorrect[Standard Auto Correction] predict[Predict] similarity[Standard Similarity Match Correction] annotation[System Annotation] number[Basic Number Recognizer] datetime[DateTime Recognizer *] languagedetect[Language Detector] pos[POS Tagger / Morphological Analyzer *] ner[Named Entity Recognizer *] end input([User Input]) subgraph settings [Input Processor Configuration] abbr[/Abbreviations/] correct[/Autocorrections/] end subgraph solution [Solution] soln[/Solution Dictionary/] end parsed([To Dialog Processing]) split --> autocorrect autocorrect --> predict predict --> similarity similarity --> annotation annotation --> number number --> datetime datetime --> languagedetect languagedetect --> pos pos --> ner input -->|User Gives Input| split abbr --> split correct --> autocorrect soln --> similarity ner --Parsed Input--> parsed classDef ip_optional stroke-dasharray:5,5; class datetime,pos,ner ip_optional; classDef external fill:#00000000,stroke-dasharray:5,5; class solution,settings external;

* The Input Processors marked with a star (*) in the above graph are currently only available as NL Analyzers for a selection of the languages; for more information on available languages, please see the specific sections.

Standard Simplifier

The Standard Simplifier is a separate processing unit which is not an input processor and which provides a method to normalize some text, usually - but not necessarily - a word. Here, "normalization" means removal of text properties that are semantically insignificant, like conversion to lower case (considering the configured language locale), removal of some accents and normalization of Unicode combining characters. By default, the Input Processors call the Simplifier when they generate a new word item. Furthermore, the Simplifier is called by the language condition parser of the Teneo Engine when it stores a language condition word (i.e., TLML syntax word) in the solution dictionary.

Simplification
The Simplifier decompose and normalize characters, for example lower casing characters and converting to Unicode.

Input Processors

Standard Splitting

The Standard Splitting Input Processor splits the user input text into sentences and words; this Input Processor generates one or more sentences with zero or more words. The generated WordData objects contain the original and the simplified form of the word; the final word form is initialized with the simplified word form.

Standard Auto Correction

The Standard Auto Correction Input Processor applies spelling correction based on a list of auto-correction mappings. The corrections are applied to the finalized form of the sentence words, and the Input Processor works on the existing sentences and words passed in. It may modify the final form of the word, but the count of sentences and words is not modified.

Automatic spelling correction

Predict

The Predict Input Processor makes use of an intent model generated when classes are available in a Teneo Studio solution to annotate user inputs with the defined classes; intent models can be generated either with Teneo Learn or CLU. Note that as of Teneo 7.3, deferred intent classification is applied and annotations are only created by Predict if references to class annotations are found during the input matching process.

When Predict receives a user input, confidence scores are calculated for each class based on the model and annotations created for the most confident class and for each other class that matches the following criteria:

the confidence is above the minimum confidence (defaults to 0.01)
the confidence is higher than 0.5 times the confidence value of the top class.

For each selected class, an annotation with the scheme <CLASS_NAME>.INTENT is created, with the value of the model's confidence in the class as well as an annotation variable specifying the used classifier (i.e., Learn, CLU or LearnFallback) and an Order variable defining the order of the selected classes (i.e., 0 for the class with the highest confidence score and 4 for the selected class with the lowest confidence score).
A special annotation <CLASS_NAME>.TOP_INTENT is created for the class with the highest confidence score.

Annotation	Variable	Variable	Variable	Description
<CLASS_NAME>.TOP_INTENT	classifier	confidence		Annotation created for the class with the highest confidence score
<CLASS_NAME>.INTENT	classifier	confidence	Order	Annotation given to each selected class with a maximum of five top classes

The Predict Input Processor creates a maximum of 5 annotations, regardless of how many classes match the criteria.

Standard Similarity Match Correction

The Standard Similarity Match Correction Input Processor applies spelling correction based on a similarity matching of sentence words against words provided by a dictionary. The corrections are applied to the final form of the sentence words.
This Input Processors works on the existing sentences and words passed in and it may modify the final form of a word; the count of sentences and words is not modified.

The dictionary used by this Input Processor is the solution dictionary which is solution specific and generated on (re)load of the Engine. It is formed of the bare words in TLML syntaxes in the solution and in referenced libraries. Adding a word to the dictionary is done by adding the word to the syntax of a new or existing Language Object, Entity, Trigger or Transition (in case of the latter two, as TLML Syntax).

Spelling tolerance

System Annotation

Teneo bundles two default collections of annotations in all language configurations: standard annotations added by the System Annotation Input Processor and special system annotations added by the Engine; the System Annotation Input Processor performs simple analysis of the sentence texts and may generate the standard annotations listed below.

Annotation	Description
_BINARY	The input consists of only 0s and 1s
_BRACKETPAIR	At least one matching pair of brackets appears in the input; possible bracket types: ( ), [ ], { }
_EXCLAMATION	At least one exclamation mark (!) appears in the input
_EM3	Three (or more) exclamation marks (!!!) appear in a row in the input
_EMPTY	The input contains no text / the sentence text is empty
_NONSENSE	The input contains nonsense text, e.g., 'asdf', 'wgwwgwg', 'xxxxxx'
_QUESTION	At least one question mark (?) appears in the input
_QT3	Three (or more) question marks (???) appear in a row in the input
_QUOTE	At least one single quotation mark (') appears in the input
_DBLQUOTE	At least one quotation mark (") appears in the input

Special System Annotations

The following two, special annotations are set by the Teneo Engine. These special system annotations are not related to individual inputs but rather to whole dialogues and are dependent on the session state.

Annotation	Description
_INIT	Indicates session start, i.e., the first input in a dialogue
_TIMEOUT	Indicates the continuation of a previously timed-out session/dialogue

Basic Number Recognizer

The Basic Number Recognizer identifies all Arabic numbers of the type 123 and 3.14 in the user input and annotates each of them with an annotation associated with a variable which holds the actual numeric value of the number found.
The Basic Number Recognizer is language dependent and each language has its own configuration defining the decimal point characters and the thousands separator character to be ignored.

Annotation	Variable	Description
NUMBER	numericValue	Annotation created for identified Arabic numbers in user inputs

For the annotation and its numeric value variable to be added, a number in the user input must meet the following syntax:

It must match the regular expression:

[,]?[0-9]+([,][0-9]+)*([.][0-9]+)?|[.][0-9]+

It must be parseable by Java's BigDecimal to ensure it is a number

The above syntax provides the following guarantees:

The sign is not included in the annotated token
The numericValue variable contains a BigDecimal representation of the number.

In the above example regex, the dot is used as a decimal marker and the comma as a regular expression; as described earlier this configuration is language dependent and therefore varies depending on the selected solution language.

Language Detector

The Language Detector uses a machine learning model to predict the language of a given user input and adds an annotation, as seen in below table, to the input together with a confidence score of the prediction.

Annotation	Variable	Description
<language label>.LANG, e.g., %$DA.LANG	Confidence	Annotation created for the predicted language

The Language Detector can predict the following 45 languages; the language label used to create the annotation name is in brackets:

Arabic (AR), Bulgarian (BG), Bengali (BN), Catalan (CA), Czech (CS), Danish (DA), German (DE), Greek (EL), English (EN), Esperanto (EO), Spanish (ES), Estonian (ET), Basque (EU), Persian (FA), Finnish (FI), French (FR), Hebrew (HE), Hindi (HI), Hungarian (HU), Indonesian-Malay (ID_MS), Icelandic (IS), Italian (IT), Japanese (JA), Korean (KO), Lithuanian (LT), Latvian (LV), Macedonian (MK), Dutch (NL), Norwegian (NO), Polish (PL), Portuguese (PT), Romanian (RO), Russian (RU), Slovak (SK), Slovenian (SL), Serbian-Croatian-Bosnian (SR_HR), Swedish (SV), Tamil (TA), Telugu (TE), Thai (TH), Tagalog (TL), Turkish (TR), Urdu (UR), Vietnamese (VI) and Chinese (ZH).

Serbian, Bosnian and Croatian are treated as one language under the label SR_HR, and Indonesian and Malay are treated as one language under the label ID_MS

A number of regexes are also in use by the Input Processor, helping the model to not predict a language for fully numerical inputs, URLs or other type of nonsense inputs.

The Language Detector will provide an annotation when the confidence prediction threshold is above 0.2 for the languages, but for the following listed languages, language annotations are always created (even for predictions below 0.2) since the Language Detector is mostly accurate when predicting them: Arabic, Bengali, Greek, Hebrew, Hindi, Japanese, Korean, Tamil, Telugu, Thai, Chinese, Vietnamese, Persian and Urdu.

Finnish Input Processors Chain

The input processing chain for Finnish language shares its Input Processors with the Standard Input Processor chain, but furthermore includes the Finnish Splitting Input Processor which comes between the Standard Splitting and Standard AutoCorrect Input Processors as displayed in the below graph.

graph TD subgraph ips [ ] split[Standard Splitting] --> finnishsplit finnishsplit[Finnish Splitting] --> autocorrect autocorrect[Standard AutoCorrection] --> predict predict[Predict] --> similarity similarity[Standard Similarity Match Correction] --> annotation annotation[System Annotation] --> number number[Basic Number Recognizer] --> datetime datetime[DateTime Recognizer*] --> languagedetect end input([User Input]) -->|User Gives Input| split subgraph settings [Input Processor Configuration] abbr[/Abbreviations/] --> split correct[/Autocorrections/] --> autocorrect end subgraph solution [Solution] soln[/Solution Dictionary/] --> similarity end languagedetect[Language Detector] -->|Parsed Input| parsed([To Dialog Processing]) classDef ip_optional stroke-dasharray:5,5; class datetime,pos,ner ip_optional; classDef external fill:#00000000,stroke-dasharray:5,5; class solution,settings external;

* The DateTime Recognizer (in the above graph) is also available in the Finnish Input Processors chain, but it is currently not supported by the Approach in the Teneo Platform for understanding and interpretation of date and time expressions.

Finnish Splitting

The Finnish Splitting Input Processor splits off suffixes from the existing sentence words passed in, using configurable word lists in its algorithm. It may modify an existing word (it is set to the word stem) and add one or more words after it (the suffixes split off). These added words all have the same original word form and begin index as the modified word. Words shorter than five characters or contained in the no-cut list will not be split. The count of sentences is not modified.

The suffixes are grouped into five lists and are searched for and split off in the order listed below:

clitic
participe
poss
cases
comparison.