Teneo Developers

Korean Input Processors Chain

Introduction

An Input Processor (IP) pre-processes inputs for the Teneo Engine to be able to perform different processes on them, such as normalization and tokenize inputs or do Part-of-Speech (POS) tagging. Each language that the Teneo Platform supports have a chain of Input Processors that know how to process that language. This page details the Input Processors chain for Korean language.

Input Processors chain setup

The following graph displays the default setup of the Korean Input Processor Chain:

graph TD subgraph ips [ ] splitting[Standard Splitting] --> morphological[Korean Morphological Analyzer] morphological[Korean Morphological Analyzer] --> annotation[System Annotation] annotation[System Annotation] --> number[Basic Number Recognizer] number[Basic Number Recognizer] --> languagedetect[Language Detector] languagedetect[Language Detector] --> predict[Predict] end input([User Input]) --User gives input--> splitting predict[Predict] --Parsed input--> parsed([To Dialog Processing])

The standard Input Processors are listed below with a short description of the IP's functionality:

  • The Standard Splitting IP divides the user input text into sentences and words, considering abbreviations that should not be split.
  • The Korean Morphological Analyzer IP
  • The System Annotation IP sets a number of annotations based on properties of the user input text.
  • The Basic Number Recognizer IP identifies all Arabic numbers of the type 123 and 3.14 in the user input, annotates each of them with the NUMBER annotation and associates a variable to this annotation called numericValue.
  • The Language Detector IP identifies the language of the input sentence provided and annotates it with the predicted language and associates the confidence score of the prediction.
  • The Predict IP classifies user input based on a machine learning model trained in Teneo Learn and annotates the user input with the predicted top intent classes and confidence score.

Korean Simplifier

The Korean Simplifier is a special kind of processor that is used to normalize the user input by:

  • converting full width Latin letters and Arabic digits into their half width version, and
  • lowercasing the uppercase Latin letters.

This Simplifier is special because it is not run as a part of the input processor chain, but rather by the Tokenizer when it puts the tokens into a Teneo data structure. Additionally, the Simplifier is also run by the condition parser inside Teneo Engine, which normalizes the Language Object condition words before adding them to the internal Engine dictionary.

Standard Splitting IP

The Standard Splitting Input Processor splits the user input text into sentences and words. Splitting is performed at configurable sentence and word delimiters. Splitting exceptions can be defined as a configurable list of abbreviations and a configurable regular expression.

This Input Processor generates one or more sentences, with zero or more words. The generated WordData objects contain the original and simplified form of the word. The final word-form is initialized with the simplified word form.

Other considerations

Extra request parameters read by this input processor: (none)
Processing options read by this input processor: (none)
Annotations generated by this input processor: (none)

Configuration properties

Defining abbreviations

NameTypeRequiredData
abbreviations.item.*Format:abbreviations.item.<n> = <abbreviation><n>: number, which must be unique within the abbreviation definitions of one file<abbreviation>: an abbreviationnonone

List of abbreviations. Abbreviations are considered in the sentence separation process. Sentence delimiters within abbreviations will not lead to separated sentences.

NameTypeRequiredDefault
abbreviations.file.namestring (filename)noempty

Filename (including path) of an extra file containing abbreviations. A relative filename relates to the location of the properties file.

NameTypeRequiredDefault
abbreviations.file.encodingstring (encoding name)noUTF-8

Encoding of the extra file containing abbreviations.

Controlling user input separation into sentences and words

NameTypeRequiredDefault
inputSeparation.sentenceDelimitersstringno. ¡ ! ¿ ? …

List of characters that are used to separate sentences (unless part of an abbreviation).

NameTypeRequiredDefault
inputSeparation.wordDelimitersstringno$ € £ % & ^ " “ ” # | ~ § ° [] () < > {} = + - ÷ \\ * / , : ; \r \n \t • . ¡ ! ¿ ? …

List of characters that are used to separate words. Delimiting characters will be kept as separate words, except for those that are listed under inputSeparation.nonWordCharacters (see below).

inputSeparation.additionalWordDelimiterRegEx may be used to specify additional or alternative word delimiting.

NameTypeRequiredDefault
inputSeparation.additionalWordDelimiterRegExstringnoempty

Additional word delimiting regular expression. This is an optional regular expression for delimiting words or defining (optionally zero width) word boundaries. It may be specified as addition or alternative to inputSeparation.wordDelimiters.

NOTE: in Java 6 & 7 a 'position look behind' construct in the regex does not work with Unicode blocks outside the BMP if the block is specified with \p{ln...} construct, probably due to a bug in the Java regex implementation. Instead the characters must be specified directly as a range.

NameTypeRequiredDefault
inputSeparation.nonWordCharactersstringno" “ ” . ¡ ! ¿ ? … , ; \t \r \n

Word separators that shall not be kept as words. The set of characters specified here should be a subset of inputSeparation.wordDelimiters and the characters matched by inputSeparation.additionalWordDelimiterRegEx.

Example (assuming defaults):

Argh$%, separate this!

will be separated into:

Argh
$
%
separate
this

NameTypeRequiredDefault
inputSeparation.excludeWordDelimitersRegExstringno(?<=([<SP><HT><CR><LF> "“”,;.¡!¿?…\d]|^))[.,](?=\d)

Regular expressions that specify exceptions to the splitting of a sentence into words.

The default regular expression prevents the characters , (comma) and . (dot) from acting as word delimiter when they appear in the context of a number.

Note: the text matched by the regular expression will be excluded from splitting, thus any word splitting characters used only as context condition should be given as zero-width look-behind/look-ahead construct.

Korean Morphological Analyzer IP

The Korean Morphological Analyzer input processor runs Komoran on every sentence from the user input, as provided by the Standard Splitting Input Processor before it. Komoran returns the root of every word in the sentence as well as a tag that contains both Part-of-Speech and morphological information. The Korean Morphological Analyzer Input Processor then converts into Teneo annotations the root, the Part-of-Speech and the morphological information for every word.

The table below lists how the tags from Komoran are mapped to annotations in Teneo:

Komoran tagDescriptionMap to Teneo annotation(s)
VAAdjectiveADJ.POS
JKGAdnominal case markerJKG.MST
MAGAdverbADV.POS
JKBAdverbial case markerJKB.MST
VCPAffirmation/positiveVCP.MST
NAAnalytical CategoryNA.MST
JXAuxiliary postpositional particleJX.MST
VXAuxiliary predicate elementVX.MST
NNBBound nounNN.POS , NNB.MST
SNCardinal numberCARDINAL.POS
JKCComplement case markerJKC.MST
JCConjunctive postpositional particleJC.MST
MAJConnective adverbADV.POS, CONNECTIVE.MST
ECConnective endingEC.MST
VCNDenial/negativeVCN.MST
MMDeterminerDET.POS
ETEnding of a wordET.MST
ETMEnding of a wordETM.MST
SLForeign wordFOREIGN.POS
SHForeign wordFOREIGN.POS
ICInterjectionINTERJ.POS
NNGNounNN.POS
NFNoun estimation categoryNF.MST
NRNumeralNR.MST
JKOObject case markerJKO.MST
JKQPostposition, postpositional particleJKQ.MST
EPPre-final endingEP.MST
XPNPrefixXPN.MST
NPPronounPRON.POS
NNPProper NounNN.POS , PROPER.POS
SFPunctuationPUNCT.POS
SPPunctuationPUNCT.POS
SSPunctuationPUNCT.POS
SEPunctuationPUNCT.POS
SOPunctuationPUNCT.POS
XRRootXR.MST
EFSentence-closing endingEF.MST
JKSSubject case markerJKS.MST
XSNSuffixXSN.MST
XSVSuffixXSV.MST
XSASuffixXSA.MST
SWSymbolSYM.POS
VVVerbVB.POS
NVVerb estimation categoryNV.MST
JKVVocative case markerJKV.MST

System Annotation IP

The System Annotation Input Processor performs simple analysis of the sentence texts to set some annotations. The decision algorithms are configurable by various properties. Further customization is possible by sub-classing this Input Processor and overriding one or more of the methods: decideBinary, decideBrackets, decideEmpty, decideExclamation, decideNonsense, decideQuestion, decideQuote.

This IP works on the sentences passed in but does not modify them.

Other considerations

Extra request parameters read by this input processor: (none)
Processing options read by this input processor: (none)
Annotations this input processor may generate:

  • _EMPTY: the sentence text is empty
  • _EXCLAMATION: the sentence text contains at least one of the characters specified with property exclamationMarkCharacters
  • _EM3: the sentence text contains three or more characters in a row of the characters specified with property exclamationMarkCharacters
  • _QUESTION: the sentence text contains at least one of the characters specified with property questionMarkCharacters
  • _QT3: the sentence text contains three or more characters in a row of the characters specified with questionMarkCharacters
  • _QUOTE: the sentence text contains at least one of the characters specified with property quoteCharacters
  • _DBLQUOTE: the sentence text contains at least one of the characters specified with property doubleQuoteCharacters
  • _BRACKETPAIR: the sentence text contains at least one matching pair of the bracket characters specified with property bracketPairCharacters
  • _NONSENSE: the sentence probably contains nonsense text as configured with properties consonants, nonsenseThreshold.absolute and nonsenseThreshold.relative
  • _BINARY: the sentence text only contains characters specified by properties binaryCharacters (at least one of them) and binaryIgnoredCharacters (zero or more of them).

Configuration properties

NameTypeRequiredDefault
consonantsstringnoBCDFGHJKLMNPQRSTVWXZ bcdfghjklmnpqrsßtvwxz ㄱㄲㄴㄷㄸㄹㅁㅂㅃㅅㅆㅇㅈㅉㅊㅋㅌㅍㅎ

Contains all letters (upper and lower case) that are considered consonants in the language. Together with the properties nonsenseThreshold.absolute and nonsenseThreshold.relative these will be used for detecting probable nonsense inputs like kljljljljjlj.

NameTypeRequiredDefault
nonsenseThreshold.absolutePositive integer numberNo6

For nonsense detection an input exclusively consisting of so many consonants without any non-consonants is considered nonsense.

NameTypeRequiredDefault
nonsenseThreshold.relativePositive integer numberno10

For nonsense detection an input containing so many consonants in a row is considered nonsense.

NameTypeRequiredDefault
exclamationMarkCharactersstringno!!

List of characters of which at least one must occur in the sentence text to set annotations _EXCLAMATION and _EM3 (in case of a sequence of at least three of the specified characters).

NameTypeRequiredDefault
questionMarkCharactersstringno??

List of characters of which at least one must occur in the sentence text to set annotations _QUESTION and _QT3 (in case of a sequence of at least three of the specified characters).

NameTypeRequiredDefault
doubleQuoteCharactersstringno"“”"『』

List of characters of which at least one must occur in the sentence text to set annotation _DBLQUOTE.

NameTypeRequiredDefault
quoteCharactersstringno''「」

List of characters of which at least one must occur in the sentence text to set annotation _QUOTE.

NameTypeRequiredDefault
binaryCharactersstringno01

List of characters recognized in the sentence text to set annotation _BINARY.

NameTypeRequiredDefault
binaryIgnoredCharactersstringno!?,.-;:# \r\n\t\"'

List of characters additionally allowed in binary text.

NameTypeRequiredDefault
bracketPairCharactersstringno()[]{}()〔〕[]{}〈〉《》〚〛〘〙〖〗【】⦅⦆

List of pairs of bracketing characters of which at least one pair (opening and closing bracket of the same type) must occur in the sentence text to set annotation _BRACKETPAIR.

Special System annotations

Two special annotations related not to individual inputs, but to whole dialogues, are added by the Teneo Engine itself:

  • _INIT: indicates session start, i.e. the first input in a dialogue
  • _TIMEOUT: indicates the continuation of a previously timed-out session/dialogue.

Basic Number Recognizer IP

The Basic Number Recognizer Input Processor identifies all Arabic numbers of the type 123 and 3.14 in the user input and annotates each of them with the NUMBER annotation and associates a variable to this annotation called numericValuewhich holds the numeric value of the number found.

This Input Processor is language independent, but every language has its own configuration file for this IP defining decimal point characters and the thousands separator character to be ignored.

For the NUMBER annotation and the variable to be added, a "number" in the user input must meet the following conditions:

It must match the regular expression:

[,]?[0-9]+([,][0-9]+)*([.][0-9]+)?|[.][0-9]+

It must be parseable by Java's BigDecimal to ensure it is a number

The above condition provides the following guarantees:

  • The sign is not included in the annotated token
  • The numericValue variable contains a BigDecimal representation of the number.

The decimal marker(s) and the thousands separator(s) can be configured; in the above regex, the dot is used as a decimal marker and the comma as a regular expression.

Configuration properties

NameDefault
decimalMarkers..

The default decimal markers in Korean.

NameDefault
charactersToIgnore,,

The default characters to ignore.

Language Detector IP

The Language Detector Input Processor uses a machine learning model that predicts the language of a given input and adds an annotation of the format %${language label}.LANG to the input as well as a confidence score of the prediction.

Language Detector annotation

The Language Detector IP can predict the following 45 languages (language label in brackets):

Arabic (AR), Bulgarian (BG), Bengali (BN), Catalan (CA), Czech (CS), Danish (DA), German (DE), Greek (EL), English (EN), Esperanto (EO), Spanish (ES), Estonian (ET), Basque (EU), Persian (FA), Finnish (FI), French (FR), Hebrew (HE), Hindi (HI), Hungarian (HU), Indonesian-Malay (ID_MS), Icelandic (IS), Italian (IT), Japanese (JA), Korean (KO), Lithuanian (LT), Latvian (LV), Macedonian (MK), Dutch (NL), Norwegian (NO), Polish (PL), Portuguese (PT), Romanian (RO), Russian (RU), Slovak (SK), Slovenian (SL), Serbian-Croatian-Bosnian (SR_HR), Swedish (SV), Tamil (TA), Telugu (TE), Thai (TH), Tagalog (TL), Turkish (TR), Urdu (UR), Vietnamese (VI) and Chinese (ZH).

Serbian, Bosnian and Croatian are treated as one language, under the label SR_HR and Indonesian and Malay are treated as one language, under the label ID_MS.

A number of regexes are also in use by the Input Processor, helping the model to not predict language for fully numerical inputs, URLs or other type of nonsense inputs.

The Language Detector will provide an annotation when the confidence prediction threshold is above 0.2 for the languages, but for Arabic (AR), Bengali (BN), Greek (EL), Hebrew (HE), Hindi (HI), Japanese (JA), Korean (KO), Tamil (TA), Telugu (TE), Thai (TH), Chinese (ZH), Vietnamese (VI), Persian (FA) and Urdu (UR) language annotations will always be created, even for predictions below 2.0, since the Language Detector is mostly accurate when predicting them.

The Predict IP

The Predict Input Processor makes use of a machine learning model generated in the Teneo Learn component when machine learning classes are available in a Teneo Studio solution. The Predict IP uses the model to annotate each user input with the machine learning classes defined.

Whenever the Predict IP receives a user input, the Input Processor calculates a confidence score for each of the classes based on the model, creating annotations for the most confident class and for each other class that matches the following criteria:

  • the confidence is above the minimum confidence (defaults to 0.01)
  • the confidence is higher than 0.5 times the confidence value of the top class.

The Predict Input Processor will create a maximum of 5 annotations, regardless of how many classes match the criteria. The numerical thresholds can be configured in the properties file of the Input Processor.

Annotations created by Predict

For each selected class, an annotation with the name <CLASS_NAME>.INTENT will be created, with the value of the model confidence in the class. A special annotation <CLASS_NAME>.TOP_INTENT is created for the class with the highest confidence score.

Configuration properties

NameTypeRequiredDefault
minConfidenceSimilarityDistancefloatno0.5

Confidence percentage of the top score confidence a class must have in order to be considered, e.g. if the top confidence class has a confidence of 0.7, classes with confidence lower than 0.5 x 0.7 = 0.35 will be discarded.

NameTypeRequiredDefault
maxNumberOfAnnotationsintno5

Maximum number of class annotations to create for each user input.

NameTypeRequiredDefault
minConfidenceThresholdfloatno0.01

Minimum value of confidence a model must have for a class in order to add it as one of the candidate annotations.

NameTypeRequiredDefault
intent.model.file.namestring (filename)noinexistent

Name of the file containing the machine learning model. It is usually set automatically by Teneo Studio, so no configuration is required.

Custom Input Processor configuration