Teneo Developers

Teneo Sentiment and Intensity NL Analyzer

Introduction

The Teneo Sentiment and Intensity NL Analyzer is Artificial Solutions' proprietary resources containing off-the-shelf building blocks to be used for detecting positive and negative sentiment as well as intensity in user inputs.

The Teneo Sentiment and Intensity NL Analyzer is distributed as a set of Language Objects available in the Teneo NLU Ontology and Semantic Network of the languages where available.

Companies have a high interest in learning about the interactions their customers make via support channels, through tweets, emails or chat with conversational AI applications. To fulfil this need, companies turn to one of the many suppliers of sentiment analysis; the sentiment analysis solutions today take mainly a statistical approach as well as using syntax analyses to make their conclusion about the content. The statistical approach has its limitations and often does not consider other important aspects of the communication, such as intent, thematic errors or intensity of the communication. At Artificial Solutions this has been attempted to resolve by presenting clear components to answer the high linguistic demands placed on such analyses.

The objects in the Teneo Sentiment and Intensity NL Analyzer can be used in Teneo Studio solutions in Listeners, Triggers, or Transitions to keep track of various occurrences during the conversation and to support various actions:

  • Flag specific dialogue events,
  • influence both proactive and direct responses or actions, and
  • compute or track user sentiment.

Available languages

The Teneo Sentiment and Intensity NL Analyzer is currently available in the below listed languages.

Language
Chinese (Mandarin)German
DutchItalian
EnglishSwedish

Teneo Sentiment

The Teneo Sentiment NL Analyzer categorizes sentiment into negative and positive categories; the category of neutral sentiment is given by an absence of negative and positive sentiment. Additionally, the determination of sentiment is scaled into two categories:

  • high confidence (clear, specific statements)
  • low confidence (partial statements or keywords that can vary according to context).

The language rules used to determine sentiment apply the complete range of Teneo Linguistic Modeling Language as well as Part-of-Speech (POS) tags, to support high-performance precisions and are varied and numerous enough to achieve industrial-level recall.

Language Objects

The main objects to use to detect sentiment, when not interested in confidence levels, are:

Sentiment objectExample variants
SENTIMENT_POSITIVE.INDICATORI like you / It’s a lovely day / You are beautiful / You make a good point
SENTIMENT_NEGATIVE.INDICATORGo to hell / I don’t like you / Fuck you / This is the worst thing that ever happened
SENTIMENT_NEUTRAL.INDICATORI am going home / It is raining / Winter is coming / You look busy

Please note that a number of Language Objects are used internally by the resource to achieve a better structured approach to recognizing sentiment, but these objects are not relevant for external use and will therefore not be documented here.

Use of the objects is straightforward: if the object is true, the applicable sentiment was detected in the input. Note that it is also possible for an input to contain both negative and positive sentiment, so both objects might be true. Sentiment is defined as neutral if neither positive nor negative sentiment was detected.

For more complex sentiment detecting involving confidence levels, the following objects are available:

Sentiment objectType of input
SENTIMENT.INDICATORUsed to set NLU variables containing sentiments and confidence levels found
SENTIMENT_NEGATIVE_HIGH_CONFIDENCE.INDICATOR SENTIMENT_POSITIVE_HIGH_CONFIDENCE.INDICATORStatements, phrases
SENTIMENT_NEGATIVE_LOW_CONFIDENCE.INDICATOR SENTIMENT_POSITIVE_LOW_CONFIDENCE.INDICATOREmojis, hashtags, keywords

The sentiment objects do not detect strength or intensity of the sentiment, e.g. I hate you versus I really really hate you!! although the latter statement is clearly more extreme than the first. The degree of sentiment can be determined by combining the detected sentiment with a detection of intensity described in the Teneo Intensity section.

Detecting sentiment

Within a Teneo solution, sentiment can be detected via Listeners and used for various purposes, for example:

  • tagging specific transactions or sessions (for later analyses)
  • defining specific actions during the session
  • tracking changes in sentiment, and
  • computation of sentiment score for the entire conversation.

Tracking sentiment score

One way to use sentiment objects is to create per sentiment (negative and positive) Listeners that keep track of the items during each transaction. The user of Teneo Studio may use these if not interested in confidence level or it is also possible to take an approach of allowing one sentiment to supersede the other.

Sentiment Listener

By maintaining a running total of sentiment hits, it is possible to register and respond to trends, for example, if a user is especially negative, handing over to a live agent might be a good option. This can be done in a Flow using a Prompt Trigger.

Prompt Trigger expression

Detecting sentiment and confidence level

If concerned about confidence levels for each sentiment, this can be detected using the Language Object SENTIMENT.INDICATOR, which checks the sentiment objects and returns the sentiments found along with a confidence score. If positive or negative sentiment were not found, it returns neutral.

The Studio user can define a single Global Pre-listener for sentiment and use the object as follows, taking care of retrieving the NLU variable that is set:

Global Pre-Listener

The variable sSentiment is a list which in some cases may contain two sentiments; the value of the entries can be as follows:

  • Positive High Confidence or Positive Low Confidence (only one of these)
  • Negative High Confidence or Negative Low Confidence (only one of these)
  • Neutral (only if positive nor negative were not found).

Customization

The sentiment analysis objects have been prepared to allow easy customization by users, for instance, if a specific product-name is erroneously classified as negative sentiment, four Language Objects can be created locally and filled with items to include or exclude from sentiment analysis. The Language Objects in question exist as placeholders in the NLU Ontology and Semantic Networks.

Language objects
SENTIMENT_NEGATIVE_EXCLUDE.PROJ
SENTIMENT_NEGATIVE_INCLUDE.PROJ
SENTIMENT_POSITIVE_EXCLUDE.PROJ
SENTIMENT_POSITIVE_INCLUDE.PROJ

Limitations

Absolute detection of sentiment is accepted as an unobtainable goal. Existing systems must deal with this limitation. Sentences containing two conflicting sentiments at the same time, e.g. Sometimes I feel so happy, sometimes I feel so sad, are especially difficult to detect.

Accuracies of 70-90% can be considered acceptable. 90% is often quoted as the level at which two independent human raters can agree on sentiment. Since sentiment is not absolute, an absolute detection of sentiment as a goal is not a wise choice. False positives and false negatives will always exist, as language is infinite and often ambiguous. In a rule-based approach each new rule that is added while decreasing the detection of false negatives will invariably increase the incidence of false positives.

Teneo Intensity

Users say things with different intensity; the statement I hate you in itself does not contain cues as to the vehemence of the sentiment, but conclusions can be drawn as to vehemence by looking for cues that enhance the statement beyond the pure informational content. If a user says I hate you!!, I despise you or I really, really hate you, it can be said that these sentiments are expressed with higher intensity. It might not be possible to know with certainty whether the expressed intensity corresponds with actual intensity of emotion, but it does correspond to the socially accepted way of representing an expression of an intense emotion.

Note that some interpretations of sentiment analysis do not separate intensity from sentiment; at Artificial Solutions this was not found a good approach because, firstly, it increases the difficulty in distinguishing categories and, secondly, it does not consider that neutral statements can also be expressed with different intensity.

Language Objects

In an examination of thousands of tweets and user inputs, six distinct categories of intensity are defined which are for the most part easy to detect with the Teneo technology.

Intensity objectDescriptionExample variants
INTENSITY.INDICATORCollects all intensity LOBs, returns a list with all types of detected intensities
INTENSITY_CAPITALIZATION.INDICATORCapitalizing single words, phrases or an entire statement to show emphasisI AM NOT HAPPY ABOUT THIS / This is NOT good / You DO NOT want to shop there
INTENSITY_DUPLICATION.INDICATORUsing repetition of words or phrases to show emphasisI know, I know / I always always liked you / Run run run
INTENSITY_EMOJIS.INDICATORUsing emojis, emoticons or emotion-abbreviations to express intensityI like you / Rotfl / Lol / 😠 / 🙂
INTENSITY_PUNCTUATION.INDICATORUsing punctuation to express intensityI love your hat!! / You’re the greatest!!!!!!!! / Why are you doing this?? / You said what!?!
INTENSITY_SENTENCES.INDICATORUsing one-word sentences to emphasize a pointIt just. won’t. work. / I am so glad. you. came. / Please. go. away.
INTENSITY_VOCABULARY.INDICATORUsing heavily weighted words, superlatives or intensity modifiersI am very happy / This is awesome / You’re the greatest

Detecting Intensity

In a solution, a listener can be created to fill a global variable with the types of intensity cues a user input contains. The INTENSITY.INDICATOR object returns a list containing the types of intensity via an NLU variable, and if no intensity is detected, an empty list is returned. The list's size may be queried in the log data to determine the degree of intensity, e.g. three types of intensity cues are present. This in turn can be combined with sentiment to determine especially happy or dissatisfied users.

The values in the intensity list can be as follows:

  • capitalization
  • duplication
  • emoji
  • punctuation
  • sentences
  • vocabulary

Global pre-listener listening for intensity

Since the definition of intensity includes six unique types, one can consider that each input will have an intensity score ranging from 0 to 6. As a rule, it is rare to find more than four types of intensity grouped into a single statement; even three different types are unusual to find. However, occurrences of two different intensities simultaneously are quite common.

When combining intensity with sentiment, it is possible to locate inputs that stand out in some way, for example if intense negative sentiments are expressed, it might be used as a trigger for human intervention via live chat, etc. or to activate feedback, allowing the user a chance to give constructive criticism. If the system takes emphatically upset users seriously, it may defuse the situation as well as deliver information about how to prevent such situations in the future.

A beneficial side-effect of combining intensity with sentiment is that the subset of inputs is likely to contain fewer false positives, as sentiment and intensity often occur together. This can be used as an additional factor in determining confidence level.

Limitations

Five of the six categories (capitalization, duplication, emoticons, punctuation and sentences) can be detected to near perfection, as they are implemented algorithmically using inline groovy scripting. Intense vocabulary is more reliably determined than sentiment, as the vocabulary is limited to a specific, well-defined mean of expressions, superlatives, etc. Capitalization is the trickiest, as acronyms are often capitalized, but are not cues to intensity, as are names and personal pronouns. But as the number of capitalized words in an input increases, so does the likelihood that it is about intensity.

Sentiment, Intensity and Teneo Query Language

In the Log Data tab (Teneo backstage > Optimization > Log Data) it is possible to use sentiment and intensity objects together with the Teneo Query Language (TQL) to perform direct queries on log data; it is also possible to adorn transactions to facilitate queries.

Adorners can be created, for example to apply to the user input the sentiment classification provided by the Sentiment and Intensity resource and could help to illuminate the causes of negative sentiments and to highlight when users aren't satisfied with a particular response given by a conversational AI application and they complain.

For information on how to create, apply and query adorners, please see the Adorners section.