Teneo Developers

Technology and deployment

SAML

SAML (Security Assertion Markup Language) can be defined as a standard to exchange security and authentication information. It enables different organizations to integrate their multiple applications to a single sign on authentication system, so that there is no more need to handle different user credentials. SAML standard basically defines how this authentication will work, what messages will be interchanged between the Identity Provided (IdP) and the Service Provided (SP) as well as the different flows it supports.

In the Teneo Platform, SAML has been implemented mainly in Teneo Manager, which is acting as a Service Provider. It can be configured to do the authentication against a standard SAML IdP. It also supports the autoconfiguration of service by means of using SAML metadata file format as well as exposing its metadata configuration through an endpoint. The rest of the components just change their login screens to adapt to the new authentication paradigm. Visually, all of them open a webpage next to the application instance, where the user can then follow the SP initiated SAML flow and finally authenticate.

Since SAML authentication requires users and groups created in advance in Teneo Manager, the TM Rest API has been extended to be able to create them. So, there is no more the need to synchronize those entities from an LDAP or create them using the UI, but rather now customers and partners can develop their own tools to feed that data in TM.

Last but not least, now it is possible to assign Studio account permissions directly to groups without the need to do it user by user. An Admin just needs to go to the Studio accounts management view in Teneo Manager and click on the group permissions icon. A new dialogue will appear allowing admin to search for groups and assign permissions. Admin simply needs to click on add group so that all users belonging to that group will inherit those permissions.

Note that it is possible to assign permissions to specific users in a similar way they are assigned for groups.

Security Improvements

Configured Password Encryption

Passwords in all configuration files are now encrypted (as per Teneo Manager) to ensure that even if a bad actor gets onto the file system the credentials for accessing the databases remain secure.

Network Structure Potential Disclosure Removed

The following status lines have been removed from the teneo-inquire-query/status?verbose and /teneo-inquire-importer/status?verbose responses as it was possible for them to expose network topology details which it is preferable to keep hidden:

properties

1Es-Cluster-Name=...
2Cs-Connection-Hosts=...
3

Unused Data Removed from Responses

  • One unused request which would return all document ids in a Studio instance has been removed as it was not used or needed and could be useful intrusion data
  • Unnecessary extra data about other users included in some responses has been completely removed
    • This data was not a risk in itself and all the requests require authorization to execute but it provided a slightly shorter path to some attack vectors - and as such has been removed.

Unknown User vs. No Accounts

The error message given upon logging in to an environment where the user has a login - but does not have access to any accounts - is now the same as that given for the wrong password / unknown user. This is a very small edge case providing a potential attack vector and has been closed.

Auth Headers follow OAuth2 Standard

Across the Platform the way authorization is passed between components has aligned with the OAuth 2.0 Standard (i.e. OAuth2: Bearer Token). Largely this will be an invisible change to users as the server and client APIs are mostly private and both ends (server and client) have been updated with this change. However if a project is calling the Inquire API REST endpoints directly (for example in JavaScript) then the format of the requests will need to be updated - if the connection to Inquire is via the Inquire java client, then this has also been updated so no changes are needed in the project code.

Add "Bearer"

The only change required for a previously working connection to Inquire API after this alignment is to change the Authorization header value:

Authorization Header Name5.1.2 Header Value6.0 Header Value
Authorizationauthentication_token_valueBearer authentication_token_value

Example (JavaScript)

For example starting a query (assuming the authentication variable contains the result of a successful login call):

javascript

1fetch(`${server}/teneo-inquire-query/rest/tql/submit?index=${encodeURI(lds)}&timeout=1200`, {
2    method: 'post',
3    headers: {
4        "Authorization": `Bearer ${authentication}`,
5        "Content-type": "application/x-www-form-urlencoded"
6    },
7    body: `query=${encodeURIComponent(query)}&index=${encodeURI(lds)}`
8})
9

Where the only difference is in Line 4 which would previously have been:

javascript

1        "Authorization": authentication,
2

Monitoring / Stability Improvements

Component Recovery on missing DB at startup

Studio, Discovery and Teneo Manager are now more resilient to situations where the database is not available at application start. This ensures that systems will startup even when there is an issue causing the database service to not start promptly - making the process of starting up a full environment from standstill a simpler process.

Retry interval and Retry attempts are configurable.

Property NameDescriptionDefault
intervalTime in ms between reconnection attempts to the DB when the backend is starting up30000 (30 s)
attemptsNumber of attempts to connect to the DB when the backend is starting up. 0 means that infinite connection attempts will be made0 (infinite attempts)

Manager Configuration

File: settings.xml

Properties

xml

1<entry key="teneo.manager.jdbc.reconnect.interval">30000</entry>
2<entry key="teneo.manager.jdbc.reconnect.attempts">0</entry>
3

Discovery Configuration

File: settings.xml

Properties

xml

1<entry key="teneo.discovery.jdbc.reconnect.interval">30000</entry>
2<entry key="teneo.discovery.jdbc.reconnect.attempts">0</entry>
3

Studio Configuration

File: teneo-studio.properties

Properties

properties

1repository.reconnect.interval=30000
2repository.reconnect.attempts=0
3

Studio 503 Unavailable when starting

When Studio Backend has started but is still initializing it reports this state to any caller. When a client connects to the status endpoint to discover component status this HTTP:200 - OK response with JSON payload was given - however this would lead monitoring tools to believe that the component was running correctly, when it was in fact not ready to receive requests.

In this release the response has been changed to instead be an HTTP:503 - Service Unavailable.

Retry Connect

The Studio Frontend has also been updated so that when this Service Unavailable response is received the Frontend will automatically retry after a (studio backend configurable) 5 seconds delay - this means that users can simply run the Studio Frontend and then when a connection is possible, they will be connected.

This delay can be configured in teneo-studio.properties configuration file.

properties

1# How long (milliseconds) before reconnect
2monitor.loginRetry.interval=5000
3

Slow Query Log - Inquire

It is now possible for Inquire Query Backend to report in the logs when a query execution time exceeds a configurable threshold. This is configured in teneo-importer-backend.yml and teneo-query-backend.yml.

The property value is defined in milliseconds and defaults to 60,000 (1 minute):

yaml

1misc:
2    slowQueryTime: 60000
3

When a query exceeds the configured threshold a message similar to this will be logged in the application log file:

SLOW QUERY detected! TQL [query detail] executed in [time taken] s

JMX Statistics included in all /status?verbose calls

Additional properties have been added to all Teneo components to show the status of the underlying JVM. These are added to enable easier monitoring of the environment as a whole through network monitoring tools.

NOTE these properties are properties read directly from the JVM - so in cases where multiple apps are running in a single VM the results will show the statistics for all running Apps, not just the Teneo component in question

properties

1JVM-Heap-Used=...
2JVM-Metaspace-Initial-Size=...
3JVM-Metaspace-Max-Size=...
4JVM-Code-Cache-Max-Size=...
5JVM-Metaspace-Committed-Size=...
6JVM-Code-Cache-Committed-Size=...
7JVM-Heap-Initial-Size=...
8JVM-Code-Cache-Used=...
9JVM-Metaspace-Used=...
10JVM-Threads-Current-Count=...
11JVM-Heap-Committed-Size=...
12JVM-GC-Count=...
13JVM-GC-Accumulated-Time=...
14JVM-Code-Cache-Initial-Size=...
15JVM-Heap-Max-Size=...
16

Published Engine reports Logging Status

Further to the work in Teneo 5.1.1 to ensure the writing to disk of sessions which do not get properly propagated on the message queue - Teneo Engine will now update the published engine status?verbose response to include an indication when this has happened

Session logs being sent successfully:

properties

1Session-Logging=OK
2

Session log has failed to send:

properties

1Session-Logging=ERROR
2

File rollover number of Log files

Following up on the migration of logging for all Teneo Components done in Teneo 5.0 to log4j2, the log file rollover number is now named in such a way that the oldest log files have the highest index in the file name. For example, for Teneo Studio:

teneo-studio.log >> teneo-studio.1.log >> teneo-studio.2.log

In the above example, teneo-studio.log is the latest/newest log file; the previous logs are stored in teneo-studio.1.log and the oldest logs are stored in teneo-studio.2.logs.

Dependency Changes

As always (especially in major releases) there have been quite some updates in the Tech Stack on which Teneo runs.

These changes are made for various reasons:

  • Stability - dependencies used by Teneo being fixed and stabilized
  • Security - potential vulnerabilities in dependencies of Teneo which have been resolved in new versions of the dependencies
  • Support / SLA - ongoing commercial (where applicable) support of the dependencies used by Teneo is important to the Enterprise market
  • Performance - general improvements to the performance on the dependencies will improve the performance also of Teneo
  • New Features - new versions of included dependencies include new features which can then be leveraged by Teneo.

Maria DB replaces MySQL

Teneo 5.1.2 was packaged to use MySQL 5.7.21

  • For commercial reasons it was not possible to package MySQL and so this was an external install dependency
  • MySQL and MariaDB at MySQL 5.7.21 were still binary compatible
    • Teneo has for some time been using the MariaDB connector to connect to MySQL
  • In the field we have seen a general move to MariaDB being the preferred implementation.

For these reasons (along with the technical benefits of MariaDB performance and project support) this Teneo Platform Release includes MariaDB 10.3.13 packaged and tested.

Additionally: Flyway updated from 5.0.7 to 6.0.1 - which is used by the upgrade of Teneo Manager and Discovery databases updated to a version compatible with this new version of MariaDB.

.NET 4.8

Teneo Studio Frontend now uses Microsoft .NET 4.8 - this provides stability improvements and a wider feature set. This will require installation on some client machines (.NET 4.8 is default on Windows 10 - older machines it will be automatically installed by running Teneo Studio Setup.exe).

Ubuntu 18.04 (LTS)

The underlying base Operating System version has been upgraded to the latest Long Term Supported version: Ubuntu 18.04.

Jackrabbit

Jackrabbit (used by Teneo Studio for content management functionalities) has seen a major upgrade from 2.6.4 to 2.19.4. This provides performance improvements as well as the mitigation of some potential vulnerabilities.

Spark 2.3.2 => 2.4.4

Apache Spark used by Teneo Learn for training the Intent Classifier model has been upgraded from 2.3.2 to the latest release 2.4.4 - this update brings general improvements and ensures we are up to date with the latest machine learning feature set offered by this component.

MLeap 0.12.0 => 0.14.0

MLeap is used to execute the model created using Apache Spark and as such has been updated to the latest version (at time of implementation) moving from 0.12.0 => 0.14.0.

Java 11

Complete rebuild of the entire platform to run on environments running Java 11. As Java 8 is coming towards End of Life and to benefit from the better support for containerization which Java 11 provides a big effort has been made to bring as much of the platform to run on Java 11 as possible. Java 11 is now the supported JVM for all components and dependencies except Cassandra, which does not yet support Java 11.

Kafka 1.1.0 => 2.3.0

Apache Kafka which is used by a Published Teneo Engine to send session logs to the configured Teneo Inquire installation.

ElasticSearch

ElasticSearch is updated from version 5.6.16 to 7.4.2.

Minor Dependency changes

Various other dependent libraries have been updated this has resolved a number of potential vulnerabilities they represented as well as benefiting the various components due to performance and stability improvements of the dependencies themselves. The details of these upgrades will not be included here, the latest list of dependencies can be found here: Dependencies and Licences.

Packaging Changes

All components are now also packaged, released and supported as RedHat .rpm (RHEL7) packages. This extends the reach of the pre-packaged Platform to a section of the Enterprise market with IT policies not supporting Ubuntu installations.

Groovy 3

In this release, the embedded Groovy version has been upgraded from Groovy 2.4.17 to Groovy 3.0.7. The upgrade allows for a more streamlined Groovy packaging in the final published solution, reducing the size of the final WAR in deployed solutions.

These changes will have implications for some solutions - though the out-of-the-box packaged modules have been chosen to limit these implication; all the packages changes and internal changes in Groovy which are likely to have an impact are listed below - full change descriptions can be found in the Groovy release notes, specifically: Groovy 2.5 and Groovy 3.0.

Included packages

New solutions will by default have access to everything in the following packages:

  • groovy
  • groovy-astbuilder
  • groovy-datetime
  • groovy-json
  • groovy-jsr223
  • groovy-macro
  • groovy-nio
  • groovy-sql
  • groovy-templates
  • groovy-xml
  • groovy-yaml

Any dependencies in use in a solution which are now in Groovy packages not listed here will require the specific packages to be included in the solution as file resources.

Package / Namespace changes

Some classes have changed package in Groovy 3 to be more aligned with the Java Platform Module System. Full details are available here (Groovy 3.0 release notes) - the changes affecting the modules included in Teneo are listed below:

Teneo 5.xTeneo 6Recommendation
Module: groovy
groovy.xml.QNamegroovy.namespaceMigrate if using QName parameters
Module: groovy-nio
org.codehaus.groovy.runtime.NioGroovyMethodsorg.apache.groovy.nio.extensions.NioExtensionsNo change
org.codehaus.groovy.runtime.WritablePathorg.apache.groovy.nio.runtimeNo change
Module: groovy-xml
groovy.utilgroovy.xmlIf used, explicitly import groovy.xml.XmlParser / groovy.xml.XmlSlurper
org.codehaus.groovy.tools.xml.DomToGroovyorg.apache.groovy.xml.toolsMigrate

The recommendations column contains the recommendation from Groovy - based on expected / standard usage. If the solution does not work correctly after following the recommendation more detail can be found in theGroovy 3.0 release notes.

Breaking Changes

LinkedList.pop() - functionally reversed

GROOVY-6396 (in Groovy 2.5) made a change to align behavior of Groovy LinkedList with that of Java. Previously Groovy pop would remove and return the last element in the list, where Java pop would return the first (ie. Groovy was LIFO with and Java was FIFO). This behavior has been changed in order to align Groovy with Java.

Check solutions for use ofpop() on a LinkedList.

Groovy 2.5

The following fixes in Groovy 2.5 could affect a solution - this is a stripped list from the full set in the Groovy release notes

  • The extension methods for the java.util.Date class are now in a separate groovy-dateutil module.
    • Migrate to the java.time JSR-310 classes (similar Groovy extension methods exist for those classes and they are included by default)
  • @ToString could output properties in a predefined order when 'includes' is used GROOVY-8014
  • @ToString could support non-field properties GROOVY-7394
  • ObjectRange iterator returns null instead of NoSuchElementException GROOVY-7961
  • IntRange iterator returns null instead of NoSuchElementException GROOVY-7960 GROOVY-7937
  • Remove synchronized methods of groovy.sql.Sql and document it as not thread-safe GROOVY-7673
  • API inconsistency between takeWhile, dropWhile and collectReplacements for CharSequences GROOVY-7433
  • Accessing private methods from public ones using categories and inheritance causes MissingMethodException GROOVY-6263
  • Have the elvis operator (?:) support the Optional type in Java 8 GROOVY-6744
  • java.util.Optional should evaluate to false if empty GROOVY-7611
Groovy 3.0

The following changes in Groovy 3.0 could affect a solution - this is a stripped list from the full set in the Groovy release notes

  • If a Groovy switch statement has a default branch, it is now required to be the last branch
  • Alpha versions of Groovy 3 incorrectly let you leave off the brackets when printing empty maps, but they are now required, e.g println([:])

Documentation

For the Teneo 6.0 release the format and delivery of the documentation for the entire Platform has been reworked. The purpose of this feature is to develop a more modern documentation which is readily available to all users of the Platform; additionally to make this documentation accessible in a centralized location to provide better possibilities for sharing and linking content with users / other systems.

The documentation now contains three sources of documentation for the Teneo Platform:

  • JavaDoc Reference: the following JavaDoc documents are coding reference documents for parts of the Platform accessible via Java/Groovy:
  • Component APIs: API reference for all the components with a Public API are available:
  • Jar files and dependencies: provides the jar files and dependency list for available clients
  • User documentation: documentation about usage and configuration of all the Platform components can be found in the remaining options in the top navigation bar of this documentation area. The documents are organized into functional groups:
    • Reference: contains reference documentation related to pre-build Teneo resources, Teneo Linguistic Modelling Language and the Teneo Query Language
    • User Guide: provides information and descriptions related to the user interfaces of the different Teneo Platform components
    • Knowledge Base: contains additional information and knowledge articles related to the Teneo Platform and its components.

Browser support

Documentation is supported on the following browsers:

  • Chrome 81 onwards
  • Firefox 76 onwards (on Windows or Ubuntu)
  • Edge 44 onwards
  • Safari 12 onwards

Besides supporting visualization of documentation on laptop screens and external screens, resolution and navigation now also permit to view documentation on mobile devices.

Upgrade to Teneo 6.0

The upgrade to Teneo 6.0 is supported from any previous 5.X version of Teneo. Even though an in place upgrade is possible, due to the considerable tech stack and configuration changes in this release, it is highly recommended to install Teneo in a separate environment and then perform a database data migration as well as re-applying any custom configuration of your current environment into the new one.

For further details on the upgrade process please refer to the migration guide provided with the Teneo 6 release.

KI Resolutions

DescriptionKI
The Transition ordering is now taken into account in the Flow's node layout/Flow graphKI-159
A solution can no longer be deleted while another user is editing itKI-162
Metadata definition: default value not being stored when turning auto-logging off is resolvedKI-218
Solution list showing Solution Version (and its last edition) but not Solution Revision is resolvedKI-426
Issue when changing solution language with a Lexical Resource assigned is resolvedKI-583
Right-click context menu for File Resources doesn't work is resolvedKI-588
Exceptions in Studio backend logs when moving a folder with subfolder is resolvedKI-589
Restoring Hybrid Flows doesn't restore linked triggers is resolvedKI-623
Behavior of the "Click here to add entry" row in Entity editor is resolvedKI-625
Example counts are now updated in the Optimization view when a Flow is savedKI-626
Using the text filter no longer crashes Studio when Entities/Language Objects are selectedKI-630
Hitting Delete key doesn't delete solution resource file is resolvedKI-631
Exported Solution XML no longer includes unused solution resource contentKI-632
File resource addition doesn't validate duplicated resources when not using the default path is resolvedKI-635
In Localization (former Master-Local) context the update buttons are now available in the My Work tab for Global users in local solutionsKI-636