The Hidden Layers of Dialog Systems

Dialogue systems or conversational AI have been a very intensive area of research since the 1950s. A major scientific breakthrough has taken place since 2013 by applying deep learning (DL) and reinforcement learning (RL) in the NLP field. We are now observing promising results both in academia and industry, as large amounts of conversational data become available for training to feed those deep networks. Thanks to the significant government funding for voice technologies and virtual assistants, the insurance industry, more specifically the quotation, underwriting, and claim processes will be the first to benefit from these applications.

Most interactions with major chatbots in the insurance industry are robotic, outdated and sometimes even frustrating. Adding Natural Language Processing (NLP) is not an option to tackle the inefficiency of the chatbots. However, chatbots are like human, they need to be trained (educated) before being able to answer to the customer needs.

Before Building Olivo, our dialog system in insurance, we had to answer the following questions:

• Is the chatbot we are building able to communicate effectively with the user textually or voice-enabled?
• Will Olivo understand multi-dimensional human communications, which are, classifying the intents by extracting the context, understanding the direction of the conversation, sensing the user’s mood?
• Should we think to call other services from Olivo, like the dematerialization of documents (insurance contracts), the anomaly detections or any web services?
• How can we gather customer’s data in insurance and how could we accelerate the building of the training data?
• What does it mean security for dialog systems and how can we scale our dialog system?

Before tackling these questions, I want to give an overview of Natural Language Processing, as we can’t talk about smart dialog systems if we are not aware of the state-of-the-art environment of Natural Language Processing.

Natural Language Processing (NLP)

The state-of-the-art Natural Language Processing (NLP) algorithms have attained levels of human-like performance. Automatic translation, sentiment analysis, and text classification are some examples of this technology. Dialog system, chatbots or question-answering systems are capable of understanding complex textual commands and respond in a human-like fashion. Such sets of algorithms, commonly called chatbots, are going to replace gradually the customer service agents, thus drastically improving how core operations are run and reduce their costs (“chatbots can help reduce customer service cost by 30%” IBM report). We have to distinguish between Retrieval-Based (easier) and Generative (harder) models approach. The first model uses a repository of predefined responses and kind of heuristic to pick an appropriate response from a fixed set based on the input and the context. The second model generates a new response from scratch. Deep Learning techniques can be used for both models but research seems to be moving into the generative direction.

Deep neural network structures can have many interconnected layers of neurons. The more layers, the deeper the network is. The notion of depth of a network of neurons implies the presence of layers arranged in such a way that neurons receive information from neurons in the previous layer until it reaches the last layer where it predicts an output. The originality of all this is that the results of the first layer of neurons will serve as input to the calculation of the following. This layered operation is what is called deep learning. The more layers and neurons the higher the number of parameters of the model and hence the more complex the task of training the model to reproduce or explain a certain phenomenon. Deep learning uses several non-linear layers to learn from data the representation of generally complex functions and this is achieved via a collection of methods and techniques that make the calibration of a high number of parameters computationally possible.

Before starting talking about models, we have to understand word vectors, which are simply vectors of numbers that represent the meaning of a word. Traditional approaches, such as one-hot encoding and bag-of-words models do not capture information about a word’s meaning or context. Reducing the number of dimensions using singular value decomposition led to the birth of latent semantic analysis [Sahlgren, Magnus. “A brief history or word embedding ]. 

Convolutional neural networks (CNN) trained on top of pre-trained word vectors (word embedding for example) for sentence (intent) classification played a key role in improving the accuracy of the conversational AI. A simple CNN with some hyperparameter settings and static vectors achieves excellent results.

Olivo as a native cloud application

Building Olivo with state-of-the-arts NLP models is good but not enough, as we deal with customers’ data, the security layer is fundamental in Olivo. The client information and conversations are encrypted. As we see in the image below, we present the strength in terms of analytics, portability, and scalability of our dialog system. To achieve that, we need to think carefully about the architecture of Olivo.

“ Microservices -also known as the microservice architecture -is an architectural style that structures an application as a collection of services that are Highly maintainable and testable: Loosely coupled, Independently deployable, Organized around business capabilities, Owned by a small team. The microservice architecture enables the rapid, frequent and reliable delivery of large, complex applications. It also enables an organization to evolve its technology stack.”[Chris Richardson]

While IoT programming is beginning to have a very significant impact on the everyday lives of consumers over processes like on-demand app development, Olivo can also track the location of any customer during the underwriting or claim process, providing status and behavior analysis updates and alerts. This will help insurance companies to improve their quality control practices.

One way to make easy the creation, running, and deployment of applications is using containers. Docker is a tool designed to virtually containerize an application, package up this application with all the dependencies, libraries and to ship it all out as one package. Docker helps enable flexibility and portability on where the application can run, whether on-premises, public or private cloud, etc …

Data augmentation with Generative Adversarial Networks

Data must be meaningful to the problem being analyzed. It is a measure of quality, data can be noisy, incomplete, biased, and so on. The use of data augmentation methods is an inevitable solution, especially when the training examples are limited. The success of the dialog systems is related to the volume and the quality of the training data (questions, answers and the history of the conversation). Without many variations, our deep networks will not be able to classify properly the intents of the user. Classifying an intent is equivalent to classify a handwritten letter. We have to label all the training examples then variate enough to be able to capture or classify the distribution of each input vector with the corresponding intent, then predict an answer based on the conversation history.

Recently, Generative Adversarial Network (GAN), normally used for image generation [Ian Goodfellow, Generative Adversarial Nets], has been proposed and shown its strength of modeling the data distribution and learning discriminative representation.

Dialogue systems must be designed to handle as many dialogue utterances as possible for a semantic dialogue field while minimizing blocked states (bugs) and erroneous responses (misclassification). However, the platforms for industrial dialogue currently on the market are mostly rules-based and rarely fully exploit the capabilities of corpus-based technologies.

Conclusion

There has been a dramatic technological progression in Natural Language Processing and this trend will surely continue. The demand for dialog systems is evolving and will follow these trends in technology. Insurance and finance are not an exception and the sector must step up to the challenge and start equipping themselves with the tools and expertise needed to deliver the next generation of dialog systems.
 

To consult the original article, click here.