AI basics

Some key concepts that will help you better understand the magical world of Artificial Intelligence.

Let's start off with the basics: AI, Machine Learning, Deep Learning, Natural Language Processing.

What is Artificial Intelligence (AI)?

AI is a field of computer science focused on creating systems capable of performing tasks that typically require human intelligence.

What is Machine Learning (ML)?

Machine Learning is a subset of AI that involves the creation of algorithms that can learn from and make predictions based on data.

What is Deep Learning (DL)?

Deep Learning is a subset of Machine Learning that uses multi-layer neural networks to learn from a large quantity of data.

What is Natural Language Processing (NLP)?

Natural Language Processing is the field of Artificial Intelligence research that deals with the understanding and processing of natural language, and in particular human-machine interaction through human language.

The NLP is used by so-called intelligent conversational systems i.e. chatbots, voice assistants, and intelligent agents capable of processing human language, which are very useful for companies to manage online and telephone conversations with their customers.

This definition of NLP includes three very important elements in the field of artificial intelligence: speech recognition, natural language understanding, and natural language generation capability.

Generative AI and Conversational AI

What is Generative AI?

Generative AI refers to the type of AI algorithms designed to create new content, to audio and video, by learning from existing data. It uses large language models and algorithms to analyze patterns in datasets and mimic the style or structure of specific content types.

Hence, generative models are AI algorithms designed to learn patterns from large data sets and generate new instances that maintain statistical consistency with the original data. Unlike other AI models, which focus on classifying or predicting data based on specific inputs, generative models aim to capture and replicate the distribution of structured and unstructured data to create something new.

For example, after analyzing thousands of facial images, a generative model could images of non-existent people that resemble real photographs. This process is carried out using techniques based on artificial neural network models.

What is Conversational AI?

Conversational AI is a set of technologies and techniques used to create chatbots and other conversational interfaces that can interact with humans in a natural and intuitive way.

It is a combination of NLP and ML techniques that enable chatbots to understand and respond to human input. Specifically, it is a way for computers to understand and respond to human language so that they can interact with humans in a more human-like way and help them with various tasks such as customer service, finding information about a specific product, and more.

Conversation AI is being applied in a variety of contexts, including customer service, virtual assistants, and chatbots for e-commerce, healthcare, and other industries. It enables companies to automate interactions with customers and provide them with quick and accurate information through chatbot interfaces, messaging apps, and voice assistants.

It can be used in different forms, such as:

• Text-based chatbots

• Voice-based chatbots

• Image-based chatbots

Software that combines these functions to conduct human-like conversations could be called bots. While bots that use text-based interfaces are commonly known as chatbots.

If you have ever interacted with ChatGPT or Amazon Alexa, then you have already experienced conversational artificial intelligence!

This technology learns from us as we learn from it, just as children learn from their parents. Deep learning enables these systems to mature their conversational experiences so that, with practice, their conversations can become more useful and richer.

Machine Learning and Generative AI

Over the course of several decades, the evolution of both Machine Learning and Generative AI has been driven by the continuous development of algorithms designed to perform specific tasks.

Over the past decade, these algorithms have grown exponentially more complex, achieving the ability to gain functionality without continued human input.

But before diving into the switch from ML to Gen AI, let's take a quick look at their main differences.

Machine Learning is a subset of AI that focuses on building systems capable of learning from data, identifying patterns, and making decisions with minimal human intervention. These systems improve over time as they are exposed to more data, honing their ability to make accurate predictions or decisions.

Generative AI is a class of AI that goes beyond analyzing data to create new content—be it text, images, music, or even video—that mimics human creations. Instead of merely making decisions or predictions based on input data, generative AI can generate novel data that wasn’t explicitly programmed into it.

With that being said, while machine learning and generative AI are both subsets of artificial intelligence, their primary distinction lies in their purpose and output.

• Purpose: Machine Learning focuses on understanding and predicting based on existing data. Generative AI is geared towards creating new data that mimics human creations.

• Output: Machine Learning outputs decisions or predictions. Generative AI produces new content, such as text, images, or audio.

• Applications: Machine Learning is used for tasks like recommendation systems, predictive analytics, and diagnostic tools. Generative AI is employed in creative domains, deepfakes, and advanced simulations.

From Machine Learning to Generative AI

As we just learned, while both Machine Learning and Generative AI are groundbreaking in their own right, they serve very different purposes and operate in unique ways.

In this regard, it should be noted that Machine Learning emerged first, focusing on identifying patterns and making data-driven decisions. Meanwhile, Generative AI built upon these foundations, introducing models capable of creating new, original content by learning from vast datasets.

But let's start from the beginning: the 2017 publication of the breakthrough paper "Attention Is All You Need". "Attention Is All You Need" is a landmark research paper in machine learning authored by eight scientists working at Google. The paper introduced a new deep learning architecture known as the transformer and is considered a foundational paper in modern artificial intelligence, as the transformer approach has become the main architecture of large language models like those based on GPT. On top of that, the authors went as far as foreseeing the technique's potential for other tasks like question answering and multimodal generative AI. This represented a huge milestone in AI's history since, for the very first time, language was represented efficiently to maintain semantics and meaning.

This paper proposed a novel approach to NLP tasks that relied solely on the self-attention mechanism, a type of attention mechanism that allows the model to weigh the importance of different words in a sentence when encoding it into a fixed-size vector representation. The transformer architecture was revolutionary in that it allowed for much faster training times and better parallelization on GPUs, since the self-attention mechanism could be computed in parallel for all words in a sequence. This made it possible to train much larger models on much larger datasets, leading to significant improvements in NLP performance.

As to the transformer architecture, it is composed of an encoder and a decoder, each of which is made up of multiple layers of self-attention and feedforward neural networks. The self-attention mechanism is the heart of the transformer, allowing the model to weigh the importance of different words in a sentence based on their affinity with each other. This is similar to how a human might read a sentence, focusing on the most relevant parts of the text rather than reading it linearly from beginning to end.

The transformer encoder architecture is used for tasks like text classification, where the model must classify a piece of text into one of several predefined categories, such as sentiment analysis, topic classification, or spam detection. The encoder takes in a sequence of tokens and produces a fixed-size vector representation of the entire sequence, which can then be used for classification.

One of the most popular transformer encoder models is BERT, which was introduced by Google in 2018. BERT is pre-trained on large amounts of text data and can be fine-tuned for a wide range of NLP tasks.

The transformer decoder architecture is used for tasks like language generation, where the model must generate a sequence of words based on an input prompt or context. The decoder takes in a fixed-size vector representation of the context and uses it to generate a sequence of words one at a time, with each word being conditioned on the previously generated words.

One of the most popular transformer decoder models is the GPT-3, which was introduced by OpenAI in 2020. The GPT-3 is a massive language model that can generate human-like text in a wide range of styles and genres.

Essentially, there was a transition from a classical Machine Learning approach, where a labeled dataset is needed, to a few-shot approach with Generative AI, where there's only an instruction in natural language.

With Generative AI you can perform tasks like analyzing the entire works of book authors to produce an original novel that seeks to simulate these authors’ styles and writing patterns. Thus, generative AI ventures well beyond traditional machine learning. By utilizing multiple forms of machine learning systems, models, algorithms, and neural networks, generative AI offers a new foray into the world of creativity.

Emerging skills

Today, advances in technology are changing the demand for skills at an accelerated pace. New technologies can not only handle a growing number of repetitive and manual tasks but also perform increasingly sophisticated kinds of knowledge-based work—such as research, coding, and writing—that have long been considered safe from disruption.

Furthermore, as the capabilities of generative AI models advanced, some researchers started seeing “sparks” of deeper intelligence in AI. In other words, they noticed how AI models can sometimes complete tasks they were not programmed to do. This is why they decided to use the word emergence to describe these surprising skills.

A skill is deemed emergent if it is not present in smaller models but appears in larger models. To be more specific, "emergent AI abilities" refer to the unexpected, novel behaviors or skills that appear in advanced artificial intelligence systems. These abilities are not pre-trained or programmed into the AI model but emerge unpredictably, particularly in large-scale models.

On this note, many of the features of Generative AI were not foreseen, but rather emerged spontaneously from very large models. Among these skills are (A) Math Word Problems, (B) Instruction Following, (C) 8-Digit Addition, and (D) Calibration.

However, there's no doubt that the most important one is the ability to follow the instructions given to the model. The emergence of instruction-following in different language models has been hailed as a key stepping stone on the path to artificial intelligence. By learning to understand and carry out open-ended requests through the power of large-scale neural networks and training data, these systems hint at a future where AI can flexibly apply knowledge and reasoning to assist humans across a vast range of domains.

Hallucinations

Now let's move on to the big question: "Are language models reliable?" When talking about LLMs, we must be aware of their downsides, including their ability to hallucinate.

The term hallucination refers to a response generated by AI that contains false or misleading information presented as fact. In other words, text that is factually incorrect or nonsensical but very likely.

Now you are probably thinking, "But how do we deal with hallucinations?" Well, at indigo.ai, we actually deploy three different methods of dealing with such cases:

1. : Using prompt chaining to check or fix a generation.

   The steps to be followed are: [Generation #1 -> Consistency Check -> Correction -> Final Generation]

   For instance, a second prompt to correct any expressions not in line with the company tone of voice.

2. Tooling: The chatbot can access external information, like for example calling APIs, and can integrate the right information into the prompt.

3. Agents: Like humans, LLMs have better performances if specialized and orchestrated by a central "brain" (the so-called mother agent). The process is: [User Request -> Mother Agent -> (relevant) Agent -> Flow -> Answer]