Transformer Model Explained: How GPT-3, BERT, and T5 Work

Original link: https://daleonai.com/transformers-explained
Author: Dale Markowitz (@Google Labs, leading generative AI advocacy and education)
Published: May 6, 2021
Translation: Data Pie THU (Translator: Wang Kehan, Proofreader: He Zhonghua), with slight modifications
Original Chinese translation: https://mp.weixin.qq.com/s/kfsW7ccYUAGp1AHWWF6c1w
[WaytoAGI (Way to AGI)] Fourth article in the "A16Z AI Collection" introductory series

You know the saying, when you have a hammer, everything looks like a nail? In machine learning, we seem to have truly discovered a magical hammer. In fact, in front of this model, everything is a nail—that model is the Transformer. The Transformer can be used to translate text, write poems, write articles, and even generate computer code. In fact, many of the amazing studies I write about on daleonai.com are built on the Transformer, such as AlphaFold 2, a model that predicts protein structures from gene sequences, and powerful natural language processing (NLP) models like GPT-3, BERT, T5, Switch, Meena, and more. You might say they didn't just encounter... well, let's move on.

If you want to keep up with machine learning, especially natural language processing, you need at least a basic understanding of Transformers. So in this article, we'll discuss what they are, how they work, and why they are so influential.

A Transformer is a type of neural network architecture. Simply put, neural networks are highly effective models for analyzing complex data types like images, video, audio, and text. There are specialized neural networks optimized for different types of data. For example, when analyzing images, we typically use convolutional neural networks. Broadly speaking, they mimic how the human brain processes visual information.

[Image]
Convolutional neural network, image by Renanar2, Wikimedia Commons

Since around 2012, we have been using CNNs quite successfully for vision tasks, such as recognizing objects in photos, identifying faces, and recognizing handwritten digits. But for a long time, language tasks (translation, text summarization, text generation, named entity recognition, etc.) lacked good methods. This was unfortunate because language is the primary way we humans communicate.

Before the Transformer was introduced in 2017, the deep learning approach to understanding text used a model called Recurrent Neural Networks (RNNs), which looked like this:

[Image]
Recurrent neural network, image by fdeloche, Wikimedia

Suppose you want to translate a sentence from English to French. An RNN takes an English sentence as input, processes one word at a time, and then outputs the corresponding French words in order. The key word here is "order." In language, word order matters; you can't just scramble them. For example, the sentence:
"Jane went looking for trouble." has a very different meaning from: "Trouble went looking for Jane."

Therefore, any model that can understand language must capture word order, and RNNs do this by processing words one at a time in a sequence.

But RNNs have problems. First, they struggle with long text sequences, like long paragraphs or articles. By the time they reach the end of a paragraph, they forget what happened at the beginning. For example, an RNN-based translation model might have difficulty remembering the gender of the subject in a long paragraph.

Worse, RNNs are hard to train. They are susceptible to the so-called vanishing/exploding gradient problem (sometimes you just have to restart training and pray). More problematic is that because RNNs process words sequentially, they are difficult to parallelize. This means you can't speed up training by adding more GPUs, which also means you can't train them on as much data.

Enter the Transformer

This is where the Transformer shines. Developed in 2017 by researchers at Google and the University of Toronto, it was originally designed for translation. But unlike RNNs, Transformers can be parallelized very efficiently. This means that with the right hardware, you can train some very large models.

How large? Extremely large!

GPT-3 is a particularly impressive text generation model, with writing ability nearly on par with humans. It was trained on 45TB of text data, including almost all public web data.

So, to sum up the Transformer in one sentence: when a highly scalable model meets a massive dataset, the results might surprise you.

How Does a Transformer Work?

[Image]

Transformer, image from the original paper: https://arxiv.org/abs/1706.03762

Although the diagram in the original paper looks a bit intimidating, the innovations behind the Transformer can be boiled down to three main concepts:

1. Positional Encodings

2. Attention

3. Self-Attention

Positional Encodings

Let's start with the first one: positional encodings. Suppose we want to translate text from English to French. Recall that RNNs—the previous translation approach—understood word order by processing words sequentially. But that's also what made them difficult to parallelize.

The Transformer sidesteps this obstacle with an innovative approach called positional encoding. The idea is to take all the words in the input sequence—in this case, an English sentence—and append a number to each word indicating its position in the sequence. So you feed your network something like this:

[("Dale", 1), ("says", 2), ("hello", 3), ("world", 4)]

Conceptually, this shifts the burden of understanding word order from the structure of the neural network into the data itself.

Initially, before any training on data, the Transformer doesn't know how to interpret these positional encodings. But as the model sees more and more sentences and their encodings, it learns how to use them effectively.

I've simplified things a bit here—the original authors used sinusoidal functions for positional encoding rather than simple integers like 1, 2, 3, 4—but the gist is the same. Store word order as data rather than relying on the network's structure, and your neural network becomes much easier to train.

Attention

The next important piece of the Transformer is called attention.

Hi!

Attention is a neural network architecture that shows up everywhere in machine learning. In fact, the title of the 2017 paper that introduced the Transformer wasn't "We Present You the Transformer." Instead, it was called "Attention Is All You Need" (Attention is All You Need).

Attention was introduced in 2015 for text translation. To understand it, consider this example sentence from the original paper:

The agreement on the European Economic Area was signed in August 1992.

Now imagine translating this sentence into French:

L'accord sur la zone économique européenne a été signé en août 1992.

One bad way to translate this sentence would be to go word-by-word through the English sentence and try to translate each word into French one at a time. This doesn't work for several reasons, but one is that some of the words in the French translation get reversed: "European Economic Area" in English becomes "la zone économique européenne" in French. On top of that, French is a language with grammatical gender. The adjectives "économique" and "européenne" must take the feminine form to match the feminine object "la zone."

Attention is a mechanism that allows a text model to "look at" every word in the source sentence when deciding how to translate a word in the output sentence. Here's a nice visualization from the original attention paper:

[Image]

Image from the paper, "Neural Machine Translation by Jointly Learning to Align and Translate" (2015), https://arxiv.org/abs/1409.0473

This is a heatmap showing where the model is "paying attention" as it outputs each word of the French sentence. As you'd expect, when the model outputs "européenne," it focuses strongly on the input words "European" and "Economic."

How does the model know which words to "attend to" at each time step? That's something it learns from training data. By looking at thousands of French and English sentences, the model learns which types of words depend on one another. It learns how to respect grammatical gender, plurality, and other grammar rules.

Since its discovery in 2015, the attention mechanism has been a highly useful tool in natural language processing, but in its original form, it was used in conjunction with recurrent neural networks. Therefore, the innovation of the 2017 "Transformer" paper lies in largely abandoning RNNs altogether. This is why the 2017 paper is titled "Attention Is All You Need."

Self-Attention Mechanism

The final—and perhaps most influential—aspect of the Transformer is a variant of attention: self-attention.

The "vanilla" attention we just discussed helps align words in English and French sentences, which is crucial for translation. But what if you're not trying to translate words, but instead aiming to build a model that understands the underlying meaning and patterns in language—a model that can be used for any number of language tasks?

In general, what makes neural networks powerful is that they often automatically build meaningful internal representations of the training data. For example, when you examine different layers of a visual neural network, you'll find that different neurons are responsible for "recognizing" different patterns, such as edges, shapes, and even high-level structures like eyes and mouths. Models trained on text data may automatically learn parts of speech, grammatical rules, and whether words are synonyms.

The better a neural network learns the internal representation of language, the better it performs on any language task. It turns out that the attention mechanism is also highly effective when applied to the input text itself.

For example, consider these two sentences:

"Server, can I have the check?"

"Looks like I just crashed the server."

Here, the word "server" means two very different things, and we humans can easily disambiguate it by looking at the surrounding words. Self-attention enables the neural network to understand a word in the context of the words around it. So, when the model processes "Server" in the first sentence, it might "attend" to the word "check," which helps disambiguate between the two meanings—waiter and machine.

In the second sentence, the model might attend to the word "crashed" to determine that this "server" refers to a machine.

Self-attention helps neural networks disambiguate words, perform part-of-speech tagging, named entity recognition, learn semantic roles, and more.

We've given a high-level summary of the Transformer here.

If you want a more in-depth technical explanation, I highly recommend checking out Jay Alammar's blog post The Illustrated Transformer.

What Can Transformers Do?

One of the most popular Transformer-based models is BERT, which stands for "Bidirectional Encoder Representations from Transformers." It was introduced by researchers at Google around the time I joined the company in 2018 and quickly made its way into nearly all NLP projects, including Google Search.

BERT refers not only to the model architecture but also to the trained model itself, which you can download and use for free here.

Google researchers trained it on a massive text corpus, and it has become a general-purpose model for natural language processing. It can be extended to solve a range of different tasks, such as:

• Text summarization

• Question answering

• Classification

• Named entity recognition

• Text similarity

• Offensive content / profanity detection

• Understanding user queries

• And more

BERT demonstrated that you can build very good language models on unlabeled data, such as text from Wikipedia and Reddit, and that these large "foundation" models can be adapted to specific domains for many different use cases.

More recently, OpenAI's GPT-3 has amazed people with its ability to generate realistic text. Google's Meena, launched last year, is a Transformer-based chatbot (ahem, conversational agent) that can hold engaging conversations on almost any topic (its authors once spent 20 minutes arguing with Meena about what it means to be human).

Transformers have also made waves beyond natural language processing, such as composing music, generating images from text descriptions, and predicting protein structures.

How to Use Transformers?

Now that you're captivated by the power of Transformers, you might be wondering how to use them in your own applications. No problem.

You can download common Transformer-based models, such as BERT, from TensorFlow Hub. For code tutorials, check out my article on building semantic language applications.

But if you want to truly stay on the cutting edge and you write Python, I highly recommend the Transformers library maintained by Hugging Face. This platform lets you train and use most of today's popular NLP models—like BERT, RoBERTa, T5, GPT-2—in a very developer-friendly way.

That's it for today's article!