What Does ChatGPT Actually Do, and Why Does It Work?

Original Title: 《What Is ChatGPT Doing … and Why Does It Work？》

Original URL: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

Original Author: Stephen Wolfram https://www.stephenwolfram.com/

Published: February 14, 2023

Translator: SIY.Z

Zhihu URL: https://zhuanlan.zhihu.com/p/607601817

《A16Z's AI Collection》Getting Started Part 3

[Figure]

[Figure]

ChatGPT Generates Text on a "Word-by-Word" Basis

ChatGPT can automatically generate text that looks remarkably like human writing—this is both impressive and unexpected. But how does it do it?

• My goal is to outline the internal process of ChatGPT, and then explore why it can successfully generate text that we consider meaningful.

• Let me state upfront that I will focus on the big picture without diving into every detail.

• Also, the key points I make apply to other "large language models" (LLMs) as well.

First, it's important to understand that ChatGPT is essentially always trying to generate a "reasonable continuation" of the existing text, where "reasonable" means "after looking at billions of web pages and other human-written texts, we might expect a person to write something like this."

So, suppose we have the existing text: "One thing AI is best at is..." Then imagine scanning billions of human-written texts (e.g., web content and digitized books), finding all instances of this text, and seeing how often each next word appears. ChatGPT does something similar, but (as I'll explain) it doesn't look at words directly—it looks for content that "matches" in a certain sense. Ultimately, it produces a ranked list of possible next words and their "probabilities":

[Figure]

Amazingly, when ChatGPT tries to write an article, it basically just asks over and over: "Given the text so far, what should the next word be?" and then adds one word at a time. (More precisely, as I'll explain, it adds a "token" [Note: token], which may be just part of a word—that's why it sometimes "invents new words.") [Note: For example, for the word "apple," ChatGPT might first generate "app" and then "le." This allows it to create nonexistent words like "bananapple." The smallest unit ChatGPT generates is roughly a Unicode character, meaning in principle it can generate text in any language on the internet, including emoji, kaomoji, and all their combinations.]

At each step, ChatGPT generates a list of words with probabilities. But which word should it choose to add to the article (or other content) it's writing? One might think it should pick the "highest-ranked" word (the one with the highest "probability"). But here, a bit of magic starts to creep in. For some reason (which we may one day understand scientifically), if we always pick the highest-ranked word, we usually get a very "flat" article that never seems to "show any creativity" (and sometimes even repeats itself). But if we occasionally (randomly) pick a lower-ranked word, we get a "more interesting" article.

This randomness means that if we use the same prompt multiple times, we may get different articles each time. And, in line with this magical thinking [Note: like alchemy, where mysterious recipes produce surprisingly good results], there is a specific parameter called "temperature" that controls how often lower-ranked words are used. For article generation, we find that a temperature of 0.8 works best. (It's worth emphasizing that no "theory" is used here—it's purely empirical. For example, the concept of "temperature" here corresponds to the temperature in exponential distributions familiar from statistical physics, but as far as we know, there is no "physical" connection between the two.)

Before moving on, I should explain that for demonstration purposes, most of this article will not use the full ChatGPT system; instead, I'll often use the simpler GPT-2 system, which has the advantage of being small enough to run on a standard desktop computer. So I'll be able to provide explicit Wolfram Language code for almost everything I show, which you can run immediately on your computer.

For example, here's how to get the probability table above. First, we need to obtain a "language model" similar to ChatGPT—it's a neural network:

[Figure]

This model is GPT-2; ChatGPT is currently based on GPT-3.5.

Later, we'll dive into this neural network and discuss how it works. But for now, we can treat this "network model" as a black box [Note: meaning we only care about input and output, and know nothing about the intermediate process] and apply it to our existing text, then ask the model for the top 5 most probable words:

[Figure]

We can convert the result into a data table for clarity:

[Figure]

What happens if we repeatedly "apply the model," each time adding the word with the highest probability (specified in the code as the model's "strategy" for generating text)?

[Figure]

What happens if we continue? In this case ("zero temperature"), the result quickly becomes chaotic and repetitive:

[Figure]

But what if, instead of always choosing the "best" word, we sometimes randomly choose a "non-best" word (with "randomness" corresponding to a temperature of 0.8)? Again, we can construct text:

[Figure]

Each time we do this, the random choices are different, and the text varies—as shown in these 5 examples:

[Figure]

It's worth noting that at each step, when generating the probability distribution for the next word, although there are many possible "next words" to choose from (at temperature 0.8), [if we sort all next words by probability] their probabilities drop quickly as the rank increases (the straight line on this log-log plot corresponds to an n^–1 "power law" decay, which is a general statistical feature of language [Note: In human natural language, the probability distribution of possible next words following a given utterance follows a power law distribution]):

[Figure]

What happens if we continue? Here's a random example. It's better than the highest-probability (zero temperature) case, but still a bit odd:

[Figure]

This was done with the simplest GPT-2 model (from 2019). Using a newer and larger GPT-3 model yields better results. Here is the highest-probability (zero temperature) text generated with the same "prompt" but using the largest GPT-3 model:

[Figure]

Here is a random example [generated by GPT-3] with "temperature 0.8":

[Figure]

So where do these probabilities come from?

ChatGPT always selects the next word based on probabilities. But where do these probabilities come from? Let's start with a simpler problem. Consider generating English text one letter at a time (instead of words). How do we calculate the probability of each letter?

We can simply sample English text and count how often different letters appear. For example, here are the letter counts from the Wikipedia article on "cats" [note: the article titled "cat", similarly below]:

[image]

And here's the same thing done for "dogs":

[image]

The results are similar, but not identical (the "o" is certainly more common in the "dogs" article because it appears in the word "dog" itself). However, if we sample enough English text, we can expect to get fairly consistent results:

[image]

Here's an example of generating a sequence of letters based solely on these probabilities:

[image]

We can break this into "words" by adding a space as a "letter" with a certain probability:

[image]

We can generate better "words" by forcing the distribution of "word lengths" to match that of English:

[image]

We don't get any "actual words" here, but the result looks slightly better. However, to go further, we need to do more than just randomly pick each letter independently. For example, we know that if there's a "q", the next letter is almost certainly "u".

Here's a plot of letter probabilities:

[image]

And here's a plot showing the probabilities of letter pairs ("2-grams") in typical English text [note: two consecutive letters as a unit, the probability of that unit occurring]. Possible first letters are shown at the top of the page, and second letters on the left:

[image]

Here we see, for example, that the "q" column is blank (zero probability) except for the "u" row. So now, instead of generating our "words" one letter at a time, we generate them two letters at a time using these "2-gram" probabilities [i.e., each time, based on the last letter, use the conditional probability to generate the next letter]. Here's an example of the result, which happens to include some "actual words":

[image]

With enough English text, we can get fairly accurate estimates not only of probabilities for single letters or letter pairs (2-grams), but also for longer sequences of letters. If we generate "random words" using progressively longer n-gram probabilities [note: the probability distribution of the nth letter depends on the previous n-1 letters], we find they become more and more "real":

[image]

But now suppose—much like what ChatGPT does—we work with whole words instead of letters. There are about 40,000 common words in English. By looking at a large amount of English text (e.g., millions of books totaling hundreds of billions of words), we can estimate the frequency of each word. Using this, we can start generating "sentences" where each word is independently chosen at random with the same probability it appears in the corpus. Here's an example:

[image]

Clearly, this is nonsense. So how do we improve? Just like with letters, we can start considering not just probabilities of single words, but also pairs or longer n-gram word probabilities. For a pair of words, here are 5 examples starting from the word "cat":

[image]

This looks more "reasonable". We can imagine that if we could use long enough n-grams, we would essentially "get a ChatGPT"—meaning we'd have something capable of generating article-length sequences of words with the "correct overall article probability". But the problem is: there isn't enough text for us to infer these probabilities.

There may be hundreds of billions of words in web crawls; another tens of billions in digitized books. But for 40,000 common words, even the possible bigrams number 1.6 billion, and trigrams number 60 trillion. So we cannot estimate the probabilities of all these possibilities from existing text. And when we get to "article snippets" of 20 words, the number of possibilities is larger than the number of particles in the universe, so in a sense, they can never all be written down.

So what do we do? The big idea is to build a model that lets us estimate the probability that a sequence should occur, even if we have never explicitly seen that sequence in the text corpus we looked at. And the core of ChatGPT is precisely such a model, known as a "large language model" (LLM), designed to do a very good job at estimating these kinds of probabilities.

What Is a Model?

Suppose you want to know (just as Galileo did in the late 1500s) how long it takes for a cannonball dropped from each floor of the Leaning Tower of Pisa to hit the ground. Well, you could measure it at each height and make a table of results. Or you could do the essence of theoretical science: make a model that can compute the answer, rather than just measuring and memorizing each individual case.

Let's imagine we have (somewhat idealized) data for how long it takes for a cannonball dropped from various floors to fall:

[Figure]

Without explicit data, how can we determine how long it takes to fall? In this particular case, we can use known physical laws to calculate it. But suppose we only have the data, and don't know the underlying rules that govern it. Then we could make a mathematical guess, perhaps suggesting we should use a straight line as a model:

[Figure]

We could choose various different lines. But the one that, on average, comes closest to our given data is this. And from this line we can estimate the falling time for any floor.

How do we know to try a straight line here? In some sense, we don't. Using a straight line is mathematically simple, and we're used to the fact that lots of measured data follows simple straight-line patterns. Of course, we could try more complex mathematical approaches, like a + bx + cx², and in this case we could fit the data even better:

[Figure]

However, things can go wrong. For instance, here's the best fit we get using a + b / x + c sin(x):

[Figure]

The key thing to understand is that there's never a "model with no model." Whatever model you use has some particular underlying structure, then a certain set of "knobs to turn" (i.e., parameters you can set) to fit your data. In the case of ChatGPT, many such "knobs" are used—in fact, 175 billion of them.

Yet the model behind ChatGPT "only" needs that many parameters to be a "good enough" model for computing the probability of the next word, even capable of generating coherent articles thousands of words long for us. To put that in perspective: the number of possible trigrams of words is already 60 trillion, and the number of possible combinations of 1,000 words (40000^1000) makes many "astronomical numbers" (like the number of atoms in the observable universe, often used for comparison) "pale in comparison." Against that, 175 billion parameters really is tiny.

Models of Human Tasks

The examples above involve making models that fit numerical data—data that basically comes from simple physics, where for centuries we've known that "simple math [Note: meaning math that can be written out on a few pages and applies generally—in this sense, things like "Maxwell's equations" are simple] works [for modeling many physical phenomena]." But for ChatGPT, we have to make a model of human language text, something like what the human brain produces. For such a thing, we don't have anything like "simple math." So what might its model look like?

Before talking about language, let's talk about another humanlike task: image recognition. As a simple example of this problem, let's consider images of digits (a classic machine learning example):

[Figure]

One thing we could do is obtain a bunch of sample images for each digit:

[Figure]

Then, to find out whether our input image corresponds to a particular digit, we could do explicit pixel-by-pixel comparison with the samples we already have. But as humans we certainly do better, because even when handwritten digits come with all sorts of modifications and distortions, we can still recognize them:

[Figure]

When we made a model for numerical data earlier, we could take the numerical value x we got, and compute a + bx for particular values of a and b. So if we treat the grayscale value of each pixel here as variables x_i, is there some function that takes all these variables as input and, when evaluated, tells us what digit the image is? As it turns out, such functions can be constructed. Not surprisingly, this function isn't particularly simple—a typical function might involve around half a million mathematical operations.

But the end result is that if we feed the pixel collection of an image into this function, we'll get the digit that corresponds to that image. Later, we'll discuss how to construct such a function and the idea of neural networks. But for now, let's treat this function as a black box: we feed in an image of a handwritten digit (as an array of pixel values) and we get out the corresponding digit:

[Figure]

But what's actually happening here? Let's progressively blur the digit. For a while, our function still "recognizes" it, and we call it a "2." But soon it "loses" it and starts giving "wrong" results:

[Figure]

Why do we call these "wrong" results? In this case, we already knew in advance that all our images were obtained by blurring a "2." But if our goal is to make a model of "how humans recognize images," the real question to ask is: if we don't know the source of the image, what would a human do?

If the results of our function generally agree with how humans would see things, then we have a "good model." And there's a nontrivial scientific fact that, in image recognition tasks like this, we now basically know how to construct functions that can do this.

Can we "mathematically prove" their effectiveness? No. Because to do that, we'd need a mathematical theory of how human cognition and perception work. Take this image of a "2" and change a few pixels—imagine if just a few pixels were "out of place," we should still recognize it as a "2." But to what extent? This is a question about human visual perception. If the one answering "does this image represent the same thing as before" weren't a human, but a bee or an octopus, the answer might be different—and for an imagined alien, it could be entirely different.

Neural Networks

So how does a typical model used for tasks like image recognition actually work? The most successful and popular approach today is to use neural networks. A neural network can be thought of as a simple idealization of how the brain seems to work.

The human brain has about 100 billion neurons (nerve cells), each of which can produce up to a thousand electrical impulses per second. These neurons are connected to each other in a complex network, with each neuron having branch-like structures that allow it to send electrical signals to other neurons. Each connection has a different weight, and whether a neuron fires an impulse at a given moment roughly depends on the impulses it receives from other neurons, with different connections having different effects on the firing.

When we "see an image," photons from the image fall onto "photoreceptor" cells at the back of the eye, generating electrical signals in nerve cells. These nerve cells connect to other nerve cells, and eventually the signal passes through an entire sequence of neuron layers. It is in this process that we "recognize" the image, ultimately "forming" the idea that we "see a 2" (and perhaps eventually doing something like saying "2" out loud).

The "black-box" function from the previous section is such a "mathematized" neural network. It happens to have 11 layers (though only 4 "core layers" [Note: layers containing the main weights]):

[Figure]

How does a neural network "recognize things"? The key lies in the concept of "attractors." Let's imagine we have images of handwritten 1s and 2s:

[Figure]

We want all the 1s to be "attracted to one place" and all the 2s to be "attracted to another place." In other words, if an image is "closer to a 1," we want it to end up in the "1 place," and vice versa.

A simple analogy: suppose we have certain positions in a plane, represented by points (in real life, they might be coffee shop locations). Then we can imagine that starting from any point on the plane, we always want to end up at the nearest point (i.e., we always go to the nearest coffee shop). We can represent this by dividing the plane into regions ("basins of attraction"), separated by idealized "watersheds":

[Figure]

We can think of this as a kind of "recognition task," where instead of identifying which digit is most similar, we directly look at which point a given point is closest to. (In the "Voronoi diagram" shown here, it separates points in two-dimensional Euclidean space; the digit recognition task can be thought of as very similar, except the space is not 2-dimensional, but a 784-dimensional space formed by the grayscale values of all pixels in each image.)

So how do we make a neural network "perform a recognition task"? Let's consider this very simple example:

[Figure]

Our goal is to transform the "input" corresponding to the position {x, y} into one of the three nearest points. In other words, we want the neural network to compute a function of {x, y} as follows:

[Figure]

How do we achieve this using a neural network? A neural network is a collection of connections made up of idealized "neurons," usually arranged in layers [Note: arranged hierarchically, meaning neurons within a layer are not connected to each other, and each layer is only connected to the layer above and below it]. A simple example looks like this:

[Figure]

Each "neuron" is actually set up to evaluate a simple numerical function [Note: i.e., it performs a simple operation, like summing inputs]. To "use" the network, we simply input numbers at the top (like our coordinates x and y), then at each layer, neurons "evaluate their function" and pass the result forward ["forward" meaning downward], eventually producing a final result at the bottom:

[Figure]

In the traditional (biologically inspired) setup, each neuron has a certain number of "input connections" from neurons in the previous layer, and each connection is assigned a "weight" (which can be positive or negative). The value of a given neuron is determined by multiplying the values of "previous layer neurons" by their corresponding weights, summing them, adding a constant, and finally applying a "threshold" (or "activation") function. Mathematically, if a neuron has inputs x = {x1, x2, …}, then this neuron computes f [w * x + b], where the weights w and constant b are typically chosen differently for each neuron in the network; the function f is usually the same.

Computing w * x + b is just a matter of matrix multiplication and addition. The "activation function" f introduces nonlinearity (ultimately leading to nontrivial behavior). There are various activation functions available; here we will only use ReLU:

[Figure]

For each task we want the neural network to perform (or equivalently, for each overall function we want to evaluate), we will have a different choice of weights. (As we will discuss later, these weights are typically determined by "training" the neural network using machine learning, i.e., from examples of the desired output.)

Ultimately, each neural network corresponds to some overall mathematical function—though it may be difficult to write down. For the example above, it would be:

[Figure]

ChatGPT's neural network also corresponds to such a mathematical function—but with billions of weights.

Let's go back to a single neuron. Here are various functions formed by a neuron with two inputs (representing coordinates x and y) using various weights and constants (and ReLU as the activation function):

[Figure]

This is what the larger network above computes:

[Figure]

It's not exactly "correct," but it's close to the "nearest point" function we showed above.

Let's see what happens with other neural networks. In each case, as we'll explain later, we use machine learning to find the best choice of weights. We then show here what the neural network computes with those weights:

[Figure]

Larger networks 【more weights, more complex structure】 generally approximate our target function better. At the center of each attractor basin, we usually get exactly the answer we want. But at the boundaries 【where the colors interleave】—where the neural network "has trouble deciding"—things can get messier.

For this simple "recognition task" that can be easily expressed mathematically, the "correct answer" is obvious. But in the problem of recognizing handwritten digits, the situation is less clear. What if someone writes a "2" that looks like a "7"? Nevertheless, in most cases the neural network can still distinguish digits well—which gives an indication 【to be continued】:

[Figure]

Can we explain "mathematically" how the network distinguishes digits? 【Since the neural network is so complex, writing out the function it corresponds to on paper might consume millions of pages,】 so in practice, no. But we know that a neural network simply "does what a neural network does," and it turns out this usually aligns fairly well with how humans make distinctions.

Let's look at a more detailed example. Suppose we have images of cats and dogs, and we have a neural network trained to distinguish between them. Here's what it might do on a few examples:

[Figure]

For this kind of problem, the "correct answer" is even less clear: for example, a cat dressed up as a dog—is that, image-wise, a cat or a dog? But regardless of the input, the neural network will produce an answer. And it turns out it does so in a fairly consistent way that resembles what humans do. As I said before, this is not something we can derive "from first principles" 【here meaning that even if you completely understand the role of every single parameter in the neural network, it's still not enough to let you "understand" or intuitively grasp why the neural network classifies correctly—you'll just see that very complex mathematical operations produce the right result】. At least in some domains, its discovery is empirical 【meaning that people just know through extensive experimentation that neural networks work well, but don't understand the underlying principles】. But this is one of the key reasons neural networks are useful: they somehow capture a "human-like" way of doing things.

If you look at a picture of a cat yourself and someone asks, "Why is that a cat?" you might say: "I can see its pointy ears and so on." But it's hard to explain how you recognize the image as a cat, because the elements of a cat are clearly not just pointy ears—many other animals have those too. Yet your brain just gives you the answer directly. However, for a brain, there's no way (at least not yet) to "go inside" and see how it makes its decision. With (artificial) neural networks, when you show a picture of a cat, it's easy to go study what each "neuron" is doing. But even doing basic visualization of all the neurons is usually extremely difficult.

In the final network we used for the "nearest point" problem above, there are 17 neurons. In the network for recognizing handwritten digits, there are 2,190. In the network we used for recognizing cats and dogs, there are 60,650. It's usually hard to imagine what a 60,650-dimensional space looks like. But because this is a network that processes images, many of its layers of neurons are organized into arrays, much like the array of pixels in the input it's looking at.

If we take a typical picture of a cat

[Figure]

then we can extract the collection of "images" output by the first layer of neurons representing the state of the first layer, many of which we can easily interpret as something like "cat with no background" or "outline of a cat":

[Figure]

By the time we look at the output of layer 10, it's already hard to interpret what's going on:

[Figure]

But in general, we can say that the neural network "picks out certain features" (perhaps pointy ears being one of them) and uses these features to determine what's in the image. But features like "pointy ears" are features we have names for; in most cases these features have neither names nor easy descriptions 【for "cat" specifically, most of the cat's features don't have well-known names—for instance, "some kind of shape formed by the cat's overall silhouette"】.

Does our brain use similar features? In most cases we don't know. What we do know is that, like the early layers of the neural networks we're showing here, the early layers seem to pick out certain features of the image (such as edges of objects), similar to the features selected in the first stages of visual processing in our brain.

But suppose we want a "theory of cat recognition" from the neural network. We can say: "Look, this particular network can recognize cats." The specific network immediately gives us some sense of "how hard the problem is" (e.g., how many neurons or layers it requires). But at least so far, we don't have a precise way, expressed in language, to describe what the network is doing. Perhaps this is because it is truly computationally irreducible 【note: computational irreducibility is a concept repeatedly mentioned in this article, meaning that certain problems are fundamentally impossible to solve with simple mathematical or computational shortcuts—for instance, by summarizing them into a form that's easy for humans to understand or compute】, and beyond tracking each step of the neural network's computation one by one, there's no general method to figure out how it operates. Or perhaps it's because we haven't yet found a systematic scientific framework or set of laws to explain the nature of neural networks.

When we talk about using ChatGPT to generate language, we'll encounter the same issue. Again, it's currently unclear whether there's a way to "summarize how it operates." But the richness and detail of language (and our experience with it) may make it easier for us to do further research than with images.

Machine Learning and the Training of Neural Networks

So far, we have been discussing neural networks that "already know" how to perform specific tasks. But neural networks (like our brains) can not only execute various tasks—they can also be progressively "trained" from examples to perform those tasks better.

When we build a neural network to distinguish cats from dogs, we don't actually have to write a program that (for instance) explicitly finds cat whiskers. Instead, we simply show it a large number of example images of cats and dogs, and let the network "machine learn" from them how to tell them apart.

The key point is that a trained network "generalizes" from the specific examples it has seen to a general mode of operation. As we saw earlier, the network doesn't just recognize the specific pixel pattern of a particular cat image it was shown; rather, it learns in some way the general properties of "cat-ness" to distinguish images, not limited to local pixel patterns.

How does neural network training work? Essentially, we are always trying to find the weights that allow the network to successfully replicate the examples we give it. Then we rely on the network to "interpolate" (or "generalize") between those examples in a "reasonable" way.

Let's look at a simpler problem than the earlier nearest-point problem. We'll just try to get a neural network to learn the following function:

[Figure]

For this task, we need a network with one input and one output, for example:

[Figure]

But what weights should we use? With every possible set of weights, the neural network computes some function. For instance, here are the results using a few randomly chosen sets of weights:

[Figure]

We can clearly see that in these cases, it fails to replicate the function we want. So how do we find the weights that can replicate that function?

The basic idea is to provide a large number of "input → output" examples for "learning," and then try to find weights that replicate those examples. Here are the results of learning with gradually increasing numbers of examples:

[Figure]

10,000 to 10,000,000 examples

At each stage of "training," the weights in the network are gradually adjusted, and we see that we eventually get a network that successfully replicates the desired function. So how do we adjust the weights? The basic idea is to look at each stage at "how far we are from the desired function," and then update the weights in a way that brings us closer.

We typically compute a "loss function" (sometimes called a "cost function") to know "how far we are from the target." Here we use a simple (L2) loss function, which is just the sum of the squares of the differences between the obtained values and the true values. We can see that as the training process proceeds, the loss gradually decreases (following different "learning curves" for different tasks), until we reach a point where the network (at least to a good approximation) successfully replicates the desired function:

[Figure]

Okay, now the last key part to explain is how to adjust the weights to reduce the loss function. As we said, the loss function gives the "distance" between the values we get (through the neural network) and the true values. But "the values we get" are determined at each stage by the current version of the neural network and the weights within it. Suppose the weights are variables, say w_i, and we want to know how to adjust these variables to minimize the loss (which depends on them).

For example, imagine (greatly simplifying a typical neural network used in practice) we have only two weights w1 and w2. Then we might get a loss function that looks something like this (contour plot / heat map):

[Figure]

Numerical analysis provides various techniques to find the minimum in such a situation. A typical approach is to start from some initial w1 and w2, and then proceed step by step along the path of steepest descent:

[Figure]

Just like water flowing down a mountain, this process is only guaranteed to eventually reach some local minimum of the surface (a "lake in a valley"); it may not reach the ultimate global minimum.

For general functions, it's not clear how to find the path of steepest descent on the "weight landscape." But calculus comes to our rescue. As mentioned above, we can always think of a neural network as computing a mathematical function—it depends on its inputs and weights. Now consider taking derivatives with respect to these weights [of course, the function needs to be "almost everywhere differentiable," but that's not hard to achieve]. It turns out that the chain rule of calculus actually lets us "unfold" the operations performed by successive layers in the neural network [i.e., we can compute derivatives layer by layer]. The result is that we can, in a local approximation sense, "invert" the operations of the neural network and gradually find weights that minimize the loss associated with the output.

The figure above shows the minimization process in the unrealistic case of only 2 weights. But it turns out that even with many more weights (ChatGPT uses 175 billion), it is still possible to perform minimization, at least to some level of approximation. In fact, the major breakthrough in "deep learning" around 2011 was related to the discovery that, in a certain sense, it can be easier to (at least approximately) minimize when there are a large number of weights, whereas with relatively few weights it is easier to get stuck in local minima ("mountain lakes") and fail to find a "way out" [Note: in fact, many theories have confirmed that when the dimensionality is high enough—i.e., when there are enough weights—local minima are not the main problem; the idea that "local minima are the main culprit for optimization non-convergence" is a low-dimensional illusion].

It is worth pointing out that, in general, there are many different sets of weights that can yield neural networks with almost identical performance. In practice, real neural network training involves many random choices that lead to "different but equivalent solutions," as shown below:

[Figure]

However, each such "different solution" will exhibit slightly different behavior. If we ask for "extrapolation" beyond the region where we give training examples, we may get strikingly different results:

[Figure]

But which of these results is the "correct" one? There is actually no way to say. They are all "consistent with the observed data." Their extrapolated outcomes are all mathematically reasonable. For example, consider the pattern: 1, 2, 4, 8, 16, ? — the last number could be many different things, each of which can be justified by a "reasonable reason." Some, however, may strike us humans as "more reasonable" than others.

The Practice and Craft of Neural Network Training

Especially over the past decade, considerable progress has been made in the art of training neural networks. Yes, it is essentially an art 【now often referred to as "炼丹" (炼丹, literally "refining the elixir," a colloquial Chinese term for the trial-and-error process of tuning models)】. Sometimes, particularly in hindsight, our successful applications can be seen to rest on certain "scientific explanations." But most of what we know has been discovered through trial and error, with ideas and tricks continually being added, gradually building up an important craft tradition of working with neural networks.

Here are a few key pieces. First, which neural network architecture should be used for a particular task. Then comes a key question: how to obtain the data needed to train the network. Increasingly today, we don't need to train a network from scratch: a new network can directly incorporate another already-trained network, or at least make use of that network to generate additional training examples for itself.

One might think that for each particular task, a different neural network architecture is required. But it turns out that the same architecture often seems to work, even for tasks that appear quite different. In some ways, this is reminiscent of the idea of universal computation (and my Principle of Computational Equivalence), but, as I will discuss later, I think it reflects more the fact that the tasks we typically try to get neural networks to do are "human-like," and neural networks can capture fairly general "human-like processes."

In the early days of neural networks, people tended to think one should "make the job as easy as possible for the network." So, for instance, in converting speech to text, it was thought that one should first analyze the audio, break it down into phonemes, etc. But it turned out that—at least for "human-like tasks"—it is usually better to simply train the network on the "end-to-end problem" 【that is, without heavy-handed intervention in data preprocessing and feature engineering—feeding the network relatively raw data directly】, and let it automatically "discover" the necessary intermediate features, encodings, and so on.

There was also the idea that one should introduce sophisticated, hand-crafted components into a neural network to have it "explicitly implement particular algorithmic ideas." But again, this is generally not worth doing; it is better instead to work with very simple components and let them "self-organize" (though usually in ways we cannot understand) into what amounts to an equivalent of those algorithmic ideas.

This is not to say that there are no relevant "structural ideas" for neural networks. For example, in the early stages of processing images, it has proved useful to use 2D arrays of neurons with local connections. And connection patterns that "look back in a sequence" 【that is, where neurons later in a layer connect back to (review) earlier neurons in the previous layer, much like "predicting later words from earlier words"】 seem to be useful—as we will see later—for handling things like human language in ChatGPT.

But like computers, a crucial feature of neural networks is that ultimately they are just manipulating data. Current neural networks—and present-day neural network training methods—are in some sense only specialized at handling arrays of numbers. But in the course of processing, these arrays can be completely rearranged and reshaped. For instance, the network we used above to recognize digits begins with a 2D "image-like" input array, quickly "thickens" into many channels, but then "concentrates" through successive layers down to a 1D array, ultimately containing elements that represent the different possible output digits:

[Figure]

But how does one determine how large a neural network needs to be for a particular task? It is, in some sense, more art than science. In part, the key is to estimate "the difficulty of the task." But for human-like tasks, this is usually hard to estimate. It is possible that there is a systematic way to complete the task in a very "mechanical" fashion using a computer. But it is hard to know whether there are what might be called tricks or shortcuts that allow one to achieve the task at "human level" with much less effort. Take games as an example: to play a game "mechanically" might require enumerating a huge game tree; but there may be simpler ("heuristic") methods that achieve "human-level play."

When working with tiny neural networks and simple tasks, one can sometimes see clearly that "this is not going to work." For example, here are the best results achievable with a few small neural networks on the task from the previous section:

[Figure]

What we see is that if the neural network is too small, it simply cannot reproduce the function we want, no matter how hard we try. But once its size exceeds a certain threshold, the problem can usually be solved, provided we train with enough examples for long enough. Also, these pictures illustrate a trick in neural networks: by deliberately designing a "bottleneck" in the middle—forcing all information to pass through a smaller number of intermediate neurons—one can compress the size of the network without greatly affecting performance. (It is worth mentioning that a neural network with "no intermediate layers" (the so-called "perceptron") can only learn functions that are essentially linear. But as long as there is at least one intermediate layer, in principle any function can be approximated as closely as desired, provided we have enough neurons—although to make training feasible, some form of regularization or normalization is usually required.)

Alright, assuming we have determined a neural network architecture, the next question is how to obtain the data needed for training. Many practical challenges in neural networks and machine learning revolve around acquiring or preparing the necessary training data. In many cases ("supervised learning"), we want explicit examples of inputs and desired outputs. For instance, we might want to label the content or other attributes of images. Perhaps we need to explicitly put in some effort to label them [e.g., a data annotation team]. But often, we can leverage work that has already been done, or use it as a proxy to solve the problem. For example, we might use the alt text of web images [the prompt text for images on a webpage] to label the image's information. Or in a different domain, we can use closed captions created for videos. Or in language translation training, we can use parallel versions of web pages or other documents that exist in different languages.

How much data needs to be shown to train a neural network? Again, it's hard to estimate from first principles. Of course, by using "transfer learning" to "transfer" a list of important features already learned in another network, the requirement can be greatly reduced. But generally speaking, neural networks need to "see many examples" to train well. For some tasks, repeated training examples can be very important. In fact, a standard strategy is to repeatedly show all existing examples to the neural network. In each "training epoch," the neural network will be in at least a slightly different state, so "reminding it of a specific example" is useful for making it "remember that example." (Yes, this is similar to the utility of repetition in human memory.)

But simply repeating the same example is not enough; we also need to show the neural network variations of the examples. And this is one of the tricks of neural networks: the variations in data augmentation don't need to be complicated. Even slight modifications to images using basic image processing can make them "as good as new images" for neural network training. Similarly, when we run out of data like actual videos needed to train a self-driving car, we can use simulated data from an environment similar to a game simulation, without needing all the details of real-world scenes.

What about models like ChatGPT? It has a nice feature: it can perform "unsupervised learning," which makes obtaining the examples needed for training much easier. Recall that ChatGPT's basic task is to figure out how to continue a given text fragment. So, to get training examples, we simply take a piece of text, mask its ending, and use it as the "input to be trained," with the "output" being the complete unmasked text. We'll discuss this further later, but the main point is: unlike learning what's in an image, ChatGPT doesn't need "explicit tokens"; ChatGPT can actually learn directly from given text examples.

The actual learning process of a neural network ultimately boils down to determining which weights best capture the given training examples. There are various detailed choices and "hyperparameter settings" (since weights can be considered "parameters") to adjust this. There are different choices of loss functions (sum of squares, sum of absolute values, etc.). There are different methods for minimizing the loss (how far to move in weight space per step, etc.). Then there are issues like the size of each consecutive learning update "batch" to reduce the loss to be minimized. We can apply machine learning (e.g., in the Wolfram Language) to automate machine learning and automatically set hyperparameters, etc. But ultimately, the entire training process can be characterized by observing how the loss gradually decreases (as shown in this Wolfram product display).

[Figure]

One typically sees the loss decrease over time, but eventually plateau at some value. If that value is small enough, the training can be considered successful; otherwise, it might mean that the network architecture should be changed.

Can we know how long it takes for the "learning curve" to plateau? As with many other things, there seems to be an approximate power-law scaling relationship, depending on the size of the neural network and the amount of data used. But the overall conclusion is that training neural networks is difficult and requires a lot of computational work. In fact, the vast majority of the work involves operations on arrays of numbers [e.g., matrix multiplication], which is what GPUs are good at. That's why neural network training is often limited by GPU availability.

In the future, will there be fundamentally better ways to train neural networks, or algorithms that are essentially as good as neural networks but more efficient [Note: The original ChatGPT text here says "or generally do what neural networks do"]? I think almost certainly yes. The basic idea of neural networks is to use a large number of simple (essentially identical) components to create a flexible "computational structure" and enable that "structure" to be incrementally modified to learn from examples. In current neural networks, people actually use ideas from calculus—applied to real numbers—to perform this incremental modification. But it's increasingly clear that high-precision numbers are not important; even with current methods, 8 bits or fewer might be sufficient.

[A more radical approach is] to use computational systems like cellular automata, operating in parallel on many individual bits. It has never been clear how to perform such incremental modification, but there is no reason to think it's impossible. In fact, just like the "deep learning breakthrough of 2012," such incremental modification might become easier in more complex situations.

Neural networks (perhaps somewhat like brains) are set up with essentially fixed networks of neurons, while the strengths of the connections between them ("weights") are modified. (Perhaps in at least young brains, a large number of entirely new connections can also grow.) However, while this may be a convenient setup for biology, it is far from clear whether it is close to the optimal implementation method for what we need. Things involving progressive network rewriting (perhaps similar to our Wolfram Physics Project) might ultimately be better.

Even within the framework of existing neural networks, there remains a key limitation: the current approach to training neural networks is essentially sequential, in batches (mini-batches), where the influence of each batch of examples is back-propagated to update the weights. In practice, even considering GPUs, current computer hardware is mostly "idle" during training, with only a fraction of the time spent on updates. In a sense, this is because our current computers tend to have memory separate from the CPU (or GPU). But in the brain, this may be different—every "memory element" (i.e., neuron) is also a potentially active computational unit. If future computer hardware could be designed in this way, training could become much more efficient.

"A sufficiently large network can do anything!"

The capabilities of technologies like ChatGPT are so impressive that one might think that if we just "keep training" larger and larger neural networks, they will eventually be able to "do anything." If one focuses on things that humans can intuitively think about, this is likely correct. But the science of the past few centuries teaches us that there are things that can be deduced through formal processes, yet are difficult for humans to solve through direct thinking.

Non-trivial mathematics is a good example. But in general, this is essentially equivalent to computation. And the ultimate issue is the phenomenon of computational irreducibility. Some computations may require many steps to complete, but they can in fact be "reduced" to very direct forms. However, the discovery of computational irreducibility means that this approach does not always work. Instead, some processes, like the example below [one-dimensional cellular automaton], inevitably require tracking every step of the computation to figure out what happens:

[Figure]

The things we typically handle with our brains may be selectively avoiding computational irreducibility. Performing (irreducible) mathematical operations in the brain requires special effort. In fact, "thinking through" the steps of any non-trivial program operation in one's mind (e.g., purely by imagining the process in the mind, generating the SHA-256 digest of a file) is practically impossible.

But we have computers. With computers, we can easily handle long, computationally irreducible tasks. The key point is that, in general, there is no shortcut for these things.

Yes, we can memorize many specific examples of what happens in a particular computational system. Perhaps we can even see some ("computationally reducible") patterns that allow us to make a few generalizations. But computational irreducibility means we can never guarantee that no surprises will occur; only by explicitly performing the computation can we tell what actually happens in any specific case.

Ultimately, there is a fundamental tension between learning and computational irreducibility. Learning is essentially compressing data by exploiting regularities. But computational irreducibility means that no matter how many regularities are grasped, there are data that cannot be compressed.

In fact, one can imagine building small computational devices (such as cellular automata or Turing machines) within a trainable system. These devices could serve as good "tools" for neural networks, just as Wolfram|Alpha can be a good tool for ChatGPT. However, computational irreducibility means we cannot expect to "go inside" these devices and make them learn.

In other words, there is a fundamental trade-off between capability and trainability: the more you want a system to fully utilize its computational power, the more it will exhibit computational irreducibility and the harder it will be to train. Conversely, the easier it is to train, the harder it is to perform complex computations.

(For the current case of ChatGPT, the situation is actually much more extreme, because the neural network used to generate each output token is a purely "feedforward" network, with no loops, and therefore cannot use any non-trivial "control flow" for computation [e.g., cannot perform recursion or loops of arbitrary length].)

Of course, one might wonder whether being able to perform irreducible computation is really important. In fact, for most of human history, it was not particularly important. But our modern technological world is built on engineering that at least utilizes mathematical computation, and increasingly, more general computation. If we look at nature, it is full of irreducible computation—and we are slowly understanding how to simulate and harness these computations for our technological purposes.

Yes, neural networks can certainly detect regularities in nature that we can also easily notice with "unaided human thinking." But if we want to solve problems that belong to the realm of mathematics or computational science, neural networks will not be able to do so unless they effectively "use" an "ordinary" computational system as a tool.

However, there may be potential confusion here. In the past, there were many tasks—including writing essays—that we thought were "basically too hard" for computers. Now that we see models like ChatGPT accomplishing these tasks, we tend to suddenly think that computers must have become much more powerful—especially surpassing what they were already basically capable of (e.g., step-by-step computation of the behavior of systems like cellular automata).

But this is not the correct conclusion. Computationally irreducible processes remain computationally irreducible, and remain fundamentally difficult for computers, even if computers can easily compute each of their individual steps. Instead, we should conclude that tasks we believed computers could not accomplish—such as writing—are in fact computationally easier than we had imagined.

In other words, the reason neural networks are able to successfully write articles is that the problem of writing is, in some sense, computationally shallower than we thought. This, in a sense, advances our theories of how we humans accomplish things like writing, or process language in general.

If you have a large enough neural network, then yes, you might be able to do anything that humans can do easily. But you will not be able to capture the general capabilities of nature, nor the capabilities of the tools we have built from nature. It is these tools—both practical and conceptual—that have allowed us to transcend the boundaries of "pure human thought" in recent centuries and capture more of the physical and computational universe for human purposes.

The Concept of Embeddings

Neural networks (at least as they are currently set up) are fundamentally based on numbers. So if we want to use them to process things like text, we need a way to represent text with numbers. Of course, we can start by (just as ChatGPT does) assigning a numerical index to each word in a dictionary. But there is an important concept called "embedding." This more powerful idea is crucial to ChatGPT. We can think of an embedding as an attempt to represent the "essence" of something by an array of numbers, such that things that are "nearby" in our assessment are represented by nearby numbers.

For example, we can think of word embeddings as an attempt to lay out words in a kind of "meaning space," where words that are similar in meaning end up placed at nearby positions in the embedding. The actual embeddings used—for instance in ChatGPT—tend to involve arrays of high-dimensional numbers. But if we project them down to two dimensions, we can show how words are embedded in "meaning space":

[Figure]

We can see that these embeddings align with our everyday impressions. But how do we construct such embeddings? The rough idea is to look at a large body of text (here 5 billion words collected from the web) and see how "similar" different words are in the different contexts in which they appear. For example, "alligator" and "crocodile" can often be interchanged in sentences with similar meaning, which means they'll be placed near each other in the embedding. But "turnip" and "eagle" will not typically appear in sentences with similar meaning, so they will end up far apart in the embedding.

But how do we actually implement this with a neural network? Let's first talk about embeddings not for words, but for images. We want to find some way to characterize images by arrays such that "images we consider similar" are assigned similar arrays.

How do we decide that "images are similar"? If our images are pictures of handwritten digits, we can "consider two images similar" if they are the same digit. Earlier we discussed a trained neural network that can recognize handwritten digits. We can think of this neural network as being set up to sort images into 10 different bins in its final output, each bin corresponding to a digit.

But what if we "intercept" what is happening inside the neural network just before the final "this is a '4'" output? [That is, we obtain the neural network's "intermediate representation."] We can expect that inside the neural network there are numbers that characterize the image, indicating that the image "mostly looks like a '4' but is a bit like a '2'," or something along those lines. Our idea is to extract such numbers and use them as the elements of an embedding.

So here is the concept. Rather than directly trying to characterize "which image is close to which image," we instead consider a well-defined task (in this case digit recognition) for which we have clear training data, and then let the neural network implicitly do the work of "approximate judgment" as it carries out this task. So we don't have to talk explicitly about "closeness of images"; we just talk about the specific digit an image represents, and "leave it to the neural network" to implicitly determine what that implies for "image closeness."

So, for the digit recognition network, how does this work in practice? We can imagine the network as consisting of 11 successive layers, which we can summarize iconically as follows (with the activation function shown as a separate layer):

[Figure]

At the beginning, we feed the actual image into the first layer, represented as a 2D array of pixel values. Finally, from the last layer, we obtain an array of 10 values, which we can think of as representing the "certainty" that the neural network assigns to the image corresponding to each of the digits 0 through 9.

Feed the image

[Figure]

in and obtain the values of the neurons in that final layer:

[Figure]

In other words, at this point the neural network is "very certain" that the image is a 4—and to actually output the digit 4, we simply pick the position of the neuron with the largest value.

But what if we look one step back? The very last operation in the network is something called a softmax, which attempts to "force the output to represent certainties, or a probability distribution, for each digit" [ChatGPT originally translated this as "force certainty"]. But before that, the values of the neurons look like this:

[Figure]

The neuron representing "4" still has the highest value. However, the values of other neurons also contain information about the image. We can expect that these lists of numbers can, to some extent, characterize the "essence" of the image—thus providing something we can use as an embedding. For example, here, each "4" has a slightly different "signature" (or "feature embedding")—all completely distinct from "8":

[Image]

Here, we essentially used 10 numbers to characterize our image. But it's usually better to use more numbers. For instance, in our digit recognition network, by tapping into an earlier layer, we can obtain an array of 500 numbers. This could be a reasonable array to use as an "image embedding."

If we want to explicitly visualize the "image space" of handwritten digits, we need to perform "dimensionality reduction," projecting the 500-dimensional vector we obtained into 3D space, resulting in:

[Image]

We just discussed how to create image features (and thus embedding vectors) based on identifying image similarity by determining whether they correspond to the same handwritten digit (according to our training set). If we had a training set capable of recognizing which of 5,000 common objects (cats, dogs, chairs, etc.) each image belongs to, we could use the same method to generate image embedding vectors. In this way, we can create an image embedding vector "anchored" by common object identities, and then "generalize" based on the behavior of the neural network. If this behavior aligns with how humans perceive and interpret images, the embedding vector will ultimately become "reasonable" and useful for tasks requiring "human-like judgment."

So how do we apply the same approach to find word embeddings? The key is to start with a word task that is easy to train. The standard task is "word prediction." Imagine we have a blank: "the ___ cat." Based on a large text corpus (e.g., the text content of the Web), what are the probabilities of different words filling the blank? Or, given "___ black ___," what are the probabilities of different words "sandwiched" in between?

How do we set up this problem for a neural network? Ultimately, we must represent everything with numbers. One approach is to assign a unique number to each of the 50,000 common words in English. For example, "the" might be 914, and " cat" (with a preceding space) might be 3542. (These numbers are the actual ones used by GPT-2.) So for the "the ___ cat" problem, our input might be {914, 3542}. The output should be a list of 50,000 numbers that effectively give the probability for each possible "fill-in" word. Similarly, to find embeddings, we want to "intercept" the "internals" of the neural network "before it reaches its conclusion," and then obtain the list of numbers that appear there, which can be seen as numbers "describing each word."

What do these features look like? Over the past 10 years, a range of different systems (word2vec, GloVe, BERT, GPT, etc.) have been developed, each based on different neural network methods. But ultimately, all these systems describe words through lists of hundreds to thousands of numbers.

In their raw form, these "embedding vectors" are fairly meaningless. For example, here are the raw embedding vectors generated by GPT-2 for three specific words:

[Image]

If we measure the distances between these vectors, we can find the "closeness" between words. We will discuss the "cognitive" significance of such embeddings in more detail later. The point is that we have a useful way to convert words into "neural network-friendly" collections of numbers.

But in fact, we can go further: not only can we describe words through collections of numbers, but we can also use the same method to describe sequences of words, or even entire blocks of text. This is how ChatGPT works. It takes existing text as input and generates an embedding vector to represent it. Then, its goal is to find the probabilities of different words that might appear next. And it expresses its answer as a series of numbers that essentially give the probability for each possible word.

(Strictly speaking, ChatGPT does not process words but "tokens"—convenient linguistic units that can be whole words or just parts like "pre," "ing," or "ized." Using tokens makes it easier for ChatGPT to handle rare, compound, and non-English words, and sometimes it can even create new words.)

Inside ChatGPT

Alright, we’re finally ready to discuss what’s going on inside ChatGPT. Yes, at its core it’s a giant neural network—currently a version of the so-called GPT-3 network [GPT-3.5] with 175 billion weights. In many ways, this neural network is very similar to the other ones we’ve talked about. But it’s a neural network particularly tailored for natural language processing, and its most notable feature is a neural network architecture called the “transformer.”

In the first neural networks we discussed above, each neuron in any given layer was basically connected to every neuron in the previous layer (at least with some weight). But if you’re dealing with data that has a particular, known structure, such fully connected networks can be overkill. That’s why, in the early stages of processing images, it’s common to use so-called convolutional neural networks—where neurons are effectively laid out on a grid analogous to the pixels in an image, and only connected to neurons nearby on that grid.

The idea of transformers is to do something similar for sequences of tokens that make up text. But instead of just defining a fixed region in the sequence over which there can be connections, transformers introduce the notion of “attention”—and the idea of paying more attention to some parts of the sequence than others. Perhaps one day it will make sense to just start from a generic neural network and do all customization through training. But at least for now, it seems to be critical in practice to “modularize” things—as transformers do, and probably as our brains do.

So what does ChatGPT (or the GPT-3 network it’s based on) actually do? Remember, its overall goal is to continue text in a “reasonable” way, based on what it’s seen from the training it’s had (which consists of looking at billions of pages of text from the web, etc.). So at any given moment, it has a certain amount of text—and its goal is to come up with an appropriate choice for the next token.

It operates in three basic stages. First, it takes the sequence of tokens that corresponds to the text so far, and finds an embedding (i.e., an array of numbers) that represents them. Then it operates on this embedding—in a “standard neural network way,” with values “rippling through layers” of a successive network—to produce a new embedding (i.e., a new array of numbers). It then takes the last part of this array and generates an array of about 50,000 values that turn into probabilities for different possible next tokens. (Yes, it so happens that about the same number of tokens as there are common words in English are used, though only about 3,000 tokens are whole words, and the rest are fragments.)

The critical point is that every part of this pipeline is implemented by a neural network, whose weights are determined by end-to-end training of the network. In other words, effectively nothing except the overall architecture is “explicitly engineered”; everything is just “learned” from training data.

There are, however, plenty of details in the way the architecture is set up—reflecting all sorts of experience and neural net lore. And even though this definitely gets into the weeds, I think it’s useful to talk about some of those details, at least to give a sense of what goes into building something like ChatGPT.

First comes the embedding module. Here’s a sketch of it in Wolfram Language representation for GPT-2:

[图]

The input is a vector of n tokens (represented as in the previous section by integers from 1 to about 50,000). Each of these tokens is converted (by a single-layer neural net) into an embedding vector (of length 768 for GPT-2 and 12,288 for ChatGPT’s GPT-3). Meanwhile, there’s a “secondary pathway” that takes the sequence of (integer) positions of the tokens, and from these integers creates another embedding vector. Finally, the embedding vectors from the token value and the token position are added together—to produce the final sequence of embedding vectors from the embedding module.

Why does one just add the token-value and token-position embedding vectors together? I don’t think there’s any particular science to this. It’s just that various different things have been tried, and this is one that seems to work. And part of the lore of neural nets is that in some sense so long as the setup you have is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself.

Here’s what the embedding module does, operating on the string “hello hello hello hello hello hello hello hello hello hello bye bye bye bye bye bye bye bye bye bye”:

[图]

The elements of the embedding vector for each token are shown down the page, and across the page we see first a run of “hello” embeddings, followed by a run of “bye” ones. The second array above is the positional embedding—with its seemingly random structure just being what “happened to be learned” (in this case in GPT-2).

OK, so after the embedding module comes the “main event” of the transformer: a sequence of so-called “attention blocks” (12 for GPT-2, 96 for ChatGPT’s GPT-3). It’s all pretty complicated, and reminiscent of typical large hard-to-understand engineering systems, or, for that matter, biological systems. But anyway, here’s a schematic representation of a single “attention block” (for GPT-2):

[图]

Within each attention block, there is a collection of "attention heads" (12 in GPT-2, 96 in ChatGPT's GPT-3), each operating independently on a different block of values within the embedding vector. (Yes, we don't know the benefit of splitting the embedding vector into several parts or what the different parts mean—this is just one of the things that has been "discovered to work.")

So what do the attention heads do? Essentially, they are a way of "looking back" at the token sequence (i.e., the text generated so far) and packaging that "past content" in a form useful for finding the next token. In the first section above, we discussed using a 2-gram to select a word based on its immediate predecessor. The "attention" mechanism in transformers allows "attention" to even earlier words, potentially "capturing" and utilizing earlier words—for example, the way a verb can refer to a noun that appeared many words before it.

More specifically, what the attention heads do is recombine blocks of the embedding vectors associated with different tokens, with certain weights. So, for example, the "weight-recombination pattern for looking back over the whole token sequence" for the first attention block (one of 12 in GPT-2) on the "hello, bye" string above would be:

[Figure]

After being processed by the attention heads, the resulting "re-weighted embedding vector" (of length 768 for GPT-2, and 12,288 for ChatGPT's GPT-3) is passed through a standard "fully connected" neural network layer. It's hard to get a handle on what this layer is doing. But here's a plot of the 768×768 weight matrix it's using (here for GPT-2):

[Figure]

Taking a 64×64 moving average, some (random walk) structure begins to emerge:

[Figure]

What determines this structure? Ultimately, it is presumably some "neural-net encoding" of features of human language. But as of now, what those features might be is unknown. Effectively, we're "opening up ChatGPT's brain" (or at least GPT-2's) and discovering, yes, it is all very complicated there, and we don't understand it—even though ultimately it's producing recognizable human language.

OK, so after one attention block we've got a new embedding vector, which then gets passed through further attention blocks (12 in total for GPT-2, 96 for GPT-3). Each attention block has its own particular pattern of "attention" and "fully connected" weights. Here is the sequence of attention weights for the first attention head for the "hello, bye" input, for GPT-2:

[Figure]

And here's the (moving-averaged) "matrix" for the fully connected layer:

[Figure]

It's interesting that even though these "weight matrices" from different attention blocks look rather similar, the distribution of weight magnitudes can be different (and not always Gaussian):

[Figure]

So, after going through one attention block, what is the net effect of a transformer? Basically, it is to transform the original collection of embeddings for the token sequence into a final collection. And ChatGPT's specific way of doing this is to take the last embedding in this collection, and "decode" it to produce a probability list for the next token.

That's basically what's going on inside ChatGPT. It may look complicated (not least because of its many, inevitably somewhat "arbitrary engineering choices"); but the ultimate elements involved are really very simple. Because in the end all we're dealing with is a neural net made of "artificial neurons", each one just performing some simple operation on a set of numbers, combining them with certain weights.

The original input to ChatGPT is an array of numbers (the embedding vectors for the tokens so far); and when ChatGPT "runs" to produce a new token, what happens is that these numbers "ripple" through the layers of the neural net, with each neuron "doing its thing" and passing the result to the next layer of neurons. There's no loop or "going back". Everything is just "fed forward" through the network.

This is a very different setup from a typical computational system (such as a Turing machine), where results get "re-processed" repeatedly by the same computational elements. At least in producing a given output token, each computational element (i.e., neuron) here is used only once.

But there is in a sense still a kind of "outer loop" going on in ChatGPT, because when it generates a new token it always "reads" (i.e., takes as input) the entire sequence of tokens that have appeared so far, including the tokens ChatGPT itself has previously "written". We can think of this setup as meaning that ChatGPT—at least in its most superficial layer—is involved in a "feedback loop", where each iteration explicitly shows up as a token appearing in the text it generates.

But let's come back to the heart of ChatGPT: the neural net that's being repeatedly used to generate each token. In some sense it is very simple: just a whole collection of identical artificial neurons. Some parts of the network consist of ("fully connected") layers of neurons, where every neuron on a given layer is connected (with some weight) to every neuron on the previous layer. But particularly because of its transformer architecture, ChatGPT has more structure, with only certain neurons being connected across layers. (Of course, one could still say that "all neurons are connected"—just that some of the weights are zero.)

Moreover, some aspects of the neural network in ChatGPT are not naturally considered to consist solely of "homogeneous" layers. For example, as shown in the iconic summary above [i.e., the transformer architecture diagram], within the attention block, there are places with "multiple copies," each processed through a different "processing path," potentially involving a different number of layers, before being recombined. However, while this may be a convenient representation of what is happening, it can always, at least in principle, be thought of as "densely filled" layers [note: as in the original text], but with some weights set to zero.

If we look at the longest path in ChatGPT, it involves about 400 (key) layers—not a huge number in some respects [compared to other neural networks]. But it has millions of neurons, with a total of 175 billion connections, and therefore 175 billion weights. One thing to realize is that every time ChatGPT generates a new token, it must perform a computation involving all these weights. From an implementation perspective, these computations can be highly parallelized and conveniently executed on GPUs. However, for each generated token, 175 billion computations (actually a bit more) must still be performed—so it is no surprise that generating a long piece of text with ChatGPT can take some time.

But ultimately, it is remarkable that all these operations—though very simple in themselves—can collectively produce text in such an impressively "human-like" manner. It must be emphasized again (at least as far as we know) that there is no "ultimate theory" predicting that ChatGPT's design would necessarily be effective for any problem like natural language [but in fact it works remarkably well]. Indeed, as we will discuss, I believe we must view this as a—potentially astonishing—scientific discovery: that in a neural network like ChatGPT, something captures the essence of what the human brain can do in generating language.

Training ChatGPT

Now we have a general understanding of how ChatGPT works once it is set up. But how is it configured? How are those 175 billion neural network weights determined? This is essentially done through large-scale training, based on a vast corpus of human-written text—including web pages, books, and more. As we said, even with this training data, it is not certain that a neural network can successfully generate "human-like" text. Moreover, achieving this requires detailed engineering design. But the biggest surprise and discovery of ChatGPT is that it is possible. In fact, a neural network with "only" 175 billion weights can generate a "reasonable model" of human-written text.

In the modern era, a vast amount of human text exists in digital form. There are at least several billion human-written pages on the public web, totaling perhaps about a trillion words of text. If non-public web pages are included, the number could be up to 100 times larger. Currently, over 5 million digitized books are available (though about 100 million have been published), providing another roughly 100 billion words of text. This does not even include text from spoken language in videos, etc. (For personal comparison, the total amount of text I have published in my lifetime is less than 3 million words; the total amount of email I have written over the past 30 years is about 15 million words; the total number of words I have typed might reach 50 million—and in just the past few years, I have spoken more than 10 million words in live streams. Yes, I would train a bot from that content.)

Alright, now that we have all this data, how do we train a neural network from it? The basic process is similar to what we discussed in the simple example earlier. You provide a batch of examples, then adjust the weights in the network to minimize the error ("loss") produced by the network on these examples. The main overhead in "backpropagation" is that each time this is done, every weight in the network typically changes by a tiny amount, and a huge number of weights must be processed. (The actual "backward computation" is usually only a small constant factor more complex than the forward computation.)

With modern GPU hardware, the results of thousands of examples can be computed in parallel. However, when it comes to actually updating the weights in the neural network, current methods essentially require doing this one batch at a time. (Yes, this may be an advantage of actual brains, which combine computation and memory elements.)

Even in the seemingly simple case of learning a numerical function we discussed earlier, we found that typically millions of examples are needed to successfully train a network, at least from scratch. So, how many examples are needed to train a "human-like language" model? Theoretically, there seems to be no fundamental way to know. But in practice, ChatGPT was successfully trained on hundreds of billions of words of text.

Some text is reused multiple times, while other text is used only once. But somehow, it "gets what it needs" from the text it sees. However, given such a large amount of text for training, how large a network is needed to "learn it well"? Again, we have no fundamental theory. Ultimately—as we will discuss further below—there may be a certain "algorithm" to human language and its typical use. But the next question is how efficiently a neural network can implement a model based on the content of that algorithm; we do not yet know, though ChatGPT's success suggests it is quite efficient.

Finally, we can note that ChatGPT achieves what it does using only about 200 billion weights. This number is comparable to the total number of words (or tokens) in the training data it received. In a sense, this may be somewhat surprising (though the same phenomenon has been observed in ChatGPT's smaller models) — namely, that the "effectively working network size" seems to be very close to the "size of the training data." After all, it's certainly not the case that "inside ChatGPT" some literal form of all the text from the web, books, and so on is directly stored. Because what ChatGPT actually contains inside is a bunch of numbers — some kind of distributed encoding of the aggregate structure of all that text, with slightly less than 10 digits of precision.

In other words, we might ask: what is the "effective information content" of human language, and what is it that people typically use it to say? There is the raw corpus of language, and there is ChatGPT's neural network representation of it. That representation is most likely far from an "algorithmically minimized" representation (as discussed below). But it is a representation that a neural network can readily use. In this representation, it appears that the training data is barely compressed at all; on average, it takes slightly less than one neural network weight to carry the "information content" of one training-data word.

When we run ChatGPT to generate text, we essentially have to use each weight once. So if there are n weights, we need about n computation steps. In practice, many of these steps can be parallelized on a GPU. But if we need about n training-data words to set those weights, then from our statement above, we can conclude that we need about n² computation steps to train the network. This is why, with current methods, we end up needing billions of dollars just for the training work.

Beyond Basic Training

In training ChatGPT, most of the work is showing it large amounts of existing text from the web, books, and so on. But it turns out there's another part that's also very important.

Once ChatGPT "raw training" from the corpus of raw text shown to it is done inside the neural network, it can start generating its own text, continuing from prompts and so on. But with longer pieces of text, especially when "wandering" in ways that are not human-like 【referring to generating text that doesn't conform to human habits and cannot interact effectively】, the results often seem plausible but aren't easily detectable through traditional textual statistical analysis. However, a human reader can readily notice this.

A key idea in ChatGPT's construction is that, after "passively reading" something like the web, there's a step needed where a human actively interacts with it, looks at what it produces, and actually provides feedback on "how to be a good chatbot." But how does the neural network use this feedback? The first step is simply having humans evaluate the neural network's outputs. But then a second neural network model is built that tries to predict those ratings. Now this prediction model can be run, effectively acting like a loss function that can be used to adjust the original network — allowing the network to be "tuned" through human feedback. In practice, this seems to contribute greatly to the system's success in producing "human-like" output.

In general, what's interesting is that for the network to develop usefully in a particular direction, it seems to need to be "nudged" only a few times. One might think that, for the network to behave as though it has "learned something new," you'd have to run the training algorithm, adjust the weights, and so on.

But that isn't what happens. Instead, it seems that all you need to do is tell ChatGPT something once, as part of the prompt you give it, and it can then successfully leverage what you've told it to generate text. Once again, this way of working is, I think, an important clue to understanding "what ChatGPT is really doing" and how it relates to the structure of human language and thought.

There is indeed something somewhat human-like about this: at least after it has done all its pretraining, you only need to tell it something once, and it can "remember" it — at least "long enough" to use it in generating text. So what exactly is going on in this case? Perhaps "everything you want to tell it is already there somewhere" — you're just steering it to the right place. But this doesn't seem plausible. Rather, it's more likely that, yes, the elements already exist, but the specifics are defined by something like the "trajectory between these elements" — which is what gets introduced when you tell it something.

In fact, much as with humans, if you tell it something strange and unexpected that doesn't fit the framework it already knows, it seems unable to successfully "integrate" it. It can only "integrate" something when it's essentially running along a framework it already possesses.

It's also worth pointing out again that there are certain inevitable algorithmic limits to what a neural network can do. Tell it "shallow" rules of the form "this goes to that", etc., and a neural network can very likely represent and reproduce such rules well — in fact, it will draw on what it "already knows" from language to provide an immediately applicable pattern. But when you try computations that involve irreducible steps, it doesn't work. (Remember that at every step it's always just "passing data forward" in its network, never looping, except insofar as generating a new token counts as a step.)

Of course, networks can learn the answers to specific "irreducible" computations. But once there is a combinatorial number of possibilities, this "table lookup" approach will fail. Therefore, just like humans, it is time for neural networks to "reach out" and use actual computational tools. (Yes, Wolfram|Alpha and the Wolfram Language are well-suited for this, as they have been built to "talk about things in the world," much like language model neural networks.)

What Makes ChatGPT Work?

Human language—and the thought processes involved in generating it—has long been regarded as the pinnacle of complexity. In fact, it seems astonishing that the human brain, with its "mere" 100 billion or so neurons (perhaps with 1 trillion connections), can handle this task. One might imagine that the brain is not just a network of neurons—that there is some undiscovered new physics at play. But now, with ChatGPT, we have gained important new information: we know that a purely artificial neural network with a number of connections comparable to that of the human brain can surprisingly generate human language.

Yes, it is still a vast and complex system—its neural network weights are roughly equal in number to the words of text currently available in the world. But in a way, it is still incredible that the richness of language and the things it can talk about can be encapsulated in such a finite system. Part of the reason is undoubtedly due to a general phenomenon (first made evident in the example of Rule 30): even when their underlying rules are simple, computational processes can actually greatly increase the surface complexity of a system. However, as we discussed above, the neural networks used in ChatGPT tend to be specifically constructed to limit the impact of this phenomenon—and the associated computational irreducibility—to make training more accessible.

So how does something like ChatGPT achieve language? I think the basic answer is that, fundamentally, language is in some ways simpler than it appears. This means that even though ChatGPT ultimately adopts a straightforward neural network architecture, it can still successfully "capture" the essence of human language and the thinking behind it. Moreover, during training, ChatGPT somehow "implicitly discovers" any regularities in language (and thought) that make this possible.

I believe the success of ChatGPT provides us with a fundamental and important scientific piece of evidence: it shows that we can expect to discover major new "laws of language"—and effective "laws of thought." In ChatGPT—as a neural network construction—these laws are at best implicit. But if we could somehow make them explicit, it might be possible to do what ChatGPT does in a more direct, efficient, and transparent way.

So what might these laws look like? Ultimately, they must provide us with guidance on how to combine language—and the things we talk about using language. Later, we will discuss how "looking inside ChatGPT" might give us some hints about this, as well as a path forward derived from the knowledge of building computational language. But first, let's discuss two examples of "laws of language" that have been known for a long time—and how they relate to the operation of ChatGPT.

The first is the grammar of language. Language is not just a random jumble of words. Instead, there are (fairly) clear grammatical rules for how different types of words can be combined: in English, for example, a noun can be preceded by an adjective and followed by a verb, but typically two nouns cannot be placed next to each other. Such grammatical structures can be captured (at least approximately) by a set of rules that define a "parse tree":

[Figure]

ChatGPT does not explicitly "know" these rules, but during training it implicitly "discovers" them, and then seems to be quite good at following them. So how does this work? It is not clear from a "macro" perspective, but it might be helpful to gain some insight through a simple example.

Consider a "language" consisting of sequences of "(" and ")", with a grammar rule that parentheses should always be balanced, as shown in the parse tree below:

[Figure]

Can we train a neural network to generate "grammatically correct" sequences of parentheses? There are various ways to handle sequences in neural networks, but we will use a transformer network, just like ChatGPT. And, given a simple transformer network, we can train it on correct parenthesis sequences as training samples. One subtlety (which also appears in ChatGPT's generation of human language) is that, in addition to "content tokens" (here "(" and ")"), we must also include an "end" token, indicating that the output should not continue (i.e., for ChatGPT, reaching the "end of the story").

If we set up a transformer network with only one attention block, 8 heads, and a feature vector length of 128 (ChatGPT also uses a feature vector length of 128, but with 96 attention blocks, each with 96 heads), it seems impossible to make it learn much about the parenthesis language. But with 2 attention heads, the learning process appears to converge—at least after being given around 10 million examples (as is common with transformer networks, showing more examples seems to only degrade performance).

So, for this network, we can do something similar to ChatGPT: ask what the probability of the next token should be—in a parenthesis sequence:

[Figure]

In the first case, the network is "very certain" that the sequence cannot end here—which is good, because if it did, the parentheses would be unbalanced. However, in the second case, it "correctly identifies" that the sequence can end here, although it also "points out" that it could "restart" by placing a "(", possibly followed by a ")". But unfortunately, even with its painstakingly trained ~400,000 weights, it says the next token has a 15% probability of being ")"—which is incorrect, as this would inevitably lead to unbalanced parentheses.

If we ask the network to generate the highest-probability completions for gradually longer sequences, we get the following:

[Figure]

Yes, the neural network performs well within a certain length range. But when it needs to execute more complex "algorithmic" tasks, such as explicitly counting whether parentheses are closed, neural networks tend to be "too shallow in computational capacity" to reliably perform them. Even the current full ChatGPT struggles to correctly match parentheses in long sequences.

So, what does this mean for applications like ChatGPT and the grammar of languages like English? The parentheses language is more "minimal"—more like an "algorithmic" language. But in English, based on local word choices and other cues, it's possible to "guess" grammatical plausibility. Yes, neural networks perform better here—even if they might miss certain "formally correct" cases, humans might miss them too. But the main point is that language has an overall grammatical structure—with all its regularities—which to some extent limits the "degree" the neural network needs to learn. The transformer architecture in neural networks seems to successfully learn the type of nested tree-like grammatical structures that appear to exist in all human languages (at least approximately).

Grammar provides a constraint for language. But there are other aspects. Sentences like "Curious electrons eat blue theories to exchange fish" are grammatically correct, but not something people would typically say. If ChatGPT generated such a sentence, it wouldn't be considered successful—because, well, it's essentially meaningless.

But is there a general way to determine whether a sentence is meaningful? Traditionally, there is no overarching theory. However, after training on billions of (presumably meaningful) web sentences, ChatGPT can be seen as implicitly "developing a theory."

What might this theory look like? Well, there's a small corner that has been known for about two thousand years: logic. Of course, in its syllogistic form discovered by Aristotle, logic is essentially a way of saying that sentences following certain patterns are reasonable, while others are not. So, for example, saying "All X are Y. This is not Y, so it's not an X." (like "All fishes are blue. This is not blue, so it's not a fish.") is reasonable. Just as one can imagine Aristotle discovering syllogistic logic through ("machine-learning-style") many rhetorical examples, we can imagine that during ChatGPT's training, by looking at vast amounts of web text, it can "discover syllogistic logic." (Yes, although one might expect ChatGPT to produce "correct inferences" based on syllogistic logic, when it comes to more complex formal logic, the situation is different—I think it would fail for the same reasons as parentheses matching.)

But beyond the narrow example of logic, is there anything about how to systematically construct (or identify) even reasonable text? Yes, there are things like Mad Libs that use very specific "phrase templates." But somehow, ChatGPT implicitly has a more general way. Perhaps it can't be explained other than "when you have 175 billion neural network weights, it just somehow happens." But I strongly suspect there is a simpler, more powerful answer.

Meaning Space and Semantic Motion Laws

We discussed above that in ChatGPT, any text is effectively represented as an array of numbers, which we can think of as coordinates of a point in some kind of "linguistic feature space." So when ChatGPT continues a text, it's equivalent to tracking a trajectory in the linguistic feature space. But now we can ask: what makes this trajectory correspond to what we consider meaningful text? Or perhaps there are some "semantic motion laws" that define or at least constrain the movement of points in the linguistic feature space while maintaining "meaningfulness"?

So what does this linguistic feature space look like? Here's an example of a 2D space where words (here, common nouns) might be placed:

[Figure]

We saw another example earlier, based on words representing plants and animals. But in both cases, "semantically similar words" are placed nearby.

Another example: words corresponding to different parts of speech are placed as follows:

[Figure]

Of course, a particular word often doesn't have just "one meaning" (or doesn't necessarily correspond to just one part of speech). By looking at the position in feature space of sentences containing a word, we can often "disambiguate" different meanings, as in this example of the word "crane" (bird or machine?):

[Figure]

So, we can think of this feature space as placing "semantically similar words" close together. But what additional structure can we identify in this space? For example, is there a concept of "parallel transport" that reflects the "flatness" of the space? One way to explore this is through analogies:

[Figure]

Yes, even when we project it into 2D, there is usually at least some "hint of flatness," though it is certainly not universal.

What about trajectories? We can look at the trajectory traced by ChatGPT's prompts in feature space, and then see how ChatGPT continues:

[Image]

There is clearly no "geometrically obvious" pattern of motion here. This is not surprising; we fully expect this to be a rather complex story. For example, even if there are "semantic laws of motion," it is not obvious in which embedding (or indeed which "variables") they would most naturally be stated.

In the figure above, we show several steps of the "trajectory," where at each step we choose the word that ChatGPT considers most likely (the "zero temperature" case). But we can also ask what words are possible next at a given point, and with what probabilities:

[Image]

In this case, we see a "fan" of high-probability words, seemingly extending in one or more well-defined directions in feature space. What happens if we go further? Here is the continuous "fan" that appears as we move "along" the trajectory:

[Image]

This is a 3D representation, covering 40 steps in total:

[Image]

Yes, it looks messy and does not particularly encourage our idea that we can identify "semantic laws of motion" analogous to "mathematical physics" by empirically studying "what ChatGPT does internally." But perhaps we are just looking at the wrong "variables" (or coordinate system). If we looked at the right ones, we would immediately see that ChatGPT is doing something "mathematically simple," like following geodesics. But so far, we are not ready to "empirically decode" from its "internal behavior" the knowledge that ChatGPT has "discovered" about the "composition" of human language.

Semantic Grammar and the Power of Computational Language

What is needed to produce "meaningful human language"? In the past, we might have thought that only the human brain could do it. But now we know that ChatGPT's neural network can do it reasonably well. However, perhaps this is as far as we can go, and there is no simpler, more understandable method to use. But I strongly suspect that ChatGPT's success hints at an important "scientific" fact: meaningful human language is actually much simpler than we thought, and there may ultimately be fairly simple rules describing how this language is composed.

As we mentioned above, syntactic grammar gives rules for how words of different parts of speech combine into human language. But to handle meaning, we need to go further. One approach is to consider not only the syntactic grammar of language but also a semantic grammar.

For syntax, we identify parts of speech like nouns and verbs. But for semantics, we need a "finer granularity." For example, we can identify the concept of "movement" and the concept of an "object" that "maintains its identity independently of location." Each of these "semantic concepts" has countless specific examples. But for our semantic grammar, we will only have some general rules, essentially saying that an "object" can "move." There are many ways to make all this work (some of which I have discussed before). But here, I will only briefly outline some potential paths forward.

It is worth noting that even if a sentence is perfectly fine according to semantic grammar, it does not mean it has been (or even can be) realized in practice. For example, "The elephant goes to the moon" would undoubtedly pass our semantic grammar, but it certainly has not yet been realized in our actual world (at least not yet), although it could absolutely be used in a fictional world.

When we start talking about "semantic grammar," we soon ask: "What lies beneath it?" What "world model" does it assume? Syntactic grammar really only deals with how to build language from vocabulary. But semantic grammar necessarily involves some kind of "world model"—a "skeleton" onto which language composed of actual vocabulary can be added.

Until recently, we might have thought that (human) language was the only general way to describe our "world model." A few centuries ago, formalization based on mathematics began for specific types of things. But now there is a more general method of formalization: computational language.

Yes, this is the large project I have been working on for over forty years (now embodied in the Wolfram Language): developing a precise symbolic representation that can talk as broadly as possible about things in the world, as well as abstract things we care about. So, for example, we have symbolic representations for cities, molecules, images, and neural networks, and we have built-in knowledge about how to compute things about them.

After decades of work, we have covered many domains in this way. But in the past, we did not specifically deal with "everyday language." In "I bought two pounds of apples," we can easily represent (and perform nutritional calculations and other computations on "two pounds of apples"), but we do not (yet) have a symbolic representation for "I bought."

All this relates to the concept of semantic grammar and the goal of having a universal symbolic "construction toolkit" of concepts that will give us rules about what can go with what, thereby determining the "flow" that we can convert into human language.

But suppose we had such a "symbolic discourse language." What would we do with it? We could start doing things like generating "locally meaningful text." But ultimately, we might want to obtain more "globally meaningful" results, which means "computing" more information about things that can actually exist or happen (or in some consistent fictional world).

In the Wolfram Language, we now have a vast amount of built-in computational knowledge about many things. But for a full symbolic discourse language, we need to establish additional "computational rules" about the world: if an object moves from A to B, and from B to C, then it has moved from A to C, and so on.

Given a symbolic discourse language, we can use it to make "standalone statements." But we can also use it to ask questions about the world in a "Wolfram|Alpha style." Or we can use it to state things we "want to make real," perhaps with some external activation mechanism. Or we can use it to make assertions—possibly about the actual world, or about some particular world we are considering, whether fictional or otherwise.

Human language is fundamentally imprecise, not only because it is not "bound" to a specific computational implementation, but also because its meaning is essentially defined only by a "social contract" among its users. However, due to the nature of computational language, it possesses a certain fundamental precision—because what it specifies can always be "unambiguously executed on a computer." Human language can often get away with a certain vagueness. (When we say "planet," does it include exoplanets, etc.?) But in a computational language, we must be precise and clearly distinguish all the distinctions we make.

It is often convenient in a computational language to use ordinary human words to construct names. But their meaning within the computational language must be precise, and may or may not cover certain specific connotations from typical human language usage.

How do we find the fundamental ontology suitable for a general symbolic discourse language? Well, it's not easy. That is perhaps why, since Aristotle's original beginnings over two thousand years ago, so little has been accomplished in this area. But now we have learned a great deal about how to think about the world computationally (and that doesn't prevent us from drawing help from the "fundamental metaphysics" of the Physics Project and the ruliad).

But what does this mean in the context of ChatGPT? From its training, ChatGPT has effectively "pieced together" a certain amount of semantic grammar (quite impressively). And its success gives us reason to think that building a more complete form of computational language is feasible. Moreover, unlike what we have discovered so far about the internals of ChatGPT, we can expect to design the computational language so that it is easily understandable by humans.

When we talk about semantic grammar, we can compare it to syllogistic logic. At first, syllogistic logic was essentially a set of rules about statements expressed in human language. But (yes, two millennia later), when formal logic was developed, the original basic structure of syllogistic logic could now be used to build vast "formal towers," including the operation of modern digital circuits. So we can expect the same for a more general semantic grammar. At first, it may only be able to handle simple patterns, such as text. But once the entire computational language framework is established, we can expect it to be used to erect tall towers of "generalized semantic logic," enabling us to deal precisely and formally with all sorts of things that were previously only accessible through the "ground level" of human language, with all its ambiguity.

We can think of the construction of a computational language and semantic grammar as representing an ultimate compression for representing things. Because it allows us to talk about the essence of possibilities without having to deal with all the "phrasal variations" that exist in ordinary human language. We can think of the great strength of ChatGPT as being similar: because it has, in a sense, "drilled down" to a level where it can "assemble language in a semantically meaningful way," without having to care about different phrasal variations.

So what happens if we apply ChatGPT to the underlying computational language? The computational language can describe possibilities. But one can still add a sense of "popularity," for example based on reading all the content on the web. However, then, operating on the underlying computational language means that a system like ChatGPT immediately gains fundamental access to the ultimate tool for leveraging potentially irreducible computation. This allows it not only to "generate plausible text," but also to be expected to understand whether that text actually makes "correct" statements about the world or whatever it is discussing, or anything else that should be discussed.

So... What Is ChatGPT Doing, and Why Does It Work?

The basic concept behind ChatGPT is, in some sense, fairly straightforward. Start with a large sample of human-created text—from the web, books, and so on. Then train a neural network to generate "similar" text. In particular, make it capable of starting from a "prompt" and then continuing with text "similar to what's in its training data."

As we've seen, the actual neural network in ChatGPT is made up of very simple elements—though billions of them. The basic operation of the neural network is also very simple: essentially, for each new word (or word part), it produces an "input" and then passes it "through its elements" (with no loops or anything like that).

Yet the fact that this process can produce text that successfully "resembles" what's on the web, in books, and so on is quite remarkable and unexpected. It doesn't just produce coherent human language—it "says things" that "follow its prompt," drawing on what it has "read." It doesn't always say something that makes sense "in a global sense" (or corresponds to a correct calculation), because—without, say, access to Wolfram|Alpha's "computational superpowers"—it simply produces "something that sounds right" based on what "sounds similar" in its training material.

The specific engineering of ChatGPT makes it quite fascinating. But in the end—at least until it can make use of external tools—ChatGPT is just pulling out "coherent threads of text" from the "statistical wisdom of the ages" it has accumulated. And yet how human-like its results are. As I've discussed, this points to something at least of great scientific importance: that the structure of human language (and the patterns of thought behind it) is simpler and more "law-like" than we might have imagined. ChatGPT has already implicitly discovered it. But we may be able to explicitly uncover it with semantic grammars, computational linguistics, and so on.

ChatGPT performs remarkably well at generating text—results often come very close to what we humans produce. So does this mean ChatGPT works like the brain? Its basic artificial neural network architecture is, after all, based on an idealized model of the brain. It's quite plausible that many aspects of how we humans generate language work in rather similar ways.

But when it comes to training (i.e., learning), the different "hardware" of brains and current computers (and possibly undeveloped algorithmic ideas) means that ChatGPT has to use a strategy that is likely quite different from—and in some ways far less efficient than—the brain's. And there's another thing: unlike even typical algorithmic computation, there's no "looping" or "recomputation" inside ChatGPT. This inevitably limits its computational capability—even compared to today's computers, but certainly compared to the brain.

It's not clear how to "fix this" while still retaining the ability to train the system with reasonable efficiency. But doing so might allow future versions of ChatGPT to do more "brain-like things." Of course, there are plenty of things brains don't do well, especially when it comes to things equivalent to irreducible computation. For these kinds of problems, both brains and things like ChatGPT have to look for "external tools"—like the Wolfram Language, for instance.

But it's exciting to see what ChatGPT has already been able to do. In a way, it's a great basic scientific fact that a large number of simple computational elements can do surprising and unexpected things. But it also offers the best possible motivation for us to arrive, within the span of two thousand years, at a much better understanding of the fundamental characteristics and principles of human language and the thinking processes behind it.

Some Reading Notes

Zhang Wuchang https://mp.weixin.qq.com/s/cpLDPDbTjarU0_PpBK_RDQ