Original: https://mccormickml.com/2022/12/21/how-stable-diffusion-works/

Author: Chris McCormick (Stanford graduate, working in computer vision, machine learning, and NLP)

Published: December 21, 2022

Translator: WaytoAGI (Way to AGI)

Fifth article in the "A16Z's AI Canon" introductory series

It's incredible that a computer can create art from just a text description! I was also very curious about what's happening "under the hood" to make this possible. So here I'll do my best to provide a fairly in-depth explanation, even for those unfamiliar with AI concepts.

Overview

In the first part, I'll give you a high-level explanation (which you might already be familiar with). It's a good start, but I know it wouldn't satisfy my curiosity. 😉 I'd ask, "Okay, great, but how does it actually do it?"

To answer that, I'll show you some of the inner workings of Stable Diffusion. The complexity inside may be more than you expect, but I at least want to show you the process more concretely, so it's no longer a complete mystery.

More specifically:

• Stable Diffusion is a huge neural network.

• Neural networks are pure mathematics.

• In fact, we don't fully know what it's doing!

• Ultimately, Stable Diffusion works because we trained it.

But let's start with the big picture!

Stable Diffusion removes noise from images

If you've ever tried to take a photo in too-dark conditions and ended up with a grainy image, that grain is an example of "noise" in an image.

We use Stable Diffusion to generate artwork, but what it's actually doing behind the scenes is "cleaning up" the image!

However, it's much more sophisticated than the noise reduction slider in a phone's image editor. It actually understands what the world looks like, understands written language, and uses these to guide the (noise removal) process.

For example, imagine I gave the image on the left below to a skilled graphic artist and told them it's a painting of an alien playing a guitar in the style of H.R. Giger (Swiss painter, sculptor, and set designer, known for the alien creature in Alien). I bet they could carefully clean it up to create something like the image on the right.

[Image]

(These are actual images from Stable Diffusion!)

The artist would use their knowledge of Giger's artwork and their understanding of the world (e.g., what a guitar should look like and how it's played) to do this. Stable Diffusion is essentially doing the same thing!

"Inference Steps"

Are you familiar with the "inference steps" slider in most art generation tools? Stable Diffusion removes noise step by step.

Here's an example running for 25 steps:

[Image]

The alien guitarist example makes more sense because you can more clearly see what it's supposed to be... but in the image above, the starting image looks completely unrecognizable!

Actually, this noisy alien example was taken from about halfway through the process—it (the very first image) actually starts from pure noise too!

[Image]

How does it start?

To generate art, we give Stable Diffusion an initial image that is actually just pure noise. But, rather cruelly 😏, we lie and say: "This is a super noisy painting of an alien playing a guitar in the style of H.R. Giger—can you clean it up for me?"

If you gave this task to a graphic artist, they'd be at a loss—"I can't help you, this image is completely unrecognizable!"

So how does Stable Diffusion do it?

At the simplest level, the answer is that it's a computer program that has no choice but to do its thing and produce something for us.

A deeper answer relates to the fact that AI models like Stable Diffusion (technically, "machine learning" models) are largely based on statistics. They estimate the probabilities of all options, and even if the correct probability for every option is extremely low, they still just choose the path with the highest probability.

For example, it has some understanding of where a guitar might appear in an image, and it can look for which part of the noise most likely resembles the edge of a guitar (even if there's no "correct" choice), then start filling in the object.

Because there is no right answer, each time you give it a different pure-noise image, it creates a different piece of art!

How Do You Program Stable Diffusion?

If I'm not familiar with machine learning and I'm trying to guess how to actually implement this, I might start thinking about how to code it. In other words, what sequence of steps does it follow?

Maybe it starts by matching keywords from the description, searches an image database for images that match the description, and then compares them to noise? And from that guy's explanation, it sounds like it might first calculate where the most obvious edges in the image are? 🤷

That's not how it works — it doesn't have a reference image database, nor does it use any image processing algorithms... It's pure mathematics.

I'm not saying "well, of course, computers are ultimately just large calculators, and everything they do boils down to math." I'm talking about the kind of math you'd see as "dazzling equations on a blackboard," like this:

[Image]

(This is one of the many building blocks of Stable Diffusion that I wrote a technical tutorial about, called "attention.")

Defining all the equations for each different building block would fill at least several pages.

Images and Text as Numbers

To apply these equations, we need to represent the initial noise image and our text description as large tables of numbers.

You may already be familiar with how images are represented, but let's look at an example. This is a long-exposure photo I took at high tide:

[Image]

This is how it's represented mathematically. It's 512 x 512 pixels, so we represent it as a table with 512 rows and 512 columns. But in reality, we need three tables to represent one image, because each pixel is made up of a mix of red, green, and blue (RGB). Here are the actual values for the image above.

[Image]

In Stable Diffusion, we also work with text. Here's a description I might write for the image:

A long exposure color photograph of decaying concrete steps leading down into the ocean, with concrete railings, head on view, symmetry, dream like, atmospheric.

This is how it gets represented as a table of numbers. Each word has a row, and each word is represented by 768 numbers. These are the actual numbers used in Stable Diffusion v1.5 to represent these words:

[Image]

How we choose the numbers to represent a word is a fascinating topic, but also quite technical. You can roughly think of these numbers as each representing a different aspect of the word's meaning.

In machine learning, we don't actually call these "tables" — we use terms like "matrix" or "vector." These come from the field of linear algebra.

However, the most important and most confusing part of all this is the concept of parameters.

1 Billion Parameters

The initial noise and our text description are what we call the inputs to Stable Diffusion, and different inputs will have different values in these tables.

We also plug a much larger set of numbers into these equations, but they're the same every time — these are called the parameters of Stable Diffusion.

Remember in high school when you used equations to draw lines, like y = 3x + 2?

[Image]

If this were Stable Diffusion, then "x" would be our input, "y" would be the final image, and the numbers 3 and 2 would be our parameters. (Of course, the actual equations are much more complicated 😝).

The input image is represented by about 790,000 values, and the 33 "tokens" in the prompt are represented by about 25,000 values.

But in Stable Diffusion, there are about 1 billion parameters. 🤯

(Can you imagine calculating all those numbers by hand?!?)

Those 1 billion numbers are distributed across roughly 1,100 matrices of varying sizes. Each matrix gets used at different stages of the mathematical operations.

If you're curious, I've printed out the complete list of these matrices here!

Again, these parameters don't change — they're the same numbers every time you generate an image.

Stable Diffusion works because we found the right value for each of those 1 billion numbers. Isn't that absolutely absurd?

Choosing 1 Billion Parameters

Obviously, the authors didn't sit down and decide what numbers to try. Especially when you consider that they aren't "whole numbers" like 1, 2, 3—they're what we computer geeks call "floating-point" values: the tiny, very precise fractions you see in the table.

Not only did we not choose these numbers—we can't even explain any single one of them! That's why we can't fully explain how Stable Diffusion works. We have some intuitive understanding of what these equations do, but a lot is hidden in the values of those numbers, and we can't fully comprehend it.

Crazy, right?

So how did we figure out these numbers?

We started by picking one billion random numbers to use. With those initial random parameter values, the model was completely useless—it couldn't do anything valuable until we found better parameter values.

So we applied a mathematical process called training, which gradually adjusts the values into useful ones.

How training works is something we fully understand—it's basic calculus (though applied to a very large equation), it's essentially guaranteed to work, and we have a clear understanding of why it works.

Training involves a huge training dataset. A single training sample consists of an input and a desired output. (I'll explain what a training sample looks like for Stable Diffusion in another article.)

When we run the first training input (with completely random parameter values), what the model spits out will be nothing like the desired output.

But using the difference between the actual output and the desired output, we can apply some very basic calculus to these equations, which tells us, for each of those one billion numbers, how much we should add or subtract. (Each individual parameter gets a different small adjustment!)

After we make those adjustments, the model is mathematically guaranteed to produce an output image closer to what we want.

So we do this many times (hundreds of millions of times) with many different training samples, and the model gets better and better. However, as training progresses, the gains diminish, and eventually we reach a point where the model can no longer benefit from further training.

Once the authors finished training the model, they released the parameter values for everyone to use for free!

Training Stable Diffusion

There's a lot about the Stable Diffusion training process that's easy to understand and fun to learn, but I've decided to save that for another blog post!

Conclusion

If you're a bit disappointed by this explanation, feeling like it's not that easy to grasp, I won't be offended—but I hope you at least feel that the veil has been lifted, and what you see is incredible and inspiring!