Types of AI Models: Dense, Mixture of Experts, and More

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Crumb & Co.'s owner is still thinking about running Biscuit, the bakery's help desk bot, on an open-weight model of her own. So she opens a model download page and starts reading names. Some are easy, like Llama-3.1-405B, a model with 405 billion parameters. Others look more like license plates:

[Image: Four model names taken apart. Llama-3.1-405B has 405 billion parameters. Mixtral-8x7B has 8 experts in every layer, on a 7B-sized design. Qwen3-235B-A22B has 235 billion parameters in total, and A22B means 22 billion are active for each token. Llama-4-Maverick-17B-128E has 17 billion active parameters and 128 experts.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_names.png)

Those extra letters describe how each model is built on the inside, and two models of similar size can do wildly different amounts of work for every token they write. In this article, we'll open up the design almost every LLM is built on, see how a mixture of experts splits it up, and figure out why one model can have two very different sizes. Then we'll sort out labels like reasoning, multimodal, and distilled, peek at designs beyond the usual one, and see where popular models land.

Onwards!

What's Inside a Transformer

Nearly every LLM in this series is a transformer, a design Google researchers introduced in the 2017 paper "Attention Is All You Need" (https://arxiv.org/abs/1706.03762). In What Is a Tensor? (https://www.kirupa.com/ai_fundamentals/what_is_a_tensor.htm), we saw that a model is a stack of layers, each with its own weight matrices. In a transformer, every layer has the same two parts, and our tokens go through them one layer after another:

[Image: The tokens of Is the sourdough vegan? go into a stack of layers. One layer is opened up to show its two parts. First, attention, where tokens look at each other. Then feed-forward, where each token's numbers go through big weight matrices on their own. Llama 3.1 405B repeats this layer 126 times, and the feed-forward parts hold about 81 percent of its parameters.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_transformer.png)

The attention part is where tokens share information. When Biscuit reads Is the sourdough vegan?, attention mixes the numbers for vegan with the numbers for sourdough, so the model can tell what's being asked about. The feed-forward part then works on each token by itself, multiplying its list of numbers by a few large weight matrices.

That feed-forward part is where most of the parameters live. Using the numbers from What Is a Tensor?, Llama 3.1 405B has 126 layers, and each feed-forward part has three matrices of 53,248 × 16,384 numbers. That's 378 matrices, or about 330 billion numbers, which is around 81% of the whole model. Keep that in mind, because the feed-forward part is exactly what a mixture of experts changes.

Dense Models: Every Weight, Every Token

Llama 3.1 405B is a dense model. Meta's Llama 3 paper (https://arxiv.org/abs/2407.21783) calls it "a dense Transformer with 405B parameters", and dense means every token goes through every parameter. When Biscuit writes Yes, all 405 billion weights take part, and they all take part again for the next token.

In Why Training Takes a Data Center (https://www.kirupa.com/ai_fundamentals/data_centers_and_training.htm), we saw that running a model takes about 2 operations per parameter for every token. So Llama 3.1 405B does about 810 billion operations per token. The usual way to make a dense model smarter is to make it bigger, and then every token gets more expensive too. A mixture of experts is a way to break that link.

Mixture of Experts: Only a Few Experts Per Token

Let's step into Crumb & Co.'s kitchen. In a dense kitchen, every baker works on every order, even if the order is one cookie. Now picture a bigger kitchen with more bakers and a head baker at the door, who sends each order to just two of them:

[Image: Two kitchens handling the same cookie order. In the dense kitchen, all 4 bakers work on it. In the mixture-of-experts kitchen, there are 8 bakers, and a head baker at the door sends the order to just 2 of them while the other 6 stay idle.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_kitchen.png)

The bigger kitchen has more combined skill, but each order only costs two bakers' time. A mixture-of-experts model (MoE for short) does this inside each layer. Instead of one big feed-forward part, an MoE layer has several smaller ones called experts, plus a small piece called the router. Here is one layer of Mistral AI's Mixtral 8x7B handling the token sourdough:

[Image: One mixture-of-experts layer in Mixtral 8x7B. The token sourdough comes out of the attention part and goes to the router, which scores all 8 experts. The 2 experts with the highest scores, expert 3 and expert 6, get the token. Their outputs are blended, weighted by their scores, and passed on to the next layer. The other 6 experts do no math for this token.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_router.png)

The router is a tiny set of weights of its own. It gives every expert a score for this token, the two highest-scoring experts each run the token through their own weights, and their outputs are blended, with the higher score counting for more. The Mixtral paper sums it up:

For every token, at each layer, a router network selects two experts to process the current state and combine their outputs.

Mistral AI, Mixtral of Experts (https://arxiv.org/abs/2401.04088)

The other six experts do no math for that token at all. Two details make a real model different from our kitchen. The choice happens at every layer, so across Mixtral's 32 layers, sourdough can visit a different pair of experts each time. And nobody tells the experts what to work on. The router and experts are trained together with the rest of the model, and the router learns whatever split of the work helps.

Experts Aren't Topic Specialists

The word experts makes it sound like one handles math and another handles biology. Mistral AI checked, and the answer surprised them:

Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.

Mistral AI, Mixtral of Experts (https://arxiv.org/abs/2401.04088)

What they found was more mechanical. In Python code, the word self tended to go to the same expert, and so did the spaces that indent each line. So basically, an expert is a chunk of weights the router has learned to send certain kinds of tokens to, and those kinds rarely match subjects a person would name. That's where our kitchen is misleading. A head baker would send sourdough orders to the bread specialist, and a router mostly doesn't work that way.

Many newer MoE models also have a shared expert that every token visits, on top of the routed ones. DeepSeek-V3 (https://arxiv.org/abs/2412.19437), for example, has 1 shared expert and 256 routed experts in each MoE layer, and its router picks 8. Others, like Alibaba's Qwen3 MoE models (https://arxiv.org/abs/2505.09388), skip the shared expert.

Total vs. Active Parameters

Now we can read those license plates. An MoE model has two sizes. Its total parameters count every weight, including every expert. Its active parameters count only what one token actually uses: the attention parts, any shared expert, and the experts the router picked. Here are four models, drawn to scale:

[Image: Total and active parameters for four mixture-of-experts models, drawn to scale. Mixtral 8x7B has 47 billion total and 13 billion active. gpt-oss-120b has 117 billion total and 5.1 billion active. Llama 4 Maverick has 400 billion total and 17 billion active. DeepSeek-V3 has 671 billion total and 37 billion active. In every case, the active part is a small slice of the total.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_total_active.png)

Mixtral has the same design as Mistral AI's 7-billion-parameter model, except that every layer has 8 feed-forward experts. That suggests 8 × 7 = 56 billion parameters, but the paper says 47 billion, because only the feed-forward parts were multiplied by eight. The attention parts are shared. With 2 of 8 experts working, each token uses about 13 billion. OpenAI's gpt-oss-120b (https://openai.com/index/introducing-gpt-oss/) goes further, using 5.1 of its 117 billion parameters per token.

For an MoE model, our rule of 2 operations per parameter applies to the active parameters. That puts DeepSeek-V3 at roughly 74 billion operations per token, about 11 times fewer than dense Llama 3.1 405B, even though DeepSeek-V3 is the bigger model. (Attention adds work that grows with the conversation, so these are rough numbers.) Google's Gemini 2.5 report (https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf) describes this as letting a model "decouple total model capacity from computation and serving cost per token".

Cheap to Run, Expensive to Hold

So is MoE a free lunch? Nope. We can't know ahead of time which experts the next token will need, so every expert has to sit in fast GPU memory, ready to go. Meta's Llama 4 announcement says it plainly:

While all parameters are stored in memory, only a subset of the total parameters are activated while serving these models.

Meta, Llama 4 announcement (https://ai.meta.com/blog/llama-4-multimodal-intelligence/)

Here is DeepSeek-V3 next to Llama 3.1 405B, with every weight stored at 2 bytes:

[Image: Llama 3.1 405B, a dense model, compared with DeepSeek-V3, a mixture-of-experts model, with every weight stored at 2 bytes. To hold its weights, Llama 3.1 405B needs about 810 gigabytes and DeepSeek-V3 needs about 1,342 gigabytes. For the math per token, Llama 3.1 405B does about 810 billion operations and DeepSeek-V3 does about 74 billion. DeepSeek-V3 needs more memory but does far less math for each token.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_memory_compute.png)

In our kitchen terms, every baker has to be in the building and on the payroll, even though only two touch each order. That's why memory-saving tricks matter so much here. In gpt-oss, the experts hold more than 90% of the parameters, and OpenAI stores them at about 4.25 bits each, which its model card (https://arxiv.org/abs/2508.10925) says lets gpt-oss-120b fit on a single 80 GB GPU. Storing weights in fewer bits is quantization (https://www.kirupa.com/ai_fundamentals/quantization.htm), which gets its own article.

Memory isn't the only catch:

- Big MoE models often put different experts on different GPUs, so tokens get shipped to wherever their experts live and back. DeepSeek-V3 limits each token to experts on at most 4 servers to keep that traffic down.
- The router can play favorites, sending most tokens to a few experts while the rest barely learn. Training adds rules to spread the work out, called load balancing.
- Training can be finicky. Google's 2021 Switch Transformer paper (https://arxiv.org/abs/2101.03961) said MoE adoption had been "hindered by complexity, communication costs and training instability".

The idea itself dates back to a 1991 paper, "Adaptive Mixtures of Local Experts" (https://doi.org/10.1162/neco.1991.3.1.79), whose authors include Geoffrey Hinton. What's new is how common it is. Since Mixtral arrived in December 2023, experts have shown up in Llama 4, Qwen3, DeepSeek-V3, gpt-oss, and Mistral Large 3 (https://mistral.ai/news/mistral-3).

Labels That Describe Something Else

Dense and MoE describe how a model is wired. Many other labels on model pages describe how it was trained or what it accepts, so one model can wear several at once:

[Image: Six models and their labels. Llama 3.1 405B is dense, with none of the other labels. Gemma 3 27B is dense, multimodal, and distilled. DeepSeek-R1-Distill-Qwen-32B is dense, a reasoning model, and distilled. DeepSeek-R1 is a mixture of experts and a reasoning model. gpt-oss-120b is a mixture of experts, a reasoning model, and distilled. Llama 4 Maverick is a mixture of experts, multimodal, and distilled.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_labels.png)

A reasoning model is trained to write out its thinking before it answers, as we saw in What Does It Mean for AI to Reason? (https://www.kirupa.com/ai_fundamentals/what_ai_reasoning_means.htm). That label is about how the model was trained, and it doesn't tell us how the model is wired. DeepSeek-R1 (https://huggingface.co/deepseek-ai/DeepSeek-R1) is built on DeepSeek-V3, so it's MoE, with 671 billion total and 37 billion active parameters. Qwen3 even puts a thinking mode and a non-thinking mode inside the same model.

A multimodal model takes in more than text. In How AI Generates Images and Video (https://www.kirupa.com/ai_fundamentals/how_ai_generates_images_and_video.htm), we saw pictures cut into patches that become tokens, and that same trick lets a language model read a photo alongside our words. Google describes Gemini 2.5 as having "native multimodal support for text, vision, and audio inputs". This says nothing about dense versus MoE. Llama 4 Maverick is a multimodal MoE model, Gemma 3 27B is a multimodal dense one, and gpt-oss is MoE but text-only.

A distilled model is a small student trained on the outputs of a bigger teacher, so it learns from the teacher's answers and not only from raw text (Small vs. Large Model Trade-offs (https://www.kirupa.com/ai_fundamentals/small_vs_large_models.htm) covers when small models are the right pick). DeepSeek trained dense models like DeepSeek-R1-Distill-Qwen-32B on about 800,000 samples written by R1. Google says its Gemma 3 models (https://arxiv.org/abs/2503.19786) "are trained with distillation", Meta's Llama 4 models learned from a bigger teacher called Llama 4 Behemoth, and OpenAI says gpt-oss used "large-scale distillation and reinforcement learning". Notice that R1 is MoE while its students are dense. A student doesn't have to copy its teacher's design.

Beyond the Standard Transformer

Attention has a cost that grows with length. Every new token looks back at every token before it, so a longer conversation means more work per token and more memory (Context Window Limits (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm) covers what that does to long chats). Some newer designs keep a running summary instead:

[Image: Two ways to handle the next token. With attention, the new token looks back at every earlier token, so there is more to look at as the text grows. With a state-space model, each token updates a fixed-size running summary, and the new token reads only that summary, so the work per token stays the same.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_attention_vs_state.png)

State-space models (SSMs) like Mamba (https://arxiv.org/abs/2312.00752) skip attention and update a fixed-size summary with each token, a bit like keeping running notes instead of rereading the whole conversation. The Mamba paper reports "linear scaling in sequence length", meaning twice the text takes about twice the work. The catch is that a fixed-size summary can't hold onto everything, and SSMs "still lag behind the performance of comparably sized Transformer language models", according to AI21 Labs. Its answer is a hybrid. Jamba (https://arxiv.org/abs/2403.19887) uses one attention layer for every seven Mamba layers and adds experts on top, for 52 billion total and 12 billion active parameters.

Diffusion language models borrow the trick from image generators. They start with a reply where every token is blanked out and fill in the blanks over several passes, many tokens at once. The research model LLaDA (https://arxiv.org/abs/2502.09992) uses this approach, and Inception Labs reports that its Mercury Coder Mini (https://arxiv.org/abs/2506.17298) writes 1,109 tokens per second on an H100. These are still the exception. Most popular LLMs today are transformers, dense or MoE.

Where Popular Models Fall

Here is where some well-known models land at the time of writing:

[Image: Popular models grouped by design. Dense: Llama 3.1 405B, Gemma 3 27B, Qwen3-32B, Ministral 3 14B, and DeepSeek-R1-Distill-Qwen-32B. Mixture of experts: Mixtral 8x7B, gpt-oss-120b, Llama 4 Maverick, Qwen3-235B-A22B, DeepSeek-V3 and R1, Mistral Large 3, and Gemini 2.5, which is closed. Hybrid: Jamba. Not published: GPT-5 and Claude, both closed.] (https://www.kirupa.com/ai_fundamentals/images/types_of_ai_models_where.png)

Open-weight models publish their sizes, since anyone can download the weights and count them. Here are the numbers, with each name linked to its source:

Model | From | Type | Total parameters | Active per token | Weights
Llama 3.1 405B (https://arxiv.org/abs/2407.21783) | Meta | Dense | 405B | 405B | Open
Qwen3-32B (https://huggingface.co/Qwen/Qwen3-32B) | Alibaba | Dense | 32.8B | 32.8B | Open
Gemma 3 27B (https://arxiv.org/abs/2503.19786) | Google | Dense | 27B | 27B | Open
Ministral 3 14B (https://mistral.ai/news/mistral-3) | Mistral AI | Dense | 14B | 14B | Open
Mixtral 8x7B (https://arxiv.org/abs/2401.04088) | Mistral AI | MoE | 47B | 13B | Open
gpt-oss-20b (https://openai.com/index/introducing-gpt-oss/) | OpenAI | MoE | 21B | 3.6B | Open
gpt-oss-120b (https://openai.com/index/introducing-gpt-oss/) | OpenAI | MoE | 117B | 5.1B | Open
Llama 4 Scout (https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) | Meta | MoE | 109B | 17B | Open
Llama 4 Maverick (https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) | Meta | MoE | 400B | 17B | Open
Qwen3-235B-A22B (https://arxiv.org/abs/2505.09388) | Alibaba | MoE | 235B | 22B | Open
DeepSeek-V3 and R1 (https://huggingface.co/deepseek-ai/DeepSeek-R1) | DeepSeek | MoE | 671B | 37B | Open
Mistral Large 3 (https://mistral.ai/news/mistral-3) | Mistral AI | MoE | 675B | 41B | Open
DeepSeek-V4-Pro (preview) (https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) | DeepSeek | MoE | 1.6T | 49B | Open
Jamba (https://arxiv.org/abs/2403.19887) | AI21 Labs | Hybrid (Transformer and Mamba) with MoE | 52B | 12B | Open
Gemini 2.5 (https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf) | Google | MoE | Not published | Not published | Closed
GPT-5 | OpenAI | Not published | Not published | Not published | Closed
Claude | Anthropic | Not published | Not published | Not published | Closed

Families often come in both kinds. Qwen3 has six dense sizes and two MoE sizes, and Mistral 3 pairs small dense Ministral models with an MoE Mistral Large 3. The biggest open models are almost all MoE, which makes sense given how much math experts save per token at that size.

Closed models are harder. OpenAI's GPT-4 technical report (https://arxiv.org/abs/2303.08774) said it "contains no further details about the architecture (including model size)", and at the time of writing, OpenAI hasn't published those details for GPT-5, nor has Anthropic for Claude. Google is a partial exception. Its Gemini 1.5 report (https://arxiv.org/abs/2403.05530) calls Gemini 1.5 Pro "a sparse mixture-of-expert (MoE) Transformer-based model", and the Gemini 2.5 report says the same of Gemini 2.5, with no parameter counts. Numbers for closed models do float around online, but they're outside estimates and leaks the companies haven't confirmed.

Conclusion

We started with the bakery owner squinting at names like Qwen3-235B-A22B, and now we can read them. Inside nearly every LLM is a stack of transformer layers, each with an attention part and a feed-forward part. A dense model sends every token through all of its weights. A mixture-of-experts model splits the feed-forward part into experts and has a router pick a few for each token, at every layer. That's why MoE models have two sizes: total parameters decide how much memory we need, and active parameters decide how much math each token costs. For Crumb & Co., a model like gpt-oss-20b, with 21 billion parameters but only 3.6 billion active, is quick for every token, but the bakery still has to hold all 21 billion, which OpenAI squeezes into 16 GB of memory with quantization.

The other labels sit on top of all this. Reasoning is a training style, multimodal is about what goes in, and distilled is about how a model learned, so one model can be all three. Past the transformer, designs like Mamba, Jamba, and diffusion language models are trying different trade-offs, even if most popular models don't use them yet.

Whatever a model is built from, every one of them shares one stubborn habit. Ask it something it doesn't know, and it might answer anyway, with total confidence. Why that happens is where we're going next in Why AI Makes Things Up (https://www.kirupa.com/ai_fundamentals/why_ai_makes_things_up.htm).