Skip to main content

Generative AI Model Architectures

Summary​

🧭 What this page covers

How Generative AI Works described the inference process as a black box: tokens in, forward pass, logits out. This page opens that black box. It covers what is inside a transformer layer, how self-attention gives the model its long-range reasoning ability, why different architecture families exist for different tasks, how vision and language are combined in multimodal models, and what Mixture of Experts means for production cost.

A model's architecture determines its fundamental reasoning capabilities—dictating how it processes information, relates concepts, and generates outputs. While training provides the knowledge, architecture defines the cognitive structure. Selecting an inappropriate architecture for a given task leads to systems that perform adequately in controlled demos but fail in complex production scenarios. For example, using a generation-optimized decoder model for semantic retrieval, or forcing a text-only architecture to process visual data, introduces significant performance bottlenecks and unnecessary computational costs on every request.

Architecture FamilyProcessing DirectionPrimary StrengthExample Models
🔁 Decoder-onlyLeft-to-right (Autoregressive)Text generation, conversational AI, code synthesisGPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro
🔎 Encoder-onlyBidirectional (Full Context)Embeddings, semantic retrieval, text classificationBERT, RoBERTa, Voyage-3
🔄 Encoder-decoderBidirectional Encoder + Autoregressive DecoderTranslation, summarization, complex sequence mappingT5, BART
🎨 DiffusionIterative DenoisingHigh-fidelity image, audio, and video generationStable Diffusion 3.5, Flux 1.1 Pro
🌐 MultimodalCross-modal Projection (e.g., Vision to Text)Analyzing and generating across mixed data typesGPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro

A standard enterprise retrieval-augmented generation (RAG) pipeline relies on two distinct model architectures working in tandem. An encoder model is used to compute embeddings for semantic retrieval, reading document chunks bidirectionally to produce rich, contextual vectors. A decoder-only LLM is then used to generate the final response, reading the retrieved context and user query left-to-right to synthesize a coherent answer. Attempting to use a single decoder model for both embedding and generation yields inferior retrieval quality at a significantly higher cost.

Self-attention is what gives transformers long-range reasoning

Before transformers, sequence models processed tokens one at a time and had to carry information across hundreds of steps to connect distant parts of a document. Self-attention lets every token directly attend to every other token in the context — a claim near the end of a contract can directly reference a definition at the start, without passing through intermediate tokens.

Diffusion models are not transformers — they work fundamentally differently

Text generation is autoregressive: one token at a time, each conditioned on all previous tokens. Image generation via diffusion is iterative denoising: starting from random noise and refining the entire image across multiple steps. The two architectures have different cost profiles, different failure modes, and different levers for quality control.

Mixture of Experts separates model capacity from inference cost

A dense model activates all its parameters for every token. An MoE model routes each token to 1–2 specialist sub-networks, leaving the rest idle. This means MoE models can have hundreds of billions of parameters — large capacity for diverse knowledge — while only using a fraction of those parameters per token, keeping latency and cost competitive with smaller dense models.

Why This Matters​

Architecture is not merely an implementation detail—it is a foundational decision that directly dictates the system's capabilities, performance boundaries, and operational costs at scale.

🔎
Wrong architecture = wrong retrieval
Decoder-only models are poor embedding models

Decoder-only LLMs read left-to-right, so each token's representation reflects only the tokens before it. Embeddings require a representation that captures the full meaning of a passage from all directions. Encoder models are purpose-built for this — they read bidirectionally. Teams that use a decoder-only LLM to generate embeddings for semantic search produce lower-quality vectors and pay 10–100× more per embedding call than purpose-built encoder models like Voyage-3 or Cohere Embed v3. Architecture selection in the retrieval layer has a larger impact on RAG quality than retrieval algorithm selection.

🖼️
Visual tokens consume context window budget
Multimodal calls are more expensive than they look

When you send an image to a multimodal model, the vision encoder converts it into hundreds of visual token embeddings that are projected into the language model's token space. A 1024×1024 image can consume 1,000–2,000 context tokens before a single word of your prompt is processed. Teams building multimodal RAG systems that extract diagrams from documents and inject them alongside text consistently hit context window limits without understanding why — the visual token overhead is invisible in API documentation but visible in token billing.

🧩
MoE changes the cost model for large deployments
Not all large models have proportionally large inference costs

Gemini 2.5 Pro is an MoE model. Its total parameter count is very large, but its active parameters per token are a fraction of that. This means a 200-billion-parameter MoE model can have inference latency and cost comparable to a 20–40 billion-parameter dense model. Teams building cost models for production deployments based on total parameter count of MoE models will significantly overestimate inference cost — and may unnecessarily downgrade to a smaller dense model when the larger MoE is both cheaper and more capable for their use case.

The key insight

Architecture families are not interchangeable. A production GenAI system typically uses at least two: a decoder-only model for generation, and an encoder model for retrieval. Understanding which architecture to use where is the first architectural decision in any serious deployment.

Core Concepts​

🧠 What is a Transformer?​

Before diving into the mechanics, it is essential to understand what a "transformer" actually is. Introduced in 2017 in the landmark paper Attention Is All You Need, the transformer is a deep learning architecture that revolutionized how machines process sequential data (like text).

Unlike older models that read text word-by-word sequentially, transformers process all words in a sequence simultaneously. They achieve this using a mechanism called self-attention, which allows the model to mathematically weight the importance of every other word in a sentence when processing a specific word, instantly mapping complex relationships regardless of how far apart the words are. This parallel processing and relationship-mapping ability makes transformers exceptionally powerful and scalable, forming the foundational backbone for nearly all modern generative AI models, including GPT, Claude, and Gemini.

🔁 Inside a Transformer Layer​

How Generative AI Works described the forward pass as a black box. Inside that black box, the model is a stack of N identical transformer layers, each performing the same two operations: self-attention and a feed-forward network, with residual connections and layer normalisation wrapping each.

①

Token embedding + positional encoding
Each token ID is mapped to a high-dimensional vector (typically 2,048–8,192 dimensions) via a learned embedding table. Since transformers process all tokens simultaneously, they have no inherent sense of word order. A positional encoding is added to inject information about where the token sits.

Example: The word "bank" gets converted into a list of numbers representing its general meaning. Positional encoding then adds data to indicate if it's the first word ("Bank on it") or the last ("river bank"), giving it spatial context.

②

Multi-head self-attention — every token attends to every other token
This is the defining operation of the transformer. For each token, three vectors are computed: a Query (what this token is looking for), a Key (what this token advertises about itself), and a Value (what information this token contributes if attended to). Attention scores are computed by comparing each token's Query against every other token's Key — tokens with high relevance to each other produce high scores. Those scores weight the Values to produce a new representation for each token that blends information from the entire context. Running multiple attention heads in parallel lets the model track several different relationship types simultaneously.

③

Residual connections preserve the original signal
After the attention operation, the original token embedding is added back to the attention output before layer normalisation. This residual connection serves two purposes: it prevents the gradient from vanishing during training across many layers, and it ensures the model can learn to "skip" the attention operation when the original representation is already sufficient. Deep transformers (GPT-4 has 96 layers) would not train stably without residual connections.

④

Feed-forward network applies per-token transformation
After attention, each token's representation passes through a small two-layer MLP independently. This is where most of the model's factual knowledge is stored — attention finds which tokens are relevant to each other, and the feed-forward network does the actual transformation. In standard dense models, this is the largest component by parameter count. In MoE models, multiple expert feed-forward networks replace this single network, and a router selects which expert to activate per token.

👁️ Why Self-Attention Matters​

Before transformers, the dominant architectures for sequence tasks were recurrent models (RNNs, LSTMs). These processed tokens one at a time, carrying a hidden state forward through the sequence. To connect a word at position 1 with a word at position 500, the information had to propagate through 499 intermediate steps — and consistently degraded in practice for long sequences.

Self-attention removes this constraint entirely. Every token in the sequence can attend directly to every other token in a single operation, regardless of distance. This is why large language models can track pronoun references, long-range logical dependencies, and clause relationships across hundreds of pages — not because they "remember" like a human, but because attention provides a direct computational path between any two positions in the context.

The cost of this capability: self-attention is quadratic in sequence length. Doubling the context from 1,000 to 2,000 tokens multiplies the attention computation by 4×, not 2×. This is the fundamental reason why longer context windows are more expensive, and why efficient attention approximations (sparse attention, sliding window attention) are active research areas.

🏗️ Architecture Families​

Not all models use the same transformer variant — and diffusion models are not transformers at all.

Decoder-only (GPT-style): Each token can only attend to previous tokens — this is called causal or masked attention. The model cannot look ahead. This makes it natural for generation: predict the next token given everything before it. At inference time, generating 100 tokens requires 100 sequential forward passes. GPT-5, Claude, Gemini, and LLaMA are all decoder-only.

Encoder-only (BERT-style): Each token attends to all tokens in both directions — full bidirectional attention. This produces a richer contextual representation of each token because its meaning is computed using the full surrounding context. Ideal for tasks that require understanding a complete passage: embedding generation, classification, named entity recognition. Cannot generate text natively. Voyage-3, Cohere Embed, and most reranker models are encoder-only.

Encoder-decoder (T5/BART-style): The encoder processes the full input with bidirectional attention, producing a rich contextual representation. The decoder then generates output token-by-token, attending to both its own previous outputs and the encoder's output via cross-attention. Optimal for tasks where the input and output are different sequences of different lengths: translation, abstractive summarisation, question answering from a fixed document set.

Diffusion models: Structurally different from all transformer variants. The training process adds Gaussian noise to real data (images, audio) across T timesteps until the data becomes indistinguishable from pure noise. The model learns to reverse this process — given a noisy image at step t, predict what it looked like at step t-1. At generation time, the model starts from pure noise and iteratively denoises across 20–50 steps. The backbone is typically a U-Net (for pixel-space diffusion) or a Diffusion Transformer (DiT), which applies transformer attention within the denoising process. Stable Diffusion 3.5, Flux 1.1 Pro, and DALL-E 3 are diffusion models.

ComparisonDecoder-onlyEncoder-onlyDiffusionExample workload
Generation methodAutoregressive (one token at a time).Not generative by default; produces representations.Iterative denoising (all pixels in parallel).Chat assistant uses decoder-only, while image synthesis pipeline uses diffusion.
Context readingLeft-to-right only.Bidirectional context understanding.Full spatial context at each denoising step.Semantic search index uses encoder-only because full input understanding matters.
Primary useText, code, chat.Embeddings, retrieval, classification.Images, audio, video.Enterprise copilot stack combines encoder retrieval plus decoder response generation.
Cost driverOutput token count dominates cost.Input token count and embedding volume dominate cost.Resolution multiplied by denoising steps dominates cost.High-resolution marketing image generation costs more than short-text support replies.
Determinism controlTemperature + top-p settings.Usually deterministic scoring outputs.Guidance scale + seed settings.QA automation sets low temperature for stable text outputs and fixed seed for reproducible visuals.

🌐 Multimodal Architectures​

Models like GPT-5.4, Claude Sonnet 4.6, and Gemini 2.5 Pro handle both text and images. The architecture that enables this is not a single unified model — it is a composition of two models joined by a projection layer.

A vision encoder (typically a Vision Transformer, or ViT) divides an image into a grid of fixed-size patches, treats each patch as a token, and processes them with transformer attention. The output is a sequence of image embeddings — one per patch. A projection layer (a small learned linear transform) maps these image embeddings into the same vector space as the language model's token embeddings. The projected visual tokens are then concatenated with the text token embeddings and fed into the decoder-only language model as if they were text tokens.

This means the language model never "sees" an image — it sees a sequence of vectors that represent image regions, mixed with vectors that represent text. It has learned during training to interpret those visual vectors in relation to the surrounding text. The practical implication: every image you send consumes context window space proportional to the number of image patches, not the byte size of the image file.

🧩 Mixture of Experts​

Standard transformers are dense — every layer uses all its parameters for every token. Mixture of Experts (MoE) is an architectural modification that replaces the dense feed-forward network in each layer with a set of N expert networks and a small router network.

For each token, the router selects the top-k experts (typically 1 or 2) and routes the token exclusively through those experts. All other experts are bypassed for that token. The result:

  • Total parameter count is large — there are many experts, each with its own weights
  • Active parameter count per token is small — only 1–2 experts fire per token per layer
  • Inference cost and latency are determined by active parameters, not total parameters

MoE models can store specialised knowledge in different expert networks — one expert may specialise in code, another in legal language, another in scientific notation — without every token paying the cost of routing through all of them. This is how Gemini 2.5 Pro achieves both high benchmark performance (large total capacity) and competitive inference cost (low active parameter count per call).

The practical implication: when evaluating models for production, compare active parameter count and measured latency, not total parameter count. A 600-billion-parameter MoE model may be faster and cheaper per token than a 70-billion-parameter dense model.

Manager vs Engineer​

👔 Manager We're building a semantic search feature on top of our document library. The team suggested using a different model for search than for the chat interface. Why can't we just use the same LLM for everything?

💻 Engineer The generation model and the retrieval model have opposite structural requirements. The chat LLM is decoder-only — it reads left-to-right and is optimised to predict what comes next. To produce an embedding for semantic search, you need a vector that captures the full meaning of a passage using context from all directions. Decoder-only models are poor at this because each token's representation only sees what came before it. Encoder models like Voyage-3 or Cohere Embed v3 read bidirectionally and are purpose-built for embeddings. Using the chat LLM for embeddings will produce lower-quality retrieval, at 10–100× the cost per document chunk compared to a dedicated encoder. The correct architecture for the retrieval layer is an encoder — it should never be the same model as the generation layer.

👔 Manager GPT-5.4 says it can analyse documents we upload. How is it reading a PDF if it's a text model?

💻 Engineer Two separate things happen. If you extract the text from the PDF and send it as text tokens, it is straightforward text processing — no vision involved. If you send the PDF as an image or as rendered pages, the multimodal vision encoder runs first: it divides each page into a grid of patches, converts them into visual token embeddings, and injects those into the language model's context alongside your text. The model then reasons over visual tokens and text tokens together. The important detail is that each page adds roughly 1,000–2,000 tokens of context window budget just for the visual representation — before your question is even counted. A 20-page document sent as rendered images can consume 20,000–40,000 tokens before you write a single word of prompt.

👔 Manager We heard Gemini is an MoE model. Does that mean it is less capable than a dense model of the same stated size?

💻 Engineer The opposite, typically. The stated parameter count for an MoE model refers to total parameters across all experts. At any given token, only 1–2 experts are active — so the active compute is much smaller than the total count suggests. What you get is large model capacity (many experts with specialised knowledge) at lower inference cost than a dense model with equivalent total parameters. The benchmarks for Gemini 2.5 Pro reflect that capacity — it performs at the level of large dense models while running cheaper per token. MoE is not a compromise on capability — it is a way to decouple model knowledge breadth from per-token inference cost.

👔 Manager Why does generating an image take more time and cost more than generating a similar-length text response?

💻 Engineer Completely different generation mechanisms. Text generation runs one forward pass per token — for a 200-word response, roughly 250–300 forward passes. Each is fast but they are sequential. Image generation via diffusion runs 20–50 full forward passes over the entire image simultaneously — each pass processes all pixels in parallel, applying the denoising step across the full image resolution. At high resolution (1024×1024), each forward pass over the full image is computationally heavier than a token-level pass in a language model. So diffusion is both more compute-per-step and runs many steps. The cost driver is resolution times number of denoising steps, not output "length" the way text is billed per token.

👔 Manager Our provider released a new model version with 64 transformer layers instead of 48. Is deeper always better?

💻 Engineer More layers increase the model's capacity to represent complex patterns, but the relationship is not linear and more layers adds latency. A deeper model requires more sequential forward passes through each layer for every generated token — depth directly adds to generation time. Wide-and-shallow models can sometimes match deep-and-narrow models on benchmark tasks while being faster. Whether the new version is actually better depends on benchmark results on tasks representative of your use case, not on layer count. The practical test: run a representative sample of your production queries through both versions, measure quality and latency, and decide based on that — not on the architecture number in the model card.

References and Next Steps​