AI Knowledge Map

Role304 nodesmap v2026-05-22-002

Take the qualification test to see what you know.Sign in

StrongPartialWeakUntestedTrend chips use dashed borders.

GPU & CUDA

45 nodes

Execution Model

CUDA Hello World Threads, Blocks, and Grids Mapping to Hardware — SMs Warps — 32 Threads in Lockstep Warp Divergence

Memory Access Patterns

Why Access Patterns Matter Coalesced Access Strided & Random Access Shared Memory Bank Conflicts Putting It Together

Memory Hierarchy

The Memory Ladder Registers & Register Spill Shared Memory & L1 Cache HBM — Where Your Model Lives NVLink & PCIe

Operator Fusion & FlashAttention

The Kernel Launch Problem Operator Fusion FlashAttention: The Problem FlashAttention: The Solution IO Complexity: Why It's Faster

Roofline Model

The Bottleneck Question Reading the Roofline Plot Where ML Operations Fall Prefill vs Decode Changing the Roof

Tensor Cores & Mixed Precision

Beyond CUDA Cores Precision Formats Precision-Throughput Tradeoff Dimension Alignment Mixed Precision Training & Inference

Tiling & Matrix Multiply

Naive Matrix Multiply The Tiling Idea Tiled Matmul Kernel Why Tiling Works Tree Reduction From Tiling to FlashAttention

Triton & torch.compile

The CUDA Expertise Gap Triton by Example How Triton Works torch.compile When to Use What

Why GPUs?

The CPU — Few Powerful Cores The GPU — Thousands of Simple Cores ML Is Matrix Math What Is CUDA?

LLM Internals

56 nodes

Self-Attention

Q / K / V Projections Why Attention?Q, K, V: Three Roles Computing Attention Scores From Scores to Output Multi-Head Attention Causal Masking

Batching

Why GPUs Need Batching Static Batching Continuous Batching Prefill vs Decode Scheduling Memory Limits the Batch Batching in Production

Embeddings

From Token IDs to Vectors What Do Dimensions Mean?Tokens in Space Measuring Similarity Positional Encoding Why Embeddings Matter

Text Generation

One Token at a Time From Logits to Probabilities Temperature Top-K and Top-P Sampling Greedy vs Creative Generation in Practice

KV Cache

The Redundancy Problem The KV Cache Solution Prefill vs Decode Memory Cost GQA: Shrinking the Cache KV Cache in Production

Paged Attention

The Fragmentation Problem Virtual Memory for GPUs Block Tables & Allocation Copy-on-Write & Prefix Sharing PagedAttention in Practice The Full Serving Stack

Quantization

INT8 vs INT4 Quantization The Size Problem How Numbers Shrink The Quantization Process The Outlier Problem Modern Methods What to Quantize

Tokenization

Why Tokenization?BPE: The Algorithm Explore the Patterns Training vs Inference Byte-Level BPE Why It Matters

Transformer Block

The Transformer Block Layer Normalization Residual Connections The Feed-Forward Network Pre-Norm vs Post-Norm Modern Variants & Scale

Trends

LLM Serving

36 nodes

CUDA Graphs

Kernel Launch Overhead Why Decode Is the Worst Case Capture & Replay The Padding Ladder Production Tradeoffs

Inference Engine

The Inference Engine The Scheduler The Memory Manager The Model Executor Continuous Batching in Action

Multi-LoRA Serving

The Cost Crisis LoRA Math at Serving Time Heterogeneous Batches and SGMV Unified Paging: Where Does Your Adapter Live?Rank Tradeoff: Adapters Steal from KV

Prefill/Decode Disaggregation

The Interference Problem Chunked Prefill Full Disaggregation The KV Cache Transfer When to Use What

Prefix Caching & RadixAttention

The Shared-Prefix Waste Why Full-Prompt Hashing Fails The Prefix Tree (RadixAttention)Block-Hash Chain (vLLM APC)Eviction and Copy-on-Write When It Helps, When It Fails

Serving Metrics & SLOs

The Metrics That Matter Percentiles & Tail Latency Throughput vs Goodput SLO-Driven System Design Capacity Planning & Saturation

Speculative Decoding

The Decode Bottleneck The Draft-Verify Idea The Verification Algorithm Draft Model Variants When to Use It

AI Agents

49 nodes

The Agent Loop & State

Why Agents?Strip Away the Peripherals The State Object Inside a Tick Why Workflows First?The Anatomy of a Harness

Capstone — Three Designs

The Task & Three Designs Trace Comparison Failure Modes + Trifecta Exposure Cost & Reliability Plot The Decision Rule

Context Engineering

Context as Scarce Resource The 4 Failure Modes The 4 Fixes Tool-Loading Strategy Subagents as Context Isolation

Evals & Diagnostics

Compounding Errors Error Analysis First Golden Cases Pass/Fail vs Score The 4 Eval Failure Modes

Planning & Reflection

The Single-Shot Failure When to Spend More Tokens When to Pause and Observe When to Retry When to Stop

Retrieval & RAG

Why Retrieval?Embeddings as Coordinates Retrieve-Then-Generate Chunking Strategies ANN — Recall vs Speed RAG Failure Modes

Security & the Lethal Trifecta

The Lethal Trifecta Draw the Data-Flow Graph Cut a Leg Capability Scoping Output Exfiltration via Tool Calls

Tool Use & Function Calling

Why Tools?The Tool Schema Tool Design — ACI + Anti-Patterns Structured Outputs MCP, Two Ways Skills as a Primitive Tools as Attack Surface

Workflow Patterns

When NOT to Use an Agent Chaining + Routing Parallelization + Voting Orchestrator-Workers + Subagents Evaluator-Optimizer

Agent Engineering

45 nodes

Agent Teams

When Teams Beat a Single Agent Supervisor / Worker Parallel Agents & Voting Handoffs Coordination Costs

Cost & Latency Engineering

The Cost Profile of an Agent Prompt Caching Result Caching Parallelizing Tool Calls Batching at the Agent Layer

Deployment & Rollout

The Prompt-as-Code Problem Canary Rollouts Rolling Releases Version Pinning Rollback Discipline

Layered Guardrails

Defense-in-Depth for Agents Input Filters Output Filters Policy Enforcement Fail-Safe vs Fail-Open

Production Harness Architecture

Why a Harness Fails in Production Idempotency — Safe to Re-run Checkpoints & Resumption Retry Policy — When to Try Again Durable Execution Platforms

Incident Handling

The First 15 Minutes Trace Replay in Anger Root-Cause Discipline Postmortem Pattern Incident Drills

Observability for Agents

The Span-per-Tick Model What to Log (and What Not To)Trace Replay Metrics That Matter Alerting on Agent Behavior

Production Evals & Shadow Mode

Online vs Offline Evals Shadow Mode A/B Harness Drift Detection Eval-Driven Rollout

Capstone — Reliability Operations

SLOs for an Agent Fleet Error Budgets On-Call Rotations Runbooks That Actually Work From Foundations to Engineering

Inside vLLM

72 nodes

Attention Backends

Three Machines, One Question The Platform Layer The Backend Registry What a Backend Demands Adding a Backend How It Fails

Capstone — Predict the PR

How This Works PR #45840 — The Narrow One PR #48011 — The Cross-Module PR PR #44165 — Where the Tests Live Reading a Diff You've Never Seen What This Track Left Out

Compilation & CUDA Graphs

What You Know One Config, Five Fields Why Split the Graph The Shape Ladder What Startup Buys You How It Fails

Distributed Execution

Three Axes, One Point Each How Do TP, PP and DP Become Processes?How Does vLLM Decide Which Ranks Exchange Data?What Actually Goes Out Over the Wire?Simplified Stand-In, or the Real Thing?What Happens When One Worker Process Dies?

KV Cache Manager

Four Layers, Not One Asking for a Cache Hit The Longest Match Allocating the Rest Where Blocks Come From How Allocation Fails

KV Connectors & Disaggregation

What You Know Two Sides The Factory Lookup and Save Wiring a P/D Pair How It Fails

Model Registry

A Name, Not a Class How Resolution Actually Runs The Loader Interface Where Weights Land The Same Checkpoint, Two Different Splits What Adding a Model Requires

Model Runner

Who Calls the GPU The Batch That Persists Building This Step's Inputs The Forward Pass Two Calls, Not One Where the Executor Varies

Output & Streaming

The Return Leg Starts in the Engine Back Across the Boundary The Output Handler Loop Token IDs Into Text Why Partial Tokens Wait Streaming to the Client

Request In, Tokens Out

The Whole Path in One Picture What vllm serve Actually Spawns In: Rendering and Tokenization Across the Boundary The Engine Loop Out: Detokenization and Streaming

Sampler

Sampling Runs on the GPU The Logits-Processor Chain Order Is the Contract Inside Sampler.forward Greedy and Random Side by Side Request-Level vs Server-Level

The Scheduler

Two Halves, One Class Inside the Budget Loop Which Queue a Request Is In The Branch That Preempts Closing the Loop What the Flags Touch

Streaming to the Client

Untested

OpenAIServingChat.chat_completion_stream_generator iterates the async generator AsyncLLM.generate() yields into, and turns each RequestOutput into one data: line, closing with data: [DONE]. Exactly one bare continue skips an output, and its condition is a conjunction of three things all being simultaneously true: no delta text, no token ids, and nothing previously sent for that choice — the comment names it directly as the chunked-prefill case, a forward pass over prompt tokens that produced no generated token to report at all. A held-back byte-fallback token is not this case: it has a real token id even though its delta text is empty, so the second clause of the conjunction is false, the continue does not fire, and an ordinary frame is written with empty delta content. On the consumer side, AsyncLLM.generate drains the per-request collector with q.get_nowait() or await q.get() — a fast path that only suspends when the collector is genuinely empty, since whatever is already sitting there already reflects everything accumulated so far. When a client disconnects, the HTTP framework's cancellation (or garbage collection of an abandoned generator) raises inside this same call stack; the except clause aborts the request and lets that abort travel back across the process boundary to free its KV blocks.

Learn from sources

moduleModule: Output & Streaming — step Streaming to the Client