Transformers → Large Language Models

Stack attention layers, train on a large fraction of the public text on the internet, and remarkable things happen.

Scale

A model’s raw weight storage is roughly parameters × bytes per parameter. One billion float32 parameters need about 4 GB before indexes, optimizer state, serving replicas, and safety systems.

Precision

float32 is easy to reason about but large. float16 and bfloat16 cut weight storage roughly in half and are common in modern training and inference stacks.

Operational cost

More parameters can improve capacity, but they also raise memory pressure, latency, energy use, deployment complexity, and evaluation burden.

Emergence

Capabilities like multi-step reasoning, code generation, and abstract analogy appear only above certain scales. They were not engineered in.

Generality

One model, many tasks. Translate, summarise, write code, answer medical questions, plan a trip — same weights.

Alignment

After pre-training, models are tuned with human preference data (RLHF) to be helpful, honest, and harmless.

Applied frame

Track tokens, context, attention, parameters, precision, and deployment cost as one system.

Diagram cue

The diagram connects sequence processing, attention flow, scale, and serving constraints.