Compiler Stages Explained: Lexer, Parser, and Code Generation for GCSE

A compiler translates high-level source code into machine code through a series of distinct stages: lexical analysis, syntax analysis, semantic analysis, intermediate code generation, optimisation, and code generation. Each stage transforms the program from one representation to another, and different categories of error are detected at different stages.

Why does compilation happen in stages?

A compiler's job is complex: it must parse human-readable text, check it conforms to the language's grammar, verify that it makes logical sense (correct variable types, defined names), and finally produce binary instructions the processor can execute. Splitting this into stages makes each part manageable and testable in isolation. It also means errors can be pinpointed precisely: a syntax error is found in stage two, a type error in stage three, each reported with the line number and cause.

Think of it like translating a book: first you tokenise the text into words, then check the grammar, then verify the meaning, and finally render it in the target language. Errors at each step are different in character.

Stage 1: Lexical analysis (the lexer or scanner)

The lexer reads source code as a sequence of characters and groups them into tokens — the smallest meaningful units of the language.

Input: area = length * width

Tokens produced:

The lexer also removes whitespace and comments (they carry no meaning for compilation) and flags any character that is not part of any valid token as a lexical error. An example would be using £ as a variable name in a language that does not permit it.

Stage 2: Syntax analysis (the parser)

The parser takes the stream of tokens and checks whether they conform to the grammar of the programming language — the rules about how tokens may be arranged. If valid, the parser builds an Abstract Syntax Tree (AST): a tree structure that represents the grammatical structure of the code.

Valid: area = length * width → assignment: identifier, operator, (expression: identifier, operator, identifier)

Syntax error example: area = * length width → the operator appears before the first operand, violating the grammar. The parser reports a syntax error at the position of *.

The AST for area = length * width has an assignment node at the root, with area as the left child and a multiplication node as the right child, which itself has length and width as its children.

Stage 3: Semantic analysis

Even if code is grammatically correct, it may still be meaningless. Semantic analysis checks for logical consistency:

• Type checking: Are operands compatible? You cannot multiply a string by a boolean.
• Declaration checking: Is every identifier declared before use?
• Scope checking: Are variables accessed within the correct scope?

Semantic error example:

area = "rectangle" * 5   # valid grammar but a type error in strict languages

The semantic analyser also builds a symbol table — a record of every identifier, its type, its scope, and where it is declared. This table is used by later stages to generate correct code.

Stages 4–6: Code generation, optimisation, and output

After semantic analysis, the compiler generates code:

Optimisation example: if the compiler detects that x = 2 * 4, it can replace this with x = 8 at compile time (constant folding) — the multiplication never runs at runtime. For Python, the "compiler" produces bytecode (.pyc files) for the Python Virtual Machine, not native machine code — the final native execution is handled by the interpreter.

How do compiler errors relate to the stages?

Understanding which stage each error comes from helps you interpret compiler messages. A "SyntaxError" in Python means the parser failed; a "TypeError" or "NameError" means semantic-style checks at runtime (Python is dynamically typed, so some semantic errors only appear at runtime rather than at compile time).

Frequently asked questions

Why does Python report some errors at runtime instead of compile time?

Python is a dynamically typed language: types are checked when operations actually execute, not before. A statically typed language (Java, C++) checks types at compile time and refuses to compile if types are incompatible. In Python, "hello" * "world" only raises a TypeError when that line runs — not when the file is first loaded. This is a trade-off: dynamic typing makes Python flexible and fast to prototype in; static typing catches more errors before any code executes.

What is the difference between a compiler and an interpreter if both translate code?

A compiler translates the entire source file into machine code (or bytecode) before any of it runs, producing a separate executable. An interpreter translates and executes the code line by line at runtime, with no separate output file. A compiler front-loads the translation work, making the resulting program faster to execute; an interpreter allows faster development cycles because you can run code immediately. Python uses a hybrid: the source is compiled to bytecode once, then the bytecode is interpreted by the Python Virtual Machine.

What is an abstract syntax tree and why is it useful?

An AST is a tree that represents the grammatical structure of source code, stripped of punctuation and whitespace. It is useful because it makes the structure of the program explicit and easy to manipulate: optimisers can traverse and transform the tree, code generators can emit instructions by walking the tree in the correct order, and analysis tools can detect patterns across the entire tree. The AST is the shared representation that passes between the front-end (lexer + parser) and back-end (optimiser + code generator) of the compiler.

Do GCSE students need to write a compiler?

No. At GCSE, you need to understand the conceptual stages of compilation (lexical analysis, syntax analysis, semantic analysis, code generation) and be able to identify what type of error each stage catches. Writing a compiler is an A-level or university project. Understanding the stages deepens your appreciation of how high-level code is transformed into something a processor can actually execute.

Understand how programming languages work from the inside with Professor Turing at aitutors.me.