A compiler engineering analysis of tokenizing source streams, building custom abstract syntax tree nodes, handling lexical scoping blocks, and step-evaluating statements in TypeScript.
Below is a custom-built JS/TS tokenizer, parser, and runtime evaluator. Select a code preset, edit the code, and click **"Execute Code"** to see tokens generated, the parsed AST hierarchy, and the variables updated in real time.
In language runtimes, source code must be translated into instructions a computer can execute. There are two primary approaches to doing this:
Most modern systems (like V8 for JavaScript or JVM for Java) use a hybrid approach called Just-In-Time (JIT) Compilation. A JIT engine interprets code quickly at first, but profiles execution and compiles frequently used "hot" functions into native machine code in the background, combining the fast start time of an interpreter with the high runtime performance of a compiler.
The first step of an interpreter is Lexical Analysis, performed by a **Lexer** or **Tokenizer**.
A lexer takes a raw stream of characters (e.g., `let x = 10;`) and splits it into a flat array of categorized units called **Tokens**. Each token represents a basic grammar symbol, such as keywords (`let`), identifiers (`x`), assignment operators (`=`), literals (`10`), and punctuation (`;`).
During tokenization, the lexer strips out irrelevant characters like white spaces and code comments, and flags syntax errors like unclosed strings or unrecognized characters before the code is sent to the parser.
Once tokenized, the tokens are passed to the **Parser** for Syntactic Analysis.
The parser reads the flat token list and maps it into a hierarchical data structure called an Abstract Syntax Tree (AST). The AST represents the grammatical structure of the program.
Parsers usually use a technique called Recursive Descent Parsing. The parser has a set of recursive methods matching the grammar rules. It looks at the current token to decide which node type to parse, builds the node, and recursively parses its child expressions.
To evaluate the AST, the interpreter needs a way to track variables and functions. This is managed by the Environment (Scope).
An environment is basically a key-value hash map mapping variable names to their active values. To support nested blocks (like functions or loop scopes), environments are chained together:
"Each scope environment holds a reference to its outer 'parent' environment. When looking up a variable, the engine checks the current local scope first. If not found, it traverses up the chain until it finds the variable or hits the global environment."
This chaining of environments is what enables lexical scoping in modern programming languages.
Below is a clean, modular TypeScript implementation mapping the tokenizing, AST node building, and environment scope lookup:
export interface Token {
type: "KEYWORD" | "IDENTIFIER" | "NUMBER" | "OPERATOR" | "PUNCTUATION";
value: string;
}
export interface ASTNode {
type: string;
[key: string]: any;
}
export class Environment {
private values: Map<string, any> = new Map();
private parent: Environment | null;
constructor(parent: Environment | null = null) {
this.parent = parent;
}
// Define variable in current scope
public define(name: string, value: any): void {
this.values.set(name, value);
}
// Lookup variable traversing parent chain
public get(name: string): any {
if (this.values.has(name)) {
return this.values.get(name);
}
if (this.parent) {
return this.parent.get(name);
}
throw new Error(`ReferenceError: ${name} is not defined`);
}
// Assign variable in scope chain where declared
public assign(name: string, value: any): void {
if (this.values.has(name)) {
this.values.set(name, value);
return;
}
if (this.parent) {
this.parent.assign(name, value);
return;
}
throw new Error(`ReferenceError: Cannot assign to undefined variable ${name}`);
}
}
export class ASTEvaluator {
/**
* Recursively evaluates AST nodes in the current Environment scope
*/
public evaluate(node: ASTNode, env: Environment): any {
switch (node.type) {
case "Program":
let lastValue = null;
for (const statement of node.body) {
lastValue = this.evaluate(statement, env);
}
return lastValue;
case "VariableDeclaration":
const initVal = node.init ? this.evaluate(node.init, env) : null;
env.define(node.id, initVal);
return initVal;
case "AssignmentExpression":
const rightVal = this.evaluate(node.value, env);
env.assign(node.id, rightVal);
return rightVal;
case "Literal":
return node.value;
case "Identifier":
return env.get(node.name);
case "BinaryExpression":
const leftVal = this.evaluate(node.left, env);
const operand = this.evaluate(node.right, env);
return this.applyOperator(node.operator, leftVal, operand);
case "ForStatement":
// Initialize loop variable in a nested loop scope environment
const loopEnv = new Environment(env);
this.evaluate(node.init, loopEnv);
while (this.evaluate(node.test, loopEnv)) {
this.evaluate(node.body, loopEnv);
this.evaluate(node.update, loopEnv);
}
return null;
default:
throw new Error(`Unknown AST Node type: ${node.type}`);
}
}
private applyOperator(op: string, left: any, right: any): any {
switch (op) {
case "+": return left + right;
case "-": return left - right;
case "*": return left * right;
case "/": return left / right;
case "<=": return left <= right;
case "==": return left == right;
default:
throw new Error(`Unsupported operator: ${op}`);
}
}
}
Compiler engineering underpins every layer of the modern web stack—from translating JavaScript in the browser to bundling assets and minifying code.
By building custom interpreters and understanding AST traversals, developers gain the skills to build developer tools, custom DSLs, and high-performance runtimes.
I engineer highly secure distributed consensus state machines, horizontal WebSocket relay networks, and performant AI vector databases. Let's build resilient systems together.