Building the Lexer

The plan.

We're building the lexer in stages, each one handling more of the language. At every step: write a test first, then make it pass.

The lexer is a struct that holds the source string, a cursor position, and the current line/column. It has one public method: NextToken() — call it repeatedly and it returns the next token until EOF.

The core loop looks at the character under the cursor, decides what kind of token is starting, reads until it's complete, then returns it. In pseudocode:

func NextToken() Token {
    skipWhitespace()

    ch := currentChar()

    switch {
    case ch == 0:      return token(EOF)
    case ch == '(':    return token(LPAREN)
    case ch == '+':    return token(PLUS)
    case isLetter(ch): return readIdentifier()
    case isDigit(ch):  return readNumber()
    case ch == '"':    return readString()
    default:           return error("unexpected character")
    }
}

Calling NextToken() repeatedly on this Monk source:

let x = 42 + 1

Produces this token stream:

When the lexer sees =, it can't immediately emit ASSIGN. It needs to peek at the next character:

This is a helper you'll use a lot: "if the next character matches, advance and return the longer token; otherwise return the shorter one." In Go, a simple if peek() == '=' conditional.

Whitespace (spaces, tabs, carriage returns) is skipped before every token. Newlines are skipped too, but they also increment the line counter and reset the column.

Comments start with // and run to the end of the line. When the lexer sees /, it peeks: next is /? Consume everything until newline. Next is =? Emit SLASH_ASSIGN. Otherwise, emit SLASH.

Monk has no multi-line comments. No /* */. This is deliberate — it keeps the lexer simple and avoids the nesting problem.

An identifier starts with a letter or underscore, then continues with letters, digits, or underscores. The scanner reads the whole word, then looks it up in the keyword table.

func readIdentifier() Token {
    start := position
    for isLetter(ch) || isDigit(ch) {
        advance()
    }
    word := source[start:position]
    kind := lookupKeyword(word) // returns IDENT if not a keyword
    return token(kind, word)
}

The same readIdentifier() call handles all three cases — it reads the characters, then the keyword table decides the kind:

The keyword lookup is a map from string to token type. 27 entries. No special logic — just a table lookup.

Number scanning has the most edge cases. The lexer needs to handle:

The strategy: start reading digits. If you hit 0x/0b/0o, switch to the appropriate digit set. If you hit a . followed by a digit, it's a float. If you hit e or E, it's scientific notation. Underscores are consumed but don't affect the value.

Two kinds: double-quoted strings and backtick template literals.

Double-quoted strings start and end with ". The lexer reads character by character, looking for the closing quote. If it hits a backslash, the next character is an escape: \n becomes newline, \" becomes a literal quote, \\ becomes a literal backslash. An unrecognized escape is an error.

Template literals start and end with backtick. Simpler — no escape sequences, multiline by default. Read until the closing backtick.

The biggest error to catch: an unterminated string (the file ends before the closing quote). Report the error with the position where the string started, not where it ran out.

A good lexer doesn't crash on bad input. It reports the error and keeps scanning — so the user sees all the problems in one pass, not one at a time.

Errors to catch:

The strategy: emit an ILLEGAL token with an error message, then continue scanning from the next character. The parser will see the ILLEGAL token and know something went wrong.

The full picture.

When you're done, the lexer handles all of Monk's syntax. Feed in any valid Monk program, get a correct token stream. Feed in garbage, get error tokens with useful positions.

Here's a real test to aim for:

const greet = (name string) string {
    return "Hello, " + name
}

let msg = greet("world")
// msg is "Hello, world"

This should produce 26 tokens (including EOF). If your lexer handles this correctly, Phase 1 is done.

Key takeaways