Goldie: class Lexer

Goldie
Parsing System (v0.8)
Goldie Home (v0.8)
Getting Started

- Requirements
- Download / Install
- Never Used a Parsing Tool?
- Goldie Overview
- Beginner's Tutorial
- Grammar Definition Language
- Contact / Support
  - Message Board
  - Issue Tracking

Goldie's Tools

- DumpCGT
- GenDocs
  - Template Format
- GRMC: Grammar Compiler
- JsonViewer
  - Parse Tree Mode
- Parse
- StaticLang
- Other Tools

GoldieLib Overview

- Static And Dynamic Styles
- Ambiguous Symbols

GoldieLib Reference

- Base
- Exception
- class Language
- class Lexer
- class Parser
- struct Symbol
- class Token
- Ver

GoldieLib Sample Apps

- Calculator Static
- Calculator Dynamic
- Sample Generic Parse

CGT File Format
FAQ
Change Log
If You Use Git Master
Future Work
Developers

- Scripts for Developers
- Library Source and StaticLang

Goldie Home (v0.8) -> GoldieLib Reference -> class Lexer

# class Lexer

- class Lexer
  - class Lexer_{languageName}: Lexer

module goldie.lexer

class Lexer
readonly @property Token[] tokens
The tokens that resulted from the last lexing.
readonly @property LexError[] errors
The lexical errors encountered during the last lexing.
string source
The original source that was last lexed.
ptrdiff_t[] lineIndicies

Indicies into source where each line starts. For example, lineIndicies[0] is the index where the first line starts (always 0), lineIndicies[1] is the index where the second line starts, etc.

See the example in lineAtIndex below.

ptrdiff_t lineAtIndex(ptrdiff_t index)

Returns the line number (zero-indexed) of a given index into the source that was lexed. As per Goldie conventions, this value should be increased by one whenever it's displayed to the user.

Example:

string src = "ABC DEF HIJ"; auto lexerUsed = someLanguage.lexCodeX(src); auto numLines = lexerUsed.lineIndicies.length; // Get everything from start of second line, to start of third line auto secondLine = src[ lexerUsed.lineIndicies[1] .. lexerUsed.lineIndicies[2] ] writeln("Number of lines: ", numLines); writeln("Second line: ", secondLine); writefln("src[9] is '%s' and is on line #%s", src[9], 1 + lexerUsed.lineAtIndex(9)); // Output: // Number of lines: 3 // Second line: DEF // // src[9] is 'I' and is on line #3
string filename

The filename (if available) of the last source lexed.

ptrdiff_t srcIndex
ptrdiff_t line

Internal lexer state. Access is provided to these for diagnostic purposes. If an exception is thrown during lexing, then these can be inspected to check the lexer's internal state at the time of the exception. This is probably only useful for debugging Goldie.

Token[] process(string source, Language lang, string filename="")

Lexes (ie, tokenizes) a source. Throws a LexException if the source contains an error.

Normally, you would use one of the Language.parse or Language.lex functions instead of calling this directly, but this direct access is provided in case you'd rather re-use the same lexer instead of instantiating a new one every time.

The filename from which the source originated can be provided so error messages can report the appropriate filename.

module {user-specified package}.lexer

class Lexer_{languageName} : Lexer
{languageName} = Name of static-style language

This is the static-style counterpart to Lexer, and is generated by the StaticLang tool.

If the name of a language is, for example, foo, then the name of this class will be Lexer_foo.

The interface is the same as Lexer.