Lexical analyzer (tokenizer)

The lexical analyzer is the part of the compiler that detects the token of the program and sends it to the syntax analyzer. The token is the smallest entity of the code, it is either a keyword, identifier, constant, string literal, or symbol. Examples of different types of tokens in C.

PDF

• Lexical Analysis (Tokenizing) - Computational Geometry Lab

Jason Pepas

• A Lisp interpreter in C, part 15: lambda
• A Lisp interpreter in C, part 14: scope (a.k.a let)
• A Lisp interpreter in C, part 13: if and cond
• A Lisp interpreter in C, part 12: quote
• A Lisp interpreter in C, part 11: apply
• A Lisp interpreter in C, part 10: bind (a.k.a define)
• A Lisp interpreter in C, part 9: nil and bool
• A Lisp interpreter in C, part 8: the environment
• A Lisp interpreter in C, part 7: comments
• A Lisp interpreter in C, part 6: lists
• A Lisp interpreter in C, part 5: escapes
• A Lisp interpreter in C, part 4: strings
• A Lisp interpreter in C, part 3: floats
• A Lisp interpreter in C, part 2: integers
• A Lisp interpreter in C, part 1: symbols