Description

The C++ Lexer Toolkit Library (LexerTk) is a simple to use, easy to integrate and extremely fast lexicographical generator - lexer. The tokens generated by the lexer can be used as input to a parser such as "ExprTk".

Capabilities

• Operators: +, -, /, *, %, ^, >>, <<, >, <, >=, <=, =, :=
• Symbols: Format [a..z|A..Z]{a..z|A..Z|0..9|_}
• Numbers: Integers, Real (scientific notation 1.234e+56)
• Strings: 'A string 1234 !@#$?'
• Brackets: (), [], {}
• Token processors and validators
• Single header implementation, no building required. No external dependencies.

The LexerTk will decompose an input character stream into a token stream compatible for the following BNF:

expression ::= term { +|- term }
term       ::= (symbol | factor) {operator symbol | factor}
factor     ::= symbol | ( '(' {-} expression ')' )
symbol     ::= number | gensymb | string
gensymb    ::= alphabet {alphabet | digit}
string     ::= "'" {alphabet | digit | operator } "'"
operator   ::= * | / | % | ^ | < | > | <= | >= | << | >> | !=
alphabet   ::= a | b | .. | z | A | B | .. | Z
digit      ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 |  7 | 8 | 9
sign       ::= + | -
edef       ::= e | E
decimal    ::= {digit} (digit [.] | [.] digit) {digit}
exponent   ::= edef [sign] digit {digit}
real       ::= [sign] decimal [exponent]
integer    ::= [sign] {digit}
number     ::= real | integer

Download

http://www.partow.net/programming/lexertk/index.html

Compatibility

The C++ Lexer Library implementation is fully compatible with the following C++ compilers:

• GNU Compiler Collection (4.1+)
• Intel® C++ Compiler (9.x+)
• Clang/LLVM (1.1+)
• PGI C++ (10.x+)
• Microsoft Visual Studio C++ Compiler (8.1+)
• IBM XL C/C++ (10.x+)

Simple Example 1

The following example will take a string representing an expression, tokenize it using the generator and then proceed to dump each of the tokens to stdout, providing information related to the token such as its type, value and position within the expression.

int main()
{
   std::string expression = "(1.1 + ( x - y * z / 1.23E+4 ) % w";

   lexertk::generator generator;

   if (!generator.process(expression))
   {
      std::cout << "Failed to lex: " << expression << std::endl;
      return 1;
   }

   lexertk::helper::dump(generator);

   return 0;
}

Expected output of the Simple Example 1 will be:

Token[00] @ 000  NUMBER  -->  '1.1'
Token[01] @ 003       +  -->  '+'
Token[02] @ 004       (  -->  '('
Token[03] @ 005  SYMBOL  -->  'x'
Token[04] @ 006       -  -->  '-'
Token[05] @ 007  SYMBOL  -->  'y'
Token[06] @ 008       *  -->  '*'
Token[07] @ 009  SYMBOL  -->  'z'
Token[08] @ 010       /  -->  '/'
Token[09] @ 011  NUMBER  -->  '1.23E+4'
Token[10] @ 018       )  -->  ')'
Token[11] @ 019       %  -->  '%'
Token[12] @ 020  SYMBOL  -->  'w'

Simple Example 2

The following example will take a string representing an expression, tokenize it using the generator. The tokens are then analyized in consequtive pairs. When a pair of tokens are determined to have an implied multiplication between themselves, a token of type multiplication is inserted between them. For example the expression '2x+1' becomes '2*x+1'.

int main()
{
   std::string expression = "1.23 (2.2variable1 / 3.3variable2) 4.56e+12 + variable3";

   lexertk::generator generator;

   if (!generator.process(expression))
   {
      std::cout << "Failed to lex: " << expression << std::endl;
      return 1;
   }

   lexertk::helper::commutative_inserter ci;
   ci.process(generator);

   lexertk::helper::dump(generator);

   return 0;
}

Expected output of the Simple Example 2 will be:

Token[00] @ 000  NUMBER  -->  '1.23'
Token[01] @ 005       *  -->  '*'
Token[02] @ 005       (  -->  '('
Token[03] @ 006  NUMBER  -->  '2.2'
Token[04] @ 007       *  -->  '*'
Token[05] @ 007  SYMBOL  -->  'variable1'
Token[06] @ 017       /  -->  '/'
Token[07] @ 019  NUMBER  -->  '3.3'
Token[08] @ 020       *  -->  '*'
Token[09] @ 020  SYMBOL  -->  'variable2'
Token[10] @ 029       )  -->  ')'
Token[11] @ 031       *  -->  '*'
Token[12] @ 031  NUMBER  -->  '4.56e+12'
Token[13] @ 040       +  -->  '+'
Token[14] @ 042  SYMBOL  -->  'variable3'

Simple Example 3

The following example will take a string representing an expression, tokenize it using the generator, then attempt to verify that the ordering of the brackets within the expression is correct.

int main()
{
   std::string expression = "{aa+(bb-[cc*dd]+ee)-ff}";

   lexertk::generator generator;

   if (!generator.process(expression))
   {
      std::cout << "Failed to lex: " << expression << std::endl;
      return 1;
   }

   lexertk::helper::bracket_checker bc;
   bc.process(generator);

   if (!bc)
   {
      std::cout << "Failed Bracket Checker!" << std::endl;
      return 1;
   }

   lexertk::helper::dump(generator);

   return 0;
}

Expected output of the Simple Example 3 will be:

Token[00] @ 000       {  -->  '{'
Token[01] @ 001  SYMBOL  -->  'aa'
Token[02] @ 002       +  -->  '+'
Token[03] @ 003       (  -->  '('
Token[04] @ 004  SYMBOL  -->  'bb'
Token[05] @ 005       -  -->  '-'
Token[06] @ 006       [  -->  '['
Token[07] @ 007  SYMBOL  -->  'cc'
Token[08] @ 008       *  -->  '*'
Token[09] @ 009  SYMBOL  -->  'dd'
Token[10] @ 010       ]  -->  ']'
Token[11] @ 011       +  -->  '+'
Token[12] @ 012  SYMBOL  -->  'ee'
Token[13] @ 013       )  -->  ')'
Token[14] @ 014       -  -->  '-'
Token[15] @ 015  SYMBOL  -->  'ff'
Token[16] @ 016       }  -->  '}'