Grabbing some text unparsed

[siakc]

I am writing a very simple parser to read a code and parse it in some conditions but return raw input string in other cases. For example:

type Sql struct {
	CreateDatabase *CreateDatabase `("CREATE DATABASE" @@`
	DropDatabase *DropDatabase ` | "DROP DATABASE" @@`
	Other string ` | (@Ident  | @Number | @Operators | @String )*)`
}

	sqlLexer = lexer.MustSimple([]lexer.SimpleRule{
		{`Keyword`, `(?i)\b(CREATE DATABASE|DROP DATABASE)\b`},
		{`Ident`, `[a-zA-Z_][a-zA-Z0-9_]*`},
		{`Number`, `[-+]?\d*\.?\d+([eE][-+]?\d+)?`},
		{`String`, `'[^']*'|"[^"]*"`},
		{`Operators`, `<>|!=|<=|>=|[-+*/%,.()=<>]`},
		{"whitespace", `\s+`},
	})

Of course this is not working but hope it gives you the idea.
I see the problem this way: After the lexer does its job what we have is tokens. We don't have the raw input accessible in parser. One way could be that our parser would somehow provide all the remaining input as an alternative pathway. I tried adding

{"All", `.*`}

But it does not seem to work I think because lexer tries to match in sequence after whitespace there is nothing left for "All" to grab. If we place it in higher order it will catch everything and nothing is left for the remaining rules. If I could do something like:

	sqlLexer = lexer.MustSimple([]lexer.SimpleRule{
                {"Rest{InParallelForEachRule}", `.*`}, // A qualifier to tell lexer to keep this for every other rule matched
		{`Keyword`, `(?i)\b(CREATE DATABASE|DROP DATABASE)\b`},
		{`Ident`, `[a-zA-Z_][a-zA-Z0-9_]*`},
		{`Number`, `[-+]?\d*\.?\d+([eE][-+]?\d+)?`},
		{`String`, `'[^']*'|"[^"]*"`},
		{`Operators`, `<>|!=|<=|>=|[-+*/%,.()=<>]`},
		{"whitespace", `\s+`},
	})

Then I could have my parser to capture the Rest anywhere I wanted.

I didn't go deep into writing custom lexers to see if it can be solved that way. But is it possible without doing so? Am I understanding the problem correctly?

[EDIT]
Well I did a trick as a workaround for the problem:

type Sql struct {
	CreateDatabase *CreateDatabase `("CREATE DATABASE" @@`
	DropDatabase *DropDatabase ` | "DROP DATABASE" @@`
	Other []string ` | (@Ident | @Number | @String | @Operators )*)` // Capture all tokens into an array of strings
}
sqlLexer = lexer.MustSimple([]lexer.SimpleRule{
        {`Keyword`, `(?i)\b(CREATE DATABASE|DROP DATABASE)\b`},
        {`Ident`, `[a-zA-Z_][a-zA-Z0-9_]*`},
        {`Number`, `[-+]?\d*\.?\d+([eE][-+]?\d+)?`},
        {`String`, `'[^']*'|"[^"]*"`},
        {`Operators`, `<>|!=|<=|>=|[-+*/%,.()=<>;]`}, // Added semicolon
        {"whitespace", `\s+`},
})

repr.Println(strings.Join(sql.Other, " ")) // Reconstruct the original query