Adding a custom lexer rule without overriding current rules.

[kafkiansky]

I want to add a custom rule, but so that I don't have to specify all existing rules. Code example:

package main

import (
	"github.com/alecthomas/participle/v2"
	"github.com/alecthomas/participle/v2/lexer"
)

type Test struct {
	Name     string `"action" @String`
	Duration string `"duration" @Duration`
}

var (
	parser = participle.MustBuild[Action](
		participle.Unquote("String"),
		participle.Lexer(
			lexer.MustSimple([]lexer.SimpleRule{
				{`Duration`, `\d+.\d+.\d+`},
			}),
		),
	)
)

However, now I get an error: mapper:(participle.Mapper)(0x10e0240)} uses unknown token "String"

[alecthomas]

There's no way to "inherit" from the default lexer as it's actually the default Go one.

[alecthomas]

That probably wasn't very clear, sorry. By "default Go one" I mean it just uses text/scanner under the hood, so it is completely unrelated to Participle's regex-based lexer.

[alecthomas]

You could probably combine them by implementing a combining lexer.

[kafkiansky]

Could you show me an example? Because here's what I came up with, but it obviously doesn't work.

package main

// imports

type CombinedLexerDefinition struct {
	custom  lexer.Definition
	scanner lexer.Definition
}

func (d CombinedLexerDefinition) Lex(filename string, r io.Reader) (lexer.Lexer, error) {
	var (
		customLxr  lexer.Lexer
		scannerLxr lexer.Lexer
		err        error
	)

	var buf bytes.Buffer
	tee := io.TeeReader(r, &buf)

	scannerLxr, err = d.scanner.Lex(filename, tee)
	if err != nil {
		return nil, err
	}

	customLxr, err = d.custom.Lex(filename, &buf)
	if err != nil {
		return nil, err
	}

	return CombinedLexer{
		scanner: scannerLxr,
		custom:  customLxr,
	}, nil
}

func (d CombinedLexerDefinition) Symbols() map[string]lexer.TokenType {
	symbols := make(map[string]lexer.TokenType)

	for symbolName, symbolType := range d.scanner.Symbols() {
		symbols[symbolName] = symbolType
	}

	for symbolName, symbolType := range d.custom.Symbols() {
		symbols[symbolName] = symbolType
	}

	return symbols
}

type CombinedLexer struct {
	custom  lexer.Lexer
	scanner lexer.Lexer
}

func (l CombinedLexer) Next() (lexer.Token, error) {
    // ?
}

var (
	parser = participle.MustBuild[Action](
		participle.Unquote("String"),
		participle.Lexer(CombinedLexerDefinition{
			scanner: lexer.TextScannerLexer,
			custom: lexer.MustSimple([]lexer.SimpleRule{
				{`Duration`, `\d+.\d+`},
			}),
		}),
	)
)

In the Next method we have to make progress for two lexers at once, but it is unclear how to do this if the custom lexer does not know about other tokens besides its own.

[dibrinsofor]

@kafkiansky were you able to get around this?