Support underscores in number literals

[smnandre]

{{ 1000 == 1_000 ? 'yes' : 'no' }}
# now: syntax error
# this PR: "yes"

As of PHP 7.4.0, integer literals may contain underscores (_) between digits, for better readability of literals. These underscores are removed by PHP's scanner.

https://www.php.net/manual/en/language.types.integer.php

This PR replicates that behaviour, using the regexp to match the literals and then remove the "_".

I'm targeting Twig4 but maybe 3.x would be ok?

I cannot think of a real case that

• does not trigger an error currently
• would work differently after this PR

[fabpot]

LGTM on 3.x
Can you also add an entry in the changelog and add a note in the docs ?

[smnandre]

@fabpot updated to target 3.x, mention syntax in documentation & add a line in changelog

[fabpot]

Maybe we can do the same as in https://github.com/symfony/symfony/blob/7.3/src/Symfony/Component/ExpressionLanguage/Lexer.php#L42-L43 where the exponent can also accept _.

[fabpot]

symfony/symfony#44073

[fabpot]

What about doing the same changes as we’ve done on Symfony? Both the regex and the logic.

[smnandre]

Yep i have been interrupted earlier.. will finish later today!

[smnandre]

Update: i missread things.. i'm sorry.

[smnandre]

Update: i missread the scanner code.. 🤕

[smnandre]

@fabpot it took time to hit me ... it should be as you expected now.

Only thing missing is the "float without integer" syntax (ie .99).

What do you think we handle this in following PR ?
I'm afraid it would require other (bigger?) changes.

So, i had to update a bit the symfony regep, to disallow the "decimal part without integer part".

    public const REGEX_NUMBER = '/
                (?(DEFINE)(?P<LNUM>[0-9]+(_[0-9]+)*))
-                (?:\.(?&LNUM)|(?&LNUM)(?:\.(?!\.)(?&LNUM)?)?)(?:[eE][+-]?(?&LNUM))?/Ax';
+                (?:(?&LNUM)(?:\.(?!\.)(?&LNUM)?)?)(?:[eE][+-]?(?&LNUM))?/Ax';

The following tests pass with the current state of this PR, but would fail if we kept the original Symfony regexp.

❌ nested.definedArray.0

Twig\Error\Error: Twig\Error\SyntaxError: Unexpected token "number" ("end of print statement" expected) in "index.twig" at line 16.

Pretty much the same thing/cause

{{ nested.definedArray.0          is     defined ? 'ok' : 'ko' }}

https://github.com/twigphp/Twig/blob/6823b64cad3ce8f9300ff69cfa7d23e20d61f10c/tests/Fixtures/tests/defined.test

❌ if items.3 is defined

Twig\Error\Error: Twig\Error\SyntaxError: Unexpected token "number" of value "0.3" ("end of statement block" expected) in "index.twig" at line 7

Twig/tests/Fixtures/tags/apply/with_if_tag.test

Line 9 in 6823b64

{% if items.3 is defined %}

I guess we are in the same situation you explained me quite recently (with a block name "cb-1")

Not sure how would fix this though 🤷‍♂️

--

[fabpot]

Thank you @smnandre.