[WIP]Feature/issue 660

[shes50103]

• Implement underscore feature
• make sure underscore using in the right way

1_2_3 => 123
1_2.3_4 => 12.34
1__2 => Error
_1 => Error
1_ => Error

This closes #660

[codecov]

Codecov Report

Merging #666 into master will decrease coverage by 0.01%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master     #666      +/-   ##
==========================================
- Coverage   82.94%   82.93%   -0.02%     
==========================================
  Files          56       56              
  Lines        7501     7507       +6     
==========================================
+ Hits         6222     6226       +4     
- Misses       1027     1028       +1     
- Partials      252      253       +1

Impacted Files	Coverage Δ
compiler/lexer/lexer.go	94.7% <77.77%> (-0.57%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c52b53e...98de941. Read the comment docs.

[st0012]

I think it's worth to mention how the wrong cases would behave in Ruby. Like 1_ in Ruby will do nothing instead of raising an error

[SD10]

IMO I think the lexer should preserve the underscores in the token. In terms of responsibility, I don't think a lexer should be responsible for any code transformation. Removing them at this stage seems sort of magical.

IMO the token should be simplified at a later stage, maybe during parsing or eval?

[st0012]

Well I think @SD10 has a good point. Keeping those information to parser allows us to provide more sophisticated error message

[SD10]

Here we are already transforming the token into a literal for Int:

goby/compiler/parser/data_type_parsing.go

Line 12 in c00ad0c

func (p *Parser) parseIntegerLiteral() ast.Expression {

Here for Float:

goby/compiler/parser/data_type_parsing.go

Line 26 in c00ad0c

func (p *Parser) parseFloatLiteral(integerPart ast.Expression) ast.Expression {

I think this is a good place to do the transformation work.

This is just my opinion. I'm a newbie 😊

[SD10]

It's also worth noting that in the Ruby lexer the underscores are maintained:

...
[[2, 7], :on_int, "1_000_000"],

[shes50103]

I will preserve the underscores in the token and handle it in parser,
in ruby irb 1_ this case will raise error at the next input so I will still remain this feature