-
Notifications
You must be signed in to change notification settings - Fork 97
Source changes history
Changeset 15610, Nov 5, 2008 - Syntax coloring
- Support for syntax coloring in customized editor control
- Grammar Explorer form is finally fixed and allows loading arbitrary grammars from compiled assemblies, instead of hard-coded list as it was before
Changeset 12333, Sept 1, 2008 - language runtime
- Implemented LanguageRuntime with dynamic operator dispatch - base class for custom language runtimes. One example is Scheme sample and SchemeRuntime class - a customization of Irony's LanguageRuntime class. Another example - CalcGrammar in Tutorial, uses default runtime implementation and provides evaluation of simple arithmetic expressions.
- Big refactoring: moved LALR parsing classes to a separate folder/namespace; making it possible to create other, non-LALR parsers with the same Grammar-construction methods and Scanner implementation. Planning to implement NLALR (Non-canonical LALR) parser algorithm. As part of refactoring, separated scanner-related data from parser data into ScannerControlData class.
- Implemented GrammarHint class, to allow adding hints for parser in grammar expressions - for example, to explicitly set an action in case of shift/reduce or reduce/reduce conflicts. Grammar.PreferShiftHere() and Grammar.ReduceThis() methods can be used to create hints inside grammar expressions. See GwBasic sample grammar, ELSE_CLAUSE_OPT and FUN_CALL elements for examples of hint use; also see SchemeGrammar.
- Cleaned up Scheme grammar, removed LALR conflicts by rearranging NonTerminals and adding grammar hints. Scheme now has real runtime implementation, supports various data types for arithmetic operations.
- Bug fix: double-escape char at the end of the string literal was not parsed correctly.
- Applied SecurityTransparent attribute to Irony assembly; more info: http://msdn.microsoft.com/en-us/library/bb397858.aspx
- RegexTerminal - enhanced following suggestions by user sakana280.
- CodePlex member "notmasteryet" submitted improved GWBasic grammar - thanks! I've tweaked it a bit to fix the conflicts.
Changeset 10877, May 16, 2008 - it's all about performance
- Changed GrammarData to use StringSets instead of StringLists for lookaheads computations - 2x+ performance improvement in parser initialization
- Dynamic operator dispatch engine - implementation of binary operators (+, -, etc.) and dynamic dispatch to implementation based on arg types. See Runtime/OperatorDispatch.cs file. Not actually used by any samples yet, only in unit test class - see OperatorDispatchTests.cs file for sample use and performance evaluation. And performance is not bad - less than 200 ns per operation on 2Hrz machine.
- Tests moved to VS Test framework - now run directly in Visual Studio, without NUnit.
- Added CreateSchemeNumber method to TerminalFactory (not quite R6RS yet though)
Changeset 10663, May 5, 2008 - a scheme for interpreter and interpeter for Scheme The first elements of interpeter engine. Implementation for trivial subset of Scheme - advanced enough to run 99 bottles. More details are coming. Note: Python and Ruby grammars are temporarily excluded from grammar samples. These grammar classes use the versions of Kleene operators (Star, Plus) that had been deprecated - so until they are updated, they are not available in samples.
Changeset 10309, April 17, 2008 - It is 2008, finally
- Solution migrated to VS 2008/.NET 3.5. Tests are still through NUnit - will be moved to MSTest later
- Added GwBasic grammar
- Improved MakeStarRule implementation with null delimiter - now introduces less conflicts
- Renamed Grammar.ExtraTerminals list to Grammar.NonGrammarTerminals. Added IsNonGrammar as TermOption; for terminals in NonGrammarTerminals this flag is set automatically; and in this case the parser ignores tokens of this type. This usually happens with Comment tokens.
- Fixed CommentTerminal to allow line comments to be the last tokens in the file, without final line break.
Changeset 10146, April 12, 2008 - Time to get real
- First REAL language grammar - c# 3.0 grammar. All language features except LINQ queries and preprocessor directives.
- Philipp Serr completed implementation of NumberTerminal with support for Complex numbers, Big Integers, and some advanced features required by c#, Python, VB
- Deprecated Kleene operators methods BnfTerm.Plus, Star(). Introduced Grammar.MakePlusRule, MakeStarRule methods that should be used for creating lists.
- Removed ReservedWord identifier, introduced Token.IsKeyword flag; renamed ReservedWords to Keywords, created internal hash table for quick keyword lookup.
- Refactored GrammarDataBuilder method for performance improvements
- Added Parser.PreviewSymbols method for resolving conflicts; method finds which of the tokens in a given list is coming first
- Add LineTerminators property to Grammar, to explicitly set characters used by Scanner for line counting
- CommentTerminal now allows multiple end symbols - used in c# LineComment terminal which can end with one of several line terminator symbols defined in c#.
- Added AstNode.Span property of type SourceSpan (StartLocation + Length) to identify the span of source code for the node
- Created AstNodeArgs struct - container for AstNode constructor arguments. The purpose is to simplify custom AstNode signatures
- Created static method CompilerContext.CreateDummy for creating dummy context for unit tests
- Create Token.CreateMultitoken method, to allow terminals to return multiple tokens packed in one.
- Improved error handling in GrammarDataBuilder & Explorer
- Added Grammar.FallbackTerminals - terminals without explicitly declared Firsts (start chars) or having start chars defined by Unicode category (like c# Identifier)
- Added skeleton classes and methods for language Runtime implementation.
Changeset 9122, Mar 7, 2008 - Terminals, part II
- Added CompoundTerminalBase class as an abstract base class for terminals having 3-part structure - prefix-body-suffix. Refactored Identifier, StringLiteral and NumberLiteral classes to inherit from this class. For more explanation of the concept see comments on top of the source file.
- Added a property Token.Details (of type ScanDetails) containing (optionally) the detailed information from scanning compound tokens: prefix, suffix, flags, etc.
- Full-featured implementation of IdentifierTerminal supporting all fancy features of c# 3.0 identifiers - prefixes (@), escaped chars, non-ASCII charsets, specifying allowed chars by unicode categories (including formatting chars)
- Removed NodeCreator delegate, added NodeCreating, NodeCreated events on NonTerminal - following standard template
- Renamed BnfElement to BnfTerm, BnfFlags to TermOptions
- Added DebuggerStepThrough attribute to properties and one-liners
- Added ISourceStream.MatchSymbol method - simplifies code in terminals
- Added Search box to GrammarExplorer for searching text in top/active textbox - contributed by Andrew Bradnan.
Changeset 8968, Mar 3, 2008 - It's all about Terminals
- Fully implemented StringLiteral, now supports escaped characters, multi-line strings; doubled quote inside as single.
- Fully implemented NumberTerminal - exponent support; custom exponent symbols; support for hex prefixes (0x...), type suffixes; support for BIG INTEGERS
- New TerminalFactory static class with methods for creating Number and String literals for c#, VB, Python
- Unit tests for terminals - many of them. See Readme file for instructions on running unit tests.
- Comment terminal - removed a special treatment of Line comments; comment now is StartSymbol...EndSymbol; line comments fit well into this, for example in c#: StartSymbol="//", EndSymbol = "\n"
- Removed CharLiteral class - for c# char literal use StringLiteral with IsChar option
- Added Script.NET grammar to the list of sample grammars
- Removed ErrorTerminal class, using base Terminal class for Grammar.Error singleton
Changeset 7937, Feb 15
- Repaired terminal sorting for scanning
- Fixed terminals sorting for display
- Fixed BnfElement.Star(delimiter) method - caused false grammar conflicts
- Fixed parsing for case-insensitive grammars - now should work correctly
- Fixed exception when NonTerminal's Rule property was not assigned - now reports error nicely and cancels the process
- Refactored Scanner to accept ISourceStream instead of SourceFile instance; a move towards using general Stream as source
- Refactored handling punctuation symbols - added IsPunctuation flag to BnfElement and removed Grammar.PunctuationSymbols symbols collection; use Grammar.RegisterPunctuation method for setting the flag in punctuation symbols and even NonTerminals
- Refactored operator symbols handling - added IsOperator flag, precedence and associativity properties directly to SymbolTerminal
- Streamlined AstNode creation process - moved most of the logic to Parser.CreateNode method, cleaned it up. (Python grammar is a bit broken as a result - AST tree shows some "extra" symbols)
- Deprecated GenericNode - AstNode is main base class for all Ast nodes
- Added facilities for VS integration: Scanner.GetNext method, returning tokens one-by-one; BraceMatchTokenFilter builds full list of brace pairs.
- Basic AST Visitor facility: AstNode.AcceptVisitor(IAstVisitor)
Changeset 6381
- Refactored internal representation of BnfExpression implementation, now without ExpressionType enumeration property
- Improved grammar data presentation by removing \b symbol from names
- Renamed NotTerminal.Expression property to Rule
- Added Values dictionary to CompilerContext
- Added Compiler property to CompilerContext
- Added Attributes property (dictionary) to AstNode class, to store any custom values associated with node/token
- Implemented basic error reporting and recovery
- Scanner error recovery: added "Delimiters" list to Grammar: on error scanner skips all until whitespace or delimiter.
- Added ErrorRule property to Non-terminal to define error recovery paths. Error expressions can also be mixed with normal expressions in Rule property
- Added ErrorTerminal and SyntaxError singleton in Grammar, to be used in error rules
- Added TryMatch method to grammar as backup for token recognition; you can use this method to create "fancy" tokens - those that use extended character sets for example and don't have limited set of prefixes.
- Added error rules to Ruby grammar and source sample with errors for Ruby
- Added Terminal parameter to MatchHandler delegate to provide reference to caller object