Skip to content

Source changes history

Roman Ivantsov edited this page Dec 12, 2013 · 1 revision

Changeset 15610, Nov 5, 2008 - Syntax coloring

  • Support for syntax coloring in customized editor control
  • Grammar Explorer form is finally fixed and allows loading arbitrary grammars from compiled assemblies, instead of hard-coded list as it was before

Changeset 12333, Sept 1, 2008 - language runtime

  • Implemented LanguageRuntime with dynamic operator dispatch - base class for custom language runtimes. One example is Scheme sample and SchemeRuntime class - a customization of Irony's LanguageRuntime class. Another example - CalcGrammar in Tutorial, uses default runtime implementation and provides evaluation of simple arithmetic expressions.
  • Big refactoring: moved LALR parsing classes to a separate folder/namespace; making it possible to create other, non-LALR parsers with the same Grammar-construction methods and Scanner implementation. Planning to implement NLALR (Non-canonical LALR) parser algorithm. As part of refactoring, separated scanner-related data from parser data into ScannerControlData class.
  • Implemented GrammarHint class, to allow adding hints for parser in grammar expressions - for example, to explicitly set an action in case of shift/reduce or reduce/reduce conflicts. Grammar.PreferShiftHere() and Grammar.ReduceThis() methods can be used to create hints inside grammar expressions. See GwBasic sample grammar, ELSE_CLAUSE_OPT and FUN_CALL elements for examples of hint use; also see SchemeGrammar.
  • Cleaned up Scheme grammar, removed LALR conflicts by rearranging NonTerminals and adding grammar hints. Scheme now has real runtime implementation, supports various data types for arithmetic operations.
  • Bug fix: double-escape char at the end of the string literal was not parsed correctly.
  • Applied SecurityTransparent attribute to Irony assembly; more info: http://msdn.microsoft.com/en-us/library/bb397858.aspx
  • RegexTerminal - enhanced following suggestions by user sakana280.
  • CodePlex member "notmasteryet" submitted improved GWBasic grammar - thanks! I've tweaked it a bit to fix the conflicts.

Changeset 10877, May 16, 2008 - it's all about performance

  • Changed GrammarData to use StringSets instead of StringLists for lookaheads computations - 2x+ performance improvement in parser initialization
  • Dynamic operator dispatch engine - implementation of binary operators (+, -, etc.) and dynamic dispatch to implementation based on arg types. See Runtime/OperatorDispatch.cs file. Not actually used by any samples yet, only in unit test class - see OperatorDispatchTests.cs file for sample use and performance evaluation. And performance is not bad - less than 200 ns per operation on 2Hrz machine.
  • Tests moved to VS Test framework - now run directly in Visual Studio, without NUnit.
  • Added CreateSchemeNumber method to TerminalFactory (not quite R6RS yet though)

Changeset 10663, May 5, 2008 - a scheme for interpreter and interpeter for Scheme The first elements of interpeter engine. Implementation for trivial subset of Scheme - advanced enough to run 99 bottles. More details are coming. Note: Python and Ruby grammars are temporarily excluded from grammar samples. These grammar classes use the versions of Kleene operators (Star, Plus) that had been deprecated - so until they are updated, they are not available in samples.

Changeset 10309, April 17, 2008 - It is 2008, finally

  • Solution migrated to VS 2008/.NET 3.5. Tests are still through NUnit - will be moved to MSTest later
  • Added GwBasic grammar
  • Improved MakeStarRule implementation with null delimiter - now introduces less conflicts
  • Renamed Grammar.ExtraTerminals list to Grammar.NonGrammarTerminals. Added IsNonGrammar as TermOption; for terminals in NonGrammarTerminals this flag is set automatically; and in this case the parser ignores tokens of this type. This usually happens with Comment tokens.
  • Fixed CommentTerminal to allow line comments to be the last tokens in the file, without final line break.

Changeset 10146, April 12, 2008 - Time to get real

  • First REAL language grammar - c# 3.0 grammar. All language features except LINQ queries and preprocessor directives.
  • Philipp Serr completed implementation of NumberTerminal with support for Complex numbers, Big Integers, and some advanced features required by c#, Python, VB
  • Deprecated Kleene operators methods BnfTerm.Plus, Star(). Introduced Grammar.MakePlusRule, MakeStarRule methods that should be used for creating lists.
  • Removed ReservedWord identifier, introduced Token.IsKeyword flag; renamed ReservedWords to Keywords, created internal hash table for quick keyword lookup.
  • Refactored GrammarDataBuilder method for performance improvements
  • Added Parser.PreviewSymbols method for resolving conflicts; method finds which of the tokens in a given list is coming first
  • Add LineTerminators property to Grammar, to explicitly set characters used by Scanner for line counting
  • CommentTerminal now allows multiple end symbols - used in c# LineComment terminal which can end with one of several line terminator symbols defined in c#.
  • Added AstNode.Span property of type SourceSpan (StartLocation + Length) to identify the span of source code for the node
  • Created AstNodeArgs struct - container for AstNode constructor arguments. The purpose is to simplify custom AstNode signatures
  • Created static method CompilerContext.CreateDummy for creating dummy context for unit tests
  • Create Token.CreateMultitoken method, to allow terminals to return multiple tokens packed in one.
  • Improved error handling in GrammarDataBuilder & Explorer
  • Added Grammar.FallbackTerminals - terminals without explicitly declared Firsts (start chars) or having start chars defined by Unicode category (like c# Identifier)
  • Added skeleton classes and methods for language Runtime implementation.

Changeset 9122, Mar 7, 2008 - Terminals, part II

  • Added CompoundTerminalBase class as an abstract base class for terminals having 3-part structure - prefix-body-suffix. Refactored Identifier, StringLiteral and NumberLiteral classes to inherit from this class. For more explanation of the concept see comments on top of the source file.
  • Added a property Token.Details (of type ScanDetails) containing (optionally) the detailed information from scanning compound tokens: prefix, suffix, flags, etc.
  • Full-featured implementation of IdentifierTerminal supporting all fancy features of c# 3.0 identifiers - prefixes (@), escaped chars, non-ASCII charsets, specifying allowed chars by unicode categories (including formatting chars)
  • Removed NodeCreator delegate, added NodeCreating, NodeCreated events on NonTerminal - following standard template
  • Renamed BnfElement to BnfTerm, BnfFlags to TermOptions
  • Added DebuggerStepThrough attribute to properties and one-liners
  • Added ISourceStream.MatchSymbol method - simplifies code in terminals
  • Added Search box to GrammarExplorer for searching text in top/active textbox - contributed by Andrew Bradnan.

Changeset 8968, Mar 3, 2008 - It's all about Terminals

  • Fully implemented StringLiteral, now supports escaped characters, multi-line strings; doubled quote inside as single.
  • Fully implemented NumberTerminal - exponent support; custom exponent symbols; support for hex prefixes (0x...), type suffixes; support for BIG INTEGERS
  • New TerminalFactory static class with methods for creating Number and String literals for c#, VB, Python
  • Unit tests for terminals - many of them. See Readme file for instructions on running unit tests.
  • Comment terminal - removed a special treatment of Line comments; comment now is StartSymbol...EndSymbol; line comments fit well into this, for example in c#: StartSymbol="//", EndSymbol = "\n"
  • Removed CharLiteral class - for c# char literal use StringLiteral with IsChar option
  • Added Script.NET grammar to the list of sample grammars
  • Removed ErrorTerminal class, using base Terminal class for Grammar.Error singleton

Changeset 7937, Feb 15

  • Repaired terminal sorting for scanning
  • Fixed terminals sorting for display
  • Fixed BnfElement.Star(delimiter) method - caused false grammar conflicts
  • Fixed parsing for case-insensitive grammars - now should work correctly
  • Fixed exception when NonTerminal's Rule property was not assigned - now reports error nicely and cancels the process
  • Refactored Scanner to accept ISourceStream instead of SourceFile instance; a move towards using general Stream as source
  • Refactored handling punctuation symbols - added IsPunctuation flag to BnfElement and removed Grammar.PunctuationSymbols symbols collection; use Grammar.RegisterPunctuation method for setting the flag in punctuation symbols and even NonTerminals
  • Refactored operator symbols handling - added IsOperator flag, precedence and associativity properties directly to SymbolTerminal
  • Streamlined AstNode creation process - moved most of the logic to Parser.CreateNode method, cleaned it up. (Python grammar is a bit broken as a result - AST tree shows some "extra" symbols)
  • Deprecated GenericNode - AstNode is main base class for all Ast nodes
  • Added facilities for VS integration: Scanner.GetNext method, returning tokens one-by-one; BraceMatchTokenFilter builds full list of brace pairs.
  • Basic AST Visitor facility: AstNode.AcceptVisitor(IAstVisitor)

Changeset 6381

  • Refactored internal representation of BnfExpression implementation, now without ExpressionType enumeration property
  • Improved grammar data presentation by removing \b symbol from names
  • Renamed NotTerminal.Expression property to Rule
  • Added Values dictionary to CompilerContext
  • Added Compiler property to CompilerContext
  • Added Attributes property (dictionary) to AstNode class, to store any custom values associated with node/token
  • Implemented basic error reporting and recovery
  • Scanner error recovery: added "Delimiters" list to Grammar: on error scanner skips all until whitespace or delimiter.
  • Added ErrorRule property to Non-terminal to define error recovery paths. Error expressions can also be mixed with normal expressions in Rule property
  • Added ErrorTerminal and SyntaxError singleton in Grammar, to be used in error rules
  • Added TryMatch method to grammar as backup for token recognition; you can use this method to create "fancy" tokens - those that use extended character sets for example and don't have limited set of prefixes.
  • Added error rules to Ruby grammar and source sample with errors for Ruby
  • Added Terminal parameter to MatchHandler delegate to provide reference to caller object
Clone this wiki locally