This files lists all changes in the CodeRay library since the 0.9.8 release.
- C++ scanner: Added C++11 keywords. [#195, thanks to Johnny Willemsen]
- SQL scanner: Allow
$
signs in SQL identifiers [#164, thanks to jasir and Ben Basson] - SQL scanner: Fix open strings [#163, thanks to Adam]
- Ruby scanner: Accept number literal suffixes
r
andi
(Ruby 2.1) - Ruby scanner: Accept quoted hash keys like
{ "a": boss }
(Ruby 2.2) - Ruby scanner: Accept save navigation operator
&.
(Ruby 2.3) - Ruby scanner: Accept squiggly heredoc
<<~
(Ruby 2.3) - Diff scanner: Prevent running out of regexp stack.
- HTML encoder: You can keep tabs intact now by setting
tab_width: false
. - Alpha style: Tweaked colors for
:function
group with:content
. - File structure: One module per file, autoload CodeRay::Version, paths follow namespace hierarchy.
New scanners:
- Go [#28, thanks to Eric Guo and Nathan Youngman]
- Lua [#21, #22, thanks to Quintus]
- Sass [#93]
- Taskpaper [#39, thanks to shimomura]
More new stuff:
.xaml
file type [#121, thanks to Kozman Bálint]- recognize
Guardfile
,Vagrantfile
, andAppraisals
as Ruby files [#121, thanks to Kozman Bálint] - new token kind
:id
for CSS/Sass [#27] - new token kind
:done
for Taskpaper [#39] - new token kind
:map
for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman] - new token kind
:unknown
for Debug scanner - new DebugLint encoder that checks for empty tokens and correct nesting
Improvements:
- CSS scanner uses
:id
and:tag
now [#27] - Diff scanner: Highlight inline changes in multi-line changes [#99]
- JavaScript scanner: Highlight multi-line comments in diff correctly
- JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
- HTML scanner displays style tags and attributes now [#145]
- Ruby scanner: Accept
%i(…)
and%I(…)
symbol lists (Ruby 2.0) [thanks to Nathan Youngman] - Ruby scanner: Accept keywords as Ruby hash keys [#126]
- performance improvements to several scanners and encoders, especially Terminal and HTML
- added
:keep_state
functionality to more scanners so they work nicely with diff now [#116] - refactoring and cleanup to achieve better “Code Climate” ratings (but I don’t really care)
- updated and cleaned up the documentation,
- documented list of TokenKinds
- Alpha style: tweaked colors for
.binary
,.local-variable
, and.predefined-type
rake generate
supports Git now instead of Subversion
Removed:
Tokens#dump
,Tokens.load
,Tokens::Undumping
, andzlib
dependency- double-click toggle handler from HTML table output
rake_helpers
,sample
directories and several other ancient garbage
Fixes:
- fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
- fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
- fixed
:docstring
token type style - fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
- fixed HTML encoder when output is a StringIO (eg. when using
-HTML
as a command line parameter) - TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
- display line numbers in HTML
:table
mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom] - override Bootstrap’s
pre { word-break: break-all }
styling for line numbers [#102, thanks to lightswitch05] - HTML encoder will not warn about unclosed token groups at the end of the stream
- fixed problem with coderay/version.rb being loaded twice
Internals:
- The Debug scanner maps unknown token kinds to
:unknown
(to avoid creating Symbols based on possibly unsafe input). - The Raydebug scanner highlights unknown token kinds as
:plain
. - The Debug encoder refactored; use DebugLint if you want strict checking now..
- The Debug encoder will not warn about errors in the token stream.
- Plugin does not warn about fallback when default is defined.
- PluginHost now works with Strings instead of Symbols internally (to avoid using
#to_sym
).
- Fix Ruby scanner: Ruby 1.9 hash syntax
{ key: value }
is highlighted correctly. [GH #106, thanks to Seth Vargo] - Fix HTML scanner: Accept DTDs. [GH #83]
- Fix PHP scanner: Accept Unicode. [GH #40, thanks to Lance Li]
- add
:string/:char
, remove:regexp/:function
color from Terminal encoder [GH #29, thanks to Kyrylo Silin] - allow
-
in line number anchor prefix for HTML encoder [GH #32, thanks to shurizzle] - Fix HTML scanner: Don’t crash if HTML in a diff contains a JavaScript tag.
- Changed license from LGPL to MIT. [GH-25, thanks to jessehu]
- Fix issue with plugin files not being loaded. [GH-20, thanks to Will Read]
- Fix HTML scanner bug: Don’t choke on boolean attributes. [GH-26, thanks to jugglinmike]
- New option
:break_lines
for the HTML encoder (splits tokens at line breaks). [GH-15, thanks to Etienne Massip] - Improved speed of
:line_numbers => :inline
option for the HTML encoder. - Fixed wrong HTML file type. (was
:page
) [GH-16, thanks to Doug Hammond] - The CSS Scanner now highlights tokens like
url(...)
as:function
instead of:string
. [GH-13, thanks to Joel Holdbrooks]
Fixes:
autoload
calls do not depend oncoderay/lib
being in the load path (GitHub issue #6; thanks to tvon, banister, envygeeks, and ConradIrwin)- avoid dark blue as terminal color (GitHub issue #9; thanks to shevegen)
Fixes in the CSS scanner:
- understands the unit “s” (seconds)
- ignores unexpected curly braces
- code inside of diffs is highlighted correctly
New:
- .tmproj files are recognized as XML.
Fixes:
- Removed files are highlighted inside diffs generated by git.
Fixes:
- .erb files are recognized as ERB.
New:
- YAML scanner allows “-” and “/” in key names
Changes:
- HTML page output has no white border anymore (alpha style)
Fixes:
- fixed warning in the output of “coderay stylesheet”
- fixed additional scrollbar in code when last line contains an eyecatcher
- minor fixes in the tests (issue github-#4)
CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.
The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.
- NEW: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.
- NEW: Extended support and usage of HTML5 and CSS 3 features.
- NEW: Direct Streaming
- NEW scanners: Clojure and HAML
- CHANGED: Token classes (used as CSS classes) are readable names; breaks you stylesheet!
- IMPROVED documentation
- IMPROVED speed: faster startup (using
autoload
), scanning, and encoding - IMPROVED Ruby 1.9 encodings support
- IMPROVED Tests: There are more of them now!
CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners,
Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with
this version.
The main benefits of this change are:
- more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)
- the ability to stream output into a pipe on the command line
- a simpler API
- less code
Changes related to the new tokens handling include:
- CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the
Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a
special case; Tokens just encodes the given tokens into an Array for later use. - CHANGED: The token actions (
text_token
,begin_group
etc.) are now public methods ofEncoder
andTokens
. - REWRITE of all Scanners, Encoders, Filters, and Tokens.
- RENAMED
:open
and:close
actions to:begin_group
and:end_group
. - RENAMED
open_token
andclose_token
methods tobegin_group
andend_group
. - NEW method
#tokens
allows to add several tokens to the stream.Tokens
andEncoders::Encoder
define this
method. - CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible
with earlier versions. - REMOVED
TokenStream
and theStreamable
API and all related features likeNotStreamableError
are now obsolete
and have been removed.
The coderay
executable was rewritten and has a few new features:
- NEW Ability to stream into a pipe; try
coderay file | more -r
- NEW help
- IMPROVED: more consistent parameter handling
- REMOVED
coderay_stylesheet
executable; usecoderay stylesheet [name]
.
- NEW methods
count
,begin_group
,end_group
,begin_line
, andend_line
. - REMOVED methods
#stream?
,#each_text_token
. - REMOVED methods
#optimize
,#fix
,#split_into_lines
along with their bang! variants. - REMOVED
#text
and#text_size
methods. Use theText
encoder instead. - REMOVED special implementation of
#each
taking a filter parameter. UseTokenKindFilter
instead.
Renamed from Tokens::ClassOfKind
(was also Tokens::AbbreviationForKind
for a while).
The term “token class” is no longer used in CodeRay. Instead, tokens have kinds.
See #122.
- CHANGED all token CSS classes to readable names.
- ADDED token kinds
:filename
,:namespace
, and:eyecatcher
. - RENAMED
:pre_constant
and:pre_type
to:predefined_constant
andpredefined_type
. - RENAMED
:oct
and:bin
to:octal
andbinary
. - REMOVED token kinds
:attribute_name_fat
,:attribute_value_fat
,:operator_fat
,interpreted
,
:tag_fat
,tag_special
,:xml_text
,:nesting_delimiter
,:open
, and:close
. - CHANGED: Don’t raise error for unknown token kinds unless in
$CODERAY_DEBUG
mode. - CHANGED the value for a token kind that is not highlighted from
:NO_HIGHLIGHT
tofalse
.
- NEW method
call
for allowing code likeCodeRay::Duo[:python => :yaml].(code)
in Ruby 1.9.
- NEW alias
:remove_comments
- NEW option
tokens
. - CHANGED: Now it simply delegates to the output.
- REMOVED
include_text_token?
andinclude_block_token?
methods.
The HTML encoder was cleaned up and simplified.
- NEW: HTML5 and CSS 3 compatible.
See #215. - ADDED support for
:line_number_anchors
.
See #208. - CHANGED the default style to
:alpha
. - CHANGED: Use double click to toggle line numbers in table mode (as single
click jumps to an anchor.) - REMOVED support for
:line_numbers => :list
. - FIXED splitting of lines for
:line_numbers => :inline
, so that the line
numbers don’t get colored, too. - RENAMED
Output#numerize
to#number
, which is an actual English word.
- CHANGED:
compile
andfinish
methods are now protected.
- RENAMED from
Encoders::Term
, added:term
alias. - CLEANUP: Use
#setup
‘ssuper
, don’t use:procedure
token class. - CHANGED:
#token
’s second parameter is no longer optional. - REMOVED colors for obsolete token kinds.
- FIXED handling of line tokens.
- FIXED default behavior of stripping the trailing newline.
- NEW: Handles token groups.
See #223. - RENAMED
include_block_token?
toinclude_group?
.
- CHANGED: Tokens actions are counted separately.
- NEW methods
#file_extension
and#encoding
. - NEW: The
#tokenize
method also takes an Array of Strings as source. The
code is highlighted as one and split into parts of the input lengths
after that usingTokens#split_into_parts
. - NEW method
#binary_string
- REMOVED helper method
String#to_unix
. - REMOVED method
#streamable?
. - REMOVED
#marshal_load
and#marshal_dump
. - RENAMED class method
normify
tonormalize
; it also deals with encoding now. - CHANGED:
#column
starts counting with 1 instead of 0
Thanks to Licenser, CodeRay now supports the Clojure language.
- NEW: Rudimentary support for the
attr
,counter
, andcounters
functions.
See #224. - NEW: Rudimentary support for CSS 3 colors.
- CHANGED: Attribute selectors are highlighted as
:attribute_name
instead of:string
. - CHANGED: Comments are scanned as one token instead of three.
- NEW: Support for line tokens (
begin_line
andend_line
represented by[
and]
.) - FIXED: Don’t send
:error
andnil
tokens for buggy input any more. - FIXED: Closes unclosed tokens at the end of
scan_tokens
. - IMPROVED: Highlight unknown tokens as
:error
. - CHANGED: Raises an error when trying to end an invalid token group.
- FIXED: Closes open string groups.
- NEW: Highlighting of code based on file names.
See ticket #52.
:highlight_code
option to turn this feature off. It’s enabled
by default.
This is a very original feature. It enables multi-language highlighting for
diff files, which is especially helpful for CodeRay development itself. The
updated version of the scanner test suite generated .debug.diff.html files
using this.
Note: This is still experimental. Tokens spanning more than one line
may get highlighted incorrectly. CodeRay tries to keep scanner states
between the lines and changes, but the quality of the results depend on
the scanner.
- NEW: Inline change highlighting, as suggested by Eric Thomas.
See ticket #227 for details.
:inline_diff
option to turn this feature off. It’s enabled by
default.
For single-line changes (that is, a single deleted line followed by a single
inserted line), this feature surrounds the changed parts with an
:eyecatcher
group which appears in a more saturated background color.
The implementation is quite complex, and highly experimental. The problem
with multi-layer tokenizing is that the tokens have to be split into parts.
If the inline change starts, say, in the middle of a string, then additional
:end_group
and :begin_group
tokens must be inserted to keep the group
nesting intact. The extended Scanner#tokenize
method and the new
Tokens#split_into_parts
method take care of this.
- NEW: Highlight the file name in the change headers as
:filename
. - CHANGED: Highlight unknown lines as
:comment
instead of:head
. - IMPROVED: Background colors for Diff output have been optimized.
It uses the new :state options of the HTML and Ruby scanners.
Some rare cases are not considered (like #{...}
snippets inside of :javascript blocks),
but it highlights pretty well.
- FIXED: Closes open string groups.
- IMPROVED: Added
NaN
andInfinity
to list of predefined constants. - IMPROVED recognition of RegExp literals with leading spaces.
- NEW: Package names are highlighted as
:namespace
.
See #210.
Nitro is dead.
- IMPROVED: Just returns the string without scanning (faster).
/\z/
in Ruby 1.8.
- CHANGED: Docstrings are highlighted as
:comment
.
See #190.
Copied from Scanners::Debug
, highlights the token dump instead of importing it. It also reacts to the .raydebug
file
name suffix now.
- ADDED more predefined keywords (see http://murfy.de/ruby-constants).
- IMPROVED support for singleton method definitions.
See #147. - FIXED: Don’t highlight methods with a capital letter as constants
(eg.GL.PushMatrix
). - NEW: Highlight buggy floats (like .5) as
:error
. - CLEANUP of documentation, names of constants and variables, state handling.
StringState
class from patterns.rb
into a separate file.
- NEW: Complicated rule for recognition of
foo=
style method names. - NEW: Handles
:keep_state
option (a bit; experimental).
[~>=]
, but not =>
following the name.
- REMOVED
EncodingError
- It is too buggy, and nobody was using it. To be added again when it’s fixed.
See #59.
- IMPROVED: Extended list of keywords and functions (thanks to
Joshua Galvez, Etienne Massip, and others).
- FIXED: Closes open string groups.
- FIXED: Words after
.
are always recognized as:ident
.
- FIXED: Allow spaces before colon in mappings.
A style that uses transparent HSLA colors as defined in CSS 3. See #199.
It also uses the CSS 3 property user-select: none
to keep the user from selecting the line numbers. This is especially
nice for :inline
line numbers. See #226.
Stripped down to 19 LOC.
- RENAMED
CaseIgnoringWordList
toWordList::CaseIgnoring
. - REMOVED caching option because it creates memory leaks.
- REMOVED block option.
- NEW: Recognizes
.gemspec
,.rjs
,.rpdf
extensions,Gemfile
, andCapfile
as Ruby.
- REMOVED
FileType#shebang
is a protected method now.
- IMPROVED:
register_for
sets theplugin_id
; it can now be aSymbol
. - ADDED
PluginHost#const_missing
method: Plugins are loaded automatically.
UsingScanners::JavaScript
in your code loadsscanners/java_script.rb
. - ADDED
#all_plugins
method to simplify getting
information about all available plugins (suggested by bnhymn). - CHANGED the default plugin key from
nil
to:default
.
- MOVED into
CodeRay
namespace. - MOVED file from
gzip_simple.rb
togzip.rb
. - REMOVED
String
extensions.
- FIXED
Encoders::HTML#token
’s second parameter is no longer optional. - CHANGED
Encoders::HTML::Output
’s API. - REMOVED lots of unused methods.
The helper classes were cleaned up; see above for details.
- CHANGED
Plugin
API was simplified and stripped of all unnecessary features. - CHANGED Moved
GZip
andFileType
libraries intoCodeRay
; cleaned them up.