-
Notifications
You must be signed in to change notification settings - Fork 1
/
file.LexerDevelopment.html
571 lines (412 loc) · 25.5 KB
/
file.LexerDevelopment.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>
File: Lexer Development
— Documentation by YARD 0.9.28
</title>
<link rel="stylesheet" href="css/style.css" type="text/css" />
<link rel="stylesheet" href="css/common.css" type="text/css" />
<script type="text/javascript">
pathId = "LexerDevelopment";
relpath = '';
</script>
<script type="text/javascript" charset="utf-8" src="js/jquery.js"></script>
<script type="text/javascript" charset="utf-8" src="js/app.js"></script>
</head>
<body>
<div class="nav_wrap">
<iframe id="nav" src="file_list.html?1"></iframe>
<div id="resizer"></div>
</div>
<div id="main" tabindex="-1">
<div id="header">
<div id="menu">
<a href="_index.html">Index</a> »
<span class="title">File: Lexer Development</span>
</div>
<div id="search">
<a class="full_list_link" id="class_list_link"
href="class_list.html">
<svg width="24" height="24">
<rect x="0" y="4" width="24" height="4" rx="1" ry="1"></rect>
<rect x="0" y="12" width="24" height="4" rx="1" ry="1"></rect>
<rect x="0" y="20" width="24" height="4" rx="1" ry="1"></rect>
</svg>
</a>
</div>
<div class="clear"></div>
</div>
<div id="content"><div id='filecontents'><h1 id="lexer-development">Lexer Development</h1>
<h2 id="overview">Overview</h2>
<p>A critical concept in the design of Rouge is the "lexer". A lexer converts
ordinary text into a series of tokens that Rouge can then process.</p>
<p>Rouge supports the languages it does by having a separate lexer for each
language. Lexer development is important both for fixing Rouge when syntax
isn't being highlighted properly and for adding languages that Rouge doesn't
support.</p>
<p>The remainder of this document explains how to develop a lexer for Rouge.</p>
<blockquote>
<p>Please don't submit lexers that are largely copy-pasted from other files.
These submissions will be rejected.</p>
</blockquote>
<h2 id="getting-started">Getting Started</h2>
<h3 id="development-environment">Development Environment</h3>
<p>To develop a lexer, you need to have set up a development environment. If you
haven't done that yet, we've got a <a href="https://rouge-ruby.github.io/docs/file.DevEnvironment.html" title="Rouge's environment setup guide">guide</a> that can
help.</p>
<p>The rest of this guide assumes that you have set up such an environment and,
importantly, that you have installed the gems on which Rouge depends to a
directory within the repository (we recommend <code>vendor/</code>).</p>
<h3 id="file-location">File Location</h3>
<p>Rouge automatically loads lexers saved in the <code>lib/rouge/lexers/</code> directory and
so if you're submitting a new lexer, that's the right place to put it. The
filename should match the name of your lexer, with the Ruby filename extension
<code>.rb</code> appended. If the name of your language is <code>Example</code>, the lexer would be
saved as <code>lib/rouge/lexers/example.rb</code>.</p>
<h3 id="subclassing-regexlexer">Subclassing <code>RegexLexer</code></h3>
<p>Your lexer needs to be a subclass of the <span class='object_link'><a href="Rouge/Lexer.html" title="Rouge::Lexer (class)">Rouge::Lexer</a></span> abstract class. Most
lexers are in fact subclassed from <span class='object_link'><a href="Rouge/RegexLexer.html" title="Rouge::RegexLexer (class)">Rouge::RegexLexer</a></span> as the simplest way to
define the states of a lexer is to use rules consisting of regular expressions.
The remainder of this guide assumes your lexer is subclassed from
<span class='object_link'><a href="Rouge/RegexLexer.html" title="Rouge::RegexLexer (class)">Rouge::RegexLexer</a></span>.</p>
<h2 id="how-to-structure">How to Structure</h2>
<p>Basically, a lexer consists of two parts:</p>
<ol>
<li>a series of properties that are usually declared at the top of the lexer; and</li>
<li>a collection of one or more states, each of which has one or more rules.</li>
</ol>
<p>There are some additional features that a lexer can implement and we'll cover
those at the end.</p>
<p>For the remainder of this guide, we'll use <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/json.rb">the JSON lexer</a> as an
example. The lexer is relatively simple and is for a language with which many
people will at least have some level of familiarity.</p>
<h3 id="lexer-properties">Lexer Properties</h3>
<p>To be usable by Rouge, a lexer should declare a <strong>title</strong>, a <strong>description</strong>, a
<strong>tag</strong>, any <strong>aliases</strong>, associated <strong>filenames</strong> and associated
<strong>mimetypes</strong>.</p>
<h4 id="title">Title</h4>
<pre class="code rb"><code class="rb">title "JSON"
</code></pre>
<p>The title of the lexer. It is declared using the <span class='object_link'><a href="Rouge/Lexer.html#title-class_method" title="Rouge::Lexer.title (method)">Rouge::Lexer.title</a></span> method.</p>
<p>Note: As a subclass of <span class='object_link'><a href="Rouge/RegexLexer.html" title="Rouge::RegexLexer (class)">Rouge::RegexLexer</a></span>, the JSON lexer inherits this method
(and its inherited methods) into its namespace and can call those methods
without needing to prefix each with <code>Rouge::Lexer</code>. This is the case with all
of the property defining methods.</p>
<h4 id="description">Description</h4>
<pre class="code rb"><code class="rb">desc "JavaScript Object Notation (json.org)"
</code></pre>
<p>The description of the lexer. It is declared using the <span class='object_link'><a href="Rouge/Lexer.html#desc-class_method" title="Rouge::Lexer.desc (method)">Rouge::Lexer.desc</a></span>
method.</p>
<h4 id="tag">Tag</h4>
<pre class="code rb"><code class="rb">tag "json"
</code></pre>
<p>The tag associated with the lexer. It is declared using the <span class='object_link'><a href="Rouge/Lexer.html#tag-class_method" title="Rouge::Lexer.tag (method)">Rouge::Lexer.tag</a></span>
method.</p>
<p>A tag provides a way to specify the lexer that should apply to text within a
given code block. In various flavours of Markdown, it's used after the opening
of a code block, such as in the following example:</p>
<pre class="code ruby"><code class="ruby">```ruby
puts "This is some Ruby"
```
</code></pre>
<p>The <code>ruby</code> tag is defined in <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/ruby.rb">the Ruby lexer</a>.</p>
<h4 id="aliases">Aliases</h4>
<p>The aliases associated with a lexer. These are declared using the
<span class='object_link'><a href="Rouge/Lexer.html#aliases-class_method" title="Rouge::Lexer.aliases (method)">Rouge::Lexer.aliases</a></span> method. Aliases are alternative ways that the lexer can
be identified.</p>
<p>The JSON lexer does not define any aliases but <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/ruby.rb">the Ruby one</a> does.
We can see how it could be used by looking at another example in Markdown. This
time, instead of specifying the tag after the opening of the code block, we'll
use an alias instead:</p>
<pre class="code ruby"><code class="ruby">```rb
puts "This is still some Ruby"
```
</code></pre>
<h4 id="filenames">Filenames</h4>
<pre class="code rb"><code class="rb">filenames "*.json"
</code></pre>
<p>The filename(s) associated with a lexer. These are declared using the
<span class='object_link'><a href="Rouge/Lexer.html#filenames-class_method" title="Rouge::Lexer.filenames (method)">Rouge::Lexer.filenames</a></span> method.</p>
<p>Filenames are declared as "globs" that will match a particular pattern. A
"glob" may be merely the specific name of a file (eg. <code>Rakefile</code>) or it could
include one or more wildcards (eg. <code>*.json</code>).</p>
<h4 id="mimetypes">Mimetypes</h4>
<pre class="code rb"><code class="rb">mimetypes "application/json", "application/vnd.api+json", "application/hal+json"
</code></pre>
<p>The mimetype(s) associated with a lexer. These are declared using the
<span class='object_link'><a href="Rouge/Lexer.html#mimetypes-class_method" title="Rouge::Lexer.mimetypes (method)">Rouge::Lexer.mimetypes</a></span> method.</p>
<h3 id="lexer-states">Lexer States</h3>
<p>The other major element of a lexer is the collection of one or more states.
For lexers that subclass <span class='object_link'><a href="Rouge/RegexLexer.html" title="Rouge::RegexLexer (class)">Rouge::RegexLexer</a></span>, a state will consist
of one or more rules with a rule consisting of a regular expression and an
action. The action yields tokens and manipulates the <em>state stack</em>.</p>
<h4 id="the-state-stack">The State Stack</h4>
<p>The state stack represents an ordered sequence of states the lexer is currently
processing. States are added and removed from the "top" of the stack. The
oldest state is on the bottom of the stack and the newest state is on the top.</p>
<p>The initial (and therefore bottommost) state is the <code>:root</code> state. The lexer
works by looking at the rules that are in the state that is on top of the
stack. These are tried <em>in order</em> until a match is found. At this point, the
action defined in the rule is run, the head of the input stream is advanced
and the process is repeated with the state that is now on top of the stack.</p>
<p>Now that we've explained the concepts, let's look at how you actually define
these elements in your lexer.</p>
<h4 id="states">States</h4>
<pre class="code rb"><code class="rb">state :root do
... # do some stuff
end
</code></pre>
<p>A state is defined using the <span class='object_link'><a href="Rouge/RegexLexer.html#state-class_method" title="Rouge::RegexLexer.state (method)">Rouge::RegexLexer.state</a></span> method.
The method consists of the name of the state as a <code>Symbol</code> and a block
specifying the rules that Rouge will try to match as it parses the text.</p>
<h4 id="rules">Rules</h4>
<p>A rule is defined using the <span class='object_link'><a href="Rouge/RegexLexer/StateDSL.html#rule-instance_method" title="Rouge::RegexLexer::StateDSL#rule (method)">Rouge::RegexLexer::StateDSL#rule</a></span> method. The
<code>rule</code> method can define either "simple" rules or "complex" rules.</p>
<h5 id="simple-rules">Simple Rules</h5>
<pre class="code rb"><code class="rb">rule /\s+/m, Text::Whitespace
rule /"/, Str::Double, :string
</code></pre>
<p>A simple rule takes:</p>
<ol>
<li>a regular expression to match against;</li>
<li>a token to yield if the regular expression matches; and</li>
<li>an optional new state to push onto the state stack if the regular expression
matches.</li>
</ol>
<p>In the above example, there are two rules. The first rule yields the token
<code>Text::Whitespace</code> but does not do anything to the state stack. The second rule
yields the token <code>Str::Double</code> and adds the new state <code>:string</code> to the top of
the state stack. The text being parsed after this point will now be processed
by the rules in the <code>:string</code> state.</p>
<p>The following code shows the definition for this state and the rules that are
defined within it:</p>
<pre class="code rb"><code class="rb">state :string do
rule /[^\\"]+/, Str::Double
rule /\\./, Str::Escape
rule /"/, Str::Double, :pop!
end
</code></pre>
<p>The last rule features the "special state" <code>:pop!</code>. This is not really a state,
rather it is an instruction to the lexer to remove the current state from the
top of the state stack. In the JSON lexer, when we encounter the double
quotation mark <code>"</code> we enter into the state of being "in a string" and when we
next encounter the double quotation mark, we leave the string and return to the
previous state (in this case, the <code>:root</code> state).</p>
<h5 id="complex-rules">Complex Rules</h5>
<p>It is possible to define more complex rules for a lexer by calling <code>rule</code> with:</p>
<ol>
<li>a regular expression to match against; and</li>
<li>a block to call if the regular expression matches.</li>
</ol>
<p>The block called can take one argument, usually written as <code>m</code>, that contains
the regular expression match object.</p>
<p>These kind of rules allow for more fine-grained control of the state stack.
Inside a complex rule's block, it's possible to call <span class='object_link'><a href="Rouge/RegexLexer.html#push-instance_method" title="Rouge::RegexLexer#push (method)">Rouge::RegexLexer#push</a></span>,
<span class='object_link'><a href="Rouge/RegexLexer.html#pop!-instance_method" title="Rouge::RegexLexer#pop! (method)">Rouge::RegexLexer#pop!</a></span>, <span class='object_link'><a href="Rouge/RegexLexer.html#token-instance_method" title="Rouge::RegexLexer#token (method)">Rouge::RegexLexer#token</a></span> and
<span class='object_link'><a href="Rouge/RegexLexer.html#delegate-instance_method" title="Rouge::RegexLexer#delegate (method)">Rouge::RegexLexer#delegate</a></span>.</p>
<p>You can see an example of these more complex rules in <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/ruby.rb">the Ruby lexer</a>.</p>
<h3 id="additional-features">Additional Features</h3>
<p>While the properties and states are the minimum elements of a lexer that need
to be implemented, a lexer can include additional features.</p>
<h4 id="source-detection">Source Detection</h4>
<pre class="code rb"><code class="rb">def self.detect?(text)
return true if text.shebang? 'ruby'
end
</code></pre>
<p>Rouge will attempt to guess the appropriate lexer if it is not otherwise clear.
If Rouge is unable to do this on the basis of any tag, associated filename or
associated mimetype, it will try to detect the appropriate lexer on the basis of
the text itself (the source). This is done by calling <code>self.detect?</code> on the
possible lexer (a default <code>self.detect?</code> method is defined in <span class='object_link'><a href="Rouge/Lexer.html" title="Rouge::Lexer (class)">Rouge::Lexer</a></span>
and simply returns <code>false</code>).</p>
<p>A lexer can implement its own <code>self.detect?</code> method that takes a
<span class='object_link'><a href="Rouge/TextAnalyzer.html" title="Rouge::TextAnalyzer (class)">Rouge::TextAnalyzer</a></span> object as a parameter. If the <code>self.detect?</code> method
returns true, the lexer will be selected as the appropriate lexer.</p>
<p>It is important to note that <code>self.detect?</code> should <em>only</em> return <code>true</code> if it
is 100% sure that the language is detected. The most common ways for source
code to identify the language it's written in is with a shebang or a doctype
and Rouge provides the <span class='object_link'><a href="Rouge/TextAnalyzer.html#shebang-instance_method" title="Rouge::TextAnalyzer#shebang (method)">Rouge::TextAnalyzer#shebang</a></span> method and the
<span class='object_link'><a href="Rouge/TextAnalyzer.html#doctype-instance_method" title="Rouge::TextAnalyzer#doctype (method)">Rouge::TextAnalyzer#doctype</a></span> method specifically for use with <code>self.detect?</code>
to make these checks easy to perform.</p>
<p>For more general disambiguation between different lexers, see <a href="#conflicting-filename-globs">Conflicting
Filename Globs</a> below.</p>
<h4 id="special-words">Special Words</h4>
<p>Every programming language reserves certain words for use as identifiers that
have a special meaning in the language. To make regular expressions that search
for these words easier, many lexers will put the applicable keywords in an
array and make them available in a particular way (be it as a local variable,
an instance variable or what have you).</p>
<p>For performance and safety, we strongly recommend lexers use a class method:</p>
<pre class="code rb"><code class="rb">module Rouge
module Lexers
class YetAnotherLanguage < RegexLexer
...
def self.keywords
@keywords ||= Set.new %w(key words used in this language)
end
...
end
end
</code></pre>
<p>These keywords can then be used like so:</p>
<pre class="code rb"><code class="rb">rule /\w+/ do |m|
if self.class.keywords.include?(m[0])
token Keyword
elsif
token Name
end
end
</code></pre>
<p>In some cases, you may want to interpolate your keywords into a regular
expression. <strong>We strongly recommend you avoid doing this.</strong> Having a large
number of rules that are searching for particular words is not as performant as
a rule with a generic pattern with a block that checks whether the pattern is a
member of a predefined set and assigns tokens, pushes new states, etc.</p>
<p>If you do need to use interpolation, be careful to use the <code>\b</code> anchor to avoid
inadvertently matching part of a longer word (eg. <code>if</code> matching <code>iff</code>)::</p>
<pre class="code rb"><code class="rb">rule /\b(#{keywords.join('|')})\b/, Keyword
</code></pre>
<h4 id="startup">Startup</h4>
<pre class="code rb"><code class="rb">start do
push :expr_start
@heredoc_queue = []
end
</code></pre>
<p>The <span class='object_link'><a href="Rouge/RegexLexer.html#start-class_method" title="Rouge::RegexLexer.start (method)">Rouge::RegexLexer.start</a></span> method can take a block that will be called when
the lexer commences lexing. This provides a way to enter into a special state
"before" entering into the <code>:root</code> state (the <code>:root</code> state is still the
bottommost state in the state stack; the state pushed by <code>start</code> sits "on top"
but is the state in which the lexer begins.</p>
<p>Why would you want to do this? In some languages, there may be language
structures that can appear at the beginning of a file.
<span class='object_link'><a href="Rouge/RegexLexer.html#start-class_method" title="Rouge::RegexLexer.start (method)">Rouge::RegexLexer.start</a></span> provides a way to parse these structures without
needing a special rule in your <code>:root</code> state that has to keep track of whether
you are processing things for the first time.</p>
<h3 id="subclassing">Subclassing</h3>
<p>If a lexer is for a language that is very similar to a language with an
existing lexer, it's possible to subclass the existing lexer. See <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/cpp.rb">the C++
lexer</a> and <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/jsx.rb">the JSX lexer</a> for examples.</p>
<h3 id="gotchas">Gotchas</h3>
<h4 id="conflicting-filename-globs">Conflicting Filename Globs</h4>
<p>If two or more lexers define the same filename glob, this will cause an
<span class='object_link'><a href="Rouge/Guesser/Ambiguous.html" title="Rouge::Guesser::Ambiguous (class)">Rouge::Guesser::Ambiguous</a></span> error to be raised by certain guessing methods
(including the one used by the <code>assert_guess</code> method used in your spec).</p>
<p>The solution to this is to define a disambiguation procedure in the
<span class='object_link'><a href="Rouge/Guessers/Disambiguation.html" title="Rouge::Guessers::Disambiguation (class)">Rouge::Guessers::Disambiguation</a></span> class. Here's the procedure for the <code>*.pl</code>
filename glob as an example:</p>
<pre class="code rb"><code class="rb">disambiguate "*.pl" do
next Perl if contains?("my $")
next Prolog if contains?(":-")
next Prolog if matches?(/\A\w+(\(\w+\,\s*\w+\))*\./)
end
</code></pre>
<p>Then, in <a href="#specs">your spec</a>, include a <code>:source</code> parameter when calling
<code>assert_guess</code>:</p>
<pre class="code rb"><code class="rb">it "guesses by filename" do
# *.pl needs source hints because it's also used by Prolog
assert_guess :filename => "foo.pl", :source => "my $foo = 1"
end
</code></pre>
<h2 id="how-to-test">How to Test</h2>
<p>When developing a lexer, it is important to have ways to test it. Rouge provides
support for three types of test files:</p>
<ol>
<li>a <strong>spec</strong> that will run as part of Rouge's test suite;</li>
<li>a <strong>demo</strong> that will be tested as part of Rouge's test suite; and;</li>
<li>a <strong>visual sample</strong> of the various language constructs.</li>
</ol>
<p>When you submit a lexer, you must also include these test files.</p>
<p>Before we look at how to run these tests, let's look at the files themselves.</p>
<h3 id="specs">Specs</h3>
<p>A spec is a list of expectations that are tested as part of the test suite.
Rouge uses the Minitest library for defining these expectations. For more
information about Minitest, refer to <a href="http://docs.seattlerb.org/minitest/">the documentation</a>.</p>
<p>Your spec should at a minimum test how your lexer interacts with Rouge's
guessing algorithm. In particular, you should check:</p>
<ul>
<li>the associated filenames;</li>
<li>the associated mimetypes; and</li>
<li>the associated sources (if any).</li>
</ul>
<p>Your spec must be saved to <code>spec/lexers/<name_of_your_lexer>_spec.rb</code>.</p>
<h4 id="filenames">Filenames</h4>
<pre class="code rb"><code class="rb">it "guesses by filename" do
assert_guess :filename => "foo.rb"
end
</code></pre>
<p>Each of the filename globs that are declared in the lexer should be tested in
the spec. <a href="#conflicting-filename-globs">As discussed above</a>, if the associated filename glob
conflicts with a filename glob defined in another lexer, you will need to write
a disambiguation.</p>
<h4 id="mimetypes">Mimetypes</h4>
<pre class="code rb"><code class="rb">it "guesses by mimetype" do
assert_guess :mimetype => "text/x-ruby"
end
</code></pre>
<p>Each of the mimetypes that are declared in the lexer should be tested in the
spec.</p>
<h4 id="sources">Sources</h4>
<pre class="code rb"><code class="rb">it "guesses by source" do
assert_guess :source => "#!/usr/local/bin/ruby"
end
</code></pre>
<p>If the lexer implements the <code>self.detect?</code> method, then each predicate that
returns true should be tested.</p>
<h3 id="demos">Demos</h3>
<p>The demo file is tested automatically as part of Rouge's test suite. The file
should be able to be parsed without producing any <code>Error</code> tokens.</p>
<p>The demo is also used on <a href="http://rouge.jneen.net/">rouge.jneen.net</a> as the default text to display
when a lexer is chosen. It should be short (less than 20 lines if possible).</p>
<p>Your demo must be saved to <code>lib/rouge/demos/<name_of_your_lexer></code>. Please note
that there is no file extension.</p>
<h3 id="visual-samples">Visual Samples</h3>
<p>A visual sample is a file that includes a representive sample of the syntax of
your language. The sample should be long enough to reasonably demonstrate the
correct lexing of the language but does not need to offer complete coverage.
While it can be tempting to copy and paste code found online, please refrain
from doing this. If you need to copy code, indicate in a comment (using the
appropriate syntax for your lexer's language) the source of the code. Avoid
including code that is duplicative of the other code in the sample.</p>
<p>If you are adding or fixing rules in the lexer, please add some examples of the
expressions that will be highlighted differently to the visual sample if
they're not already present. This greatly assists in reviewing your lexer
submission.</p>
<p>Your visual sample must be saved to <code>spec/visual/sample/<name_of_your_lexer></code>.
As with the demo file, there is no file extension.</p>
<h3 id="running-the-tests">Running the Tests</h3>
<p>The spec and the demo can be run using the <code>rake</code> command. You can run this by
typing <code>bundle exec rake</code> at the command line. If everything works, you should
see a series of dots. If you have an error, this will appear here, too.</p>
<p>You can also run <code>bundle exec rake VERBOSE=1</code> to see all warnings.</p>
<p>To see your visual sample, launch Rouge's visual test app by running
<code>bundle exec rackup</code>. You can choose your sample from the complete list by going
to <a href="http://localhost:9292">http://localhost:9292</a>.</p>
<h2 id="how-to-submit">How to Submit</h2>
<p>So you've developed a lexer (or fixed an existing one)—that's great! The basic
workflow for a lexer to be submitted is:</p>
<ol>
<li>you make a pull request;</li>
<li>a maintainer reviews the lexer;</li>
<li>the maintainer suggests any changes that need to be made;</li>
<li>you make the necessary changes;</li>
<li>the maintainer accepts the request and merges in the code; and</li>
<li>the lexer is included in a future release of the Rouge gem.</li>
</ol>
<p>Now you're on your way to fame and glory! (Maybe.)</p>
<p>If you haven't submitted a pull request before, GitHub has <a href="https://help.github.com/en/articles/about-pull-requests">excellent
documentation</a> that will help you get accustomed to the workflow.</p>
<p>We're looking forward to seeing your code!</p>
<p>You can learn a lot by reading through some of the existing lexers. A good
example that's not too long is <a href="https://github.com/rouge-ruby/rouge/blob/master/lib/rouge/lexers/json.rb">the JSON lexer</a>.</p>
</div></div>
<div id="footer">
Generated on Fri Jun 14 14:26:36 2024 by
<a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
0.9.28 (ruby-3.3.0).
</div>
</div>
</body>
</html>