Small fixes noticed while going over the when writing loose parser
This commit is contained in:
Marijn Haverbeke
20
index.html
20
index.html
@@ -92,15 +92,14 @@ reset the internal state, and invalidate existing tokenizers.</p> </
|
||||
<span class="nx">t</span><span class="p">.</span><span class="nx">type</span> <span class="o">=</span> <span class="nx">tokType</span><span class="p">;</span> <span class="nx">t</span><span class="p">.</span><span class="nx">value</span> <span class="o">=</span> <span class="nx">tokVal</span><span class="p">;</span>
|
||||
<span class="k">return</span> <span class="nx">t</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="nx">getToken</span><span class="p">.</span><span class="nx">jumpTo</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">pos</span><span class="p">)</span> <span class="p">{</span>
|
||||
<span class="nx">getToken</span><span class="p">.</span><span class="nx">jumpTo</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">pos</span><span class="p">,</span> <span class="nx">reAllowed</span><span class="p">)</span> <span class="p">{</span>
|
||||
<span class="nx">tokPos</span> <span class="o">=</span> <span class="nx">pos</span><span class="p">;</span>
|
||||
<span class="k">if</span> <span class="p">(</span><span class="nx">options</span><span class="p">.</span><span class="nx">locations</span><span class="p">)</span> <span class="p">{</span>
|
||||
<span class="nx">tokCurLine</span> <span class="o">=</span> <span class="nx">tokLineStart</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
|
||||
<span class="nx">tokLineStartNext</span> <span class="o">=</span> <span class="nx">nextLineStart</span><span class="p">();</span>
|
||||
<span class="p">}</span>
|
||||
<span class="kd">var</span> <span class="nx">ch</span> <span class="o">=</span> <span class="nx">input</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="nx">pos</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
|
||||
<span class="nx">tokRegexpAllowed</span> <span class="o">=</span> <span class="o">!</span><span class="nx">ch</span> <span class="o">||</span> <span class="sr">/[\[\{\(,;:?\/*=+\-~!|&%^<>]/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">ch</span><span class="p">)</span> <span class="o">||</span>
|
||||
<span class="sr">/[enwfd]/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">ch</span><span class="p">)</span> <span class="o">&&</span> <span class="sr">/\b(keywords|case|else|return|throw|new|in|(instance|type)of|delete|void)$/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">pos</span> <span class="o">-</span> <span class="mi">9</span><span class="p">,</span> <span class="nx">pos</span> <span class="o">+</span> <span class="mi">1</span><span class="p">));</span>
|
||||
<span class="nx">tokRegexpAllowed</span> <span class="o">=</span> <span class="nx">reAllowed</span><span class="p">;</span>
|
||||
<span class="nx">skipSpace</span><span class="p">();</span>
|
||||
<span class="p">};</span>
|
||||
<span class="k">return</span> <span class="nx">getToken</span><span class="p">;</span>
|
||||
@@ -131,7 +130,7 @@ message.</p> </td> <td class="code"> <div
|
||||
<span class="kd">var</span> <span class="nx">loc</span> <span class="o">=</span> <span class="nx">getLineInfo</span><span class="p">(</span><span class="nx">input</span><span class="p">,</span> <span class="nx">pos</span><span class="p">);</span>
|
||||
<span class="nx">message</span> <span class="o">+=</span> <span class="s2">" ("</span> <span class="o">+</span> <span class="nx">loc</span><span class="p">.</span><span class="nx">line</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nx">loc</span><span class="p">.</span><span class="nx">column</span> <span class="o">+</span> <span class="s2">")"</span><span class="p">;</span>
|
||||
<span class="kd">var</span> <span class="nx">err</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SyntaxError</span><span class="p">(</span><span class="nx">message</span><span class="p">);</span>
|
||||
<span class="nx">err</span><span class="p">.</span><span class="nx">pos</span> <span class="o">=</span> <span class="nx">pos</span><span class="p">;</span> <span class="nx">err</span><span class="p">.</span><span class="nx">loc</span> <span class="o">=</span> <span class="nx">loc</span><span class="p">;</span>
|
||||
<span class="nx">err</span><span class="p">.</span><span class="nx">pos</span> <span class="o">=</span> <span class="nx">pos</span><span class="p">;</span> <span class="nx">err</span><span class="p">.</span><span class="nx">loc</span> <span class="o">=</span> <span class="nx">loc</span><span class="p">;</span> <span class="nx">err</span><span class="p">.</span><span class="nx">raisedAt</span> <span class="o">=</span> <span class="nx">tokPos</span><span class="p">;</span>
|
||||
<span class="k">throw</span> <span class="nx">err</span><span class="p">;</span>
|
||||
<span class="p">}</span></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> <h2>Token types</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> <p>The assignment of fine-grained, information-carrying type objects
|
||||
allows the tokenizer to store the information it has about a
|
||||
@@ -195,7 +194,8 @@ in AssignmentExpression nodes.</p> </td> <td class="code
|
||||
<span class="kd">var</span> <span class="nx">_bin10</span> <span class="o">=</span> <span class="p">{</span><span class="nx">binop</span><span class="o">:</span> <span class="mi">10</span><span class="p">,</span> <span class="nx">beforeExpr</span><span class="o">:</span> <span class="kc">true</span><span class="p">};</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> <p>Provide access to the token types for external users of the
|
||||
tokenizer.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">exports</span><span class="p">.</span><span class="nx">tokTypes</span> <span class="o">=</span> <span class="p">{</span><span class="nx">bracketL</span><span class="o">:</span> <span class="nx">_bracketL</span><span class="p">,</span> <span class="nx">bracketR</span><span class="o">:</span> <span class="nx">_bracketR</span><span class="p">,</span> <span class="nx">braceL</span><span class="o">:</span> <span class="nx">_braceL</span><span class="p">,</span> <span class="nx">braceR</span><span class="o">:</span> <span class="nx">_braceR</span><span class="p">,</span>
|
||||
<span class="nx">parenL</span><span class="o">:</span> <span class="nx">_parenL</span><span class="p">,</span> <span class="nx">parenR</span><span class="o">:</span> <span class="nx">_parenR</span><span class="p">,</span> <span class="nx">comma</span><span class="o">:</span> <span class="nx">_comma</span><span class="p">,</span> <span class="nx">semi</span><span class="o">:</span> <span class="nx">_semi</span><span class="p">,</span> <span class="nx">colon</span><span class="o">:</span> <span class="nx">_colon</span><span class="p">,</span>
|
||||
<span class="nx">dot</span><span class="o">:</span> <span class="nx">_dot</span><span class="p">,</span> <span class="nx">question</span><span class="o">:</span> <span class="nx">_question</span><span class="p">,</span> <span class="nx">slash</span><span class="o">:</span> <span class="nx">_slash</span><span class="p">,</span> <span class="nx">eq</span><span class="o">:</span> <span class="nx">_eq</span><span class="p">};</span>
|
||||
<span class="nx">dot</span><span class="o">:</span> <span class="nx">_dot</span><span class="p">,</span> <span class="nx">question</span><span class="o">:</span> <span class="nx">_question</span><span class="p">,</span> <span class="nx">slash</span><span class="o">:</span> <span class="nx">_slash</span><span class="p">,</span> <span class="nx">eq</span><span class="o">:</span> <span class="nx">_eq</span><span class="p">,</span> <span class="nx">name</span><span class="o">:</span> <span class="nx">_name</span><span class="p">,</span> <span class="nx">eof</span><span class="o">:</span> <span class="nx">_eof</span><span class="p">,</span>
|
||||
<span class="nx">num</span><span class="o">:</span> <span class="nx">_num</span><span class="p">,</span> <span class="nx">regexp</span><span class="o">:</span> <span class="nx">_regexp</span><span class="p">,</span> <span class="nx">string</span><span class="o">:</span> <span class="nx">_string</span><span class="p">};</span>
|
||||
<span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">kw</span> <span class="k">in</span> <span class="nx">keywordTypes</span><span class="p">)</span> <span class="nx">exports</span><span class="p">.</span><span class="nx">tokTypes</span><span class="p">[</span><span class="nx">kw</span><span class="p">]</span> <span class="o">=</span> <span class="nx">keywordTypes</span><span class="p">[</span><span class="nx">kw</span><span class="p">];</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> <p>This is a trick taken from Esprima. It turns out that, on
|
||||
non-Chrome browsers, to check whether a string is in a set, a
|
||||
predicate containing a big ugly <code>switch</code> statement is faster than
|
||||
@@ -660,13 +660,13 @@ tests ("use strict"; 010; -- should fail).</p> </td> <td
|
||||
<span class="nx">tokPos</span> <span class="o">=</span> <span class="nx">lastEnd</span><span class="p">;</span>
|
||||
<span class="nx">skipSpace</span><span class="p">();</span>
|
||||
<span class="nx">readToken</span><span class="p">();</span>
|
||||
<span class="p">}</span></pre></div> </td> </tr> <tr id="section-79"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-79">¶</a> </div> <p>Start an AST node, attaching a start offset.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="kd">function</span> <span class="nx">node_t</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span> <span class="p">{</span>
|
||||
<span class="p">}</span></pre></div> </td> </tr> <tr id="section-79"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-79">¶</a> </div> <p>Start an AST node, attaching a start offset.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="kd">function</span> <span class="nx">node_t</span><span class="p">()</span> <span class="p">{</span>
|
||||
<span class="k">this</span><span class="p">.</span><span class="nx">type</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
|
||||
<span class="k">this</span><span class="p">.</span><span class="nx">start</span> <span class="o">=</span> <span class="nx">tokStart</span><span class="p">;</span>
|
||||
<span class="k">this</span><span class="p">.</span><span class="nx">end</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="kd">function</span> <span class="nx">node_loc_t</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span> <span class="p">{</span>
|
||||
<span class="kd">function</span> <span class="nx">node_loc_t</span><span class="p">()</span> <span class="p">{</span>
|
||||
<span class="k">this</span><span class="p">.</span><span class="nx">start</span> <span class="o">=</span> <span class="nx">tokStartLoc</span><span class="p">;</span>
|
||||
<span class="k">this</span><span class="p">.</span><span class="nx">end</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
|
||||
<span class="k">if</span> <span class="p">(</span><span class="nx">sourceFile</span> <span class="o">!==</span> <span class="kc">null</span><span class="p">)</span> <span class="k">this</span><span class="p">.</span><span class="nx">source</span> <span class="o">=</span> <span class="nx">sourceFile</span><span class="p">;</span>
|
||||
@@ -746,7 +746,7 @@ to its body instead of creating a new node.</p> </td> <t
|
||||
<span class="nx">first</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">return</span> <span class="nx">finishNode</span><span class="p">(</span><span class="nx">node</span><span class="p">,</span> <span class="s2">"Program"</span><span class="p">);</span>
|
||||
<span class="p">};</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="kd">var</span> <span class="nx">loopLabel</span> <span class="o">=</span> <span class="p">{</span><span class="nx">kind</span><span class="o">:</span> <span class="s2">"loop"</span><span class="p">},</span> <span class="nx">switchLabel</span> <span class="o">=</span> <span class="p">{</span><span class="nx">kind</span><span class="o">:</span> <span class="s2">"switch"</span><span class="p">};</span></pre></div> </td> </tr> <tr id="section-91"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-91">¶</a> </div> <p>Parse a single statement.</p>
|
||||
|
||||
@@ -1169,7 +1169,7 @@ to be a <code>[]</code> or dot subscript expression, but not a call — at
|
||||
least, not without wrapping it in parentheses. Thus, it uses the </p> </td> <td class="code"> <div class="highlight"><pre> <span class="kd">function</span> <span class="nx">parseNew</span><span class="p">()</span> <span class="p">{</span>
|
||||
<span class="kd">var</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">startNode</span><span class="p">();</span>
|
||||
<span class="nx">next</span><span class="p">();</span>
|
||||
<span class="nx">node</span><span class="p">.</span><span class="nx">callee</span> <span class="o">=</span> <span class="nx">parseSubscripts</span><span class="p">(</span><span class="nx">parseExprAtom</span><span class="p">(</span><span class="kc">false</span><span class="p">),</span> <span class="kc">true</span><span class="p">);</span>
|
||||
<span class="nx">node</span><span class="p">.</span><span class="nx">callee</span> <span class="o">=</span> <span class="nx">parseSubscripts</span><span class="p">(</span><span class="nx">parseExprAtom</span><span class="p">(),</span> <span class="kc">true</span><span class="p">);</span>
|
||||
<span class="k">if</span> <span class="p">(</span><span class="nx">eat</span><span class="p">(</span><span class="nx">_parenL</span><span class="p">))</span> <span class="nx">node</span><span class="p">.</span><span class="nx">arguments</span> <span class="o">=</span> <span class="nx">parseExprList</span><span class="p">(</span><span class="nx">_parenR</span><span class="p">,</span> <span class="kc">false</span><span class="p">);</span>
|
||||
<span class="k">else</span> <span class="nx">node</span><span class="p">.</span><span class="nx">arguments</span> <span class="o">=</span> <span class="p">[];</span>
|
||||
<span class="k">return</span> <span class="nx">finishNode</span><span class="p">(</span><span class="nx">node</span><span class="p">,</span> <span class="s2">"NewExpression"</span><span class="p">);</span>
|
||||
@@ -1192,7 +1192,7 @@ least, not without wrapping it in parentheses. Thus, it uses the </p>
|
||||
<span class="nx">isGetSet</span> <span class="o">=</span> <span class="nx">sawGetSet</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
|
||||
<span class="nx">kind</span> <span class="o">=</span> <span class="nx">prop</span><span class="p">.</span><span class="nx">kind</span> <span class="o">=</span> <span class="nx">prop</span><span class="p">.</span><span class="nx">key</span><span class="p">.</span><span class="nx">name</span><span class="p">;</span>
|
||||
<span class="nx">prop</span><span class="p">.</span><span class="nx">key</span> <span class="o">=</span> <span class="nx">parsePropertyName</span><span class="p">();</span>
|
||||
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">tokType</span> <span class="o">===</span> <span class="nx">_parenL</span><span class="p">)</span> <span class="nx">unexpected</span><span class="p">();</span>
|
||||
<span class="k">if</span> <span class="p">(</span><span class="nx">tokType</span> <span class="o">!==</span> <span class="nx">_parenL</span><span class="p">)</span> <span class="nx">unexpected</span><span class="p">();</span>
|
||||
<span class="nx">prop</span><span class="p">.</span><span class="nx">value</span> <span class="o">=</span> <span class="nx">parseFunction</span><span class="p">(</span><span class="nx">startNode</span><span class="p">(),</span> <span class="kc">false</span><span class="p">);</span>
|
||||
<span class="p">}</span> <span class="k">else</span> <span class="nx">unexpected</span><span class="p">();</span></pre></div> </td> </tr> <tr id="section-115"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-115">¶</a> </div> <p>getters and setters are not allowed to clash — either with
|
||||
each other or with an init property — and in strict mode,
|
||||
|
||||
Reference in New Issue
Block a user