<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
 <title>Jawher's characters depot</title>
 <link href="https://jawher.me/atom.xml" rel="self"/>
 <link href="https://jawher.me/"/>
 <updated>2018-01-05T13:34:22.129782Z</updated>
 <id>https://jawher.me/</id>
 <author>
   <name>Jawher Moussa</name>
   <email>firstName.lastName@gmail.com</email>
 </author>

<entry>
   <title>Golang and the G word</title>
   <link href="https://jawher.me/2015/05/21/golang-g-word/"/>
   <published>2015-05-21T00:00:00+02:00Z</published>
   <updated>2015-05-21T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2015/05/21/golang-g-word/</id>
   <content type="html">&lt;p&gt;Look, I like Go.
It's a fun language to code in.&lt;/p&gt;
&lt;p&gt;It has an extensive and mostly well-thought standard library which fits in my head &lt;em&gt;(I'm looking at you Java)&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;I like the fact that I can whip out a http service serving json in just a couple of lines using only a text editor and without the need for any external library or complex framework.&lt;/p&gt;
&lt;p&gt;But there is a problem in Go the language and the community, and it all comes down to the G word, aka &lt;em&gt;Generics&lt;/em&gt;, or the lack thereof.&lt;/p&gt;
&lt;p&gt;It is bad enough that the language doesn't have them, forcing you to either copy/paste the same code again and again to handle different types, or to resort to using &lt;code&gt;interface{}&lt;/code&gt;, losing yourself the benefits of a staticaly-typed language.  &lt;/p&gt;
&lt;p&gt;What's worse though is the reaction of at least a part of the community whenever a poor soul dares to mention the G word:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;"&lt;em&gt;generics are overrated&lt;/em&gt;"&lt;/li&gt;
&lt;li&gt;"&lt;em&gt;I've been coding in Go for 2 years and didn't even miss them&lt;/em&gt;"&lt;/li&gt;
&lt;li&gt;"&lt;em&gt;just use the built-in generic types, i.e slices, maps and channels&lt;/em&gt;"&lt;/li&gt;
&lt;li&gt;"&lt;em&gt;can we satop the trolls and get back to writing real-world software&lt;/em&gt;"&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And the list goes on and on.
Just google "golang generics" and feast your eyes on long-winded battles between the 2 camps.&lt;/p&gt;
&lt;p&gt;First: it is simply not true that you don't need generics in Go:
every program makes use of them whenever slices, maps and channels are used.&lt;br /&gt;
I'd like to see the users' reaction if Go really lacked any form of generics and instead forced you to keep casting any thing you read from the built-in containers from &lt;code&gt;interface{}&lt;/code&gt;.&lt;br /&gt;
You bet it won't be &lt;em&gt;generics are overrated&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Second: you can indeed write any software in Go even with the lack of generics.&lt;br /&gt;
The same way we used to be able to write full games in assembler with only jumps and without &lt;code&gt;for&lt;/code&gt; loops or &lt;code&gt;if/else&lt;/code&gt; constructs.&lt;br /&gt;
The same with Java before version 5 where we didn't bat an eyelid when we had casts all over our code everytime we touched a collection from the standard library.&lt;br /&gt;
Or before version 8, i.e. before the advent of lamda expressions and streams where we kept dilluting our code with for loops and/or anonymous classes whenever we wanted to filter or project a collection and finding nothing wrong with it.&lt;/p&gt;
&lt;p&gt;Programs can be limited to using the built-in generic containers only, but they shouldn't be: there are dozens of useful and efficient other datastructures besides lists and maps: tries, trees, sorted sets, heaps, and the list goes on and on.&lt;br /&gt;
Picking the right datastructure or algorithm for the problem on hand is the core of our job as developers.&lt;/p&gt;
&lt;p&gt;So there it is: Go ships with only 2 datastructures, and doesn't let you write the missing ones in a generic and type-safe way.&lt;/p&gt;
&lt;p&gt;I'm not saying that this makes Go unusable, and I shall continue using it almost daily as I've been doing for the last couple of months.&lt;br /&gt;
But enjoying the language doesn't mean it is perfect and criticizing it shouldn't be met with the scorn and vicious responses it elicits today.  &lt;/p&gt;
&lt;p&gt;Here to a future where the following snippet is valid Go:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;BTree&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="kd"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;Compare&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="nx"&gt;Comp&lt;/span&gt;
    &lt;span class="nx"&gt;Root&lt;/span&gt;    &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;Node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; 
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;Node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="kd"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;Value&lt;/span&gt;       &lt;span class="nx"&gt;T&lt;/span&gt;
    &lt;span class="nx"&gt;Left&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;Right&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;Node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;New&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="nx"&gt;compare&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;Comp&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;BTree&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nx"&gt;BTree&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]{&lt;/span&gt;&lt;span class="nx"&gt;Compare&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;compare&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;BTree&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nx"&gt;Insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;tree&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;btree&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;New&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;Account&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;y&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;Account&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;Comp&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Created&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Created&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;Eq&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Created&lt;/span&gt; &lt;span class="p"&gt;&amp;gt;&lt;/span&gt;  &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Created&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;Gt&lt;/span&gt;
        &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;Lt&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;</content>
 </entry>

<entry>
   <title>Parsing command line arguments using a finite state machine and backtracking</title>
   <link href="https://jawher.me/2015/01/18/parsing-command-line-arguments-finite-state-machine-backtracking/"/>
   <published>2015-01-18T00:00:00+01:00Z</published>
   <updated>2015-01-18T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2015/01/18/parsing-command-line-arguments-finite-state-machine-backtracking/</id>
   <content type="html">&lt;p&gt;In the &lt;a href="https://jawher.me/2015/01/13/parsing-command-line-arguments-shameless-plug-mowcli/"&gt;previous article&lt;/a&gt;, I've quickly presented &lt;a href="https://github.com/jawher/mow.cli"&gt;mow.cli&lt;/a&gt;, a command line parsing library I created in Go and compared it against &lt;a href="http://docopt.org"&gt;docopt&lt;/a&gt; and &lt;a href="https://github.com/codegangsta/cli"&gt;codegangsta/cli&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;In this article, I'll talk about the innards of mow.cli and how it uses finite state machines (FSM) and backtracking to handle some tricky usage strings like the cp test&amp;#8482; for example:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;SRC... DST
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;
&lt;p&gt;In mow.cli, you can set a spec string on a CLI application or any of its commands and sub commands.&lt;/p&gt;
&lt;p&gt;The spec string defines the call syntax, i.e. the options and arguments it accepts and their order.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://github.com/jawher/mow.cli#spec"&gt;spec string syntax&lt;/a&gt; is almost the same as docopt's and POSIX commands.&lt;/p&gt;
&lt;p&gt;For example, here's the &lt;code&gt;cp&lt;/code&gt; command spec (or usage) from its man page:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;CP(1)                     BSD General Commands Manual                    CP(1)

NAME
     cp -- copy files

SYNOPSIS
     cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
     cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In mow.cli, the second usage can be expressed as the following spec string:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;[-R [-H | -L | -P]] [-fi | -n] [-apvX] SRC... DST
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Except for some small differences, the syntax reamins mostly the same.&lt;/p&gt;
&lt;h2 id="parsing-a-spec-string"&gt;Parsing a spec string&lt;/h2&gt;
&lt;p&gt;mow.cli uses a tokenizer followed by a recursive descent parser to parse a spec string.&lt;/p&gt;
&lt;p&gt;Here's the &lt;strong&gt;simplified&lt;/strong&gt; EBNF grammar of mow.cli's spec strings:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;spec         -&amp;gt; sequence
sequence     -&amp;gt; choice*
req_sequence -&amp;gt; choice+
choice       -&amp;gt; atom (&amp;#39;|&amp;#39; atom)*
atom         -&amp;gt; (shortOpt | longOpt | group | optional) rep?
shortOp      -&amp;gt; &amp;#39;-&amp;#39; [A-Za-z]
longOpt      -&amp;gt; &amp;#39;--&amp;#39; [A-Za-z][A-Za-z0-9]*
group        -&amp;gt; &amp;#39;(&amp;#39; req_sequence &amp;#39;)&amp;#39;
optional     -&amp;gt; &amp;#39;[&amp;#39; req_sequence &amp;#39;]&amp;#39;
rep          -&amp;gt; &amp;#39;...&amp;#39;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;A conventional parser transforms the source into an abstract syntax tree which will then be traversed and acted upon.&lt;br /&gt;
For example, a compiler traverses the abstract syntax tree to generate native or byte code.&lt;/p&gt;
&lt;p&gt;However, in mow.cli, the process is a bit more &lt;em&gt;inception&lt;/em&gt; like: the spec string is parsed not into an abstract syntax tree but into a second parser which will be later used to parse the program call arguments:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-parsiong-inception.png" src="/images/graphviz/mow-parsiong-inception.png" /&gt;&lt;/p&gt;
&lt;p&gt;An FSM is composed of states and transitions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The states are abstract, they do not represent anything in particular, just a position in the command line parsing process.&lt;br /&gt;
Reaching a terminal state means that what was parsed so far is valid according to the spec string&lt;/li&gt;
&lt;li&gt;The transitions are concrete matchers.&lt;br /&gt;
For example, a transition can be triggered when encoutering the an option named &lt;code&gt;-f&lt;/code&gt; or an argument.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-big-picture"&gt;The big picture&lt;/h2&gt;
&lt;p&gt;Before delving into how the FSM is contsructed, here's a quick example:&lt;br /&gt;
Given the following spec string (from the &lt;code&gt;cp&lt;/code&gt; command man page):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;[-R [-H | -L | -P]] SRC... DST
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The corresponding FSM is:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-cp-fsm.png" src="/images/graphviz/mow-cp-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;This may seem intimidating at first, but it really isn't that hard.&lt;/p&gt;
&lt;p&gt;Simply start from the entry point &lt;code&gt;S1&lt;/code&gt; and follow the transitions until the exit point &lt;code&gt;S47&lt;/code&gt; is reached.
some possible routes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;S1 -&amp;gt; S9 -&amp;gt; S31 -&amp;gt; S41 -&amp;gt; S47&lt;/code&gt;: matches the call &lt;code&gt;cp -R -H SRC DST&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;S1 -&amp;gt; S26 -&amp;gt; S41 -&amp;gt; S41 -&amp;gt; S47&lt;/code&gt;: matches the call &lt;code&gt;cp -R -H SRC SRC DST&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In the following, I'll explain how the previous FSM was constructed from the spec string &lt;code&gt;[-R [-H | -L | -P]] SRC... DST&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="the-spec-parser"&gt;The spec parser&lt;/h2&gt;
&lt;p&gt;As &lt;a href="https://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"&gt;described in another article of mine&lt;/a&gt;, transforming an EBNF grammar into a recursive descent parser is almost a mechanical task.&lt;br /&gt;
And so, unsurprisingly, for every non-terminal in the spec grammar there is a similarly named method which will parse it.&lt;/p&gt;
&lt;p&gt;For example, the spec parser defines a &lt;code&gt;choice&lt;/code&gt; method for the &lt;code&gt;choice&lt;/code&gt; production rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;uParser&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;A state is a go struct representing a state in the FSM and contains a list of outgoing transitions to other states.&lt;br /&gt;
It also contains a boolean flag to indicate if it is terminal or not.&lt;/p&gt;
&lt;p&gt;As can be seen in the method signature above, all the parsing methods return a pair of states representing the entry and exit points of the partial FSM.&lt;br /&gt;
It is possible to only return the entry point of the FSM.
The exit points can then be retrieved by traversing the transitions until reaching the terminal states (there could be multiple exit points).&lt;br /&gt;
However, I decided to always return exactly one entry point and one exit point to make my life easier (as would be seen below).&lt;/p&gt;
&lt;p&gt;Here's how mow.cli transforms the different spec string components into FSMs.&lt;/p&gt;
&lt;h3 id="options"&gt;Options&lt;/h3&gt;
&lt;p&gt;Spec: &lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;-f
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;FSM:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-options-fsm.png" src="/images/graphviz/mow-options-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;Easy: one state with a single transition with the option to a terminal state.&lt;/p&gt;
&lt;h3 id="arguments"&gt;Arguments&lt;/h3&gt;
&lt;p&gt;Spec: &lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;SRC
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;FSM:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-args-fsm.png" src="/images/graphviz/mow-args-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;Same as before, the execution starts in the &lt;code&gt;S1&lt;/code&gt; state.&lt;br /&gt;
The only possible transition is when an argument (anything which doesn't start with a dash) is encountered.&lt;/p&gt;
&lt;h3 id="optionality"&gt;Optionality&lt;/h3&gt;
&lt;p&gt;Rendering an FSM component optional is a simple matter of creating a shortcut from its start state to its end state.&lt;/p&gt;
&lt;p&gt;For example, starting from a 2 positional arguments FSM:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-optional0-fsm.png" src="/images/graphviz/mow-optional0-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;It is a simple matter of creating a shortcut transition from &lt;code&gt;S1&lt;/code&gt; (start) to &lt;code&gt;S3&lt;/code&gt; (end):&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-optional1-fsm.png" src="/images/graphviz/mow-optional1-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;A shortcut transition is marked with the &lt;code&gt;*&lt;/code&gt; symbol and can always be followed without consuming any call arguments.&lt;/p&gt;
&lt;p&gt;This way, starting from &lt;code&gt;S1&lt;/code&gt;, the FSM can either match an &lt;code&gt;SRC&lt;/code&gt; and a &lt;code&gt;DST&lt;/code&gt; arguments, or it can directly jump to the exit point &lt;code&gt;S3&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="repetition"&gt;Repetition&lt;/h3&gt;
&lt;p&gt;To handle repetition, a shortcut transition &lt;code&gt;*&lt;/code&gt; is created from the end state back to the start state.&lt;/p&gt;
&lt;p&gt;For example, starting from the FSM for the spec string &lt;code&gt;X Y&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-repetition0-fsm.png" src="/images/graphviz/mow-repetition0-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;The FSM for &lt;code&gt;(X Y)...&lt;/code&gt; becomes:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-repetition1-fsm.png" src="/images/graphviz/mow-repetition1-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;This way, after matching a &lt;code&gt;X&lt;/code&gt; and a &lt;code&gt;Y&lt;/code&gt; arguments and reaching the exit point &lt;code&gt;S3&lt;/code&gt;, the FSM can go back to the entry point &lt;code&gt;S1&lt;/code&gt; to match more arguments.&lt;/p&gt;
&lt;h3 id="choice"&gt;Choice&lt;/h3&gt;
&lt;p&gt;We start from n possible alternatives.
Each alternative is a partial FSM.
The FSM of a choice is then constructed by:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Create a pair of start and end states &lt;code&gt;S&lt;/code&gt; and &lt;code&gt;E&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Connect &lt;code&gt;S&lt;/code&gt; to every partial FSM's start state using a shortcut &lt;code&gt;*&lt;/code&gt; transition&lt;/li&gt;
&lt;li&gt;Connect every partial FSM's end state to the &lt;code&gt;E&lt;/code&gt; state using a shortcut &lt;code&gt;*&lt;/code&gt; transition&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For example, starting from the FSM for the spec strings &lt;code&gt;-x&lt;/code&gt; and &lt;code&gt;-y&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-choice0-fsm.png" src="/images/graphviz/mow-choice0-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;The FSM for &lt;code&gt;(X Y)...&lt;/code&gt; becomes:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-choice1-fsm.png" src="/images/graphviz/mow-choice1-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;Here's an interactive animation to demonstrate the process at work:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Press the play button to start the animation&lt;br /&gt;
Press pause to pause it&lt;br /&gt;
When paused, you can advance or go back by a frame using the advance and back buttons&lt;br /&gt;
You can scroll and zoom in the animation area  &lt;/p&gt;
&lt;/blockquote&gt;
&lt;div id="fsm-choice" class="fsm-player"&gt;
    &lt;svg width=800 height=200&gt;&lt;g/&gt;&lt;/svg&gt;
&lt;/div&gt;

&lt;p&gt;As a side note, to create this interactive animation, I used the &lt;a href="https://github.com/cpettitt/dagre-d3"&gt;dagre-d3&lt;/a&gt; JS library which implements graphviz-like graph layouting algorithms and uses D3 for the rendering, plus some hand written Javascript code for the interactive part.&lt;/p&gt;
&lt;h3 id="sequence"&gt;Sequence&lt;/h3&gt;
&lt;p&gt;The last piece is handling sequences, e.g.:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;-f SRC DST
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We start from n consecutive components (&lt;code&gt;-f&lt;/code&gt;, &lt;code&gt;SRC&lt;/code&gt; and &lt;code&gt;DST&lt;/code&gt; in the example above).
As always, every component of the sequence is partial FSM with a start and an end state.  &lt;/p&gt;
&lt;p&gt;One to construct the FSM of a sequence is to connect the end state of every partial FSM to the start state of the following FSM.&lt;/p&gt;
&lt;p&gt;Here's how that process would look like:&lt;/p&gt;
&lt;div id="fsm-sequence-wrong" class="fsm-player"&gt;
    &lt;svg width=800 height=300&gt;&lt;g/&gt;&lt;/svg&gt;
&lt;/div&gt;

&lt;p&gt;This method, while it works for the example above, introduces an annoying side-effect: options order.&lt;/p&gt;
&lt;p&gt;Because it connects the partial FSMs in the order in which they appear in the spec string, the generated FSM will only accept the options in that same order.&lt;/p&gt;
&lt;p&gt;For example, given a hypothetical command &lt;code&gt;cmd&lt;/code&gt; with the following spec string:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;-f [-g] FILE
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The resulting FSM, will only accept the following invocations:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;cmd -f README.md&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;cmd -fg README.md&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;But it will reject &lt;code&gt;cmd -g -f README.md&lt;/code&gt; for example, which is annoying for the command users.&lt;br /&gt;
It only gets worse when the number of options grows, like in &lt;code&gt;cp&lt;/code&gt; for example:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] SRC... DST
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;&lt;img alt="One does not simply ask the users to remember the order of 11 options" src="/images/11-options-command.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;To solve this, mow.cli does not simply &lt;em&gt;(no pun intended)&lt;/em&gt; connect the partial FSMs in a linear fashion.&lt;br /&gt;
Instead, the following logic is applied:&lt;br /&gt;
Given 2 components FSMs &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt;, each composed of a start and an end state &lt;code&gt;(A.start, A.end)&lt;/code&gt; and &lt;code&gt;(B.start, B.end)&lt;/code&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;if &lt;code&gt;A&lt;/code&gt; or &lt;code&gt;B&lt;/code&gt; contains a positional argument, connect them using the process described above as the order of positional arguments is important&lt;/li&gt;
&lt;li&gt;else, i.e. &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; contain only options (and maybe shortcuts), construct an FSM which accepts both &lt;code&gt;A&lt;/code&gt; followed by &lt;code&gt;B&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; followed by &lt;code&gt;A&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;Create a copy &lt;code&gt;A'&lt;/code&gt; of &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B'&lt;/code&gt; of &lt;code&gt;B&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Create 2 new states &lt;code&gt;S&lt;/code&gt; and &lt;code&gt;E&lt;/code&gt; which will become the start and end states of the generated FSM&lt;/li&gt;
&lt;li&gt;Connect &lt;code&gt;S&lt;/code&gt; to &lt;code&gt;A.start&lt;/code&gt;, &lt;code&gt;A.end&lt;/code&gt; to &lt;code&gt;B.start&lt;/code&gt; and &lt;code&gt;B.end&lt;/code&gt; to &lt;code&gt;E&lt;/code&gt; using shortcuts: &lt;code&gt;S-&amp;gt;A-&amp;gt;B-&amp;gt;E&lt;/code&gt;. This will accept &lt;code&gt;A&lt;/code&gt; then &lt;code&gt;B&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Connect &lt;code&gt;S&lt;/code&gt; to &lt;code&gt;B'.start&lt;/code&gt;, &lt;code&gt;B'.end&lt;/code&gt; to &lt;code&gt;A'.start&lt;/code&gt; and &lt;code&gt;A'.end&lt;/code&gt; to &lt;code&gt;E&lt;/code&gt; using shortcuts: &lt;code&gt;S-&amp;gt;B'-&amp;gt;A'-&amp;gt;E&lt;/code&gt;. This will accept &lt;code&gt;B&lt;/code&gt; then &lt;code&gt;A&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And since it took me a couple of hours to create this FSM animation thingy, here's another interactive animation which demonstrates the process described above in action:&lt;/p&gt;
&lt;div id="fsm-sequence-ab" class="fsm-player"&gt;
    &lt;svg width=800 height=320&gt;&lt;g/&gt;&lt;/svg&gt;
&lt;/div&gt;

&lt;p&gt;The process described above works on 2 components.&lt;br /&gt;
Generalizing it to work on more is trivial:  apply it to the first 2 components &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; which results in &lt;code&gt;A+B&lt;/code&gt;, and then apply the same process on &lt;code&gt;A+B&lt;/code&gt; and the third component &lt;code&gt;C&lt;/code&gt;, etc.&lt;/p&gt;
&lt;p&gt;Here's a more complex example involving 3 options, a choice and an argument:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;-a [-b | -c] FILE
&lt;/pre&gt;&lt;/div&gt;


&lt;div id="fsm-sequence-abcarg" class="fsm-player" data-zoom="0.7"&gt;
    &lt;svg width=800 height=600&gt;&lt;g/&gt;&lt;/svg&gt;
&lt;/div&gt;

&lt;p&gt;The final FSM may seem daunting, but it can get even more scarier with more complex spec strings.&lt;/p&gt;
&lt;p&gt;However, the generation logic, as explained above, is composed of very simple generation rules which can be then be composed to produce such monstrosities.&lt;br /&gt;
And the complexity of the generated FSM is there for a reason: except for a rules engine&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;, I don't think that there is another way to correctly validate a program call arguments according to a spec string (be sure to check &lt;a href="https://jawher.me/2015/01/13/parsing-command-line-arguments-shameless-plug-mowcli/"&gt;the previous article&lt;/a&gt; for some cases where docopt either fails to accept valid cases or to reject invalid ones).&lt;/p&gt;
&lt;h2 id="the-call-arguments-parser"&gt;The call arguments parser&lt;/h2&gt;
&lt;p&gt;Once a spec string is transformed into a FSM, the latter can be used to parse the program call arguments using the following algorithm:&lt;/p&gt;
&lt;p&gt;The inputs are the current state and the call arguments.&lt;br /&gt;
Initially, the current state is the FSM start state and the call arguments are the list of arguments passed by the user to the program:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;If the call arguments list is empty and the current state is terminal, the parsing succeeds, else if fails&lt;/li&gt;
&lt;li&gt;Else, i.e. the call arguments list is not empty: List all the possible transitions from the current state:&lt;ul&gt;
&lt;li&gt;if a transition is a shortcut, it is always possible to follow&lt;/li&gt;
&lt;li&gt;if a transition is an option &lt;code&gt;-x&lt;/code&gt;, it can only be followed if the first call argument is an option with the same name&lt;/li&gt;
&lt;li&gt;if a transition is an argument, it can only be followed if the first argument is a string not starting with a dash &lt;code&gt;-&lt;/code&gt; (i.e. not an option)&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;For every possible transition:&lt;ul&gt;
&lt;li&gt;consume the matching call arguments:&lt;ul&gt;
&lt;li&gt;if the transition is a shortcut, nothing is consumed&lt;/li&gt;
&lt;li&gt;if the transition is an option, consume the first one or two arguments depending on the call syntax and the option type: consume one argument for boolean options or in case of a &lt;code&gt;=&lt;/code&gt; (e.g. &lt;code&gt;-s=42&lt;/code&gt;), and two arguments otherwise (e.g. &lt;code&gt;-s 42&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;if the transition is an argument, consume exactly one call argument&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;set the current state to the target state of the transition&lt;/li&gt;
&lt;li&gt;go back to step &lt;strong&gt;1.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;if following this transition (the call return &lt;code&gt;true&lt;/code&gt;), also return &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;If there are no possible transitions, or all of them failed, fail too&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Here's the same algorithm in Go (simplified for readability)&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;currentState&lt;/span&gt; &lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// no more call args and current is terminal, succeed&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;terminal&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;transition&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="nx"&gt;currentState&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;transitions&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="cm"&gt;/* transition.matches returns a boolean to indicate &lt;/span&gt;
&lt;span class="cm"&gt;         * if it matched or not and the number of consumed&lt;/span&gt;
&lt;span class="cm"&gt;         * arguments (0, 1 or 2)&lt;/span&gt;
&lt;span class="cm"&gt;         */&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;consumed&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;transition&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;ok&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="cm"&gt;/* recursively call parse again with the transition&lt;/span&gt;
&lt;span class="cm"&gt;             * target as the current state and with the call&lt;/span&gt;
&lt;span class="cm"&gt;             * args minus what the transition consumed&lt;/span&gt;
&lt;span class="cm"&gt;             */&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;transition&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;consumed&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="c1"&gt;// this transition succeeded, succeed too !&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="c1"&gt;// none of the transitions matched, fail&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Now for an example:&lt;/p&gt;
&lt;h3 id="successful-parse"&gt;Successful parse&lt;/h3&gt;
&lt;p&gt;Spec string:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;[-f|-g] FILE
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;FSM:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-parse-example-success-fsm.png" src="/images/graphviz/mow-parse-example-success-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;Execution with the argument list &lt;code&gt;-f X&lt;/code&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Args&lt;/th&gt;
&lt;th&gt;Comment&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there are two possible transitions to &lt;code&gt;F1&lt;/code&gt; and &lt;code&gt;G1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;try the first one leading to &lt;code&gt;F1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;F2&lt;/code&gt; and it consumes one argument&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;A&lt;/code&gt; and it doesn't consume anything&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;&lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;E&lt;/code&gt; and it consumes one argument&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;``&lt;/td&gt;
&lt;td&gt;args is empty and current state is terminal, success !&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="failed-parse"&gt;Failed parse&lt;/h3&gt;
&lt;p&gt;Same spec string but with the argument list &lt;code&gt;-f -g X&lt;/code&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Args&lt;/th&gt;
&lt;th&gt;Comment&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there are two possible transitions to &lt;code&gt;F1&lt;/code&gt; and &lt;code&gt;G1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;try the first one leading to &lt;code&gt;F1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;F2&lt;/code&gt; and it consumes one argument&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;A&lt;/code&gt; and it doesn't consume anything&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;there are no possible transitions, fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;no more transitions to try, fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;F1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;no more transitions to try, fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;try the second transition to &lt;code&gt;G1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;G1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;there are no possible transitions, fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;td&gt;&lt;code&gt;-f&lt;/code&gt; &lt;code&gt;-g&lt;/code&gt; &lt;code&gt;X&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;no more transitions to try, fail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="simplification"&gt;Simplification&lt;/h2&gt;
&lt;p&gt;We're not done yet.&lt;br /&gt;
The FSMs generated using the rules described above, while semantically correct, can cause infinite loops due to the shortcut transitions &lt;code&gt;*&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For example, given this spec string:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;[-e]...
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;i.e. a repeatable options string, like the &lt;code&gt;-e&lt;/code&gt; flag in the &lt;code&gt;docker run&lt;/code&gt; command for example.&lt;/p&gt;
&lt;p&gt;mow.cli would generate the following FSM for such a spec:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-infinite-loop-fsm.png" src="/images/graphviz/mow-infinite-loop-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;-e&lt;/code&gt; option gets a start and an end states &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; with the option name as a transition.&lt;br /&gt;
Because it is optional, a shortcut is added from &lt;code&gt;A&lt;/code&gt; to &lt;code&gt;B&lt;/code&gt;.&lt;br /&gt;
Finally, because it is repeatable, another shortcut from &lt;code&gt;B&lt;/code&gt; to &lt;code&gt;A&lt;/code&gt; is also added.&lt;/p&gt;
&lt;p&gt;In some situations, the algorithm described above will run forever (or until the call stack explodes) bouncing back and forth taking the shortcut transitions between &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The shortcuts are very handy during the spec parsing phase to easily construct semantically correct FSM, so we'd rather keep them during that phase.  &lt;/p&gt;
&lt;p&gt;The solution is to let the spec parsing phase generate shortcuts but to then apply a transformation on the FSM to get rid of them.&lt;/p&gt;
&lt;p&gt;Here's how this is done in mow.cli:&lt;/p&gt;
&lt;p&gt;For every state &lt;code&gt;S&lt;/code&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Bail out if &lt;code&gt;S&lt;/code&gt; was already visited&lt;/li&gt;
&lt;li&gt;For every transition, recursively apply the algorithm on the next state&lt;/li&gt;
&lt;li&gt;While &lt;code&gt;S&lt;/code&gt; has a shortcut transition &lt;code&gt;tr&lt;/code&gt; leading to a state &lt;code&gt;T&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;Add all of &lt;code&gt;T&lt;/code&gt;'s transitions to &lt;code&gt;S&lt;/code&gt; (without adding duplicates)&lt;/li&gt;
&lt;li&gt;If &lt;code&gt;T&lt;/code&gt; is terminal, mark &lt;code&gt;S&lt;/code&gt; as also terminal&lt;/li&gt;
&lt;li&gt;remove the transition &lt;code&gt;tr&lt;/code&gt; from &lt;code&gt;S&lt;/code&gt;'s transitions&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And here's how this algorithm would behave when applied to the FSM of the spec string &lt;code&gt;[-e]...&lt;/code&gt; (which causes an infinite loop as shown above):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;St&lt;/th&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Remark&lt;/th&gt;
&lt;th&gt;FSM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Initial state&lt;/td&gt;
&lt;td&gt;&lt;div class="wide"&gt;&lt;img src="/images/graphviz/mow-simplify-0-fsm.png"&gt;&lt;/div&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;A&lt;/code&gt; has a transition &lt;code&gt;t1&lt;/code&gt; to &lt;code&gt;B&lt;/code&gt;, simplify &lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; has not already been visited&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; has a transition &lt;code&gt;t3&lt;/code&gt; to &lt;code&gt;A&lt;/code&gt;, but &lt;code&gt;A&lt;/code&gt; was already visited, NOP&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; has one shortcut &lt;code&gt;t3&lt;/code&gt; to &lt;code&gt;A&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.1&lt;/td&gt;
&lt;td&gt;add all of &lt;code&gt;A&lt;/code&gt;'s transitions (&lt;code&gt;t1&lt;/code&gt; and &lt;code&gt;t2&lt;/code&gt;) to &lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;A&lt;/code&gt; is not terminal, NOP&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.3&lt;/td&gt;
&lt;td&gt;remove &lt;code&gt;t3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;div class="wide"&gt;&lt;img src="/images/graphviz/mow-simplify-1-fsm.png"&gt;&lt;/div&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; has one shortcut &lt;code&gt;t1'&lt;/code&gt; to &lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.1&lt;/td&gt;
&lt;td&gt;Nothing to add as &lt;code&gt;B&lt;/code&gt; already has all of the target transitions&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; is already terminal&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.3&lt;/td&gt;
&lt;td&gt;remove &lt;code&gt;t1'&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;div class="wide"&gt;&lt;img src="/images/graphviz/mow-simplify-2-fsm.png"&gt;&lt;/div&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; has no more shortcuts&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;3.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;A&lt;/code&gt; has one shortcut &lt;code&gt;t1&lt;/code&gt; to &lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.1&lt;/td&gt;
&lt;td&gt;add all of &lt;code&gt;B&lt;/code&gt;'s transitions to &lt;code&gt;A&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt; is terminal, mark &lt;code&gt;A&lt;/code&gt; as also terminal&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3.3&lt;/td&gt;
&lt;td&gt;remove &lt;code&gt;t1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;div class="wide"&gt;&lt;img src="/images/graphviz/mow-simplify-3-fsm.png"&gt;&lt;/div&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;A&lt;/code&gt; has no more shortcuts&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;We went from:&lt;/p&gt;
&lt;p&gt;&lt;img alt="" src="/images/graphviz/mow-simplify-0-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;to:&lt;/p&gt;
&lt;p&gt;&lt;img alt="" src="/images/graphviz/mow-simplify-3-fsm.png" /&gt;&lt;/p&gt;
&lt;p&gt;Both will accept and reject exactly the same set of call args, but the second form doesn't contain any shortcuts and so is more suitable for parsing as it can't cause infinite loops.&lt;/p&gt;
&lt;h2 id="backtracking"&gt;Backtracking&lt;/h2&gt;
&lt;p&gt;The backtracking is already implemented in the algorithm above because all the possible transitions will be tried and not only the first that matches.&lt;/p&gt;
&lt;p&gt;Here's how that would work for the cp test&amp;#8482;, i.e. with the following spec &lt;code&gt;SRC... DST&lt;/code&gt; and FSM:&lt;/p&gt;
&lt;p&gt;&lt;img alt="mow-backtracking-0-fsm.png" src="/images/graphviz/mow-backtracking-0-fsm.png" /&gt;&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Args&lt;/th&gt;
&lt;th&gt;Comment&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;A&lt;/code&gt; &lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there is one possible transition to &lt;code&gt;S2&lt;/code&gt; and it consumes one argument&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;args is not empty, so check all the possible transitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;there are 2 possible transitions to &lt;code&gt;S2&lt;/code&gt; and to &lt;code&gt;S3&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;try the first one which loops back to &lt;code&gt;S2&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;`                    | args is empty but&lt;/code&gt;S2` is not terminal, fail&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;try the second transition leading to &lt;code&gt;S3&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;`                    | args is empty and&lt;/code&gt;S3` is terminal, success&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;This simple algorithm scales to arbitrarily complex cases.&lt;/p&gt;
&lt;h2 id="fin"&gt;Fin&lt;/h2&gt;
&lt;p&gt;To summarize:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;mow.cli uses a recursive descent parser to transform a spec string into an FSM&lt;/li&gt;
&lt;li&gt;the FSM is then simplified to get rid of the shortcuts because they may lead to infinite loops&lt;/li&gt;
&lt;li&gt;the resulting FSM is then used to validate the program call arguments using a backtracking algorithm&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This only covers the high-level concepts and does not cover everything mow.cli does, like collecting, converting and storing the various options and arguments values for the user to retrieve them later.&lt;/p&gt;
&lt;p&gt;&lt;link rel="stylesheet" type="text/css" href="/extra/mow-fsm/styles.css"&gt;
&lt;script src="/extra/jquery-2.1.3.min.js" charset="utf-8"&gt;&lt;/script&gt;
&lt;script src="/extra/d3.v3.min.js" charset="utf-8"&gt;&lt;/script&gt;
&lt;script src="/extra/dagre.min.js"&gt;&lt;/script&gt;
&lt;script src="/extra/dagre-d3.min.js"&gt;&lt;/script&gt;
&lt;script src="/extra/mow-fsm/app.js"&gt;&lt;/script&gt;&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;I'm sure somebody in the internet will prove me wrong on this point: I'd love you to !&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>Parsing command line arguments and a shameless plug of mow.cli</title>
   <link href="https://jawher.me/2015/01/13/parsing-command-line-arguments-shameless-plug-mowcli/"/>
   <published>2015-01-13T00:00:00+01:00Z</published>
   <updated>2015-01-13T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2015/01/13/parsing-command-line-arguments-shameless-plug-mowcli/</id>
   <content type="html">&lt;p&gt;Parsing command line arguments is a deceptively complex task.
It looks simple at first:
Just iterate over the arguments, if it starts with a dash (&lt;code&gt;-&lt;/code&gt;) then it is an option, otherwise it is an argument.&lt;/p&gt;
&lt;p&gt;But it's a bit more trickier than that.
A &lt;a href="https://github.com/search?utf8=✓&amp;amp;q=cli"&gt;search on Github&lt;/a&gt; with the keyword &lt;code&gt;cli&lt;/code&gt; yields &lt;strong&gt;55905&lt;/strong&gt; repositories at the time when this article was written.
Granted, not all of these hits are libraries for parsing command line arguments, but it still gets the idea across.&lt;/p&gt;
&lt;h2 id="anantomy-of-a-command-line-program"&gt;Anantomy of a command line program&lt;/h2&gt;
&lt;p&gt;Writing a command line program can be divided in two macro tasks:&lt;/p&gt;
&lt;h3 id="call-arguments-parsing"&gt;Call arguments parsing&lt;/h3&gt;
&lt;p&gt;Most programming languages expose a way to retrieve the arguments with which the currently running program was executed as an array of strings.&lt;/p&gt;
&lt;p&gt;These strings need to be analyzed and grouped into options, commands and arguments and their values stored to be accessed later.&lt;/p&gt;
&lt;p&gt;This step, while not particularly hard to implement is very tedious as it has to handle many cases:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Short options: e.g. &lt;code&gt;-f&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Boolean flags (without a value) as in &lt;code&gt;rm -f&lt;/code&gt; and options with values as in &lt;code&gt;dd if=/dev/null of=/tmp/out&lt;/code&gt; or without the &lt;code&gt;=&lt;/code&gt; sign: &lt;code&gt;head -n 5 readme.txt&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Long optios: &lt;code&gt;docker run --name super/container&lt;/code&gt; and &lt;code&gt;docker run --name=super/container&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Option grouping/folding, e.g. &lt;code&gt;rm -rf *.png&lt;/code&gt; besides &lt;code&gt;rm -r -f *.png&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Option end marker &lt;code&gt;--&lt;/code&gt;: &lt;code&gt;touch -- -f&lt;/code&gt; to treat &lt;code&gt;-f&lt;/code&gt; as a filename instead of an option&lt;/li&gt;
&lt;li&gt;etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A rudimentary form of validation can be performed in this, mainly rejecting unknown options or commands or downright incorrect call syntax.&lt;/p&gt;
&lt;p&gt;Almost all command line parsing libraries handle this parsing part with varying degrees of completeness (not all libraries handle option folding for example).&lt;/p&gt;
&lt;h3 id="routing"&gt;Routing&lt;/h3&gt;
&lt;p&gt;Once the call arguments are parsed, the running program needs to select a code path based on the set of options, commands and arguments passed by the program user.&lt;/p&gt;
&lt;p&gt;A higher level validation can take place in this step, e.g. reject conflicting options for the task in hand or ensure that the correct number of arguments is present.&lt;/p&gt;
&lt;p&gt;Not many command line parsing libraries handle this step, and it is left to the program writer to write the &lt;code&gt;if/else&lt;/code&gt; soup to make sense of what the user wants to do.&lt;/p&gt;
&lt;h2 id="naval-fate"&gt;Naval fate&lt;/h2&gt;
&lt;p&gt;In the following, I'll be using two open source command line parsing libraries as examples for the state of the art, namely:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/codegangsta/cli"&gt;codegangsta/cli&lt;/a&gt; written in Go&lt;/li&gt;
&lt;li&gt;&lt;a href="http://docopt.org"&gt;docopt&lt;/a&gt; written in Python (with &lt;a href="https://github.com/docopt/"&gt;ports to many other languages&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The example app that will be used is the naval fate game (shamelessly copied from the &lt;a href="http://try.docopt.org"&gt;try docopt&lt;/a&gt; page).&lt;/p&gt;
&lt;p&gt;It's a simple game where you can position and move ships and mines in a grid using a command line interface:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;naval_fate ship new HMS
naval_fate ship move HMS 1 3
naval_fate mine set 4 2 --drifting
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;On a side note, I had to modify the move ship command syntax from:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;naval_fate.py ship &amp;lt;name&amp;gt; move &amp;lt;x&amp;gt; &amp;lt;y&amp;gt; [--speed=&amp;lt;kn&amp;gt;]
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;To:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;naval_fate.py ship move &amp;lt;name&amp;gt; &amp;lt;x&amp;gt; &amp;lt;y&amp;gt; [--speed=&amp;lt;kn&amp;gt;]
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I.e swap the &lt;code&gt;&amp;lt;name&amp;gt;&lt;/code&gt; argument and &lt;code&gt;move&lt;/code&gt; command order because the former syntax is not &lt;em&gt;(easily)&lt;/em&gt; implementable in &lt;code&gt;codegangsta/cli&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;I also restricted the game to 3 commands: &lt;code&gt;ship new&lt;/code&gt;, &lt;code&gt;ship move&lt;/code&gt; and &lt;code&gt;mine set&lt;/code&gt; as they are sufficient for the purpose of this article.&lt;/p&gt;
&lt;h3 id="docopt"&gt;docopt&lt;/h3&gt;
&lt;p&gt;The idea behind this library is simply genious: use the help message you would want to be shown by the app to describe its syntax:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;Naval Fate.&lt;/span&gt;
&lt;span class="sd"&gt;Usage:&lt;/span&gt;
&lt;span class="sd"&gt;  naval_fate.py ship new &amp;lt;name&amp;gt;...&lt;/span&gt;
&lt;span class="sd"&gt;  naval_fate.py ship move &amp;lt;name&amp;gt; &amp;lt;x&amp;gt; &amp;lt;y&amp;gt; [--speed=&amp;lt;kn&amp;gt;]&lt;/span&gt;
&lt;span class="sd"&gt;  naval_fate.py mine set &amp;lt;x&amp;gt; &amp;lt;y&amp;gt; [--moored|--drifting]&lt;/span&gt;

&lt;span class="sd"&gt;Options:&lt;/span&gt;
&lt;span class="sd"&gt;  --speed=&amp;lt;kn&amp;gt;  Speed in knots [default: 10].&lt;/span&gt;
&lt;span class="sd"&gt;  --moored      Moored (anchored) mine.&lt;/span&gt;
&lt;span class="sd"&gt;  --drifting    Drifting mine.&lt;/span&gt;
&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;docopt&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__doc__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Naval Fate 2.0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ship&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;new&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;create ships {}&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;name&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;move&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;name&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;x&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
                &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;y&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
                &lt;span class="n"&gt;speed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;--speed&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;print&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Incorrect Usage&amp;#39;&lt;/span&gt;
                &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="k"&gt;print&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;move ship named {} to {}:{} with speed {}&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;speed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;mine&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;set&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;x&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
                &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;y&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;print&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Incorrect Usage&amp;#39;&lt;/span&gt;
                &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;moored&amp;#39;&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;--moored&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
                &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;moored&amp;#39;&lt;/span&gt;
            &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;--drifting&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
                &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;drifting&amp;#39;&lt;/span&gt;

            &lt;span class="k"&gt;print&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;set a {} mine in {}:{}&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The help message syntax respects the conventions used in the majority of CLI apps in POSIX systems.&lt;/p&gt;
&lt;p&gt;On the positive side:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Docopt usage syntax is very flexible and can be used to build very complex command syntaxes&lt;/li&gt;
&lt;li&gt;Docopt does the low and high level call args validation for you. In the code above for example, you can't call the &lt;code&gt;mine set&lt;/code&gt; command with both &lt;code&gt;--moored&lt;/code&gt; and &lt;code&gt;--drifting&lt;/code&gt; options.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;However, docopts is not without its warts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It doesn't do the routing part:&lt;br /&gt;
it only returns a hash containing the call commands, arguments and options, and it is up to the developer to branch on its values to select and execute the correct code path as can be seen in the code above with the nested &lt;code&gt;if/else&lt;/code&gt; blocks&lt;/li&gt;
&lt;li&gt;Non contexual help messages: either everything is correct and the program runs, or the program exits and the whole usage string is dumped, no matter the commands and options the program called specified&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="codegangstacli"&gt;codegangsta/cli&lt;/h3&gt;
&lt;p&gt;This is one very popular project (2095 stars in Github at the time where this article was written).&lt;/p&gt;
&lt;p&gt;Here's a possible implementation of the naval fate subset using this library:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;package&lt;/span&gt; &lt;span class="nx"&gt;main&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;&amp;quot;fmt&amp;quot;&lt;/span&gt;
    &lt;span class="s"&gt;&amp;quot;os&amp;quot;&lt;/span&gt;
    &lt;span class="s"&gt;&amp;quot;strconv&amp;quot;&lt;/span&gt;

    &lt;span class="s"&gt;&amp;quot;github.com/codegangsta/cli&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;app&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;NewApp&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Name&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;naval_fate&amp;quot;&lt;/span&gt;
    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Commands&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;ship&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="nx"&gt;Subcommands&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="s"&gt;&amp;quot;new&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;NAME...&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;newShips&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;},&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;move&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;NAME X Y&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Flags&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Flag&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                        &lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntFlag&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                            &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;speed&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                            &lt;span class="nx"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                            &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Speed in knots&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                        &lt;span class="p"&gt;},&lt;/span&gt;
                    &lt;span class="p"&gt;},&lt;/span&gt;
                    &lt;span class="nx"&gt;Action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;moveShip&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="nx"&gt;Subcommands&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;set&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;X Y [--moored|--drifting]&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="nx"&gt;Flags&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Flag&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                        &lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BoolFlag&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                            &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                            &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Moored (anchored) mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                        &lt;span class="p"&gt;},&lt;/span&gt;
                        &lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BoolFlag&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                            &lt;span class="nx"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                            &lt;span class="nx"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Drifting mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                        &lt;span class="p"&gt;},&lt;/span&gt;
                    &lt;span class="p"&gt;},&lt;/span&gt;
                    &lt;span class="nx"&gt;Action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;setMine&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;newShips&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;create ships %#v\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;moveShip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;name&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;strconv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Atoi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;strconv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Atoi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;speed&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;speed&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;move ship named %v to %d:%d with speed %d\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;speed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;setMine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;strconv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Atoi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;strconv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Atoi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Incorrect usage\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;
    &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;set a %s mine in %d:%d\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This clocks at 118 lines of code.&lt;/p&gt;
&lt;p&gt;Here's a quick rundown of the code above:
Structs are used to describe the application structure.
Starting from an application, a list of top-level commands are defined used an slice of structs.
Each command can in turn have a list of sub commands, and so on.&lt;/p&gt;
&lt;p&gt;Each command has a name, a usage string, a list of flags (options) and an action, the code to be called when the command is invoked.&lt;/p&gt;
&lt;p&gt;The good:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The library does the routing part for you: no need for nested &lt;code&gt;if/else&lt;/code&gt; blocks&lt;/li&gt;
&lt;li&gt;Declarative and readable code&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The bad:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;This library suffers from a major weakness: it doesn't offer any facilities to handle arguments.&lt;br /&gt;
As can be seen in the code blob above, it is up to the library user to ensure that the correct number and values of arguments have been specified.&lt;/li&gt;
&lt;li&gt;No high level validation. For example, it is up to the developper to ensure that &lt;code&gt;--moored&lt;/code&gt; and &lt;code&gt;--drifting&lt;/code&gt; can't both be set at the same time&lt;/li&gt;
&lt;li&gt;Type repetition for flags: a flag first needs to be declared with its type (&lt;code&gt;cli.IntFlag(...)&lt;/code&gt;) when describing the commands. Then, to get its value in the &lt;code&gt;Action&lt;/code&gt; func, you need to call &lt;code&gt;c.Int(name)&lt;/code&gt; where the type is specified again.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="summary"&gt;Summary&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th align="right"&gt;&lt;/th&gt;
&lt;th align="center"&gt;docopt&lt;/th&gt;
&lt;th align="center"&gt;codegangsta/cli&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td align="right"&gt;LOC&lt;/td&gt;
&lt;td align="center"&gt;45&lt;/td&gt;
&lt;td align="center"&gt;118&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;routing&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;high level validation&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;argument support&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;contexual help&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="the-cp-test"&gt;The cp test&lt;/h2&gt;
&lt;p&gt;As said earlier, docopt's usage syntax makes it possible to describe very complex and flexible syntaxes.&lt;/p&gt;
&lt;p&gt;The implementation though isn't up to par.&lt;/p&gt;
&lt;p&gt;Take this innocent looking application:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;cp&lt;/span&gt;
&lt;span class="sd"&gt;Usage:&lt;/span&gt;
&lt;span class="sd"&gt;  cp.py &amp;lt;src&amp;gt;... &amp;lt;dst&amp;gt;&lt;/span&gt;
&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;docopt&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__doc__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;cp 42.0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Running this program, no matter what arguments you provide, will fail with no error message: the usage string will simply be printed.&lt;/p&gt;
&lt;p&gt;What happens is that &lt;code&gt;&amp;lt;src&amp;gt;...&lt;/code&gt; will eagerly consume all the provided arguments leaving nothing for &lt;code&gt;&amp;lt;dst&amp;gt;&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This problem is not limited to the repetition operator &lt;code&gt;...&lt;/code&gt;.
Here's another case where docopt fails:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;x&lt;/span&gt;
&lt;span class="sd"&gt;Usage:&lt;/span&gt;
&lt;span class="sd"&gt;  x.py [&amp;lt;src&amp;gt;] &amp;lt;dst&amp;gt;&lt;/span&gt;
&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;docopt&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;docopt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__doc__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;x 42.0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;this will reject a perfectly valid call like &lt;code&gt;x.py a&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;And that's why command line parsing is anything but trivial.
A correct solution must handle these (and other) tricky cases.&lt;/p&gt;
&lt;p&gt;In the following, I'll present how &lt;a href="https://github.com/jawher/mow.cli"&gt;mow.cli&lt;/a&gt;, a go library for parsing command line arguments written my humble self solves this problem using finite state machines and backtracking.&lt;/p&gt;
&lt;h2 id="introducing-mowcli"&gt;Introducing mow.cli&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://github.com/jawher/mow.cli"&gt;mow.cli&lt;/a&gt; takes a hybrid approach between docopt and codegangsta/cli, while also taking inspiration from the &lt;code&gt;flag&lt;/code&gt; package of the go standard library:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It resembles codegangsta/cli in that it provides a structured API to describe commands and subcommands to be able to implement contextual help&lt;/li&gt;
&lt;li&gt;It also lets you use the usage syntax of docopt but on a command by command basis to enable high level validation&lt;/li&gt;
&lt;li&gt;Options and arguments get stored in variables like the standard flag package to avoid the type repetition&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here's how naval fate would be implemented in mow.cli:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;package&lt;/span&gt; &lt;span class="nx"&gt;main&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;&amp;quot;fmt&amp;quot;&lt;/span&gt;
    &lt;span class="s"&gt;&amp;quot;os&amp;quot;&lt;/span&gt;

    &lt;span class="s"&gt;&amp;quot;github.com/jawher/mow.cli&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;app&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;App&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;naval_fate&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;ship&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ship&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Cmd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;ship&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;new&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Create new ships&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;newShips&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;ship&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;move&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Move a ship by name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;moveShip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;

    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;mine&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Cmd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;mine&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;set&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Set a mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;setMine&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;

    &lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;newShips&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cmd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Cmd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Spec&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;NAME...&amp;quot;&lt;/span&gt;
    &lt;span class="nx"&gt;names&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;StringsArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;NAME&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Action&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;create ships %#v\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;names&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;moveShip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cmd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Cmd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Spec&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;NAME X Y [--speed]&amp;quot;&lt;/span&gt;
    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nx"&gt;name&lt;/span&gt;  &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;StringArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;NAME&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;x&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;X&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;y&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;speed&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntOpt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;speed&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Speed in knots&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Action&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;move ship named %v to %d:%d with speed %d\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;speed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nx"&gt;setMine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cmd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Cmd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Spec&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;X Y [--moored|--drifting]&amp;quot;&lt;/span&gt;
    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nx"&gt;x&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;X&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;y&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;IntArg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;moored&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BoolOpt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Moored (anchored) mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nx"&gt;drifting&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BoolOpt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Drifting mine&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nx"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Action&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;
        &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;moored&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;moored&amp;quot;&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;drifting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;drifting&amp;quot;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="nx"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;set a %s mine in %d:%d\n&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;It clocks at &lt;strong&gt;64&lt;/strong&gt; lines of code, almost half of that of codegangsta/cli (&lt;strong&gt;118&lt;/strong&gt;) and one third more than docopt (&lt;strong&gt;45&lt;/strong&gt;).&lt;/p&gt;
&lt;p&gt;Feature-wise, here's an updated summary table including mow.cli:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th align="right"&gt;&lt;/th&gt;
&lt;th align="center"&gt;docopt&lt;/th&gt;
&lt;th align="center"&gt;codegangsta/cli&lt;/th&gt;
&lt;th align="center"&gt;jawher/mow.cli&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td align="right"&gt;naval fate LOC&lt;/td&gt;
&lt;td align="center"&gt;45&lt;/td&gt;
&lt;td align="center"&gt;118&lt;/td&gt;
&lt;td align="center"&gt;64&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;routing&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;high level validation&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;argument support&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;contexual help&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;passes the cp test&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;manual&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td align="right"&gt;Most flexible&lt;/td&gt;
&lt;td align="center"&gt;✓&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;td align="center"&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Docopt is still the most flexible and poweful of the lot:
like codegangsta/cli, the original &lt;code&gt;ship move&lt;/code&gt; syntax cannot &lt;em&gt;(at least easily)&lt;/em&gt; be expressed using mow.cli.&lt;/p&gt;
&lt;p&gt;Also, mow.cli's API is, for the lack of a better way to describe it, &lt;em&gt;weirder&lt;/em&gt; the the other two: you have to use function to configure a command (its spec, options and arguments)&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;But it passes the cp test&amp;#8482;&lt;/p&gt;
&lt;h3 id="usage-string"&gt;Usage string&lt;/h3&gt;
&lt;p&gt;mow.cli uses almost the same syntax as docopt for usage strings.
The differences are (in mow.cli):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;You can define a usage string (spec) on a per command basis, there is no global usage description&lt;/li&gt;
&lt;li&gt;Options are described using the mow.cli API and not in the usage string&lt;/li&gt;
&lt;li&gt;A usage string is in a single line&lt;/li&gt;
&lt;li&gt;Argument names are restricted to uppercased words, the bracketed syntax is not supported &lt;em&gt;(for the time being at least)&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For a complete documentation on mow.cli's usage syntax, please refer to the the &lt;a href="https://github.com/jawher/mow.cli#spec"&gt;project's readme&lt;/a&gt; or &lt;a href="https://godoc.org/github.com/jawher/mow.cli#hdr-Spec"&gt;godoc&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="fsm-and-backtracking"&gt;FSM and backtracking&lt;/h2&gt;
&lt;p&gt;Without going into too much detail, mow.cli is able to handle the tricky cases listed above by using a finite state machine with backtracking to parse the call arguments according to a spec string.&lt;/p&gt;
&lt;p&gt;What this means is that mow.cli will not simply iterate over the spec string components (options, arguments, commands, choices, sequences, etc.) and naively select the first matching alternative.&lt;br /&gt;
Instead, it will try all the possible routes until one matches or none does.&lt;/p&gt;
&lt;p&gt;For example, given the following spec string (cp test&amp;#8482;):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;SRC... DST
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Here's a rundown of the parsing process when the program is called with the arguments &lt;code&gt;["a", "b"]&lt;/code&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The first argument &lt;code&gt;a&lt;/code&gt; is matched as a &lt;code&gt;SRC&lt;/code&gt; argument&lt;/li&gt;
&lt;li&gt;Next, there are 2 possible routes to take for the value &lt;code&gt;b&lt;/code&gt;: it can either be treated as &lt;code&gt;SRC&lt;/code&gt; (because it is repeatable) or &lt;code&gt;DST&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Say the first route is taken, and so &lt;code&gt;b&lt;/code&gt; is consumed as part of the &lt;code&gt;SRC&lt;/code&gt; argument&lt;/li&gt;
&lt;li&gt;All values were consumed but there's still the &lt;code&gt;DST&lt;/code&gt; argument which didn't match anything.&lt;br /&gt;
The previous step will be undone and &lt;code&gt;b&lt;/code&gt; will be restored&lt;/li&gt;
&lt;li&gt;mow.cli will then try the second route by consuming &lt;code&gt;b&lt;/code&gt; as the &lt;code&gt;DST&lt;/code&gt; argument&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And that's how mow.cli manages to handle the tricky cases you throw at it.&lt;/p&gt;
&lt;p&gt;I'll go into much more details &lt;a href="https://jawher.me/2015/01/18/parsing-command-line-arguments-finite-state-machine-backtracking/"&gt;in a follow up article&lt;/a&gt; on how an FSM is constructed from a spec string and how it is then applied to the call arguments.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;This is not arbitrary: since options and arguments have to be stored in variables, defining them in a function helps to scope them to that function instead of polluting the global scope.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The universal package format</title>
   <link href="https://jawher.me/2015/01/11/universal-package-format/"/>
   <published>2015-01-11T00:00:00+01:00Z</published>
   <updated>2015-01-11T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2015/01/11/universal-package-format/</id>
   <content type="html">&lt;p&gt;There are dozens of package formats: apk, deb, rpm, jar, war, whl, egg, dmg, and the list goes on.&lt;/p&gt;
&lt;p&gt;Some are language agnostic, like deb and rpm.
They still are system specific, e.g. rpm have to be installed in a fedora/redhat system whereas debs go to debian/ubuntu systems &lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;Others are language/framework specific, like jar and war for Java apps, wheel and egg for Python, gems for Ruby, etc.&lt;/p&gt;
&lt;p&gt;What would it take to have a system and language agnostic universal package format ?&lt;/p&gt;
&lt;h2 id="no-or-universal-runtime"&gt;No or universal runtime&lt;/h2&gt;
&lt;p&gt;Apart form the auto-extracting packages, almost all others need a runtime to function:
debs and rpms needs something like apt-get or yum, eggs and wheels need Python, jars and wars need a JVM and possibly a Java server, etc.&lt;/p&gt;
&lt;p&gt;A universal package format would ideally also provde a universal &lt;em&gt;runner&lt;/em&gt; to bootstrap the package.&lt;/p&gt;
&lt;h2 id="flexible"&gt;Flexible&lt;/h2&gt;
&lt;p&gt;A universal package format would need to be damn flexible to be able to handle anything you throw at it, be it a WSGi Python application or a WAR file already deployed into a servlet container or something else alongside a specific version of a native library or program installed (image magick or ffmeg for example).&lt;/p&gt;
&lt;h2 id="isolated"&gt;Isolated&lt;/h2&gt;
&lt;p&gt;Since a universal package could embed native programs and libraries, it should not simply be unpacked into the host system or risk conflicting with other versions of said program or library.&lt;/p&gt;
&lt;p&gt;A possible solution would be to unpack in a package exclusive directory, but this won't work with unwilling or unknowing programs and libraries which will look for their dependencies in their standard and usual location (e.g. &lt;code&gt;/usr/lib/&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;Ideally, when such a universal package is installed, the programs it contains shouldn't have to be modified to look for their dependencies in an installation specific location.&lt;/p&gt;
&lt;p&gt;A virtual machine is one possible solution for this.&lt;/p&gt;
&lt;h2 id="lightweight"&gt;Lightweight&lt;/h2&gt;
&lt;p&gt;VMs are too heavyweight.&lt;/p&gt;
&lt;p&gt;The universal package format should strive for higher density on the same hardware.&lt;/p&gt;
&lt;h2 id="configurable"&gt;Configurable&lt;/h2&gt;
&lt;p&gt;A program inside a universal package might listen on TCP or UDP connections on a specific port.&lt;/p&gt;
&lt;p&gt;It also might read or write file from and to a specific directory.&lt;/p&gt;
&lt;p&gt;Also, the package user might want to control the environment inside of it.&lt;/p&gt;
&lt;p&gt;To be usable, a universal package format must let its users map the ports of the package to other ports in the host (to be able to run two apps listening on the same port).&lt;/p&gt;
&lt;p&gt;It also must handle mounting files and directories from the host into the packaged program (to be able to collect the files produced by the program or override the files it embeds).&lt;/p&gt;
&lt;p&gt;And finally, it must be possible to inject environment variables into the package.&lt;/p&gt;
&lt;h2 id="goodies"&gt;Goodies&lt;/h2&gt;
&lt;p&gt;Being able to step inside the package to interact with the programs it embeds is invaluable.&lt;/p&gt;
&lt;p&gt;Log collecting is also a very useful feature to have.&lt;/p&gt;
&lt;h2 id="solution-x"&gt;Solution X&lt;/h2&gt;
&lt;p&gt;Now, imagine a solution where you start from a pristine base system.
Say a linux distribution, debian wheezy for example.&lt;/p&gt;
&lt;p&gt;You then install the packages required by your application: nginx or haproxy, OpenJDK, a specific Ruby version, etc.&lt;/p&gt;
&lt;p&gt;You can then interact and modify the configuration files of the system, e.g. adding a frontend and backend declaration in the HAProxy configuration, or a virtual host in your httpd config, etc.&lt;/p&gt;
&lt;p&gt;You can also copy files from your machine to the package, e.g. your application compiled code, its static assets and configuration files, etc.&lt;/p&gt;
&lt;p&gt;Once you're satisfied with the result, you package the system into an artifact. Let's call it an image since that's what it is: an image of a system.&lt;/p&gt;
&lt;p&gt;You copy the resulting image to the target system, say the staging environment and run it there.&lt;/p&gt;
&lt;p&gt;We won't be getting on the how of running the image, I'll just frantically wave my hands and tell you it is like running a virtual machine but not really, because it is much more lightweight and without any hardware emulation.&lt;/p&gt;
&lt;p&gt;The target system must have a tiny program: the package runtime we talked about earlier.&lt;/p&gt;
&lt;p&gt;This program lets me run an image in a configurable way: it lets me remap an image port to a different host port, mount files and directories from the host into the running image and set environment variables inside the running image.&lt;/p&gt;
&lt;p&gt;Wouldn't such a solution be great?&lt;/p&gt;
&lt;p&gt;It is language agnostic.
I can use it to package PHP applications as well as Java or native apps.&lt;/p&gt;
&lt;p&gt;It is much more flexible than the said languages custom package formats since I have total control of the Linux system it contains.&lt;/p&gt;
&lt;p&gt;It also is system agnostic: all you'd need is a system capable of running the solution X runtime.&lt;/p&gt;
&lt;p&gt;And finally, such a solution is very reliable since it doesn't depend on the host system to have the required dependencies in the correct versions.&lt;/p&gt;
&lt;h2 id="such-a-solution-already-exists"&gt;Such a solution already exists&lt;/h2&gt;
&lt;p&gt;Many of you must have already guessed it:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Dockah dockah dockah&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I won't bore you with details on how Docker works or how to use it, the internet is teeming with talks, slides and articles on this subject.&lt;/p&gt;
&lt;p&gt;I'll just say this: among other things, Docker defines a universal package format, the docker images which can handle any stack you throw at it.
Docker went even further and implemented a smart image format where instead of being a ginormous opaque blob (VM images for example) are instead a set of layers which can be reused across other images to reduce the download bandwidth.&lt;/p&gt;
&lt;p&gt;Docker also provides a very simple yet powerful tool to automate the image building: Dockerfiles.&lt;/p&gt;
&lt;p&gt;And finally, the Docker daemon is the runtime part of the solution X and should run in most reasonably recent linux distros.&lt;/p&gt;
&lt;p&gt;While they are an overkill sometimes, Docker images are better WARs than WARs and better X than X in general, for X in language specific deployable package formats.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;There are workarounds in the form of deb to rpm and vice-versa converters like alien for example.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>Using XCode for AVR C developement</title>
   <link href="https://jawher.me/2014/03/21/using-xcode-avr-c/"/>
   <published>2014-03-21T00:00:00+01:00Z</published>
   <updated>2014-03-21T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2014/03/21/using-xcode-avr-c/</id>
   <content type="html">&lt;p&gt;In this post, I'll show you how to setup your XCode for AVR developement (in the C programming language) using &lt;a href="https://github.com/jawher/xavr"&gt;X-AVR&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;X-AVR was born out of my frustration with eclipse as an IDE to program for AVRs.
While there is the excellent &lt;a href="https://github.com/keestux/avr-eclipse-plugin"&gt;avr-eclipse plugin&lt;/a&gt; (to which I contributed), I wasn't satisfied with CDT.
The devil is in the small details, like the auto-complete propositions order for example.&lt;/p&gt;
&lt;p&gt;The (official) Arduino editor was also not an option:
it's basically a barebone text editor with (a rudimentary) syntax coloring.
I also prefer to code against avr-libc and not use the arduino's high level library as the latter results in bloated hex files that won't fit in smaller and cheaper AVR MCUs (with 1, 2 or 4 Kb flash like the ATtiny13, ATtiny45 or ATmega44 for example).&lt;/p&gt;
&lt;p&gt;So I started looking for an alternative.
After playing a bit with XCode for general C development, I decided to see if I could also use it for AVR C development.&lt;/p&gt;
&lt;p&gt;After a bit of googling, the general consensus seemed to be to create a C project with an external build system (make) and to manually create some targets to invoke the make targets.&lt;/p&gt;
&lt;p&gt;There were also a couple of XCode project templates lying around which eased the project creation and setup a bit.&lt;/p&gt;
&lt;p&gt;Both approaches present a couple of serious limitations&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;: no auto-complete, no navigate to declaration/definition, no real time errors highlighting, etc.
It's back to the &lt;em&gt;coding in notepad&lt;/em&gt; days.&lt;/p&gt;
&lt;p&gt;Enter &lt;a href="https://github.com/jawher/xavr"&gt;X-AVR&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="x-avr"&gt;X-AVR&lt;/h2&gt;
&lt;p&gt;From the project's &lt;a href="https://github.com/jawher/xavr/blob/master/README.md"&gt;readme file&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;X-AVR is an XCode template for generating AVR C projects.&lt;/p&gt;
&lt;p&gt;Now the meta version: X-AVR is a python script which uses the installed avr-gcc and avrdude to generate and install an XCode TemplateInfo.plist file. This template can be used to create AVR C XCode projects with a Makefile to build and upload the program to an AVR chip.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id="installation"&gt;Installation&lt;/h2&gt;
&lt;p&gt;Head to the project's &lt;a href="https://github.com/jawher/xavr/blob/master/README.md"&gt;readme file&lt;/a&gt; for instructions on how to install the X-AVR project template.&lt;/p&gt;
&lt;h2 id="usage"&gt;Usage&lt;/h2&gt;
&lt;p&gt;Once the template was correctly installed, select the xavr entry in the create project wizard:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/wizard-1.png" /&gt;&lt;/p&gt;
&lt;p&gt;And hit next.&lt;/p&gt;
&lt;p&gt;The following screen lets you configure the project:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/wizard-2.png" /&gt;&lt;/p&gt;
&lt;p&gt;Fill in the project name and select the target MCU, its frequency and the programmer and hit next.&lt;/p&gt;
&lt;p&gt;XCode will then create and open the project:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/targets.png" /&gt;&lt;/p&gt;
&lt;p&gt;The X-AVR template creates the following files:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;main.c: your program entry point&lt;/li&gt;
&lt;li&gt;Makefiles/Makefile: well, the project's makefile !&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And the following targets:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;All: performs a clean build and uploads the program the the target MCU&lt;/li&gt;
&lt;li&gt;Build: performs a clean build&lt;/li&gt;
&lt;li&gt;Upload: performs the upload the the target MCU&lt;/li&gt;
&lt;li&gt;Clean&lt;/li&gt;
&lt;li&gt;Index: this target makes it possible for XCode to parse, index and verify your program, in addition to the auto-complete. You shouldn't have to use this target&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here's a screenshot of the auto-complete in action:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/autocomplete-1.png" /&gt;&lt;/p&gt;
&lt;p&gt;In the previous screenshot, you can see that XCode proposes &lt;code&gt;DDRB&lt;/code&gt;, &lt;code&gt;DDRC&lt;/code&gt; and &lt;code&gt;DDRD&lt;/code&gt;.
That's because the project was created with ATmega328P as a target.
Here's what XCode proposes if it was an ATtiny85 for example:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/autocomplete-2.png" /&gt;&lt;/p&gt;
&lt;p&gt;Only &lt;code&gt;DDRB&lt;/code&gt; is present, which matches the MCU capabilities&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2" rel="footnote"&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;You also get error highlighting in the editor:&lt;/p&gt;
&lt;p&gt;&lt;img alt="wat" src="/images/xavr/errors.png" /&gt;&lt;/p&gt;
&lt;p&gt;Cmd+Click also works, even for the avr-libc symbols.&lt;/p&gt;
&lt;h2 id="closing-words"&gt;Closing words&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://github.com/jawher/xavr"&gt;X-AVR&lt;/a&gt; is released as an opensource software under the MIT license.&lt;/p&gt;
&lt;p&gt;I've published it in the hop that it may prove useful to others besides me.
I'd love to get any feedback on it (via Github issues or by leaving a comment)&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;Except for &lt;a href="http://embedxcode.weebly.com/"&gt;embedXcode&lt;/a&gt; which unfortunately generates a C++ project using the Arduino library.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;X-AVR sets the correct &lt;code&gt;__AVR_*__&lt;/code&gt; macro in the &lt;code&gt;GCC_PREPROCESSOR_DEFINITIONS&lt;/code&gt; configuration option.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The Litil chronicle - Chapter 3.3, Expression Parsing</title>
   <link href="https://jawher.me/2013/08/26/creation-langage-programmation-litil-3-3-expression-parsing/"/>
   <published>2013-08-26T00:00:00+02:00Z</published>
   <updated>2013-08-26T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2013/08/26/creation-langage-programmation-litil-3-3-expression-parsing/</id>
   <content type="html">&lt;p&gt;In &lt;a href="http://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"&gt;the previous post&lt;/a&gt;, I showed how to write a recursive descent parser starting from an EBNF grammar.&lt;/p&gt;
&lt;p&gt;The grammar used as an example described a minimal expression language that could only handle numeric and boolean literals and if/else expressions.&lt;/p&gt;
&lt;p&gt;In this post, I'll expand that language to handle more types of expressions, notably:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;arithmetic operators: addition (&lt;code&gt;+&lt;/code&gt;), subtraction (&lt;code&gt;-&lt;/code&gt;), multiplication (&lt;code&gt;*&lt;/code&gt;), division (&lt;code&gt;/&lt;/code&gt;) and modulu (&lt;code&gt;%&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;comparison operators: &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;&amp;lt;&lt;/code&gt;, &lt;code&gt;&amp;lt;=&lt;/code&gt;, &lt;code&gt;&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;gt;=&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;boolean operators: &lt;code&gt;and&lt;/code&gt;, &lt;code&gt;or&lt;/code&gt; and &lt;code&gt;not&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;if/else expressions: &lt;code&gt;if x &amp;gt;= y then x else y&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;function application: &lt;code&gt;sin x&lt;/code&gt; for example&lt;/li&gt;
&lt;li&gt;parenthesized expressions: &lt;code&gt;( ... )&lt;/code&gt; &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let's try and write down an EBNF grammar for such a language.&lt;/p&gt;
&lt;h2 id="first-go"&gt;First go&lt;/h2&gt;
&lt;p&gt;Let's start with additions and multiplications first:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → atom ( &amp;#39;*&amp;#39; | &amp;#39;+&amp;#39;) expr | atom
atom → NUM | BOOL | NAME
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;According to the grammar above, an expression is an &lt;code&gt;atom&lt;/code&gt;, optionally followed by a &lt;code&gt;*&lt;/code&gt; or &lt;code&gt;+&lt;/code&gt; operator and another &lt;code&gt;atom&lt;/code&gt;.
An &lt;code&gt;atom&lt;/code&gt; can be either a name or a numeric or boolean literal.&lt;/p&gt;
&lt;p&gt;Here are the AST nodes types for this grammar:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;abstract&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ENum&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EBool&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EName&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EOp&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;Op&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And the recursive descent parser that recognizes it:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unexpected input&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Looks promising right ?&lt;/p&gt;
&lt;p&gt;Let's test it with the following input &lt;code&gt;1 + 2 * 3&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Remember that the lexer produces a &lt;code&gt;NEWLINE&lt;/code&gt; in the beginning of a line, se we need to consume it before calling the &lt;code&gt;expr&lt;/code&gt; method:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;Reader&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StringReader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1 + 2 * 3&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;ExprParser&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ExprParser&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LookaheadLexerWrapper&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StructuredLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;BaseLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;  &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;
&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NEWLINE&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And here's the resulting AST:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast0.png" src="/images/graphviz/litil-pratt-parser-ast0.png" /&gt;&lt;/p&gt;
&lt;p&gt;So far so good.
But before trying to implement the rest of the expression language, let's first implement an evaluator that walks an expression's AST and computes its value.
Such a tool could come in handy to ensure that the parser is producing the correct AST.&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ENum&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;EOp&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;UnsupportedOperationException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Not impelmented&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Running this evaluator on the AST of &lt;code&gt;1 + 2 * 3&lt;/code&gt; returns &lt;code&gt;7&lt;/code&gt;, which is the correct answer.&lt;/p&gt;
&lt;p&gt;How about &lt;code&gt;5 * 2 + 3&lt;/code&gt; ?
We would expect the evaluation result to be &lt;code&gt;13&lt;/code&gt;.
However, if you run the evaluator on the AST produced by the parser given above, you would get &lt;code&gt;25&lt;/code&gt; instead.&lt;/p&gt;
&lt;p&gt;Looking at the resulting AST should explain it:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast1.png" src="/images/graphviz/litil-pratt-parser-ast1.png" /&gt;&lt;/p&gt;
&lt;p&gt;Here's a rundown of how the parsing went for the &lt;code&gt;5 * 2 + 3&lt;/code&gt; expression:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;expr&lt;/code&gt; calls &lt;code&gt;atom&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;atom&lt;/code&gt; matches the numeric literal &lt;code&gt;5&lt;/code&gt; and returns &lt;code&gt;ENum(5)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Now back to &lt;code&gt;expr&lt;/code&gt;, which stores the result in &lt;code&gt;left&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;It will then match the &lt;code&gt;+&lt;/code&gt; operator&lt;/li&gt;
&lt;li&gt;And so it will recursively call itself the get the &lt;code&gt;right&lt;/code&gt; operand&lt;ol&gt;
&lt;li&gt;This is a new invocation of &lt;code&gt;expr&lt;/code&gt;. &lt;code&gt;atom&lt;/code&gt; is called again&lt;/li&gt;
&lt;li&gt;&lt;code&gt;atom&lt;/code&gt; matches the numeric literal &lt;code&gt;2&lt;/code&gt; and returns &lt;code&gt;ENum(2)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ENum(2)&lt;/code&gt; is stored in &lt;code&gt;left&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;*&lt;/code&gt; operator is matched&lt;/li&gt;
&lt;li&gt;and so &lt;code&gt;expr&lt;/code&gt; is again recursively called to get the value of the right operand&lt;ol&gt;
&lt;li&gt;&lt;code&gt;atom&lt;/code&gt; is called and returns &lt;code&gt;ENum(3)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;There is nothing left in the input, no operator is found and so &lt;code&gt;expr&lt;/code&gt; returns &lt;code&gt;left&lt;/code&gt;, i.e. &lt;code&gt;ENum(3)&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;expr&lt;/code&gt; returns &lt;code&gt;EOp(operator, left, right)&lt;/code&gt;, i.e. &lt;code&gt;EOp('*', ENum(2), ENum(3))&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;expr&lt;/code&gt; returns &lt;code&gt;EOp(operator, left, right)&lt;/code&gt;, i.e. &lt;code&gt;EOp('+', ENum(5), EOp('*', ENum(2), ENum(3)))&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In its current form, the parser doesn't know anything about precedence rules, and would simply parse expressions from left to right, always giving precedence to rightmost operator regardless of arithmetic rules.&lt;/p&gt;
&lt;h2 id="precedence"&gt;Precedence&lt;/h2&gt;
&lt;p&gt;One possible solution to the precedence problem is to encode them in the grammar itself.&lt;/p&gt;
&lt;p&gt;What we need to do is to rewrite the grammar in a way that forces the parser to handle the operators with more precedence before those with less precedence.&lt;/p&gt;
&lt;p&gt;In our particular case, this means coaxing the parser into parsing the multiplications before the additions.&lt;/p&gt;
&lt;p&gt;Here's a grammar that does just that:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add
add → mul (&amp;#39;+&amp;#39; add)?
mul → atom (&amp;#39;*&amp;#39; mul)?
atom → NUM | NAME
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;A grammar written like the one above forces the parser to try and parse multiplications before additions, even though the first rule (&lt;code&gt;expr → add&lt;/code&gt;) might indicate otherwise.
You have to move to the next rule, &lt;code&gt;add → mul ('+' add)?&lt;/code&gt;, to see that the first non-terminal is &lt;code&gt;mul&lt;/code&gt;, which ensures the precedence rules are respected.&lt;/p&gt;
&lt;p&gt;Here's the parser rewritten to match this modified grammar:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;mul&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unexpected input&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And here's the AST this parser produces for the same problematic expression &lt;code&gt;5 * 2 + 3&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast2.png" src="/images/graphviz/litil-pratt-parser-ast2.png" /&gt;&lt;/p&gt;
&lt;p&gt;Contrast it to the problematic AST:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Bad AST" src="/images/graphviz/litil-pratt-parser-ast1.png" /&gt;&lt;/p&gt;
&lt;h3 id="more-arithmetic-operators"&gt;More arithmetic operators&lt;/h3&gt;
&lt;p&gt;Now that we know how to handle the precedence rules, let's extend the grammar to handle more operators and expression types.&lt;/p&gt;
&lt;p&gt;One easy addition is to handle subtraction and division (with both the quotient and remainder operators).
We simply add the desired operators to the production rules:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) add_rem)?
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) mul_div)?
atom → NUM | NAME
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Which translates to &lt;em&gt;ored&lt;/em&gt; &lt;code&gt;found&lt;/code&gt; calls in the parser code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;add_rem&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add_rem&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We'll also need to update the evaluator to handle these new operators:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ENum&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;EOp&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;UnsupportedOperationException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Not impelmented&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="associativity"&gt;Associativity&lt;/h2&gt;
&lt;p&gt;Now that we've added subtractions to the grammar, if we try and evaluate the expression &lt;code&gt;3 - 2 + 1&lt;/code&gt;, which should evaluate to &lt;code&gt;2&lt;/code&gt;, we will get &lt;code&gt;0&lt;/code&gt; instead.&lt;/p&gt;
&lt;p&gt;Here's the AST for the problematic expression:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast-bad-assoc.png" src="/images/graphviz/litil-pratt-parser-ast-bad-assoc.png" /&gt;&lt;/p&gt;
&lt;p&gt;Which explains why we're getting the wrong result: the &lt;code&gt;2 + 1&lt;/code&gt; sub expression ends up in its own &lt;code&gt;EOp&lt;/code&gt; node.&lt;br /&gt;
Thus, the evaluator have to evaluate it first, which returns &lt;code&gt;3&lt;/code&gt;, and subtracts that from &lt;code&gt;3&lt;/code&gt;, which results in the wrong result &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;To fix this this, we'll need to modify the parser so that it handles operators associativity.&lt;/p&gt;
&lt;p&gt;Operator associativity comes into play when operators of the same precedence appears more than once in a sequence, as in &lt;code&gt;3 - 2 + 1&lt;/code&gt;.
There are 2 ways to evaluate such an expression:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;(3 - 2) + 1&lt;/code&gt;: which is the correct one&lt;/li&gt;
&lt;li&gt;&lt;code&gt;3 - (2 + 1)&lt;/code&gt;: which is the wrong one&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;An operator is said to be &lt;strong&gt;left-associative&lt;/strong&gt; when the operations are grouped from the left (the first way of evaluating the expression above), and &lt;strong&gt;right-associative&lt;/strong&gt; when the operations are grouped from the right (the second way of evaluating the expression above).&lt;br /&gt;
Some operators are non-associative, meaning that they cannot appear in sequence, e.g. comparison operators: &lt;code&gt;4 &amp;lt; x &amp;gt;= 23&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Here's our problematic grammar:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) add_rem)?
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) mul_div)?
atom → NUM | NAME
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the &lt;code&gt;add_rem&lt;/code&gt; production rule, there are 2 things going on:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Calling &lt;code&gt;mul_div&lt;/code&gt; to get the first operand, which ensures operator precedence rules are respected&lt;/li&gt;
&lt;li&gt;Recursively calling itself to get the second operand, which handles operations sequences (&lt;code&gt;3 + x + 1&lt;/code&gt;)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The associativity &lt;em&gt;bug&lt;/em&gt; is caused by the second point:
to handle operations sequences, we forced the operators to be right-associative, whereas the arithmetic operators &lt;code&gt;+&lt;/code&gt;, &lt;code&gt;-&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt; and &lt;code&gt;/&lt;/code&gt; are left-associative.&lt;/p&gt;
&lt;p&gt;To fix this, we'll have to handle operations sequences differently:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) mul_div)*
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) atom)*
atom → NUM | NAME
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In this fixed version of the grammar, recursivity was replaced by the usage of the &lt;code&gt;*&lt;/code&gt; operator, which in EBNF means &lt;em&gt;zero or more repetitions&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;With this grammar, and when parsing the problematic expression &lt;code&gt;3 - 2 + 1&lt;/code&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;expr&lt;/code&gt; calls &lt;code&gt;add_rem&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;add_rem&lt;/code&gt; calls &lt;code&gt;mul_div&lt;/code&gt; to get its first operand&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mul_div&lt;/code&gt; calls &lt;code&gt;atom&lt;/code&gt; to get its first operand&lt;/li&gt;
&lt;li&gt;&lt;code&gt;atom&lt;/code&gt; matches the &lt;code&gt;3&lt;/code&gt; literal&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mul_div&lt;/code&gt; yields because the input doesn't match any of the operators it expects&lt;/li&gt;
&lt;li&gt;&lt;code&gt;add_rem&lt;/code&gt; has &lt;code&gt;3&lt;/code&gt; as its first operand. It then starts the repetition cycle because the input matches one of the operators it handles (&lt;code&gt;+&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;add_rem&lt;/code&gt; calls &lt;code&gt;mul_div&lt;/code&gt; again to get its second operand, which will eventually return &lt;code&gt;2&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;It can't be seen in the grammar, but the parser should now create an &lt;code&gt;EOp&lt;/code&gt; node with what was matched so far. This way the left-associativity of &lt;code&gt;-&lt;/code&gt; is honored. Also, this produced node now becomes the first operand for the next operation&lt;/li&gt;
&lt;li&gt;The repetition cycle loops again, as the next input is &lt;code&gt;-&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mul_div&lt;/code&gt; calls &lt;code&gt;atom&lt;/code&gt; which returns the &lt;code&gt;1&lt;/code&gt; literal&lt;/li&gt;
&lt;li&gt;another &lt;code&gt;EOp&lt;/code&gt; is created, with the first subtraction as its first argument and &lt;code&gt;1&lt;/code&gt; as its second.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Here's the modified parser code for &lt;code&gt;add_rem&lt;/code&gt; and &lt;code&gt;mul_div&lt;/code&gt; that implements this logic:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;add_rem&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This modified version of the parser produces the following AST for &lt;code&gt;3 - 2 + 1&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast-good-assoc.png" src="/images/graphviz/litil-pratt-parser-ast-good-assoc.png" /&gt;&lt;/p&gt;
&lt;p&gt;Again, constrast it to the &lt;em&gt;bad&lt;/em&gt; AST:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Bad AST" src="/images/graphviz/litil-pratt-parser-ast-bad-assoc.png" /&gt;&lt;/p&gt;
&lt;h3 id="parenthesized-expressions"&gt;Parenthesized expressions&lt;/h3&gt;
&lt;p&gt;Next the parenthesized expressions.&lt;br /&gt;
Such expressions must have precedence over the others.
Now remember how we solved the precedence problem between additions and multiplications:
we first try the expression with less precedence, and have it call the one with more precedence to parse its operands.
For parenthesized expressions, they have more precedence than additions and multiplications.
Actually, they have the same precedence as atoms (numeric literals and names for example):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) add_rem)?
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) mul_div)?
atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And the parser code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;(&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unexpected input&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Here's the AST produced from parsing &lt;code&gt;5 * (2 + 3)&lt;/code&gt;, which shows that it is correctly parsed:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast-par.png" src="/images/graphviz/litil-pratt-parser-ast-par.png" /&gt;&lt;/p&gt;
&lt;h3 id="unary-minus"&gt;Unary minus&lt;/h3&gt;
&lt;p&gt;We already handled the binary minus (&lt;code&gt;-&lt;/code&gt;) in subtractions, e.g. &lt;code&gt;5 - x&lt;/code&gt;.&lt;br /&gt;
We'd also like to handle the unary minus to support negative numbers, e.g. &lt;code&gt;-3&lt;/code&gt;, or in an expression like &lt;code&gt;-3 - x&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The unary minus has a higher precedence than binary operators (like &lt;code&gt;+&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt;, etc.).
We'll handle it in the &lt;code&gt;atom&lt;/code&gt; production rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) mul_div)*
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) atom)*
atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39; | `-` atom
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Here's the revised &lt;code&gt;atom&lt;/code&gt; parsing method:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;(&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
            &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unexpected input&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Notice that unary minus produces an operation with the &lt;code&gt;-&lt;/code&gt; operator and no right operand.&lt;/p&gt;
&lt;p&gt;Here's the updated &lt;code&gt;evaluate&lt;/code&gt; method that handles unary minus:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ENum&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;EOp&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;Integer&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;UnsupportedOperationException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Not impelmented&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="comparison-operators"&gt;Comparison operators&lt;/h3&gt;
&lt;p&gt;We'd like to handle the following comparison operators: &lt;code&gt;&amp;lt;&lt;/code&gt;, &lt;code&gt;&amp;lt;=&lt;/code&gt;, &lt;code&gt;&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;gt;=&lt;/code&gt; and &lt;code&gt;=&lt;/code&gt;.&lt;br /&gt;
We'd like these operators to have less precedence than the arithmetic ones, so that expressions like &lt;code&gt;i - 1 &amp;lt; length&lt;/code&gt; get parsed correctly.&lt;br /&gt;
Again, the trick is to stick the production rule for such expressions &lt;code&gt;comp&lt;/code&gt; before the &lt;code&gt;add_rem&lt;/code&gt; rule, and have &lt;code&gt;comp&lt;/code&gt; call &lt;code&gt;add_rem&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → comp
comp → add_rem ((&amp;#39;&amp;lt;&amp;#39;|&amp;#39;&amp;lt;=&amp;#39;|&amp;#39;&amp;gt;&amp;#39;|&amp;#39;&amp;gt;=&amp;#39;|&amp;#39;=&amp;#39;) add_rem)?
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Also notice that &lt;code&gt;comp&lt;/code&gt; is not recursive, unlike &lt;code&gt;add_rem&lt;/code&gt; and &lt;code&gt;mul_div&lt;/code&gt;.
That's because comparisons cannot be chained like arithmetic operations.
For instance, &lt;code&gt;1 + 2 - 3&lt;/code&gt; is a valid expression, whereas &lt;code&gt;1 &amp;gt; 2 &amp;lt;= 3&lt;/code&gt; is not.&lt;/p&gt;
&lt;p&gt;Here's the parser code for &lt;code&gt;expr&lt;/code&gt; and &lt;code&gt;comp&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;comp&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;comp&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add_rem&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;lt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;lt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;gt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add_rem&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And here's the AST produced from parsing &lt;code&gt;i -1 &amp;lt; length&lt;/code&gt; which shows that it is correctly parsed:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast-comp.png" src="/images/graphviz/litil-pratt-parser-ast-comp.png" /&gt;&lt;/p&gt;
&lt;p&gt;We'll also need to update the &lt;code&gt;evaluate&lt;/code&gt; method to handle the comparison operators.&lt;br /&gt;
The &lt;code&gt;evaluate&lt;/code&gt; method should be modified to return &lt;code&gt;Object&lt;/code&gt; instead of &lt;code&gt;int&lt;/code&gt;, as we'll be dealing with booleans besides integers:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ENum&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;EOp&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;lt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;lt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;UnsupportedOperationException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Not impelmented&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="binary-boolean-operators"&gt;Binary boolean operators&lt;/h3&gt;
&lt;p&gt;The binary boolean operators &lt;code&gt;and&lt;/code&gt; and &lt;code&gt;or&lt;/code&gt;:
&lt;em&gt; are left-associative: and so we'll use the same trick we used for the arithmetic expressions (using the * repetition)
&lt;/em&gt; should have lower precedence than comparison operators in order to be able to handle expressions like &lt;code&gt;x &amp;gt; 0 and x &amp;lt; 10&lt;/code&gt;. We'll declare their production rule &lt;code&gt;and_or&lt;/code&gt; before &lt;code&gt;comp&lt;/code&gt; and have &lt;code&gt;and_or&lt;/code&gt; call &lt;code&gt;comp&lt;/code&gt; for its operands
* &lt;code&gt;and&lt;/code&gt; have a higher precedence than &lt;code&gt;or&lt;/code&gt; as is the case in many programming languages&lt;/p&gt;
&lt;p&gt;Here's the revised EBNF grammar:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → or
or → and (`or` and)*
and → comp (&amp;#39;and&amp;#39; comp)*
comp → add_rem ((&amp;#39;&amp;lt;&amp;#39;|&amp;#39;&amp;lt;=&amp;#39;|&amp;#39;&amp;gt;&amp;#39;|&amp;#39;&amp;gt;=&amp;#39;|&amp;#39;=&amp;#39;) add_rem)?
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) mul_div)*
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) atom)*
atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39; | `-` atom
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The parser code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;or&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;or&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;and&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;or&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;and&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;comp&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;and&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;comp&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And the evaluator:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ENum&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;eNum&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;EOp&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;lt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;lt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;and&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;or&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;UnsupportedOperationException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Not impelmented&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="unary-boolean-operator-aka-not"&gt;Unary boolean operator, aka &lt;em&gt;not&lt;/em&gt;&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;not&lt;/code&gt; closely resembles the unary minus:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39; | `-` atom | `not` atom
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The parser:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;(&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;not&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;not&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operand&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unexpected input&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And the evaluator:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;not&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;!(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="if-expressions"&gt;If expressions&lt;/h3&gt;
&lt;p&gt;We've already implemented the parsing of &lt;code&gt;if&lt;/code&gt; expressions in the previous post, so I'll just reproduce the code here:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39; | `-` atom | `not` atom | &amp;#39;if&amp;#39; expr &amp;#39;then&amp;#39; expr &amp;#39;else&amp;#39; expr
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The AST type:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EIf&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;thenExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;elseExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The parser (in the &lt;code&gt;atom&lt;/code&gt; method):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;then&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;else&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And the evaluator:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;EIf&lt;/span&gt; &lt;span class="n"&gt;eIf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;eIf&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;eIf&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;eIf&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="function-application"&gt;Function application&lt;/h3&gt;
&lt;p&gt;This is one of the trickiest parts of the language to parse.&lt;br /&gt;
Here's how the syntax looks like:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;function&lt;/span&gt; &lt;span class="n"&gt;arg1&lt;/span&gt; &lt;span class="n"&gt;arg2&lt;/span&gt; &lt;span class="n"&gt;arg3&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt; 
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Where in a language like Java &lt;code&gt;function&lt;/code&gt; would be the function name, in litil &lt;code&gt;function&lt;/code&gt; can be an expression that returns a function.&lt;/p&gt;
&lt;p&gt;Here are some actual examples from litil:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;print&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- simplest case&lt;/span&gt;
&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;incBy&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- incBy is a function that returns a function&lt;/span&gt;
&lt;span class="o"&gt;(\&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt; &lt;span class="c"&gt;-- applying an anonymous lambda &lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Parsing function applications would consist of:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Parse an &lt;code&gt;atom&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Look ahead to see if the next token is the start of an &lt;code&gt;atom&lt;/code&gt;.&lt;ul&gt;
&lt;li&gt;If that's the case, then we're parsing a function application where the first &lt;code&gt;atom&lt;/code&gt; is the function expression and the other &lt;code&gt;atom&lt;/code&gt;s following it are its arguments&lt;/li&gt;
&lt;li&gt;Otherwise, it is no function appliation, just a regular &lt;code&gt;atom&lt;/code&gt; instead&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's a small catch though with the unary minus:
consider the following program:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Should it be parsed as &lt;em&gt;f minus one&lt;/em&gt; or &lt;em&gt;the f function applied to minus 1&lt;/em&gt; ?
I've decided to pick the the first form, i.e. &lt;em&gt;f minus one&lt;/em&gt;, because otherwise regular arithmetic exceptions would fail to parse whenever the minus operator is involved, which would be a &lt;em&gt;PITA&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Parenthesis can be used to express the second form:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;(-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the grammar, we'll add a new production rule &lt;code&gt;app&lt;/code&gt; for function application after &lt;code&gt;mul_div&lt;/code&gt; and before &lt;code&gt;atom&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → or
or → and (`or` and)*
and → comp (&amp;#39;and&amp;#39; comp)*
comp → add_rem ((&amp;#39;&amp;lt;&amp;#39;|&amp;#39;&amp;lt;=&amp;#39;|&amp;#39;&amp;gt;&amp;#39;|&amp;#39;&amp;gt;=&amp;#39;|&amp;#39;=&amp;#39;) add_rem)?
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) mul_div)*
mul_div → app ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;|&amp;#39;%&amp;#39;) app)*
app → atom atom*
atom → NUM | NAME | &amp;#39;(&amp;#39; expr &amp;#39;)&amp;#39; | `-` atom | `not` atom | &amp;#39;if&amp;#39; expr &amp;#39;then&amp;#39; expr &amp;#39;else&amp;#39; expr
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Notice that &lt;code&gt;mul_div&lt;/code&gt; now calls &lt;code&gt;app&lt;/code&gt; instead of &lt;code&gt;atom&lt;/code&gt; for its operands.&lt;/p&gt;
&lt;p&gt;We'll also need to create a new AST node type, &lt;code&gt;EAp&lt;/code&gt;, for function applications.
&lt;code&gt;EAp&lt;/code&gt; should consist of a function expression and a list of the call arguments' expressions.&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EAp&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;EAp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;fn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The parser needs to be updaed to add the &lt;code&gt;app&lt;/code&gt; parsing method and to modify &lt;code&gt;mul_div&lt;/code&gt; so that it calls the new function instead of &lt;code&gt;atom&lt;/code&gt; for its operands:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;mul_div&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EOp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;app&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ArrayList&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;(&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;not&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;add&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;atom&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;isEmpty&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EAp&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Notice the usage of the &lt;code&gt;peek&lt;/code&gt; method.
&lt;code&gt;peek&lt;/code&gt; is pretty much the same as &lt;code&gt;found&lt;/code&gt;, except that it doesn't consume the token even if it matches its arguments.
Here's its code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="nf"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="nf"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;As explained earlier, &lt;code&gt;app()&lt;/code&gt; starts by parsing an &lt;code&gt;atom&lt;/code&gt;.
Afterwards, if the next token &lt;strong&gt;can be the start of an &lt;code&gt;atom&lt;/code&gt;&lt;/strong&gt; (hence the the &lt;em&gt;ored&lt;/em&gt; calls to &lt;code&gt;peek&lt;/code&gt;), &lt;code&gt;atom&lt;/code&gt; gets called again to parse it and the returned expression is stored in the arguments list.&lt;br /&gt;
At the end, if we didn't parse any arguments, then it was just a &lt;em&gt;regular&lt;/em&gt; &lt;code&gt;atom&lt;/code&gt; and its expression is returned.&lt;br /&gt;
Otherwise, it was a function call, and so an &lt;code&gt;EAp&lt;/code&gt; node is constructed and returned.&lt;/p&gt;
&lt;p&gt;The unary minus ambiguity was handled by not considering the &lt;code&gt;-&lt;/code&gt; symbol as a start of an atom (in the calls to &lt;code&gt;peek&lt;/code&gt; from &lt;code&gt;app&lt;/code&gt;).&lt;/p&gt;
&lt;h2 id="closing-words"&gt;Closing words&lt;/h2&gt;
&lt;p&gt;And we're done !
We just finished writing a &lt;em&gt;not-too-shabby&lt;/em&gt; expression parser that handles arithmetic, comparison and boolean operators, function applications and &lt;code&gt;if&lt;/code&gt; expressions in approximately 100 lines of Java code&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;How able is it ?
It can parse this expression &lt;code&gt;(x + 2) * y &amp;lt; max and -(f x y) - 1 &amp;gt;= min or not h&lt;/code&gt; into this AST&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2" rel="footnote"&gt;2&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-pratt-parser-ast-big.png" src="/images/graphviz/litil-pratt-parser-ast-big.png" /&gt;&lt;/p&gt;
&lt;p&gt;However, parsing a very simple expression composed of a single numeric literal, like &lt;code&gt;5&lt;/code&gt; for example, requires 8 method calls:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;expr()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;or()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;and()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;comp()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;add_rem()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mul_div()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;app()&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;atom()&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Not very efficient, but necessary in a recursive descent parser if we want to respect the operators precedence rules.&lt;br /&gt;
That's why I ended up parsing expressions in litil using a &lt;a href="http://en.wikipedia.org/wiki/Pratt_parser"&gt;Pratt parser&lt;/a&gt; instead.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;excluding the imports and the parsing support code (&lt;code&gt;found&lt;/code&gt;, &lt;code&gt;expect&lt;/code&gt; and &lt;code&gt;peek&lt;/code&gt;) as can be seen in &lt;a href="https://gist.github.com/jawher/67f77ba0574496837e41"&gt;this gist&lt;/a&gt;&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;yes, I love graph images and footnotes too !&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The Litil chronicle - Chapter 3.2, Recursive Descent Parsing</title>
   <link href="https://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"/>
   <published>2013-08-19T00:00:00+02:00Z</published>
   <updated>2013-08-19T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/</id>
   <content type="html">&lt;p&gt;In &lt;a href="http://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"&gt;The previous post&lt;/a&gt;, I talked about some of the theory behind parsing.
This post however will be about how to actually write down a parser, which given a list of tokens generated by the lexer would generate an AST.&lt;/p&gt;
&lt;p&gt;For example, given the following Litil program:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;bool2int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Feeding it to the lexer would generate the following token stream:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;KEYWORD(let)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(bool2int)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(x)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;SYM(=)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;KEYWORD(if)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(x)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;KEYWORD(then)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NUM(1)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;KEYWORD(else)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NUM(0)&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Feeding this token stream to the parser we're going to create would generate this AST&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-parser-ast0.png" src="/images/graphviz/litil-parser-ast0.png" /&gt;&lt;/p&gt;
&lt;p&gt;As we've seen in the previous post, there are different kinds of parsers: Top-down (LL for example), Bottom-up (LR for example) and many concrete methods and algorithms to implement them.&lt;/p&gt;
&lt;p&gt;I chose to use the &lt;a href="http://en.wikipedia.org/wiki/Recursive-descent_parser"&gt;recursive descent parsing&lt;/a&gt; in litil for the following reasons:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Such parsers are easy to write by hand&lt;/li&gt;
&lt;li&gt;The resulting code is readable and easy to evolve and maintain&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="from-grammar-to-parser"&gt;From grammar to parser&lt;/h2&gt;
&lt;p&gt;What follows is how to create a recursive descent parser from an EBNF grammar.&lt;/p&gt;
&lt;p&gt;We're going to create a parser for a subset of litil.
Here's its EBNF grammar:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;program → letBinding*

letBinding → &amp;#39;let&amp;#39; NAME NAME* &amp;#39;=&amp;#39; expression
expression → &amp;#39;if&amp;#39; expression &amp;#39;then&amp;#39; expression &amp;#39;else&amp;#39; expression | NAME | &amp;#39;true&amp;#39; | &amp;#39;false&amp;#39; | NUM
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the language defined by the grammar above, a program is a set of let bindings.&lt;br /&gt;
A let binding starts with the &lt;code&gt;let&lt;/code&gt; keyword followed by a name, an optional list of arguments, the &lt;code&gt;=&lt;/code&gt; symbol followed by an expression.  &lt;/p&gt;
&lt;p&gt;An expression can either be a name (referencing a variable by its name), a numeric literal (5, 42, etc.), a boolean literal (&lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;) or an &lt;code&gt;if&lt;/code&gt; expression.&lt;br /&gt;
An &lt;code&gt;if&lt;/code&gt; expression starts with the &lt;code&gt;if&lt;/code&gt; keyword followed by a condition (which is an expression).&lt;br /&gt;
The branch values are specified as an expression after the &lt;code&gt;then&lt;/code&gt; and &lt;code&gt;else&lt;/code&gt; keywords.&lt;/p&gt;
&lt;p&gt;Some examples of programs written in this language:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;true&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="bp"&gt;true&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="mi"&gt;45&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Granted, it is not a very useful language (it can't even add 2 numbers together or compare them).&lt;br /&gt;
However, its grammar is compact and simple and it will serve for the purpose of demonstrating how to write a recursive descent parser.&lt;/p&gt;
&lt;h3 id="the-framework"&gt;The framework&lt;/h3&gt;
&lt;p&gt;We'll start by creating a class for the parser that takes a lexer as its constructor argument:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LitilParser&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;LookaheadLexer&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;LitilParser&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LookaheadLexer&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;lexer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Next, we'll add a field that will hold the current token the lexer just read and a couple of functions that operate on it:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LitilParser&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;LookaheadLexer&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;LitilParser&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LookaheadLexer&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;lexer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="nf"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;pop&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
            &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="nf"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;pop&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
            &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tk&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The &lt;code&gt;found&lt;/code&gt; function checks if the upcoming token's type from the lexer matches its argument.&lt;br /&gt;
If that's the case, the token is consumed and stored in the &lt;code&gt;token&lt;/code&gt; field, and &lt;code&gt;true&lt;/code&gt; is returned.&lt;br /&gt;
Else, no token gets consumed and &lt;code&gt;false&lt;/code&gt; is returned.&lt;/p&gt;
&lt;p&gt;Here's an example of how this function can be used:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;There's a second variant that also checks the upcoming token's value besides its type:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// try and parse a let binding&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We'll also need &lt;code&gt;expect&lt;/code&gt;, which behaves like &lt;code&gt;found&lt;/code&gt;, except that it throws an exception instead of returning false:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LitilParser&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(!&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Was expecting a token of type &amp;#39;&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;#39; but got &amp;quot;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(!&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Was expecting &amp;#39;&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;&amp;#39;(&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;ParseException&lt;/span&gt; &lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ParseException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getCurrentLine&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;col&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The &lt;code&gt;error&lt;/code&gt; function constructs a &lt;code&gt;ParseException&lt;/code&gt;, which holds an error message, the current line being parsed and the parse coordinates (row and column):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LitilParser&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;ParseException&lt;/span&gt; &lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;peek&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ParseException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lexer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getCurrentLine&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;col&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Here's the &lt;code&gt;ParseException&lt;/code&gt; declaration:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ParseException&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;RuntimeException&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;ParseException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;super&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%s (@%d:%d)\n%s\n%s^\n&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ParseException&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="nf"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;StringBuilder&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StringBuilder&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;append&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot; &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;&lt;code&gt;ParseException&lt;/code&gt; is a runtime exception (unchecked) that formats its arguments into a handy display format.&lt;br /&gt;
For instance, given this invalid input (due to a missing let binding name):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;true&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The &lt;code&gt;ParseException&lt;/code&gt; would generate a message resembling the following:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nl"&gt;ParseException&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Was&lt;/span&gt; &lt;span class="n"&gt;expecting&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="n"&gt;of&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt; &lt;span class="n"&gt;but&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="n"&gt;SYM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sc"&gt;&amp;#39;=&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;@&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;@2&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;let&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;true&lt;/span&gt;
    &lt;span class="o"&gt;^&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="coding-the-production-rules"&gt;Coding the production rules&lt;/h3&gt;
&lt;p&gt;Once we have the basic framework I showed above, converting the EBNF grammar into a parser becomes as easy as:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Create a method for every production rule using the left hand side non terminal's name as the method's name&lt;/li&gt;
&lt;li&gt;For the right hand part:&lt;ol&gt;
&lt;li&gt;A terminal maps to a call to &lt;code&gt;expect&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;A non-terminal maps to a call to its production rule method&lt;/li&gt;
&lt;li&gt;A choice (&lt;code&gt;|&lt;/code&gt;) or an optional part (&lt;code&gt;?&lt;/code&gt;) is handled with the &lt;code&gt;found&lt;/code&gt; primitive and the host language's &lt;code&gt;if&lt;/code&gt; construct&lt;/li&gt;
&lt;li&gt;A repetition is handled with the host language's &lt;code&gt;while&lt;/code&gt; construct&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Let's apply this to the grammar we defined above:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;program → letBinding*
letBinding → &amp;#39;let&amp;#39; NAME NAME* &amp;#39;=&amp;#39; expression
expression → &amp;#39;if&amp;#39; expression &amp;#39;then&amp;#39; expression &amp;#39;else&amp;#39; expression | NAME | &amp;#39;true&amp;#39; | &amp;#39;false&amp;#39; | NUM
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Applying the rule 1, we create 3 methods, one for every product rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;program&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// TODO&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;


&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// TODO&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// TODO&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Let's implement the &lt;code&gt;program&lt;/code&gt; method.&lt;br /&gt;
As a reminder, here's the right hand part of its production rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;program → letBinding*
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;i.e. zero or more repetitions of the letBinding non terminal.&lt;br /&gt;
Applying the rule 2.4, we know we'll use a &lt;code&gt;while&lt;/code&gt; loop the handle the repetition.
We'll also use a special feature in litil's lexer, which produces a &lt;code&gt;NEWLINE&lt;/code&gt; token  for every non-empty line in the program.&lt;br /&gt;
Here's how &lt;code&gt;program&lt;/code&gt; should be implemented:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;program&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NEWLINE&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Or in english:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;as long a we get a &lt;code&gt;NEWLINE&lt;/code&gt; token: that's the &lt;code&gt;while(found(NEWLINE))&lt;/code&gt; part&lt;/li&gt;
&lt;li&gt;parse a let binding: this is done by simply calling the &lt;code&gt;letBinding&lt;/code&gt; method (rule 2.2)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;How about &lt;code&gt;letBinding&lt;/code&gt; ?&lt;/p&gt;
&lt;p&gt;Here's its production rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;letBinding → &amp;#39;let&amp;#39; NAME NAME* &amp;#39;=&amp;#39; expression
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This cleanly maps to the following java code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// NOP&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We start by handling the 'let' terminal.
We simply apply the rule 2.1 by calling &lt;code&gt;expect&lt;/code&gt; to &lt;em&gt;expect&lt;/em&gt; the &lt;code&gt;let&lt;/code&gt; keyword.&lt;br /&gt;
Same for the name that follows it. We simply call &lt;code&gt;expect(NAME)&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;We then have to handle &lt;code&gt;NAME*&lt;/code&gt; (the optional arguments list). This is achieved with a &lt;code&gt;while&lt;/code&gt; loop and the &lt;code&gt;found&lt;/code&gt; primitive (rule 2.4).&lt;br /&gt;
We then &lt;em&gt;expect&lt;/em&gt; the &lt;code&gt;=&lt;/code&gt; symbol and finally, parse an expression by calling the &lt;code&gt;expr&lt;/code&gt; method.&lt;/p&gt;
&lt;p&gt;Here's the production rule for an expression:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expression → &amp;#39;if&amp;#39; expression &amp;#39;then&amp;#39; expression &amp;#39;else&amp;#39; expression | NAME | &amp;#39;true&amp;#39; | &amp;#39;false&amp;#39; | NUM
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;To handle the choice, we apply rule 2.3 and use the &lt;code&gt;if&lt;/code&gt; construct with the &lt;code&gt;found&lt;/code&gt; primitive:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;then&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;else&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;//NOP&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;//NOP&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;//NOP&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unrecognized expression&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;That's it.
Seriously, that's it.
We just coded a recursive descent parser for the grammar given above.&lt;/p&gt;
&lt;p&gt;To actually run this parser, we can use this snippet:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="kd"&gt;throws&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let y = if true then 45 else x&amp;quot;&lt;/span&gt;
    &lt;span class="n"&gt;Reader&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StringReader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;LitilParser&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LitilParser&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LookaheadLexerWrapper&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StructuredLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;BaseLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;  &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;program&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Simply instantiate a lexer and a parser, and call the &lt;code&gt;program&lt;/code&gt; method to start the parsing process.&lt;/p&gt;
&lt;h3 id="ast"&gt;AST&lt;/h3&gt;
&lt;p&gt;There still one thing missing though:
in its current form, the parser can only ensure that a program is valid according to a grammar or throw a &lt;code&gt;ParseException&lt;/code&gt; if it isn't.  &lt;/p&gt;
&lt;p&gt;That's useful in itself, but what we'd really like is to have it produce an AST so we can do more useful stuff later, like type checking or executing it for example.&lt;/p&gt;
&lt;p&gt;In this case, and according the EBNF grammar defined above, we'll define these AST node types:&lt;/p&gt;
&lt;h4 id="program"&gt;Program&lt;/h4&gt;
&lt;p&gt;A program is a list of let bindings:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Program&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;Program&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;instructions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h4 id="letbinding"&gt;LetBinding&lt;/h4&gt;
&lt;p&gt;A let binding can be described with an object containing the following properties:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;name&lt;/td&gt;
&lt;td&gt;&lt;code&gt;String&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the name of the the variable or function being defined&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;args&lt;/td&gt;
&lt;td&gt;&lt;code&gt;List&amp;lt;String&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the list of argument names&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;body&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Expression&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the expression forming the value of the let binding&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;or in Java:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LetBinding&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h4 id="expression"&gt;Expression&lt;/h4&gt;
&lt;p&gt;Now, expressions are a bit trickier because of the choice in the production rule:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expression → &amp;#39;if&amp;#39; expression &amp;#39;then&amp;#39; expression &amp;#39;else&amp;#39; expression | NAME | &amp;#39;true&amp;#39; | &amp;#39;false&amp;#39; | NUM
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;So we're going to need a base type:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;abstract&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And a sub class for every choice:&lt;/p&gt;
&lt;h5 id="eif"&gt;EIf&lt;/h5&gt;
&lt;p&gt;&lt;code&gt;EIf&lt;/code&gt; should contain the following properties:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;cond&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Expression&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the if condition&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;thenExpr&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Expression&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the expression to return when the &lt;code&gt;then&lt;/code&gt; branch matches&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;elseExpr&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Expression&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the expression to return when the &lt;code&gt;else&lt;/code&gt; branch matches&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Which translates to the following Java code:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EIf&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;thenExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;elseExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h5 id="ename"&gt;EName&lt;/h5&gt;
&lt;p&gt;Just a class with a name field:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EName&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h5 id="ebool"&gt;EBool&lt;/h5&gt;
&lt;p&gt;Same for boolean literals expressions:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EBool&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h5 id="enum"&gt;ENum&lt;/h5&gt;
&lt;p&gt;And also the same for numeric literals expressions:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ENum&lt;/span&gt; &lt;span class="kd"&gt;extends&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="generating-the-ast"&gt;Generating the AST&lt;/h3&gt;
&lt;p&gt;Now that we have defined our AST node types, we need to make the parser construct it.&lt;/p&gt;
&lt;p&gt;We'll start by modifying the parse methods to return the correct AST type instead of &lt;code&gt;void&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For example, where &lt;code&gt;expr()&lt;/code&gt; used to return void:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="o"&gt;:&lt;/span&gt;
    &lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We'll make it return &lt;code&gt;Expression&lt;/code&gt; instead:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="o"&gt;:&lt;/span&gt;
    &lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;We'll also need to modify its implementation to actually return an &lt;code&gt;Expression&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Let's start with the &lt;code&gt;if&lt;/code&gt; expressions parsing.&lt;br /&gt;
Here's what we have got so far:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// parse the if condition&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;then&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// parse the then result&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;else&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// parse the else result&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; 
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;What we'll need to do is construct an &lt;code&gt;EIf&lt;/code&gt; node using the condition and branches expressions:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;then&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;else&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; 
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;That was the hardest one.&lt;br /&gt;
The remaining ones (names, numeric and boolean literals) can be implemented as follow:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Long&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseLong&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; 
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Now, here's how the complete &lt;code&gt;expr&lt;/code&gt; method:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;if&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;then&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;else&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EIf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cond&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thenExpr&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;elseExpr&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EName&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ENum&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Long&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseLong&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;EBool&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Boolean&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseBoolean&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Unrecognized expression&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;How about the &lt;code&gt;let&lt;/code&gt; binding construct ?
As a reminder, here's how we coded the &lt;code&gt;let&lt;/code&gt; parsing mthod:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// NOP&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;To get it to produce an AST node, we start by changing its return type from &lt;code&gt;void&lt;/code&gt; to &lt;code&gt;LetBinding&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;LetBinding&lt;/span&gt; &lt;span class="nf"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And then, same as with the &lt;code&gt;if&lt;/code&gt; parsing method, we're going to collect the necessary information to construct the result:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the &lt;code&gt;let&lt;/code&gt; binding name&lt;/li&gt;
&lt;li&gt;the arguments names&lt;/li&gt;
&lt;li&gt;the body's expression&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The &lt;code&gt;let&lt;/code&gt; binding name was parsed using the &lt;code&gt;expect&lt;/code&gt; method:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The &lt;code&gt;expect&lt;/code&gt; function, as explained earlier, reads the next token produced by the lexer, and checks that its type and value match its arguments.
If a match is found, the consumed token is then stored in the &lt;code&gt;token&lt;/code&gt; field.&lt;/p&gt;
&lt;p&gt;Hence, the &lt;code&gt;let&lt;/code&gt; binding's name can be retrieved after the &lt;code&gt;expect&lt;/code&gt; call in &lt;code&gt;token.text&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The same applies to the &lt;code&gt;found&lt;/code&gt; method, i.e. if it matches, the cosumed token is stored in the &lt;code&gt;token&lt;/code&gt; field.&lt;br /&gt;
Thus, to collect the arguments names, we need to modify the empty &lt;code&gt;while&lt;/code&gt; loop's body to add the current token's text into a list:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ArrayList&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;();&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;add&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;It only remains to store the &lt;code&gt;let&lt;/code&gt; bindings's body expression into a variable:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And finally to construct and return a &lt;code&gt;LetBinding&lt;/code&gt; instance using the collected name, argument list and body:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And we're done !
Here's the complete listing of the &lt;code&gt;let&lt;/code&gt; parsing method:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;LetBinding&lt;/span&gt; &lt;span class="nf"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ArrayList&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;add&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;=&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Expression&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Easy peasy !&lt;/p&gt;
&lt;p&gt;I think you got how it works by now.
So I'll simply paste the modified version of the program parsing method below:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;Program&lt;/span&gt; &lt;span class="nf"&gt;program&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;instructions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;ArrayList&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;LetBinding&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NEWLINE&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;add&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;letBinding&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Program&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Now that the different parsing methods, and mainly &lt;code&gt;program&lt;/code&gt; return the AST, we can go back to the &lt;code&gt;main&lt;/code&gt; function to use the return value:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;String&lt;/span&gt;&lt;span class="o"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="kd"&gt;throws&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;let y = if true then 45 else x&amp;quot;&lt;/span&gt;
    &lt;span class="n"&gt;Reader&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StringReader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;LitilParser&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LitilParser&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;LookaheadLexerWrapper&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;StructuredLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;BaseLexer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;  &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;

    &lt;span class="n"&gt;Program&lt;/span&gt; &lt;span class="n"&gt;program&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;program&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Instruction&lt;/span&gt; &lt;span class="n"&gt;instr&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;program&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;instruction&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Do amazing things with the instructions !&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And there you have it, a recursive descent parser that ensures the validaity of a program according to a grammar and produces an AST representing the structure of the program.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;As a side note, the AST tree images in this post were generated using a visitor that transforms a parsed litil AST into a dot file.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The Litil chronicle - Chapter 3.1, Introduction to parsing</title>
   <link href="https://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"/>
   <published>2013-08-11T00:00:00+02:00Z</published>
   <updated>2013-08-11T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/</id>
   <content type="html">&lt;p&gt;After a &lt;a href="http://jawher.me/2012/10/20/creation-langage-programmation-litil-1/"&gt;teaser about the Litil language&lt;/a&gt;,
and a &lt;a href="http://jawher.me/2012/10/21/creation-langage-programmation-litil-2-lexer/"&gt;&lt;em&gt;(boring)&lt;/em&gt; chapter about the lexing component&lt;/a&gt;,
now we finally get to the interesting parts, starting with the parsing.&lt;/p&gt;
&lt;p&gt;There's one catch though: we'll have to go a bit into the theory of grammars and parsing.
But bear with me: I'll try to keep it short and not bore you with too much theory and technical jargon.&lt;/p&gt;
&lt;h2 id="grammar"&gt;Grammar&lt;/h2&gt;
&lt;p&gt;A grammar is a formal way to define the syntax of a language.&lt;/p&gt;
&lt;p&gt;Let's suppose I wanted to present a programming language's syntax to a friend.
I would start from the top-level language constructs, and from there go into their details.&lt;/p&gt;
&lt;p&gt;In the case of litil for example, I would start by telling him that a litil program is a list of let bindings.&lt;br /&gt;
A let binding is a fancy way to say a variable or a function declaration.&lt;br /&gt;
A let binding starts with the &lt;code&gt;let&lt;/code&gt; keyword followed by the variable or function name.&lt;br /&gt;
For variables, next comes the &lt;code&gt;=&lt;/code&gt; symbol followed by its value which is an expression, e.g. &lt;code&gt;let x = 5&lt;/code&gt;.&lt;br /&gt;
For functions, next comes the argument list and then the &lt;code&gt;=&lt;/code&gt; symbol followed by the function's body (a list of instructions), e.g. &lt;code&gt;let f x = x&lt;/code&gt;.&lt;br /&gt;
I would then go on to talk about expressions, their different types (arithmetic, if/then/else, etc.) and how they are constructed.
etc.&lt;/p&gt;
&lt;p&gt;Using natural languages to describe a programming language works, but is clunky, verbose and often imprecise.&lt;br /&gt;
That's why computer scientists have come up with formal notations for a language's grammar.
One such a notation is &lt;a href="http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form"&gt;BNF&lt;/a&gt;/&lt;a href="http://en.wikipedia.org/wiki/EBNF"&gt;EBNF&lt;/a&gt; (Backus Naur Form and Extended Backus Naur Form).&lt;/p&gt;
&lt;h2 id="bnf"&gt;BNF&lt;/h2&gt;
&lt;p&gt;When describing a language, we use these 2 following concepts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;terminals: keywords, values, symbols etc. that appear in a program text and usually map to a lexer generated token, e.g. &lt;code&gt;let&lt;/code&gt;, &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;34&lt;/code&gt;, etc.&lt;/li&gt;
&lt;li&gt;non-terminals: the "to be defined" concepts, like a function, a let binding, an expression, etc.
  They don't appear in the program's text.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A BNF grammar is a list of production rules (that the technical term).&lt;br /&gt;
A production rule explains how a non-terminal is constructed from other terminals and non-terminals by combining them.&lt;br /&gt;
Combining can be as a sequence, i.e. &lt;code&gt;A followed by B&lt;/code&gt; or as a choice &lt;code&gt;A or B&lt;/code&gt;.&lt;br /&gt;
That's it.&lt;/p&gt;
&lt;p&gt;Here is the litil language presentation given earlier in english:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In the case of litil for example, I would start by telling him that a litil program is a list of let bindings.&lt;br /&gt;
A let binding is a fancy way to say a variable or a function declaration.&lt;br /&gt;
A let binding starts with the &lt;code&gt;let&lt;/code&gt; keyword followed by the variable or function name.&lt;br /&gt;
For variables, next comes the &lt;code&gt;=&lt;/code&gt; symbol followed by its value which is an expression, e.g. &lt;code&gt;let x = 5&lt;/code&gt;.&lt;br /&gt;
For functions, next comes the argument list and then the &lt;code&gt;=&lt;/code&gt; symbol followed by the function's body (a list of instructions), e.g. &lt;code&gt;let f x = x&lt;/code&gt;.&lt;br /&gt;
I would then go on to talk about expressions, their different types (arithmetic, if/then/else, etc.) and how they are constructed.
etc.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;From the description, we can extract the following terminals: &lt;code&gt;let&lt;/code&gt;, &lt;code&gt;=&lt;/code&gt; and names.&lt;br /&gt;
The concepts that don't appear in the program's text but that represent that language's building blocks, i.e. the non-terminals are: &lt;code&gt;letBinding&lt;/code&gt; and &lt;code&gt;expression&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Let's write the production rule for a let binding:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;letBinding → &amp;#39;let&amp;#39; NAME (&amp;#39;=&amp;#39; expression | arguments &amp;#39;=&amp;#39; expression)
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the above, &lt;code&gt;letBinding&lt;/code&gt; (to the left of &lt;code&gt;→&lt;/code&gt;) is the non-terminal being defined, whereas the part to the right of &lt;code&gt;→&lt;/code&gt; is the recipe describing how to construct it.  &lt;/p&gt;
&lt;p&gt;The recipe starts with a terminal (&lt;code&gt;let&lt;/code&gt;) followed by a &lt;code&gt;NAME&lt;/code&gt;  token.&lt;br /&gt;
Then there are 2 possibilities:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;In case of a variable declaration: a terminal &lt;code&gt;=&lt;/code&gt; followed by a non-terminal &lt;code&gt;expression&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;In case of a function declaration: a list of arguments (the non terminal &lt;code&gt;arguments&lt;/code&gt;), followed by the terminal &lt;code&gt;=&lt;/code&gt; and the non-terminal &lt;code&gt;expression&lt;/code&gt;, i.e. the expression that forms the body of the function.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The choice is expressed using the &lt;code&gt;|&lt;/code&gt; operator. Also note that we can use parenthesis to group strings of terminal and non terminals.&lt;/p&gt;
&lt;p&gt;So, to define the &lt;code&gt;letBinding&lt;/code&gt; non-terminal, we used 2 new non-terminals (&lt;code&gt;expression&lt;/code&gt; and &lt;code&gt;arguments&lt;/code&gt;) that we now have to define.&lt;/p&gt;
&lt;p&gt;Let's start with &lt;code&gt;arguments&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;arguments → NAME (arguments | ε)
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The above production rule introduces 2 new concepts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;recursivity: a non-terminal's definition can reference itself in the right-hand part. This is a very powerful feature of BNF that enables expressing repeating patterns&lt;/li&gt;
&lt;li&gt;ε : means &lt;em&gt;nothing&lt;/em&gt;. It is usually used to expression optionality or as a stop point for recursive patterns&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So an argument list is defined to be either a name followed by an argument list or &lt;em&gt;nothing&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Let's tackle expressions now. For the sake of simplicity, I'll limit this post to very simple expressions, mainly names (of other variables) and numeric literals:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expression → NAME | NUM
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The production rule is a simple choice between a &lt;code&gt;NAME&lt;/code&gt; or a &lt;code&gt;NUM&lt;/code&gt; token.&lt;/p&gt;
&lt;h2 id="ebnf"&gt;EBNF&lt;/h2&gt;
&lt;p&gt;EBNF extends BNF with a couple operators to ease the writing of production rules.&lt;br /&gt;
There is nothing we can encode with one but not with the other.
It's just that it is easier and less verbose with EBNF than with BNF.&lt;/p&gt;
&lt;p&gt;The operators added by EBNF are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;?&lt;/code&gt;: to indicate an optional part&lt;/li&gt;
&lt;li&gt;&lt;code&gt;*&lt;/code&gt;: to indication a repetition (&lt;strong&gt;zero&lt;/strong&gt; or more) of a part&lt;/li&gt;
&lt;li&gt;&lt;code&gt;+&lt;/code&gt;: to indication a repetition (&lt;strong&gt;one&lt;/strong&gt; or more) of a part&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Armed with these operators, we can now rewrite the production rule for an argument list from:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;arguments → NAME (arguments | ε)
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;to:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;arguments → NAME+
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I could have used a more contrived example so that the gain from using EBNF would have been more spectacular, but you get the idea.&lt;/p&gt;
&lt;p&gt;Here are some examples of EBNF production rules and how they can be expressed in BNF:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;EBNF&lt;/th&gt;
&lt;th&gt;BNF&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;s → a?&lt;/td&gt;
&lt;td&gt;s → a &amp;#124; ε&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;s → a*&lt;/td&gt;
&lt;td&gt;s → (a s) &amp;#124; ε&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;s → a+&lt;/td&gt;
&lt;td&gt;s → a (s &amp;#124; ε)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="how-to-parse"&gt;How to parse&lt;/h2&gt;
&lt;p&gt;There are essentialy 2 ways to parse an input according to a grammar:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Top-down&lt;/strong&gt;: Starting from the first production rule, repeatedly expand every non-terminal (by replacing it with its definition) until there are no non-terminals left and the generated terminals string matches the input. If we consume the input from &lt;strong&gt;L&lt;/strong&gt;eft to right and always expand the &lt;strong&gt;L&lt;/strong&gt;eft-most non terminal, we end up with an &lt;strong&gt;LL&lt;/strong&gt; parser.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Bottom-up&lt;/strong&gt;: Try to recognize patterns (non-terminals) in the input and from there go back up to the first production rule. If we consume the input from &lt;strong&gt;L&lt;/strong&gt;eft to right and always expand the &lt;strong&gt;R&lt;/strong&gt;ight-most non terminal, we end up with an &lt;strong&gt;LR&lt;/strong&gt; parser.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Top-down parsing is arguably more intuitive and easier to understand, implement and debug.
Tools like ANTLR or JavaCC generate Top-down parsers.&lt;/p&gt;
&lt;p&gt;Bottom-up parsing, however, is also arguably more powerful as it can parse more grammars and languages than Top-down.
Tools like yacc/bison generate Bottom-up parsers.&lt;/p&gt;
&lt;p&gt;Here's an example to show Top-down parsing at work:&lt;/p&gt;
&lt;p&gt;Given this grammar describing arithmetic expressions:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;expr → add_rem
add_rem → mul_div ((&amp;#39;+&amp;#39;|&amp;#39;-&amp;#39;) add_rem)?
mul_div → atom ((&amp;#39;*&amp;#39;|&amp;#39;/&amp;#39;) mul_div)?
atom → NUM | NAME | &amp;#39;(&amp;#39; add_rem &amp;#39;)&amp;#39;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And this input:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;x + 15 * (b + 23)
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Here's a rundown of of the Top-down parsing algorithm:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Start with the first rule: &lt;code&gt;expr → add_rem&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;No much choice here: there is a single non-terminal to the right so we expand &lt;code&gt;add_rem&lt;/code&gt; with its defintion to get &lt;code&gt;mul_div (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;We start with the left-most non terminal, i.e. &lt;code&gt;mul_div&lt;/code&gt; and expand it: &lt;code&gt;atom (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Again, we expand the first non-terminal, &lt;code&gt;atom&lt;/code&gt;. Since its definition contains a choice, we need to choose one of the alternatives based on the input. The first token in our input is &lt;code&gt;x&lt;/code&gt; which is of type &lt;code&gt;NAME&lt;/code&gt;, so we pick that alternative and replace &lt;code&gt;atom&lt;/code&gt; with &lt;code&gt;NAME&lt;/code&gt; and advance a virtual cursor in the input. We end up with &lt;code&gt;NAME (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Now the left-most non-terminal is &lt;code&gt;mul_div&lt;/code&gt;. It is located in an optional segment (due to the use of the &lt;code&gt;?&lt;/code&gt; operator). The next token in the input is &lt;code&gt;+&lt;/code&gt;, which doesn't match &lt;code&gt;'*'|'/'&lt;/code&gt;, so we skip the optional part and end up with: &lt;code&gt;NAME (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Now the left-most non-terminal is &lt;code&gt;add_rem&lt;/code&gt;. Again, it is in an optional segment, so we need to check against the first token of the input (&lt;code&gt;+&lt;/code&gt;) to decide if it is to be skipped or kept. We're in luck, as the optional segment also starts with &lt;code&gt;+&lt;/code&gt;. The production rule is now &lt;code&gt;NAME ('+'|'-') add_rem&lt;/code&gt; and we need to expand &lt;code&gt;add_rem&lt;/code&gt;: &lt;code&gt;NAME ('+'|'-') mul_div (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Now the left-most non-terminal is &lt;code&gt;mul_div&lt;/code&gt;. After expanding it, we get: &lt;code&gt;NAME ('+'|'-') atom (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Again, to expand &lt;code&gt;atom&lt;/code&gt; we need to check against the input to pick an alternative. The next token is &lt;code&gt;15&lt;/code&gt;, i.e. a &lt;code&gt;NUM&lt;/code&gt;, so that's the alternative to pick: &lt;code&gt;NAME ('+'|'-') NUM (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;The next token in the input is &lt;code&gt;*&lt;/code&gt;, so we keep the optional part that comes after &lt;code&gt;NUM&lt;/code&gt;: &lt;code&gt;NAME ('+'|'-') NUM ('*'|'/') mul_div (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;After expanding &lt;code&gt;mul_div&lt;/code&gt;: &lt;code&gt;NAME ('+'|'-') NUM ('*'|'/') atom (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;After expanding &lt;code&gt;atom&lt;/code&gt; (by replacing it with &lt;code&gt;'(' add_rem ')'&lt;/code&gt;, since &lt;code&gt;(&lt;/code&gt; is the next token in the input): &lt;code&gt;NAME ('+'|'-') NUM ('*'|'/') '(' add_rem ')' (('*'|'/') mul_div)? (('+'|'-') add_rem)?&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;:&lt;/li&gt;
&lt;li&gt;:&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You get the idea: just continue expanding the left-most non-terminal, using the input to choose among the laternatives until there are no more non-terminals left.
If we get stuck mid-way, then the input is invalid according to our grammar.
Otherwise, the input is considered to be valid.&lt;/p&gt;
&lt;h2 id="concrete-syntax-tree"&gt;Concrete syntax tree&lt;/h2&gt;
&lt;p&gt;Now, if we keep track of all the intermediate steps, and represent every terminal and non-terminal with a node in a graph, and its expansion as its children, we end up with a tree like structure resembling this:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-parser-topdown-tree.png" src="/images/graphviz/litil-parser-topdown-tree.png" /&gt;&lt;/p&gt;
&lt;p&gt;This tree is called the &lt;a href="http://en.wikipedia.org/wiki/Parse_tree"&gt;&lt;em&gt;(concrete)&lt;/em&gt; parse tree&lt;/a&gt;: a structured representation of an input according to grammar where the leaf nodes are terminals and the other nodes represent non-terminals.&lt;/p&gt;
&lt;p&gt;We could write down the Java types representing this tree structure and then a couple of algorithms that operate on it.
For instance, we can represent the tree nodes with the following recursive type:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Node&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;children&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Once the input is transformed into a parse tree, the subsequent operations become much easier as they operate on a structured represetnation instead of the orginal raw text.
Think of an arithmetic evaluator for expressions like the one given above for example.
Compare this with having to implement the said evaluator that directly operates on the input string, without any structured representation.&lt;/p&gt;
&lt;h2 id="abstract-syntax-tree"&gt;Abstract syntax tree&lt;/h2&gt;
&lt;p&gt;The Concret syntax tree is already a very useful form of representing a parsed input.&lt;br /&gt;
However, it contains a lot of noise and could be trimmed a bit to only the useful and important parts:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;For instance, we could get rid of the useless terminals like the parenthesis.
   They were useful in the input string to encode the correct precedence.
   No so much in the syntax tree as the tree structure itself suffices.&lt;/li&gt;
&lt;li&gt;Whenever a node contains a single child, we can get rid of it.
   For instance, for the &lt;code&gt;b&lt;/code&gt; token, we don't really care what path we took to parse it.
   We're only interested in the fact that it is a &lt;code&gt;NAME&lt;/code&gt; and the the value is &lt;code&gt;b&lt;/code&gt;.
   Hence, we can get rid of &lt;code&gt;atom&lt;/code&gt; and &lt;code&gt;mul_div&lt;/code&gt; from its parents chain.
   The same applies for all the other terminals.
   A general rule of thum is: &lt;strong&gt;if a node has only one child, we can replace it with that child&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Applying these 2 optimisations, we end up with this tree:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-parser-topdown-tree-2.png" src="/images/graphviz/litil-parser-topdown-tree-2.png" /&gt;&lt;/p&gt;
&lt;p&gt;Just by removing the useless nodes we get a much leaner tree.&lt;/p&gt;
&lt;p&gt;But we can go even further to make traversing and operating on the tree easier.  &lt;/p&gt;
&lt;p&gt;As it it stands now, while traversing the tree visiting a node at a time, a hypothetical evaluator's code would look like this: &lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;NUM&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Integer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;parseInt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;add_rem&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;operand1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
        &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;operand2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;+&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;operand1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;operand2&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;operand1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;operand2&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Ugly stuff ! String comparisons all over the place, hard-coded list indices, etc.&lt;/p&gt;
&lt;p&gt;A much cleaner and easier to work with and to maintain (once it's done) approach would be to have specialized node types, instead of a single generic &lt;code&gt;Node&lt;/code&gt; type &lt;em&gt;(to rule them all)&lt;/em&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;node&lt;/th&gt;
&lt;th&gt;description&lt;/th&gt;
&lt;th&gt;fields&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Num&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a numeric literal&lt;/td&gt;
&lt;td&gt;value: int&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a name reference&lt;/td&gt;
&lt;td&gt;name: String&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Add&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;an addition&lt;/td&gt;
&lt;td&gt;left, right: Node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Sub&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a subtraction&lt;/td&gt;
&lt;td&gt;left, right: Node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Mul&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a multiplication&lt;/td&gt;
&lt;td&gt;left, right: Node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Div&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a division&lt;/td&gt;
&lt;td&gt;left, right: Node&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Since some of these nodes do reference other nodes (like the &lt;code&gt;Add&lt;/code&gt; which references 2 operands), we'll need to define a common base type &lt;code&gt;Node&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Now in Java:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Num&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Name&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Add&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Sub&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Mul&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Div&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Note that we'll also need to write down the algorithm that transforms the concret parse tree to this representation.&lt;/p&gt;
&lt;p&gt;Using this strongly-typed tree representation, here's how the parse tree would like like:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-parser-ast-tree.png" src="/images/graphviz/litil-parser-ast-tree.png" /&gt;&lt;/p&gt;
&lt;p&gt;Also, the evaluator can be rewritten to this much cleaner form (it could be made cleaner still using the visitor pattern for example):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;Num&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="n"&gt;Num&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Add&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;left&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;right&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="o"&gt;:&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Another approach would be to have the evaluate method in the node types:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Num&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Name&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="c1"&gt;//Lookup the value somewhere&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Add&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Node&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;evaluate&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;But as you'll see in the next chapters, I chose to use the first approach (visitor based) instead of the second (polymorphic) because I would like to be able to parse litil in litil &lt;em&gt;(inception)&lt;/em&gt;.
litil isn't an OOP language and doesn't have subtyping polymorphism, but instead has pattern matching which makes writing the visitor pattern a breeze.&lt;/p&gt;
&lt;p&gt;Oh, by the way: this type of tree with specialized node types is what we call the Abstract Syntax Tree or &lt;em&gt;AST&lt;/em&gt;.&lt;/p&gt;
&lt;h2 id="closing-words"&gt;Closing words&lt;/h2&gt;
&lt;p&gt;That's all the theory we're going to need: grammars, BNF/EBNF, Top-down parsing, Concrete and Abstract Syntax Trees.&lt;/p&gt;
&lt;p&gt;But it is still theory: we still haven't talked about how to write the parser that creates the parse tree.
This is going to be fixed in the &lt;a href="http://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"&gt;next installment&lt;/a&gt; where I'm going to show you how to hand-write a Top-down parser that directly generates an AST using the Recursive descent approach.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>I created my first PCB !</title>
   <link href="https://jawher.me/2013/08/04/first-pcb/"/>
   <published>2013-08-04T00:00:00+02:00Z</published>
   <updated>2013-08-04T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2013/08/04/first-pcb/</id>
   <content type="html">&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/achievement-pcb.png" /&gt;&lt;/p&gt;
&lt;p&gt;Earlier this morning, I received 10 of these beauties in my mailbox:&lt;/p&gt;
&lt;p&gt;Top:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/tiny85-devboard-pcb-top.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;(Overly pessimistic)&lt;/em&gt; Bottom:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/tiny85-devboard-pcb-bottom.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;This is a development board I designed to ease prototyping with Atmel's &lt;a href="http://www.atmel.com/devices/attiny85.aspx"&gt;tiny 85&lt;/a&gt; micro-controllers.
It's a rather basic design with the tiny 85 in a DIP8 package in the center with traces to expose 5 of its digital pins (PB0 to PB4) plus ground and VCC via a header to the top.&lt;br /&gt;
To the bottom, other traces connect to a 10-pin ISP connector for programming.&lt;br /&gt;
There is also a power indicator led, a reset button and decoupling caps, and that's about it.&lt;/p&gt;
&lt;p&gt;Here's an assembled board connected to the programmer and plugged in a breadboard:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/tiny85-devboard-breadboard.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;I've created this board using Eagle CAD. Here's the schematic:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/tiny85-devboard-schematic.png" /&gt;&lt;/p&gt;
&lt;p&gt;The board layout:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/tiny85-devboard-board.png" /&gt;&lt;/p&gt;
&lt;p&gt;And the part list:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Part&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Device&lt;/th&gt;
&lt;th&gt;Package&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;BRAINZ&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;ATTINY85&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C1&lt;/td&gt;
&lt;td&gt;0.1uF&lt;/td&gt;
&lt;td&gt;Ceramic disc capacitor&lt;/td&gt;
&lt;td&gt;0805&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C2&lt;/td&gt;
&lt;td&gt;100uF&lt;/td&gt;
&lt;td&gt;Electrolytic capacitor&lt;/td&gt;
&lt;td&gt;CPOL-RADIAL-100UF-25V&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PWR&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;LED&lt;/td&gt;
&lt;td&gt;0805&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PWR_R&lt;/td&gt;
&lt;td&gt;1K&lt;/td&gt;
&lt;td&gt;Resistor&lt;/td&gt;
&lt;td&gt;0805&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RESET&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;Tactile switch&lt;/td&gt;
&lt;td&gt;TACTILE-PTH&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RST_R&lt;/td&gt;
&lt;td&gt;10K&lt;/td&gt;
&lt;td&gt;Resistor&lt;/td&gt;
&lt;td&gt;0805&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;This PCB design was fabbed using &lt;a href="http://www.seeedstudio.com/depot/fusion-pcb-service-2-layers-p-835.html"&gt;Seeed Studio's fusion service&lt;/a&gt;: an incredibly cheap offer where you get 10 5cmx5cm PCBs for only 9.99$.&lt;/p&gt;
&lt;p&gt;I soldered components on 2 of them, and after a bit of stumbling and yak shaving, mainly due to a buggy USBasp programmer, I got the boards to work beautifully.&lt;/p&gt;
&lt;p&gt;It was also my first time soldering SMD devices (0805 resistors, capacitors and leds).&lt;br /&gt;
I was shocked the first time I hold a 0805 resistor in one hand with the iron in the other.
It was tiny.
Really tiny.
I was like &lt;em&gt;no way I'm going to be able to solder this&lt;/em&gt;.
But it wasn't that bad.&lt;br /&gt;
The only part I struggled with was the power indicator led, and it turned out I was trying to solder the wrong package (a 1206) which was bigger than the pads.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;/me very proud !&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Humble beginnings:&lt;/p&gt;
&lt;p&gt;&lt;img alt="image" src="/images/tiny85-devboard/arduino-humble-beginnings.jpg" /&gt;&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;&lt;em&gt;Achievement unlocked badges generated using http://www.achievementgen.com/360/&lt;/em&gt;&lt;/p&gt;</content>
 </entry>

<entry>
   <title>Une génération de "select * => tableau"</title>
   <link href="https://jawher.me/2012/11/10/generation-select-tableau/"/>
   <published>2012-11-10T00:00:00+01:00Z</published>
   <updated>2012-11-10T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2012/11/10/generation-select-tableau/</id>
   <content type="html">&lt;p&gt;Nous sommes des développeurs.
Du moins, si vous lisez ce blog, vous l'êtes très probablement.
On est des millions dans le monde.
Certains d'entre nous sommes passionnés.
On lit des blogs, on fait de la veille techno, on télécharge et teste les nouveaux frameworks dès leur alpha.
On peut s'appeler des consultants, développeurs artisans, développeurs passionnés, des &lt;em&gt;ninjas&lt;/em&gt;, des &lt;em&gt;rockstars&lt;/em&gt;, tout ce que vous voulez.
La majorité du temps, ce qu'on fait à longueur de journée se résume à:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;select * from uneTable&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;dessiner un tableau qui liste les résultats, avec des liens/boutons pour voir les détails, modifier ou supprimer ces entrées&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Après, en pratique, on gère effectivement beaucoup plus de choses, comme par exemple:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Séparer ça en 3 &lt;em&gt;(ou plus)&lt;/em&gt; couches: isoler le code d'accès à la base de données de la couche métier de la partie web&lt;/li&gt;
&lt;li&gt;Ajouter peut être quelques façades. Sait-t-on jamais !&lt;/li&gt;
&lt;li&gt;Se battre avec le &lt;em&gt;DBA&lt;/em&gt; pour qu'il ajoute une colonne à telle table&lt;/li&gt;
&lt;li&gt;Décider si on va utiliser un &lt;em&gt;ORM&lt;/em&gt; (&lt;em&gt;Hibernate&lt;/em&gt;) ou pas&lt;/li&gt;
&lt;li&gt;Faire le choix entre &lt;em&gt;Spring&lt;/em&gt; et &lt;em&gt;JavaEE&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;Quelques uns iront même jusqu'à s'inquiéter de comment leur appli de gestion va &lt;em&gt;scaler&lt;/em&gt;. L'appli en question sera utilisée par 10 utilisateurs qui sont habitués à des temps de réponse de 50 secondes. Soit, mais que se passera t'il le jour où l'o aura à gérer 1000 utilisateurs ? Ne devrait-t-on pas héberger notre appli sur le &lt;em&gt;cloud&lt;/em&gt; et passer au &lt;em&gt;nosql&lt;/em&gt;, avec un zeste de &lt;em&gt;realtime&lt;/em&gt; ?&lt;/li&gt;
&lt;li&gt;Le tableau qui affiche les résultats du &lt;em&gt;select&lt;/em&gt; devrait être triable sur n'importe quelle colonne, filtrable, paginé, et joli.&lt;/li&gt;
&lt;li&gt;etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On fait que ça.
Oui, des fois, il y'a de la logique métier à implémenter.
Quelques &lt;code&gt;if&lt;/code&gt; par si et par là.
Ou des fois, c'est de l'&lt;strong&gt;illogique&lt;/strong&gt; métier qu'il faut gérer.
Mais on reste toujours dans le &lt;em&gt;workflow&lt;/em&gt; &lt;code&gt;select * from table =&amp;gt; afficher dans un tableau&lt;/code&gt;.
Bienvenue dans le monde des applications de gestion.&lt;/p&gt;
&lt;p&gt;C'est une conséquence inévitable du succès de l'informatique.
C'est devenu un outil indispensable dans tous les domaines.
Du bancaire à la GED en passant par la gestion de stock, tout le monde veut automatiser et informatiser leur &lt;em&gt;process&lt;/em&gt;.
Je m'en plains pas, c'est ce qui fait qu'il y ait si peu de chômeurs parmi les développeurs.&lt;/p&gt;
&lt;p&gt;Mais je n'en suis pas complètement satisfait.
Je dis pas complètement parce que je prend un eu de plaisir à bien implémenter une &lt;em&gt;feature&lt;/em&gt; qui je sais va simplifier la vie aux utilisateurs de l'application.
Mais ça ne me fait plus rêver.&lt;/p&gt;
&lt;p&gt;Je ne suis pas le seul à ressentir ça.
Mais ce que je constate c'est que la majorité de ceux qui veulent s'en sortir vont vers des &lt;em&gt;startups&lt;/em&gt; pour travailler sur des variantes d'un réseau social.
Des développeurs, qui pour la majorité sont très doués, qui passent des milliers d'heures à créer des produits mort-nés qui se résument à une zone de texte qui permet de publier un &lt;em&gt;statut/tweet/update/post&lt;/em&gt; et que les &lt;em&gt;amis/followers&lt;/em&gt; peuvent &lt;em&gt;liker/retweeter/reblogger/favoriser&lt;/em&gt;.
Quel gâchis !&lt;/p&gt;
&lt;p&gt;D'un autre côté, il y'a des développeurs qui travaillent sur des sujets, qui pour moi, sont beaucoup plus intéressants et passionnants:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Les gens qui créent et optimisent les outils qu'on utilise tous les jours: l'os, la base de données, l'éditeur/IDE, etc. Récemment, le toplink sur &lt;a href="http://news.ycombinator.com/item?id=4763879"&gt;HackerNews&lt;/a&gt; parle de &lt;a href="http://www.rethinkdb.com/"&gt;RethinkDb&lt;/a&gt;: une bande de passionnés&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt; qui travaillent pendant des années pour résoudre des problèmes très durs, comme comment ne pas bloquer la base pendant que l'on exécute du &lt;em&gt;map/reduce&lt;/em&gt; là-dessus par exemple.&lt;br /&gt;
  Ou même les gens qui bossent sur les systèmes moins &lt;em&gt;sexy&lt;/em&gt;, comme l'optimisateur de requêtes de Postgres ou Oracle par exemple.&lt;/li&gt;
&lt;li&gt;Les gens qui créent les langages qu'on utilise pour développer nos applis.
  Surtout de nos jours où il ne se passe pas une semaine sans que l'on annonce la sortie d'un nouveau langage: &lt;a href="https://mail.mozilla.org/pipermail/rust-dev/2012-October/002489.html"&gt;Rust&lt;/a&gt;, Go, ou même les petits langages jouets dont parlent les gens sur des sites spécialisés comme &lt;a href="http://lambda-the-ultimate.org/"&gt;λ the ultimate&lt;/a&gt;
  Je suis jaloux de ces gens qui passent leurs journées à réfléchir sur la syntaxe à choisir, la sémantique derrière, l'interaction entre les différents &lt;em&gt;constructs&lt;/em&gt; du langage, puis aux détails d'implémentation: comment inférer et vérifier des types, générer du code optimisé, implémenter le &lt;em&gt;garbage collector&lt;/em&gt;, etc.&lt;/li&gt;
&lt;li&gt;Les gens qui travaillent sur l'intelligence artificielle par exemple, ou le &lt;em&gt;machine-learning&lt;/em&gt; avec les différentes applications tel que l'analyse et compréhension du langage naturel (&lt;em&gt;Siri&lt;/em&gt; et Google), &lt;a href="http://www.youtube.com/watch?feature=player_embedded&amp;amp;v=i3ernrkZ91E"&gt;collaboration entre plusieurs robots &lt;em&gt;(vid)&lt;/em&gt;&lt;/a&gt;, etc.&lt;/li&gt;
&lt;li&gt;Les gens qui ont écrit &lt;a href="http://compass.informatik.rwth-aachen.de/ws-slides/havelund.pdf"&gt;le soft &lt;em&gt;(pdf)&lt;/em&gt;&lt;/a&gt; qui pilote le &lt;em&gt;rover Curiosity&lt;/em&gt;.
  Ou ceux qui ont &lt;a href="http://www.jpl.nasa.gov/news/news.php?release=2010-151"&gt;diagnostiqué et corrigé un problème software sur la sonde voyager&lt;/a&gt; alors qu'elle était à plus de 13 milliards de Km de la terre.&lt;/li&gt;
&lt;li&gt;Les gens qui créent les algorithmes de compression (jpeg, mp3, lzw, etc.), les moteurs de recherche capable de retourner le bon résultat parmi des milliards de pages indexées en quelques millisecondes, les softs de pilotage automatique d'avions ou les voitures sans conducteur, etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Je suis jaloux.
J'aimerais travailler sur des sujets qui auront un impact positif sur l'humanité.
Mais ce n'est pas le cas.
J'arrive à survivre en m'intéressant à l'aspect technique du métier.&lt;/p&gt;
&lt;p&gt;Dans mon temps libre, j'essaie de travailler sur &lt;a href="https://github.com/jawher"&gt;des projets&lt;/a&gt; qui n'ont rien à voir avec les applications de gestion.
Je travaille surtout sur des projets de types "socle technique" (bibliothèque, framework).
Je passe aussi beaucoup de temps à lire d'obscurs papiers de recherche sur la théorie de compilation et l'inférence de types et à travailler sur mon &lt;a href="http://jawher.me/2012/06/04/creation-langage-programmation-litil-1/"&gt;&lt;em&gt;toy-language&lt;/em&gt;&lt;/a&gt;.
J'essaie aussi de ne pas me cantonner au sphère Java en allant souvent sur des sites comme HackerNews et Reddit et à me forcer à élargir mes lectures pour inclure d'autres langages/technos comme Python, ML, Haskell, Clojure, Ruby, Erlang, etc.&lt;/p&gt;
&lt;p&gt;Mais pendant mes journées, je continue à faire du &lt;code&gt;select * =&amp;gt; tableau&lt;/code&gt;.&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;C'est cette couverture du &lt;em&gt;MIT Technology Review&lt;/em&gt; du mois de Novembre qui a motivé ce post:&lt;/p&gt;
&lt;p&gt;&lt;img src="http://images.digital.technologyreview.com/rvimageserver/Technology%20Review/Technology%20Review/November%20December%202012/page0000001.jpg" alt="Damn" width="500" /&gt;&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;Dont Salava Akhmechet, qui tient un &lt;a href="http://www.defmacro.org/"&gt;excellent blog&lt;/a&gt; que je vous conseillerais de prendre le temps de lire.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The new bookmarks page</title>
   <link href="https://jawher.me/2012/10/24/new-bookmarks-page/"/>
   <published>2012-10-24T00:00:00+02:00Z</published>
   <updated>2012-10-24T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/10/24/new-bookmarks-page/</id>
   <content type="html">&lt;p&gt;Just a quick heads up: I've added a new bookmarks page where I'll be posting interesting&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt; stuff as I stumble upon it on the internets™.&lt;/p&gt;
&lt;p&gt;&lt;a href="http://jawher.me/bookmarks.html"&gt;Bookmarks⇾&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;A link to this page was added in the first position in the index page's menu, and in the second position in the other pages, just after the home link.&lt;/p&gt;
&lt;h2 id="the-rationale"&gt;The rationale&lt;/h2&gt;
&lt;p&gt;as most of you must be already doing,
I click on dozens of links every day, a few of which are very interesting and worth keeping track of.&lt;/p&gt;
&lt;p&gt;I use many means to try and collect such links,
like Instapaper, twitter, the save functionality on reddit, starred posts in Google Reader, the browsers bookmarks, and what's not.&lt;/p&gt;
&lt;p&gt;Most of these are online services, out of my control and could disappear overnight.
Even if they stay around, I might (as I already have done many times already) abandon a service for another.&lt;/p&gt;
&lt;p&gt;And so I've decided I'd use my own means of collecting my bookmarks.&lt;/p&gt;
&lt;h2 id="the-how-it-works"&gt;The how &lt;em&gt;(it works)&lt;/em&gt;&lt;/h2&gt;
&lt;p&gt;I'm glad &lt;a href="http://jawher.me/2012/10/19/moving-from-jekyll-to-pelican/"&gt;I switched to Pelican&lt;/a&gt; and to a static site generator in general to power this blog,
as it was very easy to extend it with a &lt;a href="https://github.com/jawher/jawher.me/blob/master/pelican_bookmarks_generator.py"&gt;a plugin&lt;/a&gt;&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2" rel="footnote"&gt;2&lt;/a&gt;&lt;/sup&gt; to generate this new page.&lt;/p&gt;
&lt;p&gt;Each bookmark is materialized as a file in the &lt;code&gt;content/bookmarks&lt;/code&gt; folder.
A bookmark file looks like a regular post, albeit with custom metadata (namely the bookmark url) and an optional (markdown) content.
The plugin I wrote parses and collects these files into a list of bookmarks objects, and passes them to a Jinja2 template.
It is then simply a matter of iterating over them and outputting the appropriate HTML.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;to me at least, but hopefully to you too.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;not claiming it is state of the art, but at least it gets the job done.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</content>
 </entry>

<entry>
   <title>The Litil chronicle - Chapter 2, The text chopper, aka Lexer</title>
   <link href="https://jawher.me/2012/10/21/creation-langage-programmation-litil-2-lexer/"/>
   <published>2012-10-21T00:00:00+02:00Z</published>
   <updated>2012-10-21T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/10/21/creation-langage-programmation-litil-2-lexer/</id>
   <content type="html">&lt;p&gt;In this second post of the the Litil chronicle, I'm going to talk about lexing.
Lexing is generally the first step in a compiler's pipeline.
What it does is transform the source code from its textual representation to a token sequence that'll be consumed by the next stage, i.e. the parser.&lt;/p&gt;
&lt;p&gt;How the text is broken down into tokens varies from one language to another.
Usually though, at least with conventional languages (in the Algol family), we could define the following token groups:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;symbols: operators (&lt;code&gt;+&lt;/code&gt;, &lt;code&gt;-&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt;, &lt;code&gt;/&lt;/code&gt;, etc.), punctuation signs (&lt;code&gt;,&lt;/code&gt;, &lt;code&gt;;&lt;/code&gt;, &lt;code&gt;(&lt;/code&gt;, &lt;code&gt;[&lt;/code&gt;, etc.)&lt;/li&gt;
&lt;li&gt;literals: numerals (&lt;code&gt;42&lt;/code&gt;, &lt;code&gt;3.14&lt;/code&gt;, etc.), strings (&lt;code&gt;"kthxbye"&lt;/code&gt;), booleans (&lt;code&gt;true&lt;/code&gt;, &lt;code&gt;false&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;keywords: &lt;code&gt;if&lt;/code&gt;, &lt;code&gt;while&lt;/code&gt;, etc.&lt;/li&gt;
&lt;li&gt;identifiers: language specific, but generally they start with a letter or an underscore (&lt;code&gt;_&lt;/code&gt;) and optionally a sequence of letters, digits and some other symbols&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A token is comprised of a &lt;strong&gt;type&lt;/strong&gt; (whether it is a symbol, a keyword or a literal, etc.) and a &lt;strong&gt;value&lt;/strong&gt; (its textual value).
A token could also be extended to include contextual data, like the row and column where it appears in the source text.
This could be useful to indicate where in the source an error occurred.&lt;/p&gt;
&lt;p&gt;The lexer could also hide parts of the input deemed useless to the following stage, like whitespace and comments for example.&lt;/p&gt;
&lt;h2 id="whitespace-handling"&gt;Whitespace handling&lt;/h2&gt;
&lt;p&gt;Unlike most other languages, we won't be discarding whitespaces in Litil.
Indeed, and like Python or Haskell for example, whitespace is used in Litil to mark bloc boundaries.
A bloc is started by increasing the indentation level and ended by decreasing it.&lt;/p&gt;
&lt;p&gt;Example:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
  &lt;span class="n"&gt;print&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The Java equivalent would be:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the previous code snippet the &lt;code&gt;then&lt;/code&gt; and &lt;code&gt;else&lt;/code&gt; branches are indented.
This indentation is optional though, and is merely a code convention.
The java lexer and parser couldn't care less if it is there or not.
The bloc boundaries are marked with the opening and closing braces.
So the following code snippet is strictly equivalent:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;  
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;or even:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This hurts the code readability, but doesn't change anything from the compiler's perspective.
This won't be the case with Litil.
As said earlier, indentation level controls the blocs boundaries.&lt;/p&gt;
&lt;p&gt;Also, whereas in Java the &lt;code&gt;;&lt;/code&gt; is used to separate the statements in the same bloc, Litil uses the line breaks instead.
So &lt;code&gt;;&lt;/code&gt; are not just optional in Litil (like in Javascript for example).
They aren't even recognised as a valid symbol.
The rationale behind this choice was to create a clean and clutter-free syntax.
We'll cover the syntax in more detail in upcoming posts, but here's a quick primer on the syntax:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;No braces to mark blocs&lt;/li&gt;
&lt;li&gt;No &lt;code&gt;;&lt;/code&gt; to separate instructions&lt;/li&gt;
&lt;li&gt;No parentheses around function arguments: &lt;code&gt;sin 5&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;No parentheses around &lt;code&gt;if&lt;/code&gt; conditions.&lt;/li&gt;
&lt;li&gt;etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In a nutshell, the lexer we'll be developing won't completely ignore whitespace.
It should rather produce tokens for line breaks (to indicate the end of an instruction) and for line indentations (to indicate an increase or a decrease in indentation level).&lt;/p&gt;
&lt;h2 id="implementation"&gt;Implementation&lt;/h2&gt;
&lt;p&gt;We'll start with the &lt;code&gt;Token&lt;/code&gt; definition:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Token&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;enum&lt;/span&gt; &lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;NEWLINE&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;INDENT&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DEINDENT&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;STRING&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CHAR&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;EOF&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;row&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;col&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;A token is defined by:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;type&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: to indicate a line breaks&lt;/li&gt;
&lt;li&gt;&lt;code&gt;INDENT&lt;/code&gt;: to indicate that the indentation level increased relative to the previous line, and thus marks the start of a new bloc&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DEINDENT&lt;/code&gt;: to indicate that the indentation level decreased relative to the previous line, and thus marks the bloc end&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME&lt;/code&gt;: used for identifiers&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NUM&lt;/code&gt;, &lt;code&gt;STRING&lt;/code&gt;, &lt;code&gt;CHAR&lt;/code&gt;, &lt;code&gt;BOOL&lt;/code&gt;: used for literals&lt;/li&gt;
&lt;li&gt;&lt;code&gt;KEYWORD&lt;/code&gt;: used for keywords&lt;/li&gt;
&lt;li&gt;&lt;code&gt;EOF&lt;/code&gt;: produced when the end of the source text is reached&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;text&lt;/code&gt;: a string containing the token's textual value&lt;/li&gt;
&lt;li&gt;&lt;code&gt;row&lt;/code&gt; et &lt;code&gt;col&lt;/code&gt;: the token's position in the source file&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here's now the lexer's interface:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;Lexer&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="kd"&gt;throws&lt;/span&gt; &lt;span class="n"&gt;LexingException&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="nf"&gt;getCurrentLine&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This interface defines 2 methods:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;pop&lt;/code&gt;: returns the next token&lt;/li&gt;
&lt;li&gt;&lt;code&gt;getCurrentLine&lt;/code&gt;: returns the current line in the source file&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Notice that there's no method that indicates if there are still more tokens in the source file.
Rather, when the lexer reaches the end of the source file, it generates an &lt;code&gt;EOF&lt;/code&gt; token.
So I deemed it unnecessary to add another method, à la iterator's &lt;code&gt;hasNext&lt;/code&gt; to keep the interface minimal.&lt;/p&gt;
&lt;p&gt;One other thing: I'm no fan of defining interfaces just for the sake of it.
The lexer was defined as an interface because there'll be more than one implementation, as we'll see further below.&lt;/p&gt;
&lt;h3 id="how-it-workstm"&gt;How it works™&lt;/h3&gt;
&lt;p&gt;What follows is a quick overview of how the base implementation of the lexer works (&lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/BaseLexer.java"&gt;The source of &lt;code&gt;BaseLexer.java&lt;/code&gt; (on Github) for those who'd like to check it out&lt;/a&gt;):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The lexer's constructor takes a &lt;code&gt;java.io.Reader&lt;/code&gt; argument that points to the source text&lt;/li&gt;
&lt;li&gt;The lexer defines a &lt;code&gt;currentLine&lt;/code&gt; field to hold the line being lexed, plus 2 more fields &lt;code&gt;row&lt;/code&gt; and &lt;code&gt;col&lt;/code&gt; to hold the position&lt;/li&gt;
&lt;li&gt;When &lt;code&gt;pop&lt;/code&gt; is called, we check the value of &lt;code&gt;currentLine&lt;/code&gt;. If none is set, we try to read a whole line from the reader. If the latter returns &lt;code&gt;null&lt;/code&gt;, then we've reached the end of the source text. The lexer switches to a mode where it'll always return an &lt;code&gt;EOF&lt;/code&gt; token&lt;/li&gt;
&lt;li&gt;Else, and after handling the line indentation (we'll come back to this part later), we examine the first non-blank character&lt;/li&gt;
&lt;li&gt;If it is a letter, we consume the next characters as long as they are letters or digits. the consumed characters are appended to a string&lt;ul&gt;
&lt;li&gt;if this string equals &lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;, we return a &lt;code&gt;BOOL&lt;/code&gt; token&lt;/li&gt;
&lt;li&gt;if not, we check if the string is in the list of the keywords and return a &lt;code&gt;KEYWORD&lt;/code&gt; token if that's the case&lt;/li&gt;
&lt;li&gt;else, it must be an identifier. We return a &lt;code&gt;NAME&lt;/code&gt; token.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;If the first character is a digit, we consume the following digits and return a &lt;code&gt;NUM&lt;/code&gt; token&lt;/li&gt;
&lt;li&gt;If it's rather a &lt;code&gt;"&lt;/code&gt; or &lt;code&gt;'&lt;/code&gt;, we try to read a string or a char, and return a &lt;code&gt;STRING&lt;/code&gt; or &lt;code&gt;CHAR&lt;/code&gt; token. The implementation logic is rather simple. What complicates the matter is the handling of escapes (like &lt;code&gt;\"&lt;/code&gt; or &lt;code&gt;\n&lt;/code&gt; for example)&lt;/li&gt;
&lt;li&gt;If none of the above applies, we try to read a symbol among the list of the known symbols. I'll get back to this part further down in this article, but the trick here is to use an automaton to try and match the longest symbol sequence (one single &lt;code&gt;-&amp;gt;&lt;/code&gt; instead of &lt;code&gt;-&lt;/code&gt; and &lt;code&gt;&amp;gt;&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Finally, and if we've reached the end of the line, we reset the &lt;code&gt;currentLine&lt;/code&gt;. This way, the next call &lt;code&gt;pop&lt;/code&gt; would move to the next line. Else, we throw an exception is we're faced with unrecognised input&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="whitespace-handling_1"&gt;whitespace handling&lt;/h3&gt;
&lt;p&gt;After reading a new line, and before executing the algorithm presented above, the lexer computes the line indentation level (the number of blanks in the beginning of the line). The lexer then compares this value to the previous line's indentation level (which is stored in a field initialised with 0):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;If the 2 values are equal, the lexer returns a &lt;code&gt;NEWLINE&lt;/code&gt; token&lt;/li&gt;
&lt;li&gt;If the current line's indentation level is greater than the previous', first an &lt;code&gt;INDENT&lt;/code&gt; token is returned then a &lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Else, i.e. if the indentation level decreased, a &lt;code&gt;DEINDENT&lt;/code&gt; token is returned first, then a &lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In a previous version of the lexer, I used to return only &lt;code&gt;INDENT&lt;/code&gt; when the indentation level increases, &lt;code&gt;DEINDENT&lt;/code&gt; is it decreases, and &lt;code&gt;NEWLINE&lt;/code&gt; otherwise. this caused me no end of troubles in the next stage (the parser) to correctly handle blocs.
Afterwards, I've stumbled upon &lt;a href="http://stackoverflow.com/questions/232682/how-would-you-go-about-implementing-off-side-rule/946398#946398"&gt;this answer in Stackoverflow&lt;/a&gt; which suggests the 2 tokens trick (&lt;code&gt;INDENT&lt;/code&gt; + &lt;code&gt;NEWLINE&lt;/code&gt;).
This trick made the parsing part much easier and more solid.
I'll explain the why in more details in the upcoming posts.&lt;/p&gt;
&lt;p&gt;Here's a concrete example to clarify things. 
Given this input:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
  &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The lexer is supposed to generate the following tokens:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(a)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;INDENT&lt;/code&gt;: The 2nd line's indentation level increased&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: always produced after line breaks&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(b)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: the indentation level didn't change between the 2nd and 3rd lines&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(c)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DEINDENT&lt;/code&gt;: the indentation level decreased in the 4th line&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(d)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;EOF&lt;/code&gt;: well, the end …&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;There's a catch with the algorithm described above: consider what happens with this input:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The lexer won't generate a &lt;code&gt;DEINDENT&lt;/code&gt; token after producing &lt;code&gt;NAME(b)&lt;/code&gt;, but rather an &lt;code&gt;EOF&lt;/code&gt;because there's no new line after &lt;code&gt;b&lt;/code&gt;.
We could try to work around this in the parser by using &lt;code&gt;EOF&lt;/code&gt; besides &lt;code&gt;DEINDENT&lt;/code&gt; to mark the end of bloc.
Besides complicating the parser logic, there are some more corner cases.
Given this input:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;One single &lt;code&gt;DEINDENT&lt;/code&gt; token would be generated after &lt;code&gt;NAME©&lt;/code&gt;, instead of the 2 we really need to correctly detect the end of the bloc started by &lt;code&gt;c&lt;/code&gt; but also the one started by &lt;code&gt;b&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;To handle these corner cases, and to keep the base lexer implementation simple, I've created &lt;code&gt;StructuredLexer&lt;/code&gt; (&lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/StructuredLexer.java"&gt;source code in Github&lt;/a&gt;), a second &lt;code&gt;Lexer&lt;/code&gt; implementation (See now why there is &lt;code&gt;Lexer&lt;/code&gt; interface ?).&lt;/p&gt;
&lt;p&gt;This implementation uses the decorator pattern &lt;em&gt;(I'm a professional Java developer, I know my design patterns)&lt;/em&gt;.
It delegates to &lt;code&gt;BaseLexer&lt;/code&gt; to produce tokens from the source text, while at the same time extending it to do the following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It maintains the current indentation level in a field. This level is computed by dividing the number of blanks in the beginning of a line by an indentation unit. This unit was arbitrarily fixed to be &lt;strong&gt;2 spaces&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;In &lt;code&gt;pop&lt;/code&gt;, when the base lexer returns &lt;code&gt;INDENT&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;Ensure that the number of spaces in that &lt;code&gt;INDENT&lt;/code&gt; token is a multiple of the unit (2). Throw an exception if that's not the case&lt;/li&gt;
&lt;li&gt;Also ensure that the indentation level can only increase by a step of &lt;strong&gt;1&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Update the field holding the current indentation level&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Still in &lt;code&gt;pop&lt;/code&gt;, if the base lexer returns &lt;code&gt;DEINDENT&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;As with &lt;code&gt;INDENT&lt;/code&gt;, ensure that the number of spaces is multiple of the unit. If all is well, compute the indentation level&lt;/li&gt;
&lt;li&gt;If this level decreased by &lt;strong&gt;more than 1&lt;/strong&gt; (like in the previous example), produce as many virtual &lt;code&gt;DEINDENT&lt;/code&gt; tokens as necessary&lt;/li&gt;
&lt;li&gt;Update the current indentation level&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Finally, if the base lexer returns an &lt;code&gt;EOF&lt;/code&gt;, and as with &lt;code&gt;DEINDENT&lt;/code&gt;, produce as many virtual &lt;code&gt;DEINDENT&lt;/code&gt; tokens as necessary, until the indentation level reaches &lt;strong&gt;0&lt;/strong&gt;, before returning an &lt;code&gt;EOF&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thus, with that same problematic example:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The structured lexer would produce an extra &lt;code&gt;DEINDENT&lt;/code&gt; token (besides the one produced by the base lexer) between &lt;code&gt;c&lt;/code&gt; and &lt;code&gt;d&lt;/code&gt;. The next stage (i.e. the parser) would correctly detect that 2 blocs end after &lt;code&gt;c&lt;/code&gt; and that &lt;code&gt;d&lt;/code&gt; has the same indentation level as &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="comments-handling"&gt;Comments handling&lt;/h3&gt;
&lt;p&gt;Litil's syntax supports Haskell-like one line comments prefixed by &lt;code&gt;--&lt;/code&gt;.
I've chosen to handle them in the lexer stage by swallowing them, i.e. not producing any tokens for them.
But I could have chosen to produce &lt;code&gt;COMMENT&lt;/code&gt; tokens and ignore them &lt;em&gt;(or not)&lt;/em&gt; in the parser stage.&lt;/p&gt;
&lt;p&gt;Comments are handled in 2 locations in the lexer:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;When a line is read from the source text. If it starts with &lt;code&gt;--&lt;/code&gt;, the whole line is discarded&lt;/li&gt;
&lt;li&gt;If the symbol matching logic matches a &lt;code&gt;--&lt;/code&gt;, it and the line remainder are discarded&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Examples:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c"&gt;-- compute max x y … NOT !&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- It is a well known fact that x always wins ! &lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="symbols-handling"&gt;Symbols handling&lt;/h3&gt;
&lt;p&gt;Except maybe for the indentation handling, this is, &lt;em&gt;imho&lt;/em&gt;, the most interesting part of the lexer.&lt;/p&gt;
&lt;p&gt;Before proceeding, I'll introduce &lt;a href="http://fr.wikipedia.org/wiki/Automate_fini"&gt;finite automata&lt;/a&gt;, a most important topic in compilation and languages theory.&lt;/p&gt;
&lt;p&gt;Here's a metaphor that I use to explain finite automata:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;You wake up to find yourself in an empty room with many closed doors.
You're holding a strange device in your hand.
It resembles a credit card, except that it has a display on it with showing the word &lt;code&gt;The cake&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;You check the first door, and you notice that it is marked with the word &lt;code&gt;The reaper&lt;/code&gt;.
Also, there is a slot to its right, and it looks like the card you hold would fit in there.&lt;/p&gt;
&lt;p&gt;You're smart enough to shy away from a door marked with &lt;code&gt;The reaper&lt;/code&gt;, so you go check on the second door.&lt;/p&gt;
&lt;p&gt;This one is also marked with a word, though it is &lt;code&gt;An Angel&lt;/code&gt; this time, and it also has a slot to its right.&lt;/p&gt;
&lt;p&gt;You must have guessed it by now, but you try it anyway, and insert the card in the slot.
A red led blinks, it buzzes, and it ejects your card.&lt;/p&gt;
&lt;p&gt;So you go to the next door, which is marked with &lt;code&gt;The cake&lt;/code&gt;, the same as what the display in your card shows.
You like cakes.
Everybody does.
So you insert it in the slot: a green led blinks and the door opens.
You go through it, and it closes behind you.&lt;/p&gt;
&lt;p&gt;You turn around and notice that there is no slot, no word, nothing.
You can't go back through that door.&lt;/p&gt;
&lt;p&gt;You also notice that the word shown on your card changed.
It now shows &lt;code&gt;is&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;By now, you've understood the rules of the game: you can only go through a door if is marked with the word you have on your card.
Once you go through a door, you can't go back.
If the word on your card doesn't match any of the room's doors, you're dead meat.
You'll be stuck there till you die of thirst, hunger or boredom.&lt;/p&gt;
&lt;p&gt;So you look around for a door marked with &lt;code&gt;is&lt;/code&gt;, and luckily for you, there is one to the left, so you go through it.&lt;/p&gt;
&lt;p&gt;The word on your card changes to &lt;code&gt;a lie&lt;/code&gt;, and it must be your lucky day, the fifth door has that same word.&lt;/p&gt;
&lt;p&gt;After going through that door, something strange happens.
You card display starts flickering before going out.
Game over, no more words for you sir, and so no more doors: you're stuck.&lt;/p&gt;
&lt;p&gt;But it could be worse, you could have ended up in a room with no doors, if that's a consolation to you.&lt;/p&gt;
&lt;p&gt;But what you don't know is that some lucky souls could end up in special rooms.
These special rooms have huge refrigerators full of pizza and beers, an XBox, F1 2012, Skyrim, a computer with a working internet connection, and everything you could wish for.
You won sir. Achievement unlocked.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Now, back to finite automata. A finite automaton is a set of states and transitions.
States are the rooms, and transitions are the doors.
given a sequence of inputs (the words on your card), it consumes them one at a time while following the adequate transition (and hence moving from one state to another) until there are no more inputs (you card display flickers and dies), or it reaches a state with no possible transitions (room without doors), or it reaches a final state (special room, pizza and beers).&lt;/p&gt;
&lt;p&gt;Here's what an automaton looks like:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa0.png" src="/images/graphviz/litil-lexer-dfa0.png" /&gt;&lt;/p&gt;
&lt;p&gt;This automaton is composed of:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;An initial state &lt;code&gt;S0&lt;/code&gt; (the first room you wake up in). That's the unique entry point.&lt;/li&gt;
&lt;li&gt;3 other states (rooms): &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt; and &lt;code&gt;C&lt;/code&gt;.
  Notice the double edges around them.
  This indicates that they are final states (pizza, beers)&lt;/li&gt;
&lt;li&gt;Transitions (doors) between these states marked with symbols (words on the doors)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let's apply this automaton to the input &lt;code&gt;-&amp;gt;a&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;We start from the entry point &lt;code&gt;S0&lt;/code&gt; and with the first symbol &lt;code&gt;-&lt;/code&gt;.
We try to find a transition annotated with that symbol.
We're in luck: the transition from &lt;code&gt;S0&lt;/code&gt; to &lt;code&gt;A&lt;/code&gt; is just what the doctor ordered.
We consume the first symbol &lt;code&gt;-&lt;/code&gt;, which leaves us with &lt;code&gt;&amp;gt;&lt;/code&gt; as the current symbol, and we follow that transition to reach &lt;code&gt;A&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;A&lt;/code&gt; is a terminal state, so we could just interrupt the operation and say that we successfully matched &lt;code&gt;-&lt;/code&gt; &lt;em&gt;(since it led us to pizza and beers, or more formally to a final state)&lt;/em&gt;.
But in Litil, and in most other languages, we try to match the longest possible sequence.
Otherwise, it wouldn't be possible to have &lt;code&gt;&amp;lt;&lt;/code&gt; and &lt;code&gt;&amp;lt;=&lt;/code&gt; for example.&lt;/p&gt;
&lt;p&gt;So we don't give up just yet. &lt;code&gt;A&lt;/code&gt; has a transition annotated with &lt;code&gt;&amp;gt;&lt;/code&gt;.
Following this transition leads us to &lt;code&gt;C&lt;/code&gt;.
The latter is also final, by by the same token, we continue trying to match a longer sequence.
Except that there is no transition out of &lt;code&gt;C&lt;/code&gt; annotated with the next character &lt;code&gt;a&lt;/code&gt;.
So we stop the flow at &lt;code&gt;C&lt;/code&gt; and say that we matched the string &lt;code&gt;-&amp;gt;&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The algorithm described above, which is the one currently used by Litil, is rather simplistic and is far from the state of the art.
For one thing, it doesn't support backtracking:
in its quest to match the longest sequence, it might get stuck in between states.&lt;/p&gt;
&lt;p&gt;Here's an example:
Given this automaton, which is supposed to recognise the sequences &lt;code&gt;-&lt;/code&gt; and &lt;code&gt;--&amp;gt;&lt;/code&gt;, and with the input &lt;code&gt;--a&lt;/code&gt;, we end up with a no match, whereas it was supposed to recognise 2 times &lt;code&gt;-&lt;/code&gt; (the why is left as an exercise to the reader):&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa1.png" src="/images/graphviz/litil-lexer-dfa1.png" /&gt;&lt;/p&gt;
&lt;p&gt;In its current version, the operators or symbols recognised by Litil are: &lt;code&gt;-&amp;gt;&lt;/code&gt;, &lt;code&gt;.&lt;/code&gt;, &lt;code&gt;+&lt;/code&gt;, &lt;code&gt;-&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt;, &lt;code&gt;/&lt;/code&gt;, &lt;code&gt;(&lt;/code&gt;, &lt;code&gt;)&lt;/code&gt;, &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;%&lt;/code&gt;, &lt;code&gt;&amp;lt;&lt;/code&gt;, &lt;code&gt;&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;=&lt;/code&gt;, &lt;code&gt;&amp;gt;=&lt;/code&gt;, &lt;code&gt;:&lt;/code&gt;, &lt;code&gt;,&lt;/code&gt;, &lt;code&gt;[&lt;/code&gt;, &lt;code&gt;]&lt;/code&gt;, &lt;code&gt;|&lt;/code&gt;, &lt;code&gt;_&lt;/code&gt;, &lt;code&gt;=&amp;gt;&lt;/code&gt;, &lt;code&gt;\\&lt;/code&gt;, &lt;code&gt;--&lt;/code&gt;, &lt;code&gt;::&lt;/code&gt;, &lt;code&gt;{&lt;/code&gt; and &lt;code&gt;}&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Now, just for the fun of it &lt;em&gt;(I like big graphs, they make this blog post look formal and everything)&lt;/em&gt;, here's the automaton that recognises them:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa-litil.png" src="/images/graphviz/litil-lexer-dfa-litil.png" /&gt;&lt;/p&gt;
&lt;p&gt;Now, back to how Litil handles symbols.
I use automata that's how.
In Litil, the symbols automaton is built at runtime from a list of symbols.
Also, Litil's automaton implementation doesn't handle backtracking: remember being too eager and the getting stuck in between states we've seen above ?.
Which is fine, as that problem doesn't occur unless our language has 3-letters long operators.
Litil is no Scala, so we won't be running into that problem.&lt;/p&gt;
&lt;p&gt;On the other side, the automaton based operators recognition implementation is barely 50 lines long.
We're talking about Java here, so putting aside all the clutter &lt;em&gt;(imports, constructor, getters, toString, etc.)&lt;/em&gt;, the essence of the algorithm is barely 12 lines long.
I'm quite proud of it actually.
Here's the &lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/LexerStage.java"&gt;source code&lt;/a&gt; if you're interested.&lt;/p&gt;
&lt;h3 id="lookahead"&gt;Lookahead&lt;/h3&gt;
&lt;p&gt;Another useful feature to have in a lexer is the ability to look ahead,
i.e. being able to peek into the upcoming tokens without actually consuming them (which would have been the case with &lt;code&gt;pop&lt;/code&gt;).
I'll get back to the why in the parser post.&lt;/p&gt;
&lt;p&gt;As with &lt;code&gt;StructuredLexer&lt;/code&gt;, I've implemented this feature in &lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/LookaheadLexerWrapper.java"&gt;&lt;code&gt;LookaheadLexerWrapper&lt;/code&gt;&lt;/a&gt; as a decorator.&lt;/p&gt;
&lt;p&gt;The logic behind is rather simple:
This class adds a new method, &lt;code&gt;lookahead&lt;/code&gt; which takes a number as a parameter that indicate how far ahead we'd like to peek (e.g. 1 for the next token)
When this method is called, it calls &lt;code&gt;pop&lt;/code&gt; on it's delegate lexer as many times as necessary and stores the returned tokens in a list.
When &lt;code&gt;pop&lt;/code&gt; is called, and before delegating to the concrete lexer, this class checks if the list of tokens is empty or not.
If it isn't, it returns the first token of the list and removes it.
This way, the next call to &lt;code&gt;pop&lt;/code&gt; would return the next token.
When this list is empty, it delegates to the concrete lexer.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;In this post, I showcased some of the techniques I used to implement the lexing part of Litil.
They might not be the proper way of doing it, nor the fastest.
They're rather easy to code and to extend, and hava a rather reasonable memory footprint:
we're not reading the whole file in a string as many of the examples you find in the internets do.&lt;/p&gt;
&lt;p&gt;This part is also the least &lt;em&gt;fun&lt;/em&gt; to code.
I was tempted to use a tool (like SableCC) to generate its code.
But I decided that I'll stay true to my original goal of hand rolling everything &lt;em&gt;(except for the parts that I won't)&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Now that this part is over, the real fun is ahead of us: &lt;a href="http://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"&gt;the parser&lt;/a&gt; and the evaluator.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>The Litil chronicle - Chapter 1, The inception of a programming language</title>
   <link href="https://jawher.me/2012/10/20/creation-langage-programmation-litil-1/"/>
   <published>2012-10-20T00:00:00+02:00Z</published>
   <updated>2012-10-20T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/10/20/creation-langage-programmation-litil-1/</id>
   <content type="html">&lt;p&gt;I've always been fascinated by compilers and programming languages.
I was fascinated by the fact that I could take control of the computer to make it do what I want.
But I was even more impressed with the compilers themselves.
The fact that they could &lt;em&gt;understand&lt;/em&gt; and execute what I've fed them was comparable to magic to me at that time.&lt;/p&gt;
&lt;p&gt;I still remember the first time I tried to write a parser.
I was still in school back then.
I didn't even know that they were named parsers.
I've just started learning Pascal on my commodore.
But I absolutely wanted to write &lt;em&gt;something&lt;/em&gt; that could evaluate arithmetic expressions input as a string.
I didn't have any theoretical background on the subject.
Hell, I didn't even have an internet connection.
I don't remember if it had worked out or not.
But I still remember some of the hacks I've come up with, like the one to handle prioritising parenthesised expressions.
The solution I came up with was to look for the first closing parenthesis. Then go back to find the first opening parenthesis.
This way I could isolate the part to evaluate first.
Then repeat the same process till there were no more parentheses.&lt;/p&gt;
&lt;p&gt;I've learned a lot since then.
I've taken 2 courses on the subject in the college.
I've also read lots of articles and research papers, and keep visiting sites like &lt;a href="http://lambda-the-ultimate.org/"&gt;LtU&lt;/a&gt;.
I was lucky enough to put modest &lt;em&gt;modest&lt;/em&gt; knowledge into practice at work, where I had to write DSL parsers in 2 separate projects.&lt;/p&gt;
&lt;p&gt;Recently, I've decided that like everybody else seems to be doing, I'd create my own programming language, aka JavaSlayer™.
To make the task more challenging, I chose not to use a parser generator (à la ANTLR) like I usually would, but rather hand roll everything.
Like in the old days, without using any frameworks nor libraries.
My goal was to learn for myself how compilers internals work.
I've also been influenced by &lt;a href="http://en.wikipedia.org/wiki/Niklaus_Wirth"&gt;Niklaus Wirth&lt;/a&gt; (the man behind Pascal, Modula and Oberon languages), who has hand written all of his compilers.&lt;/p&gt;
&lt;p&gt;And like that, after 2 or 3 months of occasional work, I ended up with a working parser and evaluator of a half-decent language.
Say hello to &lt;code&gt;Litil&lt;/code&gt; !&lt;/p&gt;
&lt;p&gt;Since I'm not in the mood for blogging about "How to create a &lt;code&gt;&amp;lt;web framework #1&amp;gt;&lt;/code&gt; app with &lt;code&gt;&amp;lt;orm #2&amp;gt;&lt;/code&gt; and &lt;code&gt;&amp;lt;js framework #3&amp;gt;&lt;/code&gt; using &lt;code&gt;&amp;lt;an IDE&amp;gt;&lt;/code&gt; to handle &lt;code&gt;&amp;lt;hip subject of the day&amp;gt;&lt;/code&gt; and host it on &lt;code&gt;&amp;lt;PaaS|SaaS|IaaS of the day&amp;gt;&lt;/code&gt;", I'm going instead to start a series of posts covering a more exotic and interesting subject: parsing.&lt;/p&gt;
&lt;p&gt;You need to be aware though that there are lots of tools to &lt;strong&gt;easily&lt;/strong&gt; generate a working parser, like SableCC, ANTLR, JavaCC, lex/yacc, flex/bison, etc.
I've already used most of these, and that's not my goal here.
As I said earlier, I wanted to understand how things really worked.&lt;/p&gt;
&lt;h1 id="litil"&gt;Litil&lt;/h1&gt;
&lt;p&gt;This is the first post of a series where I'll talk about the different techniques I've used to create a parser and un evaluator for the Litil™  programming language in Java.
However, please keep the following in mind:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;This is still &lt;em&gt;work in progress&lt;/em&gt;, but I'm honouring the "release early, release often" mantra. I'm also still learning a lot while I'm working on it.
  So inevitably, I might say dumb or plain wrong things.&lt;/li&gt;
&lt;li&gt;I no Matt Might™.
  I've had some courses about this subject, and I've read a couple research papers, but that doesn't make me an expert on the subject.&lt;/li&gt;
&lt;li&gt;I've started working on compiling to byte code.
  I might talk about it in a future post, but it's still too early, so I'm not promising anything.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here's a quick overview of the Litil language&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A functional language heavily inspired by ML, (O)Caml, Haskell and &lt;a href="http://roy.brianmckenna.org/"&gt;Roy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Full &lt;a href="http://en.wikipedia.org/wiki/Hindley-Milner_type_inference"&gt;Hindley Milner&lt;/a&gt; type inference&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Off-side_rule"&gt;Python like use of indentation to define blocs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Supported types: numerics, strings, characters and boolean, &lt;a href="http://en.wikipedia.org/wiki/Tuple"&gt;tuples&lt;/a&gt;, &lt;a href="http://en.wikipedia.org/wiki/Record_(computer_science)"&gt;records&lt;/a&gt; and &lt;a href="http://en.wikipedia.org/wiki/Algebraic_data_type"&gt;ADTs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Pattern_matching"&gt;pattern matching&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Currying"&gt;curried functions&lt;/a&gt; by default&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Closure_(computer_science)"&gt;closures&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;exceptions (try/catch/throw)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now, some teaser examples to give you a feel for the language we'll be creating.&lt;/p&gt;
&lt;h2 id="assignment-expressions-functions"&gt;assignment, expressions &amp;amp; functions&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    &lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="k"&gt;else&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;f5&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="tuples-records"&gt;tuples &amp;amp; records&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;a&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;person&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lpe&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="destructuring"&gt;destructuring&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;d&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;true&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;test&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="sc"&gt;&amp;#39;c&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bool&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="algebraic-data-types"&gt;algebraic data types&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;Option&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Some&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;None&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Some&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;thing&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Null&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="pattern-matching"&gt;pattern matching&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;
    &lt;span class="bp"&gt;[]&lt;/span&gt;     &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;

&lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="partial-application"&gt;partial application&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;inc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;three&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;inc&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="closures-higher-order-functions"&gt;closures &amp;amp; higher-order functions&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;xs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="n"&gt;xs&lt;/span&gt;
    &lt;span class="bp"&gt;[]&lt;/span&gt;     &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;double&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;

&lt;span class="c"&gt;-- pass a function by name&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;double&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;

&lt;span class="c"&gt;-- or simply a lambda&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="o"&gt;(\&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;\&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- f captures the lexical value of a, i.e. 4&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h1 id="the-subjects-thatll-be-covered"&gt;The subjects that'll be covered&lt;/h1&gt;
&lt;p&gt;To create a programming language, and Litil in particular, we'll need to cover these topics:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2012/10/21/creation-langage-programmation-litil-2-lexer/"&gt;Lexing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Parsing&lt;ul&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"&gt;BNF/EBNF&lt;/a&gt; &lt;/li&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"&gt;Recursive descent parsing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Pratt_parser"&gt;Pratt parsing&lt;/a&gt; pour les expressions&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Type inference and checking (Hindley Milner)&lt;/li&gt;
&lt;li&gt;Evaluation&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These subjects won't necessarily be covered in this order.
What we'll be doing instead is gradually implementing the different parts from the parsing to the evaluation.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>Moving from Jekyll to Pelican</title>
   <link href="https://jawher.me/2012/10/19/moving-from-jekyll-to-pelican/"/>
   <published>2012-10-19T00:00:00+02:00Z</published>
   <updated>2012-10-19T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/10/19/moving-from-jekyll-to-pelican/</id>
   <content type="html">&lt;p&gt;I just finished migrating this blog from Jekyll to &lt;a href="http://blog.getpelican.com/"&gt;Pelican&lt;/a&gt;.
I've stumbled upon (no pun intended) Pelican a couple of months ago on &lt;code&gt;/r/python&lt;/code&gt;.
I thought it was cool but that was it.
But recently, I've decided I'd be translating some Litil posts from French to English in the next weeks.
I've checked, and there wasn't any straitforward way to handle translation in Jekyll.
I could create the translations as new posts, but that would just be messy.
That's when I remembered Pelican, and the fact that it handled translations out of the box.
The fact that Pelican is written in Python didn't hurt, as I'm much more comfortable with Python than with Ruby.&lt;/p&gt;
&lt;p&gt;Since I was stranded at home for a couple of days after getting injured in a football game, I've decided to give Pelican a try.
Here's how I've proceeded:&lt;/p&gt;
&lt;h2 id="content"&gt;Content&lt;/h2&gt;
&lt;p&gt;I started with the content.
Luckily &lt;em&gt;(or rather unfortunately ?)&lt;/em&gt;, I only had 3 posts in this new blog.
I'm using markdown as my posts format. I had to modify the metadata from the Jekyll format to Pelican's.
All I had to do was to delete the lines (&lt;code&gt;-&lt;/code&gt;) surrounding the metadata and change a couple of key names.
So it went from:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;---
date: &amp;#39;2012-06-27 21:45:00&amp;#39;
layout: post
slug: creation-langage-programmation-litil-2-lexer
published: true
title: Le conte de Litil - Chapitre 2, Le dépeceur du texte, aka Lexer
---
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;To:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2012&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;
&lt;span class="n"&gt;slug&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;creation&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;langage&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;programmation&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;litil&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;lexer&lt;/span&gt;
&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Le&lt;/span&gt; &lt;span class="n"&gt;conte&lt;/span&gt; &lt;span class="n"&gt;de&lt;/span&gt; &lt;span class="n"&gt;Litil&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;Chapitre&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Le&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="err"&gt;é&lt;/span&gt;&lt;span class="n"&gt;peceur&lt;/span&gt; &lt;span class="n"&gt;du&lt;/span&gt; &lt;span class="n"&gt;texte&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;aka&lt;/span&gt; &lt;span class="n"&gt;Lexer&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I also had to change the code highlighting parts. Whereas in Jekyll it was done using a Liquid block:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;{% highlight java}
public static final int answer = 42;
{% endhighlight %}
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In Pelican you use fenced code blocks à-la Github instead:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;```java
public static final int answer = 42;
```
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I was left with one last problem regarding the content: 
with Jekyll, I have developed a Liquid extension that generates an image from a graph defintion (in the dot language) and inserted it into the content (I used it in &lt;a href="http://jawher.me/2012/06/27/creation-langage-programmation-litil-2-lexer/"&gt;this post&lt;/a&gt; for example).
Here's how the source looked like:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;{% graphviz litil-lexer-dfa0.png %}
digraph G {
  rankdir=LR
  S0 -&amp;gt; A [label=&amp;quot;-&amp;quot;]
  A -&amp;gt; B [label=&amp;quot;-&amp;quot;]
  A -&amp;gt; C [label=&amp;quot;&amp;gt;&amp;quot;]

  A [peripheries=2]
  B [peripheries=2]
  C [peripheries=2]
}
{% endgraphviz %}
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;So I had to come up with something similar in Pelican.
Even worse, I couldn't use something off the shelf as it had to work tightly with the site generator (e.g. place the generated images in a specific directory).
A quick googling didn't turn anything useful, so I came up with &lt;a href="https://github.com/jawher/markdown-dot"&gt;my own version&lt;/a&gt;.
I've done it differently that with Jekyll though.
This new version is a markdown extension rather than a custom tag (of the templating system), as in Pelican, and unless I'm mistaken, posts do not get processed by the templating engine.
Here's how it looks like now &lt;em&gt;(ugly syntax, I know)&lt;/em&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;{% dot litil-lexer-dfa0.png
digraph G {
  rankdir=LR
  S0 -&amp;gt; A [label=&amp;quot;-&amp;quot;]
  A -&amp;gt; B [label=&amp;quot;-&amp;quot;]
  A -&amp;gt; C [label=&amp;quot;&amp;gt;&amp;quot;]

  A [peripheries=2]
  B [peripheries=2]
  C [peripheries=2]
}
%}
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="settings"&gt;Settings&lt;/h2&gt;
&lt;p&gt;I've then started tweaking the settings.
The existing urls had to continue working.
It's as simple as that.
Luckily, that wasn't hard at all using these 2 lines:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;ARTICLE_URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;{date:%Y}/{date:%m}/{date:&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s1"&gt;}/{slug}/&amp;#39;&lt;/span&gt;
&lt;span class="n"&gt;ARTICLE_SAVE_AS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;{date:%Y}/{date:%m}/{date:&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s1"&gt;}/{slug}/index.html&amp;#39;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I also had to tweak the feed generation part:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;FEED_ATOM&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;atom.xml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;CATEGORY_FEED_ATOM&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;span class="n"&gt;FEED_DOMAIN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SITEURL&lt;/span&gt;
&lt;span class="n"&gt;TRANSLATION_FEED&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I couldn't keep the same urls for everything though, e.g. the css files and images.
Pelican has some hard-coded paths for this kind of assets.
Hopefully it'll get more flexible in the future.&lt;/p&gt;
&lt;h2 id="theming"&gt;Theming&lt;/h2&gt;
&lt;p&gt;The theming part was the trickiest.
Theming is handled differently between Jekyll and Pelican.
Jekyll's approach is more &lt;em&gt;abstract&lt;/em&gt;, e.g. you used metadata in the template files to indicate inheritance, and special vars to inject the content.
Pelican's way of doing things is closer to the metal:
it used Jinja2 as a templating engine, and that was it.&lt;/p&gt;
&lt;p&gt;For example, the article template went from:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;---
layout: default
---

&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;nav&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;← all posts&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="ni"&gt;&amp;amp;mdash;&lt;/span&gt;
  &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/my-projects&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;my projects&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
   &lt;span class="ni"&gt;&amp;amp;mdash;&lt;/span&gt;
  &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;https://twitter.com/#!/jawher&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;twitter&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
   &lt;span class="ni"&gt;&amp;amp;mdash;&lt;/span&gt;
  &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;https://github.com/jawher&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;github&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;article&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;post&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;meta&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;day&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ page.date | date: &amp;quot;%d&amp;quot; }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;month&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ page.date | date: &amp;quot;%b&amp;quot; }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;year&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ page.date | date: &amp;quot;%Y&amp;quot; }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="c"&gt;&amp;lt;!--&amp;lt;div&amp;gt;&amp;lt;a href=&amp;quot;{{ page.url }}#disqus_thread&amp;quot; data-disqus-identifier=&amp;quot;{{ page.url }}&amp;quot;&amp;gt;Count&amp;lt;/a&amp;gt;&amp;lt;/div&amp;gt;--&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;

{{ content }}
&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;article&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;To:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;{% extends &amp;quot;base.html&amp;quot; %}
{% block extra_links %}
  &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;li&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;← home&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;a&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="ni"&gt;&amp;amp;#9733;&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;li&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
{% endblock %}
{% block content %}

&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;article&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;post&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;meta&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;day&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ article.date.strftime(&amp;quot;%d&amp;quot;) }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;month&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ article.date.strftime(&amp;quot;%b&amp;quot;) }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;year&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ article.date.strftime(&amp;quot;%Y&amp;quot;) }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;span&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;h1&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{article.title}}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;h1&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
{{ article.content }}
&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;article&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I have to admit that I like Pelican's approach much mor than Jekyll's.
While it was more low level, I found it to be much more intuitive.&lt;/p&gt;
&lt;p&gt;And while I was at it, I've also tweaked the design a bit.
Mostly changing the post and page templates design to match the index page.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Pelican's documentation, while rich, is often imprecise and lacks details.
For example, it'll often tell you to place static assets in a images directory so that they'll be automatically copied to the output.
What it doesn't tell you though is that this directory must be placed in the content directory, and not in the site root.&lt;/p&gt;
&lt;p&gt;But all in all, it wasn't such a bad experience, and went rather smoothly. I've grown to love the &lt;code&gt;make devserver&lt;/code&gt; command, and the generation speed (light years ahead of Jekyll).&lt;/p&gt;
&lt;p&gt;Now that the migration is done, I have no more excuses for not publishing more often. See you in the next post.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>Le conte de Litil - Chapitre 2, Le dépeceur du texte, aka Lexer</title>
   <link href="https://jawher.me/2012/06/27/creation-langage-programmation-litil-2-lexer/"/>
   <published>2012-06-27T00:00:00+02:00Z</published>
   <updated>2012-06-27T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/06/27/creation-langage-programmation-litil-2-lexer/</id>
   <content type="html">&lt;p&gt;Dans ce deuxième post du conte de Litil, je vais parler de la phase de lexing. C'est généralement la première étape dans la construction d'un compilateur (ou évaluateur) d'un langage donné. Cette phase sert à transformer le texte du code source (séquence de caractères) vers une séquence de &lt;code&gt;tokens&lt;/code&gt;, qui seront consommés par le parseur à l'étape suivante.&lt;/p&gt;
&lt;p&gt;Les règles suivant lesquelles le texte est découpé en tokens varient d'un langage à un autre, mais en général (avec les langages conventionnels de la famille Algol par exemple), on peut définir les familles de tokens suivants:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;symboles: les opérateurs (&lt;code&gt;+&lt;/code&gt;, &lt;code&gt;-&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt;, &lt;code&gt;/&lt;/code&gt;, etc.), les signes de ponctuation (&lt;code&gt;,&lt;/code&gt;, &lt;code&gt;;&lt;/code&gt;, &lt;code&gt;(&lt;/code&gt;, &lt;code&gt;[&lt;/code&gt;, etc.).&lt;/li&gt;
&lt;li&gt;les littéraux: nombres (&lt;code&gt;42&lt;/code&gt;, &lt;code&gt;3.14&lt;/code&gt;, etc.), chaînes de caractères (&lt;code&gt;"kthxbye"&lt;/code&gt;), booléens (&lt;code&gt;true&lt;/code&gt;, &lt;code&gt;false&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;les mots clés: &lt;code&gt;if&lt;/code&gt;, &lt;code&gt;while&lt;/code&gt;, etc.&lt;/li&gt;
&lt;li&gt;les identifiants: cela dépend du langage, mais généralement un identifiant commence par une lettre ou un tiret-bas (&lt;code&gt;_&lt;/code&gt;), puis optionnellement une séquence de lettres, chiffres et quelques autres symboles.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On peut donc voir un token comme un couple &lt;strong&gt;type&lt;/strong&gt; (symbole, mot clé, littéral, etc.) et &lt;strong&gt;valeur&lt;/strong&gt; (le contenu textuel de ce token). Optionnellement, on peut enrichir un token pour qu'il contienne aussi le numéro de la ligne et de la colonne où il apparait dans le texte source, ce qui peut s'avérer utile pour signaler l'emplacement d'une erreur.&lt;/p&gt;
&lt;p&gt;Enfin, le lexer se charge de cacher ou ignorer le contenu inutile dans le fichier source, comme les blancs, retours à la ligne et autres.&lt;/p&gt;
&lt;h2 id="gestion-des-blancs"&gt;Gestion des blancs&lt;/h2&gt;
&lt;p&gt;Dans le langage que nous allons implémenter, et à l'encontre de la majorité des autres langages, les blancs sont importants car il servent à démarquer le début et la fin des blocs comme dans Python (et Haskell) par exemple. Le début d'un bloc est signalé par une augmentation du niveau de l'indentation tandis que sa fin est marquée par une dé-indentation.&lt;/p&gt;
&lt;p&gt;Exemple:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
  &lt;span class="n"&gt;print&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;L'équivalent Java serait:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Bien que les corps des branches &lt;code&gt;then&lt;/code&gt; et &lt;code&gt;else&lt;/code&gt; du code java sont bien indentés, cette indentation est optionnelle et est carrément ignorée par le lexer. Ce sont les accolades ouvrantes et fermantes qui démarquent le début et la fin d'un bloc. On aurait pu obtenir le même résultat avec:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;  
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;ou encore:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;print&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Ohai&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;);}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Exception&lt;/span&gt;&lt;span class="o"&gt;();}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;La lisibilité du code en souffre, mais cela ne change rien du point de vue du compilateur. Ce n'est pas le cas avec Litil, où comme dit plus haut, l'indentation du code n'est pas optionnelle car elle sert à définir sa structure. De plus, là où dans Java on utilisait le &lt;code&gt;;&lt;/code&gt; pour séparer les instructions d'un même bloc, Litil utilise plutôt les retours à la ligne. Les &lt;code&gt;;&lt;/code&gt; ne sont pas optionnels, ils ne sont pas reconnues. Mon but était de s'inspirer des langages comme Haskell et Python pour créer une syntaxe épurée avec un minimum de bruit et de décorum autour du code utile. Je reveniendrai la dessus dans le&lt;em&gt;(s)&lt;/em&gt; post&lt;em&gt;(s)&lt;/em&gt; à venir quand je vais détailler la syntaxe de Litil, mais pour vous donner quelques exemples:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Pas d'accolades pour délimiter les blocs&lt;/li&gt;
&lt;li&gt;Pas de &lt;code&gt;;&lt;/code&gt; pour séparer les instructions&lt;/li&gt;
&lt;li&gt;Pas de parenthèse pour les arguments d'une fonction: &lt;code&gt;sin 5&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Pas de parenthèse pour les conditions des &lt;code&gt;if&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Donc, pour résumer, le lexer que nous allons développer ne va pas complètement ignorer les blancs. Plus précisément, le lexer devra produire des tokens pour signaler les retours à la ligne (pour indiquer la fin d'une instruction) et les espaces (ou leur absence) en début de lignes (pour indiquer le début ou la fin d'un bloc).&lt;/p&gt;
&lt;h2 id="implementation"&gt;Implémentation&lt;/h2&gt;
&lt;p&gt;Pour commencer, voici la définition d'un Token:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Token&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;enum&lt;/span&gt; &lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;NEWLINE&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;INDENT&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DEINDENT&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NUM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;STRING&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CHAR&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SYM&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;BOOL&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYWORD&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;EOF&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;Token&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Type&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;row&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;col&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Un token est composé de:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;type&lt;/code&gt;: le type du token:&lt;ul&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: pour indiquer un retour à la ligne&lt;/li&gt;
&lt;li&gt;&lt;code&gt;INDENT&lt;/code&gt;: pour indiquer que le niveau d'indentation a augmenté par rapport à la ligne précédente, et donc qu'on entre dans un nouveau bloc&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DEINDENT&lt;/code&gt;: pour indiquer que le niveau d'indentation a diminué par rapport à a ligne précédente, et donc qu'on sort d'un bloc&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME&lt;/code&gt;: une clé pour indiquer qu'il s'agit d'un identifiant&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NUM&lt;/code&gt;, &lt;code&gt;STRING&lt;/code&gt;, &lt;code&gt;CHAR&lt;/code&gt;, &lt;code&gt;BOOL&lt;/code&gt;: une clé pour indiquer qu'il s'agit d'un littéral&lt;/li&gt;
&lt;li&gt;&lt;code&gt;KEYWORD&lt;/code&gt;: une clé pour indiquer qu'il s'agit d'un mot clé&lt;/li&gt;
&lt;li&gt;&lt;code&gt;EOF&lt;/code&gt;: produit quand on a atteint la fin du texte source&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;text&lt;/code&gt;: une chaîne qui contient le texte correspondant à ce token&lt;/li&gt;
&lt;li&gt;&lt;code&gt;row&lt;/code&gt; et &lt;code&gt;col&lt;/code&gt;: indique la position du token dans le texte source&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Voici maintenant l'interface qui décrit le lexer:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;Lexer&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Token&lt;/span&gt; &lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="kd"&gt;throws&lt;/span&gt; &lt;span class="n"&gt;LexingException&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;String&lt;/span&gt; &lt;span class="nf"&gt;getCurrentLine&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Cette interface définit les 2 méthodes suivantes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;pop&lt;/code&gt;: retourne le token suivant&lt;/li&gt;
&lt;li&gt;&lt;code&gt;getCurrentLine&lt;/code&gt;: retourne la ligne courante dans le texte source&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Notez l'absence d'une méthode qui indique la présence ou pas d'un token suivant. En effet, quand le lexer atteint la fin du fichier, il se place en un mode où tous les appels à &lt;code&gt;pop&lt;/code&gt; retournent un token de type &lt;code&gt;EOF&lt;/code&gt;. J'ai donc estimé inutile d'alourdir l'interface pour ajouter une méthode à la &lt;code&gt;hasNext&lt;/code&gt; d'un itérateur par exemple.&lt;/p&gt;
&lt;p&gt;Notez aussi que si j'ai défini une interface pour le lexer, c'est parce qu'il y aurait plusieurs implémentations que nous allons voir par la suite.&lt;/p&gt;
&lt;h3 id="comment-ca-fonctionne"&gt;Comment ça fonctionne&lt;/h3&gt;
&lt;p&gt;Voici une présentation rapide du fonctionnement de l'implémentation de base du lexer (&lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/BaseLexer.java"&gt;Code source de &lt;code&gt;BaseLexer.java&lt;/code&gt; (sur Github) pour ceux qui voudront suivre avec le code sous les yeux&lt;/a&gt;):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Le constructeur du lexer prend en paramètre un &lt;code&gt;java.io.Reader&lt;/code&gt; qui pointe vers le texte source&lt;/li&gt;
&lt;li&gt;Le lexer définit un champ &lt;code&gt;currentLine&lt;/code&gt; qui contient la ligne courante et 2 autres champs &lt;code&gt;row&lt;/code&gt; et &lt;code&gt;col&lt;/code&gt; pour la position&lt;/li&gt;
&lt;li&gt;Quand &lt;code&gt;pop&lt;/code&gt; est appelée, on teste si la ligne courante est renseignée ou pas. Si elle ne l'est pas, on essaie de lire une ligne complète du &lt;code&gt;Reader&lt;/code&gt;. Si la méthode retourne &lt;code&gt;null&lt;/code&gt;, c'est que la fin du texte source est atteinte, et dans ce cas le lexer se place en un mode où il retourne toujours un token de type &lt;code&gt;EOF&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Sinon, et après avoir traité les indentations au début de la ligne (je vais revenir là dessus plus tard), le lexer examine la première lettre de la ligne pour choisir la branche à suivre&lt;/li&gt;
&lt;li&gt;Si c'est une lettre, alors il continue à consommer la ligne un caractère à la fois jusqu'à ce qu'il trouve autre chose qu'une lettre ou un chiffre, en accumulant les caractères lus dans une chaîne&lt;ul&gt;
&lt;li&gt;Si cette chaîne est égale à &lt;code&gt;true&lt;/code&gt; ou &lt;code&gt;false&lt;/code&gt;, il retourne un token de type &lt;code&gt;BOOL&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Si cette chaîne figure dans la liste des mots clés, il retourne un token de type &lt;code&gt;KEYWORD&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Sinon, c'est que c'est un identifiant. Il retourne alors un token de type &lt;code&gt;NAME&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Si le premier caractère est un chiffre, le lexer continue de consommer les caractères tant qu'il trouve des chiffres, puis il retourne un token de type &lt;code&gt;NUM&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Si c'est plutôt &lt;code&gt;"&lt;/code&gt; ou &lt;code&gt;'&lt;/code&gt;, le lexer lit la chaîne ou le caractère et retourne un token de type &lt;code&gt;STRING&lt;/code&gt; ou &lt;code&gt;CHAR&lt;/code&gt;. Ce n'est pas très compliqué comme traitement, si ce n'est pour gérer les échappements (&lt;code&gt;\"&lt;/code&gt; ou &lt;code&gt;\n&lt;/code&gt; par exemple)&lt;/li&gt;
&lt;li&gt;Le lexer tente ensuite de lire un symbole parmi la liste des symboles qu'il reconnait. Je vais revenir sur cette partie plus tard, mais l'idée est d'utiliser un automate en essayant de matcher le symbole le plus long (par exemple, matcher un seul token avec la valeur &lt;code&gt;-&amp;gt;&lt;/code&gt; plutôt que 2 tokens &lt;code&gt;-&lt;/code&gt; et &lt;code&gt;&amp;gt;&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Enfin, si on a atteint la fin de la ligne, on remet &lt;code&gt;currentLine&lt;/code&gt; à &lt;code&gt;null&lt;/code&gt;. De cette manière, le prochain appel à &lt;code&gt;pop&lt;/code&gt; va passer à la ligne suivante. Sinon, on lance une erreur car on est face à une entrée non reconnue&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="gestion-des-blancs_1"&gt;Gestion des blancs&lt;/h3&gt;
&lt;p&gt;A lecture d'une nouvelle ligne, et avant d'exécuter l'algorithme décrit dans la section précédente, le lexer consomme les espaces en début de la ligne en calculant leur nombre, ce qui définit le niveau d'indentation de la ligne. Il compare ensuite cette valeur au niveau d'indentation de la ligne précédente (qu'il maintient dans un champ initialisé à 0):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Si les 2 valeurs sont égales, il retourne un token de type &lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Si le niveau d'indentation de la ligne courante est supérieur à celui de la ligne précédente, il retourne un token de type &lt;code&gt;INDENT&lt;/code&gt; mais il se met aussi dans un mode où le prochain appel à &lt;code&gt;pop&lt;/code&gt; retourne &lt;code&gt;NEWLINE&lt;/code&gt;. Dans une première version du lexer de Litil, je générais uniquement &lt;code&gt;INDENT&lt;/code&gt; ou &lt;code&gt;DEINDENT&lt;/code&gt; quand le niveau d'indentation changeait, &lt;code&gt;NEWLINE&lt;/code&gt; sinon. Mais cela posait plein de problèmes dans la phase suivante (le parseur) pour délimiter correctement les blocs, jusqu'à ce que je tombe sur &lt;a href="http://stackoverflow.com/questions/232682/how-would-you-go-about-implementing-off-side-rule/946398#946398"&gt;cette réponse sur Stackoverflow&lt;/a&gt;. En suivant la méthode décrite dans cette réponse, j'ai fini avec une implémentation beaucoup plus simple et surtout solide du parseur. Je reviendrai là-dessus dans le post qui va traiter du parsing.&lt;/li&gt;
&lt;li&gt;Sinon, il retourne un token de type &lt;code&gt;DEINDENT&lt;/code&gt; et se met en un mode pour retourner &lt;code&gt;NEWLINE&lt;/code&gt; à l'appel suivant&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Un exemple pour clarifier un peu les choses. Etant donné ce texte en entrée:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
  &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Le lexer est censé générer les tokens suivants:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(a)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;INDENT&lt;/code&gt;: le niveau d'indentation a augmenté à la 2ième ligne&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: toujours produit pour une nouvelle ligne&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(b)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;: le niveau d'indentation n'a pas changé entre les 2ième et 3ième lignes&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(c)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DEINDENT&lt;/code&gt;: le niveau d'indentation a diminué&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NEWLINE&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NAME(d)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;EOF&lt;/code&gt;: fin de l'entrée&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Seulement, l'algorithme décrit jusqu'ici n'est pas suffisant pour que le parseur arrive à gérer proprement l'indentation. En effet, avec l'exemple suivant:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Le lexer ne va pas produire un token de type &lt;code&gt;DEINDENT&lt;/code&gt; après le &lt;code&gt;NAME(b)&lt;/code&gt; mais plutôt un &lt;code&gt;EOF&lt;/code&gt; car il n'y a pas de nouvelle ligne après le &lt;code&gt;b&lt;/code&gt;. On pourrait imaginer une solution où le parseur utilise &lt;code&gt;EOF&lt;/code&gt; en plus de &lt;code&gt;DEINDENT&lt;/code&gt; pour détecter la fin d'un bloc, mais ce n'est pas suffisant. En effet, avec l'exemple suivant:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;L'implémentation décrite ici va générer un seul token &lt;code&gt;DEINDENT&lt;/code&gt;après &lt;code&gt;NAME(c)&lt;/code&gt; alors que cette position dans le source marque la fin de 2 blocs et non pas un seul.&lt;/p&gt;
&lt;p&gt;Pour gérer ce type de situations, et ne voulant pas complexifier encore le code du lexer de base, j'ai décidé de gérer ça dans une autre implémentation de l'interface &lt;code&gt;Lexer&lt;/code&gt;, &lt;code&gt;StructuredLexer&lt;/code&gt;. Cette dernière implémente le pattern décorateur en délégant à &lt;code&gt;BaseLexer&lt;/code&gt; pour générer les tokens du texte source, mais en l'enrichissant avec les traitements suivants:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;On maintient le niveau courant d'indentation dans un champ. Le niveau d'indentation est calculé en divisant le nombre d'espaces en début d'une ligne par une taille d'unité d'indentation, fixée à *&lt;em&gt;2 espaces&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Dans &lt;code&gt;pop&lt;/code&gt;, si le lexer de base retourne un &lt;code&gt;INDENT&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;Vérifier que le nombre d'espaces est un multiple de l'unité. Si ce n'est pas le cas, retourner une erreur&lt;/li&gt;
&lt;li&gt;S'assurer aussi que le niveau d'indentation n'augmente qu'avec des pas de &lt;strong&gt;1&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Mettre à jour le champ niveau d'indentation&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Toujours dans &lt;code&gt;pop&lt;/code&gt;, et quand le lexer de base retourne un &lt;code&gt;DEINDENT&lt;/code&gt;:&lt;ul&gt;
&lt;li&gt;Idem que pour &lt;code&gt;INDENT&lt;/code&gt;, s'assurer que le nombre de blancs est un multiple de l'unité d'indentation&lt;/li&gt;
&lt;li&gt;Si le niveau d'indentation a diminué de plus d'une unité (comme dans l'exemple précédent), générer autant de tokens &lt;code&gt;DEINDENT&lt;/code&gt; virtuels que nécessaires, tout en mettant à jour le champ niveau d'indentation&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Si dans &lt;code&gt;pop&lt;/code&gt; le lexer de base retourne &lt;code&gt;EOF&lt;/code&gt;, produire des &lt;code&gt;DEINDENT&lt;/code&gt; virtuels jusqu'à ce que le niveau d'indentation atteigne 0, puis retourner &lt;code&gt;EOF&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Ainsi, avec l'exemple suivant:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
  &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Le lexer structuré génère 1 &lt;code&gt;DEINDENT&lt;/code&gt; virtuel, en plus du &lt;code&gt;DEINDENT&lt;/code&gt; généré par le lexer de base entre &lt;code&gt;c&lt;/code&gt; et &lt;code&gt;d&lt;/code&gt;. Comme ça, le parseur au dessus pourra détecter la fin de 2 blocs et détecter correctement que &lt;code&gt;d&lt;/code&gt; a le même niveau que &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Enfin, &lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/StructuredLexer.java"&gt;le code source&lt;/a&gt; qui implémente cet algorithme est disponible dans le repo Github de Litil pour les intéressés.&lt;/p&gt;
&lt;h3 id="gestion-des-commentaires"&gt;Gestion des commentaires&lt;/h3&gt;
&lt;p&gt;Dans Litil, les commentaires sont préfixés par &lt;code&gt;--&lt;/code&gt; (double &lt;code&gt;-&lt;/code&gt;) et s'étendent sur une seule ligne uniquement. J'ai &lt;em&gt;(arbitrairement)&lt;/em&gt; choisi de les gérer au niveau du lexer en les ignorant complètement. Mais j'aurais aussi pu produire des tokens de type &lt;code&gt;COMMENT&lt;/code&gt; et les ignorer &lt;em&gt;(ou pas)&lt;/em&gt; dans le parseur.&lt;/p&gt;
&lt;p&gt;Les commentaires sont gérés à 2 endroits dans le lexer:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Dans le code qui lit une ligne du texte source. Si elle commence par &lt;code&gt;--&lt;/code&gt;, on ignore la ligne entière et on passe à la ligne suivante (pour gérer les commentaires en début d'une ligne)&lt;/li&gt;
&lt;li&gt;Dans le code qui gère les symboles. Si le symbole &lt;em&gt;matché&lt;/em&gt; correspond à &lt;code&gt;--&lt;/code&gt;, on passe à la ligne suivante (pour gérer les commentaires à la fin d'une ligne)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Exemples:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c"&gt;-- compute max x y … NOT !&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;max&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- It is a well known fact that x always wins ! &lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h3 id="gestion-des-symboles"&gt;Gestion des symboles&lt;/h3&gt;
&lt;p&gt;Avec la gestion des indentations, c'est la partie la plus intéressante &lt;em&gt;(amha)&lt;/em&gt; dans le code du lexer.&lt;/p&gt;
&lt;p&gt;Avant de rentrer dans les détails d'implémentation, je vais d'abord parler un peu d'&lt;a href="http://fr.wikipedia.org/wiki/Automate_fini"&gt;automates finis&lt;/a&gt;, qui sont une notion centrale dans la théorie des langages et la compilation.&lt;/p&gt;
&lt;p&gt;Un automate fini est un ensemble d'états et de transitions. On peut le voir comme un système de classification: étant donnée une séquence en entrée, il consomme ses éléments un à un en suivant les transitions adaptés (et donc en passant d'un état à un autre) jusqu'à ce qu'il ait consommé toute l'entrée ou encore qu'il arrive dans un état sans aucune transition  possible. Quelques états peuvent être marqués comme terminaux ou finals, une façon de dire que ça représente un succès. Donc étant donnée un automate et une entrée, si le traitement s'arrête dans un état terminal, on peut dire qu'on a prouvé une propriété donnée sur l'entrée. Cette propriété va dépendre de l'automate.&lt;/p&gt;
&lt;p&gt;Ok, j'explique comme un pied. Un exemple concrêt:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa0.png" src="/images/graphviz/litil-lexer-dfa0.png" /&gt;&lt;/p&gt;
&lt;p&gt;L'automate présenté dans la figure précédente se compose de:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Un état initial (conventionnellement appelé &lt;code&gt;S0&lt;/code&gt;). C'est l'unique point d'entrée de l'automate&lt;/li&gt;
&lt;li&gt;3 autres états &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt; et &lt;code&gt;C&lt;/code&gt;. Notez le double contour de ces états. Ca sert à indiquer que ce sont des états terminaux ou d'acceptation&lt;/li&gt;
&lt;li&gt;Des transitions entre ces états qui sont annotées par des caractères&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Maintenant, appliquons cet automate à la séquence de caractères &lt;code&gt;-&amp;gt;a&lt;/code&gt;.
Comme dit plus haut, on se positionne initialement au point d'entrée &lt;code&gt;S0&lt;/code&gt;. On examine le premier caractère de la séquence d'entrée (&lt;code&gt;-&amp;gt;a&lt;/code&gt;) et on cherche une transition qui part de cet état et qui est étiquetée avec ce caractère. Ca tombe bien, le premier caractère &lt;code&gt;-&lt;/code&gt; correspond bien à une transition entre &lt;code&gt;S0&lt;/code&gt; et &lt;code&gt;A&lt;/code&gt;. On suit donc cette transition pour arriver à l'état &lt;code&gt;A&lt;/code&gt; et on passe au caractère suivant (&lt;code&gt;&amp;gt;&lt;/code&gt;). L'état &lt;code&gt;A&lt;/code&gt; est terminal. On pourrait donc interrompre le traitement et dire et qu'on a réussi à matcher le caractère &lt;code&gt;-&lt;/code&gt;. Cependant, dans Litil et presque tous les autres langages, le lexer essai plutôt de matcher la séquence la plus longue. L'alternative est qu'il ne faut jamais avoir plusieurs opérateurs qui commencent par le même caractère, ce qui serait trop contraignant.&lt;/p&gt;
&lt;p&gt;On continue donc le traitement. &lt;code&gt;A&lt;/code&gt; dispose bien d'une transition étiquetée avec &lt;code&gt;&amp;gt;&lt;/code&gt;. On suit donc cette transition et on arrive à l'état &lt;code&gt;C&lt;/code&gt; qui est aussi terminal. Mais par la même logique, on n'abandonne pas tout de suite et on tente de voir s'il est possible de matcher une séquence plus longue. Ce n'est pas le cas ici car l'état &lt;code&gt;A&lt;/code&gt; n'a aucune transition étiquetée avec le caractère &lt;code&gt;a&lt;/code&gt;. Le traitement s'arrête donc ici, et comme c'est un état terminal, on retourne un succès avec la chaîne &lt;em&gt;matchée&lt;/em&gt; qui ici est &lt;code&gt;-&amp;gt;&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Notez que l'algorithme que je viens de décrire (et qui est utilisé par l'implémentation actuelle du lexer de Litil) est plutôt simpliste et incomplet par rapport à l'état de l'art car il ne gère pas le &lt;em&gt;backtracking&lt;/em&gt;. Par exemple, cet algorithme échoue avec l'automate suivante (censé reconnaitre les symboles &lt;code&gt;-&lt;/code&gt; et &lt;code&gt;--&amp;gt;&lt;/code&gt;) avec la chaîne &lt;code&gt;--a&lt;/code&gt; comme entrée alors qu'il devrait réussir à retourner deux fois le symbole &lt;code&gt;-&lt;/code&gt; (le pourquoi est laissé comme exercice au lecteur):&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa1.png" src="/images/graphviz/litil-lexer-dfa1.png" /&gt;&lt;/p&gt;
&lt;p&gt;Dans sa version actuelle, les symboles reconnus par Litil sont: &lt;code&gt;-&amp;gt;&lt;/code&gt;, &lt;code&gt;.&lt;/code&gt;, &lt;code&gt;+&lt;/code&gt;, &lt;code&gt;-&lt;/code&gt;, &lt;code&gt;*&lt;/code&gt;, &lt;code&gt;/&lt;/code&gt;, &lt;code&gt;(&lt;/code&gt;, &lt;code&gt;)&lt;/code&gt;, &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;%&lt;/code&gt;, &lt;code&gt;&amp;lt;&lt;/code&gt;, &lt;code&gt;&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;=&lt;/code&gt;, &lt;code&gt;&amp;gt;=&lt;/code&gt;, &lt;code&gt;:&lt;/code&gt;, &lt;code&gt;,&lt;/code&gt;, &lt;code&gt;[&lt;/code&gt;, &lt;code&gt;]&lt;/code&gt;, &lt;code&gt;|&lt;/code&gt;, &lt;code&gt;_&lt;/code&gt;, &lt;code&gt;=&amp;gt;&lt;/code&gt;, &lt;code&gt;\\&lt;/code&gt;, &lt;code&gt;--&lt;/code&gt;, &lt;code&gt;::&lt;/code&gt;, &lt;code&gt;{&lt;/code&gt; et &lt;code&gt;}&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Maintenant, juste pour la science, voici l'automate fini correspondant à ces symboles:&lt;/p&gt;
&lt;p&gt;&lt;img alt="litil-lexer-dfa-litil.png" src="/images/graphviz/litil-lexer-dfa-litil.png" /&gt;&lt;/p&gt;
&lt;p&gt;L'implémentation de cet automate dans Litil est dynamique, dans la mesure où l'automate est construit au &lt;em&gt;runtime&lt;/em&gt; à partir d'une liste des symboles à reconnaitre. Aussi, cette implémentation ne gère pas le &lt;em&gt;backtracking&lt;/em&gt;, qui est inutile pour le moment car, et à moins que je ne dise de bêtises, le problème décrit plus haut n'arrive que si on a des symboles à 3 caractères (et qui ont le même préfixe qu'un symbole à un caractère), ce qui n'est pas le cas dans Litil (ce n'est pas Scala tout de même. Enfin, pas encore). Par contre, l'implémentation tient en 50 lignes de code Java, et si on ignore le décorum de Java (les imports, les &lt;em&gt;getters&lt;/em&gt;, le &lt;code&gt;toString&lt;/code&gt;, le constructeur, etc.), l'essence de l'algorithme tient juste dans une douzaine de lignes. &lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/LexerStage.java"&gt;Voici son code source sur Github&lt;/a&gt; pour les intéressés.&lt;/p&gt;
&lt;h3 id="lookahead"&gt;Lookahead&lt;/h3&gt;
&lt;p&gt;Une fonctionnalité utile à avoir dans le lexer est le &lt;code&gt;lookahead&lt;/code&gt;, i.e. la possibilité de voir les tokens suivants sans pour autant les consommer (comme c'est le cas avec &lt;code&gt;pop&lt;/code&gt;). On va revenir la dessus dans le(s) post(s) à propos du parseur.&lt;/p&gt;
&lt;p&gt;Tout comme &lt;code&gt;StructuredLexer&lt;/code&gt;, j'ai décidé d'implémenter cette fonctionnalité dans un décorateur autour d'un lexer concrêt, pour ne pas complexifier le code de ce dernier. Il s'agit de la classe &lt;code&gt;LookaheadLexerWrapper&lt;/code&gt; (&lt;a href="https://github.com/jawher/litil/blob/master/src/main/java/litil/lexer/LookaheadLexerWrapper.java"&gt;dont voici le code source&lt;/a&gt;). L'implémentation derrière est plutôt simple. En effet, l'astuce est juste de maintenir une liste de tokens (en plus du lexer concrêt). Quand la méthode &lt;code&gt;lookahead&lt;/code&gt; est appelé (avec un paramètre qui indique le niveau du lookahead: 1 pour le token suivant, etc.), on récupère autant de tokens que nécessaire du lexer et on les stocke dans cette liste. Quand &lt;code&gt;pop&lt;/code&gt; est appelée, et avant de déléger au lexer concrêt, on vérifie si la liste de tokens n'est pas vide. Si c'est le cas, on retourne le premier élément de cette liste et en prenant soin de l'enlever, comme ça, l'appel suivant à &lt;code&gt;pop&lt;/code&gt; va retourner le token suivant. Si cette liste est vide, alors on délègue au lexer.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Dans ce post, j'avais présenté rapidement et sans trop entrer dans les détails quelques techniques de lexing que j'ai utilisé dans Litil. Ce n'est en aucune façon la meilleure façon de faire ni la plus performante. C'était plus une implémentation relativement facile à coder et à étendre tout en restant raisonnable pour tout ce qui est consommation mémoire (le fait de ne pas lire la totalité du fichier dans une seule &lt;code&gt;String&lt;/code&gt; d'un coup par exemple).&lt;/p&gt;
&lt;p&gt;Il faut aussi noter que c'est la partie la moins &lt;em&gt;marrante&lt;/em&gt; à coder dans un langage. J'ai même été tenté de la faire générer par un outil comme AntLR ou SableCC, mais au final j'ai décidé de rester fidèle à ma vision de vraiment tout coder à la main &lt;em&gt;(sauf les parties que je ne coderai pas à la main, NDLR)&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Maintenant que cette partie est derrière nous, nous allons enfin pouvoir attaquer les sujets intéressants, e.g. &lt;a href="http://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"&gt;le parseur&lt;/a&gt; et l'évaluateur, dans les posts suivants.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>Le conte de Litil - Chapitre 1, Génèse d'un langage de programmation</title>
   <link href="https://jawher.me/2012/06/04/creation-langage-programmation-litil-1/"/>
   <published>2012-06-04T00:00:00+02:00Z</published>
   <updated>2012-06-04T00:00:00+02:00Z</updated>
   <id>https://jawher.me/2012/06/04/creation-langage-programmation-litil-1/</id>
   <content type="html">&lt;p&gt;Comme beaucoup d'autres passionnés d'informatique, j'ai toujours été fasciné par les compilateurs et les langages de programmation, et ce depuis que j'ai appris à programmer. Mais ma fascination ne s'arrêtait pas au fait que je pouvais commander ma machine en écrivant des programmes pour produire des résultats &lt;em&gt;(parfois)&lt;/em&gt; utiles. En effet, j'étais encore plus fasciné par les compilateurs eux-mêmes. Je trouvais magique qu'un programme réussisse à &lt;em&gt;"lire"&lt;/em&gt; et à exécuter ce que je venais de saisir.&lt;/p&gt;
&lt;p&gt;Je me rappelle encore de ma première tentative d'écrire un parseur. A l'époque (j'étais encore au lycée), je ne savais même pas que c'est comme ça que ça s'appelait. Je venais à peine d'apprendre Pascal sur un Commodore, et je voulais absolument écrire un &lt;em&gt;truc&lt;/em&gt; qui évaluerait une expression arithmétique sous forme d'une chaine de caractères, sans le moindre &lt;em&gt;background&lt;/em&gt; théorique sur le sujet, ni même d'accès aux internets®. Je ne me rappelle plus si j'avais réussi à le faire ou pas, mais je n'ai pas oublié quelques rustines que j'avais pondu à l'époque, comme par exemple le fait d'évaluer en priorité les expressions parenthésées. Ma solution était de chercher la première parenthèse fermante, puis de partir dans l'autre sens jusqu'à la première parenthèse ouvrante pour délimiter le bout à évaluer en premier, puis répéter le processus jusqu'à ce qu'il ne reste plus de parenthèses &lt;em&gt;(Non, ce n'est pas comme ça qu'il faut faire ;)&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Depuis, j'en ai pas mal appris sur le sujet lors de mes études universitaires, mais surtout dans un cadre professionnel, où j'ai eu la chance de travailler sur 2 projets nécéssitant l'écriture d'un parseur pour un DSL métier. J'essayais aussi d'en apprendre un peu plus en lisant l'occasionnel papier de recherche, ou en fréquentant des sites comme &lt;a href="http://lambda-the-ultimate.org/"&gt;LtU&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Récemment, j'ai décidé que, comme tout le monde, je vais créer mon langage de programmation, aka JavaSlayer™, et pour rendre l'exercice encore plus stimulant, j'ai décidé de ne pas utiliser un générateur de parseurs (à la ANTLR) mais plutôt de tout faire à la main et à l'ancienne sans la moindre bibliothèque ni framework. Le but était de vraiment comprendre comment fonctionnait un compilateur. J'avais aussi été influencé par &lt;a href="http://en.wikipedia.org/wiki/Niklaus_Wirth"&gt;Niklaus Wirth&lt;/a&gt; (le créateur de Pascal, Modula et Oberon) qui a écrit tous ses compilateurs à la main.&lt;/p&gt;
&lt;p&gt;Et je ne sais pas comment, mais après 2 ou 3 mois de travail occasionnel, je me suis retrouvé avec un parseur et évaluateur pour langage plutôt puissant, dites bonjour à &lt;code&gt;Litil&lt;/code&gt;!&lt;/p&gt;
&lt;p&gt;Comme je n'avais plus envie de blogger sur "Comment créer une application &lt;code&gt;&amp;lt;framework web #1&amp;gt;&lt;/code&gt; avec &lt;code&gt;&amp;lt;orm #2&amp;gt;&lt;/code&gt; et &lt;code&gt;&amp;lt;framework js #3&amp;gt;&lt;/code&gt; sur &lt;code&gt;&amp;lt;un ide&amp;gt;&lt;/code&gt; pour gérer &lt;code&gt;&amp;lt;sujet hip du moment&amp;gt;&lt;/code&gt; et l'héberger sur &lt;code&gt;&amp;lt;fournisseur PaaS|SaaS|IaaS&amp;gt;&lt;/code&gt;", je me lance dans une série (enfin j'espère) de billets sur le sujet de parsing, sans plan particulier, quoi que j'essairais de maintenir une semblance d'ordre logique, où j'exposerais les techniques que j'utilise (au fur et à mesure que je les apprenne).&lt;/p&gt;
&lt;p&gt;Il faut préciser que bien qu'il existe pleins d'outils qui permettent de générer &lt;strong&gt;facilement&lt;/strong&gt; des parseurs, comme SableCC, ANTLR, JavaCC, lex/yacc, flex/bison, etc., mais le but ici est justement de tout faire à la main, l'objectif étant de comprendre comment ça fonctionne à l'intérieur(™).&lt;/p&gt;
&lt;h1 id="litil"&gt;Litil&lt;/h1&gt;
&lt;p&gt;Comme dit plus haut, ce post entame une série où je présenterai et expliquerai les différentes techniques que j'avais utilisées pour créer un parseur et un évaluateur pour un langage de programmation, Litil™, le tout en Java avec toutefois quelques notices:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;C'est du &lt;em&gt;work in progress&lt;/em&gt;. On n'arrête pas de nous le dire: &lt;em&gt;release early, release often&lt;/em&gt;, mais je tenais à le préciser quand même: j'avançais en tatonnant, en apprenant au fur et à mesure, et donc, inévitablement, il y'aurait plusieurs coquilles.&lt;/li&gt;
&lt;li&gt;J'en ai parlé dans le point précédent: je ne prétends pas être une autorité dans ce sujet. J'ai eu un début de formation théorique dessus et j'ai lu pas mal de littérature.&lt;/li&gt;
&lt;li&gt;J'ai récemment commencé à travaillé sur la compilation vers du bytecode. Il se peut que j'en parle dans un futur post, mais c'est encore assez tôt et je ne promet rien.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Une présentation rapide de Litil en quelques points:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;langage fonctionnel fortement inspiré de ML, (O)Caml et Haskell et &lt;a href="http://roy.brianmckenna.org/"&gt;Roy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;inférence de types totale avec &lt;a href="http://en.wikipedia.org/wiki/Hindley-Milner_type_inference"&gt;Hindley Milner&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Off-side_rule"&gt;les blocs sont délimités avec l'indentation (à la Python)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;types supportés: entiers, chaines, booléens, caractères, &lt;a href="http://en.wikipedia.org/wiki/Tuple"&gt;tuples&lt;/a&gt;, &lt;a href="http://en.wikipedia.org/wiki/Record_(computer_science)"&gt;records&lt;/a&gt; et &lt;a href="http://en.wikipedia.org/wiki/Algebraic_data_type"&gt;ADTs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Pattern_matching"&gt;pattern matching&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;em&gt;&lt;a href="http://en.wikipedia.org/wiki/Currying"&gt;curried functions&lt;/a&gt;&lt;/em&gt; par défaut&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Closure_(computer_science)"&gt;closures&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;exceptions (try/catch/throw)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Voilà. Maintenant un petit &lt;em&gt;teaser&lt;/em&gt; pour vous donner une idée sur à quoi ressemble le langage qu'on va créer.&lt;/p&gt;
&lt;h2 id="affectations-expressions-et-fonctions"&gt;affectations, expressions et fonctions&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    &lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="k"&gt;else&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;f5&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="tuples-et-records"&gt;tuples et records&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;a&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;person&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lpe&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="destructuring"&gt;destructuring&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;d&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;true&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;test&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="sc"&gt;&amp;#39;c&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bool&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="types-algebriques"&gt;types algebriques&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;Option&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Some&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;None&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Some&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;thing&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Cons&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Null&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Tree&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Node&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Leaf&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="pattern-matching"&gt;pattern matching&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;
    &lt;span class="bp"&gt;[]&lt;/span&gt;     &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;

&lt;span class="n"&gt;len&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="application-partielle"&gt;application partielle&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;inc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;three&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;inc&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="closures-et-higher-order-functions"&gt;closures et higher-order functions&lt;/h2&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;xs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="n"&gt;xs&lt;/span&gt;
    &lt;span class="bp"&gt;[]&lt;/span&gt;     &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Nil&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;double&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;

&lt;span class="c"&gt;-- pass a function by name&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="n"&gt;double&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;

&lt;span class="c"&gt;-- or simply a lambda&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;l2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;map&lt;/span&gt; &lt;span class="o"&gt;(\&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;

&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;\&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="c"&gt;-- f capture la valeur lexicale de a, i.e. 4&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h1 id="les-themes-qui-seront-abordes"&gt;Les thèmes qui seront abordés&lt;/h1&gt;
&lt;p&gt;La création d'un langage de programmation en général, et de Litil en particulier, nécessite d'aborder les thèmes suivants:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2012/06/27/creation-langage-programmation-litil-2-lexer/"&gt;Lexing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Parsing&lt;ul&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2013/08/11/creation-langage-programmation-litil-3-1-introduction-parsing/"&gt;BNF/EBNF&lt;/a&gt; &lt;/li&gt;
&lt;li&gt;&lt;a href="http://jawher.me/2013/08/19/creation-langage-programmation-litil-3-2-recursive-descent-parsing/"&gt;Parsing à descente récursive&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://en.wikipedia.org/wiki/Pratt_parser"&gt;Pratt parsing&lt;/a&gt; pour les expressions&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Inférence et vérification des types (Hindley Milner)&lt;/li&gt;
&lt;li&gt;Evaluation&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Pour les posts qui vont suivre, je ne vais pas forcément traiter de ces sujets dans l'ordre, le but étant d'implémenter graduellement les différentes composantes du langage du parsing jusqu'à l'évaluation. Du coup, la suite va consister à faire des allers-retours sur ces différents thèmes au fur et à mesure de l'avancement du développement du langage.&lt;/p&gt;</content>
 </entry>

<entry>
   <title>Running multiple python apps with nginx and uwsgi in emperor mode</title>
   <link href="https://jawher.me/2012/03/16/multiple-python-apps-with-nginx-uwsgi-emperor-upstart/"/>
   <published>2012-03-16T00:00:00+01:00Z</published>
   <updated>2012-03-16T00:00:00+01:00Z</updated>
   <id>https://jawher.me/2012/03/16/multiple-python-apps-with-nginx-uwsgi-emperor-upstart/</id>
   <content type="html">&lt;p&gt;This is a recipe on how to &lt;strong&gt;easily&lt;/strong&gt; run multiple Python web applications using uwsgi server (in emperor mode) and behind nginx.
Most existing docs and blogs would show how to manually start uwsgi to run a single app. In this post, I'll show how to configure uwsgi as a system service (with upstart) capable of serving multiple python WSGI compliant web applications by simply placing them in a standard location and adding an standard xml file.&lt;/p&gt;
&lt;p&gt;It took me many many hours and sleepless nights to get this setup to work. I was on the verge of giving up on uwsgi/nginx and falling back to apache/mod python. I'm not proud of it, but when it worked, I had (and still don't have) no idea why it did. So I would advice the readers not to change too much at a time, and rather work incrementally, with small changes at a time and immediate testing afterwards.&lt;/p&gt;
&lt;p&gt;I'm using this setup on a couple EC2 machines running Ubuntu 11.10, nginx 1.0.5-1, Python 2.7.2, uwsgi 0.9.8.1-1 and using the bottle web framework, but the same &lt;strong&gt;should&lt;/strong&gt; work for any other WSGI compliant framework.&lt;/p&gt;
&lt;h2 id="1-installing-the-required-software"&gt;1. Installing the required software&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;We'll (of course) need Python. It comes pre-installed on Ubuntu and other linuxes. Nothing to do here.&lt;/li&gt;
&lt;li&gt;uwsgi: &lt;code&gt;sudo aptitude install uwsgi uwsgi-plugin-python&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;bottle (optional): Since I'll be using Bottle framework for the test applications, we'll need to have it installed for this recipe to work, but if you're using another framework, you could skip this step. Just run this command: &lt;code&gt;sudo easy_install bottle&lt;/code&gt;. If &lt;code&gt;easy_install&lt;/code&gt; isn't already installed, run &lt;code&gt;sudo aptitude install python-setuptools&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;nginx: &lt;code&gt;sudo aptitude install nginx-full&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="2-configuration"&gt;2. Configuration&lt;/h2&gt;
&lt;h3 id="uwsgi"&gt;uwsgi&lt;/h3&gt;
&lt;p&gt;We'll run uwsgi using upstart so that it is automatically started on startup.&lt;/p&gt;
&lt;p&gt;We'll create a &lt;code&gt;uwsgi.conf&lt;/code&gt; file in &lt;code&gt;/etc/init&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo emacs /etc/init/uwsgi.conf&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;With this content:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# uWSGI - manage uWSGI application server                                                                                                                                                                &lt;/span&gt;
&lt;span class="c1"&gt;#                                                                                                                                                                                                    &lt;/span&gt;

description     &lt;span class="s2"&gt;&amp;quot;uWSGI Emperor&amp;quot;&lt;/span&gt;

start on &lt;span class="o"&gt;(&lt;/span&gt;filesystem and net-device-up &lt;span class="nv"&gt;IFACE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lo&lt;span class="o"&gt;)&lt;/span&gt;
stop on runlevel &lt;span class="o"&gt;[&lt;/span&gt;!2345&lt;span class="o"&gt;]&lt;/span&gt;

respawn

env &lt;span class="nv"&gt;LOGTO&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/var/log/uwsgi.log
env &lt;span class="nv"&gt;BINPATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/bin/uwsgi

&lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nv"&gt;$B&lt;/span&gt;INPATH --emperor /home/ubuntu/apps/vassals/ --logto $LOGTO
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Notice the path telling uwsgi emperor mode where to look for applications' configuration files (We'll be using the xml syntax here). &lt;strong&gt;You'll need to modify this to match your setup&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Then run this command to refresh the services configuration:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo initctl reload-configuration&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;We also want to make sure that uwsgi is using the system installed python (2.7 in this case) and not another version (2.6), so we need to run this:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo update-alternatives --set uwsgi /usr/bin/uwsgi_python27&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;And start the uwsgi service:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo service uwsgi start&lt;/code&gt;&lt;/p&gt;
&lt;h3 id="nginx"&gt;nginx&lt;/h3&gt;
&lt;p&gt;We'll create a &lt;code&gt;apps.conf&lt;/code&gt; file in the &lt;code&gt;/etc/nginx/sites-enabled&lt;/code&gt; directory to tell nginx to pass through requests to uwsgi:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo emacs /etc/nginx/sites-enabled/apps.conf&lt;/code&gt;&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;server&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kn"&gt;listen&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kn"&gt;server_name&lt;/span&gt; &lt;span class="s"&gt;jawher.me&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kn"&gt;index&lt;/span&gt; &lt;span class="s"&gt;index.html&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kn"&gt;location&lt;/span&gt; &lt;span class="s"&gt;/app1/&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;include&lt;/span&gt; &lt;span class="s"&gt;uwsgi_params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_param&lt;/span&gt; &lt;span class="s"&gt;SCRIPT_NAME&lt;/span&gt; &lt;span class="s"&gt;/app1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_modifier1&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_pass&lt;/span&gt; &lt;span class="s"&gt;unix://tmp/app1.sock&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kn"&gt;location&lt;/span&gt; &lt;span class="s"&gt;/app2/&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;include&lt;/span&gt; &lt;span class="s"&gt;uwsgi_params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_param&lt;/span&gt; &lt;span class="s"&gt;SCRIPT_NAME&lt;/span&gt; &lt;span class="s"&gt;/app2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_modifier1&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;uwsgi_pass&lt;/span&gt; &lt;span class="s"&gt;unix://tmp/app2.sock&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;You'll need to customize this file to match your setup (server name, apps locations, socket names, etc.)&lt;/p&gt;
&lt;p&gt;Also, there's a bit of magic here, the &lt;code&gt;uwsgi_param SCRIPT_NAME /app?&lt;/code&gt; and &lt;code&gt;uwsgi_modifier1 30&lt;/code&gt; thingies. Without them, requesting &lt;code&gt;jawher.me/app1/index&lt;/code&gt; would end up calling &lt;code&gt;app1/index&lt;/code&gt; on the bottle web application (i.e. the nginx location gets passed to the wsgi application, which is not what we want most of the time). With this modifier and script param thingies, the nginx location is stripped before invoking the python application.&lt;/p&gt;
&lt;p&gt;Start the nginx server if it is not already running:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo service nginx start&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Also, whenever you change the nginx config files, remember to run the following command:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sudo service nginx reload&lt;/code&gt;&lt;/p&gt;
&lt;h2 id="3-the-test-applications"&gt;3. The test applications&lt;/h2&gt;
&lt;p&gt;We'll create to test web applications to test our setup:&lt;/p&gt;
&lt;p&gt;An app in this case is a python file that defines a bottle app:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;emacs /home/ubuntu/apps/app1/app1.py&lt;/code&gt;&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;bottle&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;

&lt;span class="nd"&gt;@bottle.get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/index&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;App 1 live and kicking&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reloader&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8080&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;application&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;default_app&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This file can be executed with python, in which case the main bloc will launch the application in debug mode, or with uwsgi without a change to the file. Don't change the variable name &lt;code&gt;application&lt;/code&gt;, as that's what uwsgi will be looking for (unless you tell it otherwise).&lt;/p&gt;
&lt;p&gt;We'll also need an xml file to tell uwsgi emperor mode how to handle our app:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;emacs /home/ubuntu/apps/vassals/app1.xml&lt;/code&gt;&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;uwsgi&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;plugins&amp;gt;&lt;/span&gt;python&lt;span class="nt"&gt;&amp;lt;/plugins&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;master&amp;gt;&lt;/span&gt;true&lt;span class="nt"&gt;&amp;lt;/master&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;processes&amp;gt;&lt;/span&gt;1&lt;span class="nt"&gt;&amp;lt;/processes&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;vaccum&amp;gt;&lt;/span&gt;true&lt;span class="nt"&gt;&amp;lt;/vaccum&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;chmod-socket&amp;gt;&lt;/span&gt;666&lt;span class="nt"&gt;&amp;lt;/chmod-socket&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;socket&amp;gt;&lt;/span&gt;/tmp/%n.sock&lt;span class="nt"&gt;&amp;lt;/socket&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;uid&amp;gt;&lt;/span&gt;www-data&lt;span class="nt"&gt;&amp;lt;/uid&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;gid&amp;gt;&lt;/span&gt;www-data&lt;span class="nt"&gt;&amp;lt;/gid&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;pythonpath&amp;gt;&lt;/span&gt;%d../%n&lt;span class="nt"&gt;&amp;lt;/pythonpath&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;%n&lt;span class="nt"&gt;&amp;lt;/module&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/uwsgi&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This config file is completely generic, and we could reuse the exact same content for any other app. In order to do this, it has to heavily rely on these conventions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the xml config file has to have the same name as the python module that exports the wsgi application (&lt;code&gt;app1.xml&lt;/code&gt; and &lt;code&gt;app1.py&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;the python module file name has to match its directory name (&lt;code&gt;app1.py&lt;/code&gt; in an &lt;code&gt;app1&lt;/code&gt; dir)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If this is not acceptable for your setup, you'll need to change the values of the &lt;code&gt;&amp;lt;pythonpath&amp;gt;&lt;/code&gt; and &lt;code&gt;&amp;lt;module&amp;gt;&lt;/code&gt; tags to match your setup.&lt;/p&gt;
&lt;p&gt;The second application is exactly the same, except for 'app 1' which gets replaced by 'app 2':&lt;/p&gt;
&lt;p&gt;&lt;code&gt;emacs /home/ubuntu/apps/app2/app2.py&lt;/code&gt;&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;bottle&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;

&lt;span class="nd"&gt;@bottle.get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/index&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;App 2 live and kicking&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reloader&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8080&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;application&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bottle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;default_app&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;&lt;code&gt;emacs /home/ubuntu/apps/vassals/app2.xml&lt;/code&gt;&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;uwsgi&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;plugins&amp;gt;&lt;/span&gt;python&lt;span class="nt"&gt;&amp;lt;/plugins&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;master&amp;gt;&lt;/span&gt;true&lt;span class="nt"&gt;&amp;lt;/master&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;processes&amp;gt;&lt;/span&gt;1&lt;span class="nt"&gt;&amp;lt;/processes&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;vaccum&amp;gt;&lt;/span&gt;true&lt;span class="nt"&gt;&amp;lt;/vaccum&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;chmod-socket&amp;gt;&lt;/span&gt;666&lt;span class="nt"&gt;&amp;lt;/chmod-socket&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;socket&amp;gt;&lt;/span&gt;/tmp/%n.sock&lt;span class="nt"&gt;&amp;lt;/socket&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;uid&amp;gt;&lt;/span&gt;www-data&lt;span class="nt"&gt;&amp;lt;/uid&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;gid&amp;gt;&lt;/span&gt;www-data&lt;span class="nt"&gt;&amp;lt;/gid&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;pythonpath&amp;gt;&lt;/span&gt;%d../%n&lt;span class="nt"&gt;&amp;lt;/pythonpath&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;%n&lt;span class="nt"&gt;&amp;lt;/module&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/uwsgi&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;h2 id="4-troubleshooting"&gt;4. Troubleshooting&lt;/h2&gt;
&lt;p&gt;When it doesn't work, you'll usually end up with an unhelpful 502 error. To diagnose the problem:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Check that the sockets were created: &lt;code&gt;ls /tmp&lt;/code&gt; and that nginx's process can read, write and execute them. This shouldn't happen though as we specified &lt;code&gt;chmod 666&lt;/code&gt; in the applications config file.&lt;/li&gt;
&lt;li&gt;Check that the apps were correctly loaded in &lt;code&gt;/var/log/uwsgi.log&lt;/code&gt;: if you find this message &lt;code&gt;\*\*\* no app loaded. going in full dynamic mode \*\*\*&lt;/code&gt;, then go back and double check the module name and path in the app xml file, or try to run the app with python, as in &lt;code&gt;python app1.py&lt;/code&gt; to see if there aren't any missing imports or other errors. You may also run into this if the user under which uwsgi is running doesn't have read permissions on the app python files.&lt;/li&gt;
&lt;li&gt;Make sure that your app runs with the same version of python that uwsgi uses. Usually, you'll just want to configure uwsgi to use the system-wide python install.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/var/log/nginx/error.log&lt;/code&gt; might turn out some useful info too.&lt;/li&gt;
&lt;/ul&gt;</content>
 </entry>

</feed>
