the blog
Back to -Blog

Antlr 3 and the Dynamic Language Runtime

Programming Dynamite!
by Dermot Hogan
Monday 4 February 2008.

I’ve just had one of those strange moments where nothing has changed – and yet everything has changed. I’d been meaning to get into Microsoft’s Dynamic language Runtime (aka DLR) for some time but, as usual, hadn’t got round to it. Then a rainy day (torrential rain and gales) meant that I couldn’t get out and work outside. So I downloaded the DLR via the IronPython distribution.

The first question was (and is) where to start? The IronRuby implementation is too early to be really useful. It’s difficult to know whether something is a feature or a bug. That’s not desirable if you are just starting out on figuring out what the DLR is about! A quick look at IronPython convinced me that some knowledge of Python might be desirable: but I have none. So how about the ‘example’ language included with IronPython, ToyScript? Nope. Still far too complicated. Not much on the web, either. Just some (to me) incomprehensible articles by the DLR designer, Jim Hugunin, and that’s about it.

Then I hit gold. In the Ukraine, believe it or not. I downloaded Piter Protsky’s Simple DLR Calculator and got going. The DLR Calculator really is a great introduction to the DLR for absolute beginners such as myself. The version I downloaded used version 400 of the DLR, whereas the version downloaded with IronPython was 700. So I set about modifying Piter’s calculator for my own purposes and to work with the 700 DLR version.

A couple of hours later I had re-written DLR calculator with a nice new ‘print’ function to print out the result (it seems you have to implement this differently in the 700 DLR version) and now it pretty closely resembles the way things are done in ToyScript. At this stage I had kept Piter’s original C# lexer and parser, but now I wanted to use Antlr3. So after another hour ot two, I had rewritten the original calculator in Antlr3 (lexer, parser and - best of all - a ‘tree grammar’) and connected it all up to the DLR. The lexer and parser are standard stuff. It’s the tree grammar that’s the killer feature. To give you a flavour here’s the tree grammar - all of it:

prog returns [MyLStatement result]
        : e=expr  {$result = new MyLPrint(e.Span, e);}
        ;

expr returns [MyLExpression result]
        : ^(op=(PLUS|MINUS|MULTIPLY|DIVIDE) left=expr right=expr) {$result = new MyLBinary(new SourceSpan(), op.Token.Type, left, right);}
        | ^(op=(UNARY_PLUS|UNARY_MINUS) arg=expr) {$result = new MyLUnary(new SourceSpan(), op.Token.Type, arg);}
        | INT {$result = new MyLConstant(new SourceSpan(), double.Parse($INT.Text));}
        ;

This the important bit: the tree grammar is what connects my DLR classes MyLPrint, MyLBinary, etc into the DLR. Here’s one of my DLR classes (referenced in the above tree grammar) that prints out the result:

class MyLPrint : MyLStatement {
 private readonly MyLExpression _expression;
 
public MyLPrint(SourceSpan span, MyLExpression expression) :base(span) {
   _expression = expression;
}

 protected internal override Statement Generate() {
   return Ast.Statement( Span, Ast.Call(typeof(MyLHelpers).GetMethod("Print"), Ast.ConvertHelper(_expression.Generate(), typeof(object))));
 }
}

I’ll go into more detail about what’s going on here at a later date – but the point I’m trying to get over is that this is simple stuff. None of the code I’ve written is more than a few lines long and all in all, there are about half a dozen small files. And at the end of the process, I had a functioning calculator. I’d written my first DLR language – from soup to nuts - in a few hours from a starting point of zero knowledge about the DLR.

Now the problem I’ve always had with Antlr is that having got the AST – what do you do next? Getting a simple grammar in Antlr is easy – but then you need a miracle to occur to actually do anything with it – emitting CLR code isn’t simple. But connecting the Antlr AST to the DLR via the tree grammar is a trivial operation – see the code above. As is writing the DLR’s ‘adapter’ classes.

So one more step. I needed to implement unary operators. Having spent a little time wrestling with Antlr3 to do this (the sooner Terrence Parr produces an Antlr3 ‘cookbook’ the better) I found that the DLR didn’t like my unary ‘+’. No problem. I looked at the DLR source code, saw that for some reason the unary ‘+’ hadn’t been implemented – and added it to the DLR. Wow! Another hour gone!

Now either I’m a genius at this stuff (sadly I’m not) or this is simple and easy to do. It’s pretty clear to me that Jim Hugunin has done something exceptional here with the DLR. Combined with the power and expressiveness of Antlr3, this stuff is dynamite.

So nothing has changed: I have a trivial ‘calculator’ program. And everything has changed: I know I can now write any language I want. Without a miracle happening too.

Bookmark and Share   Keywords:  .NET  DLR
  • Antlr 3 and the Dynamic Language Runtime
    8 February 2008, by Piter

    Nicely done. I am glad to hear that my posts help someone. I have prepared a more sophisticated example of DLR language with generated parser (than DLR calculator). Enjoy!

  • Antlr 3 and the Dynamic Language Runtime
    5 February 2008, by Tony

    Any chance we can get a complete source code listing? I’ve been meaning to get into ANTLR and get into the guts of the DLR. This would be a great jump start on both. Thanks.

    • Antlr 3 and the Dynamic Language Runtime
      5 February 2008

      Yes I’ll do that. I need to do a few bits of tidying up and I’m planning to write about it in more detail in a few weeks, but I’ll post an ’as is’ zip file in a day or so. I’ll put a note in the blog when its ready.

      The only thing to be aware of is that I’m using a pre-release of Antlr 3.1 which I built from the source code a few weeks ago, so I don’t know how this will work with Antlr 3.0.1. (I think that the 3.1 version I’m using might be a bit more forgiving syntactically than any ’official’ version - see the other comment).

      Dermot

  • Antlr 3 and the Dynamic Language Runtime
    4 February 2008, by Kay Roepke

    Hi!

    One thing I noticed was your usage of the ’e’ in

    prog returns [MyLStatement result] : e=expr $result = new MyLPrint(e.Span, e); ;

    It’s better to use $e to access labels, since then ANTLR can do error checking and will ensure that the property you use (in this case Span) is actually valid. Also, that it works with the plain ’e’ is coincidence, since the actual naming of the variable in the generated code is an implementation detail - it could be different from ’e’, say ’_e’.

    cheers,
    - k

    • Antlr 3 and the Dynamic Language Runtime
      4 February 2008

      Yes - that’s right! Thanks for pointing this out.

      The main point is though, that Antlr tree grammars are a very concise way of linking into the DLR. That’s why I posted the code - just to show how it’s done. I think I’m still stunned by Antlr3.

      Dermot

© SapphireSteel Software 2014