the blog
Back to -Blog

IntelliSense and Parsing Ruby

by Dermot Hogan
Monday 24 July 2006.

I’ve been working on getting IntelliSense into RubyInSteel. IntelliSense covers several areas, the ones I’m looking at right now are ‘member completion’ and ‘parameter info’. The first is where you type a ‘.’ and typically get a drop down list of member functions of a class, the second is additional parameter data after you type an opening parenthesis.

Getting the basic IntelliSense operational proved to be relatively easy - I used the Iron Python project as a guide and with a good bit of cutting and pasting and nipping and tucking got a dummy IntelliSense system working pretty quickly. I’m not normally a great fan of Iron Python as a ‘how-to’ - it’s way too complicated. But as a reference project incorporating everything in the Visual Studio SDK pantheon it’s getting pretty good. However, I do think that a simpler MyC end-to-end example would be handy.

But then the next problem was how to get the information on a particular member - what are the methods of ‘self’ for example? After a false start - I tried to collect the class/def information myself - I hit on what looked like the ideal answer: Antlr! I’ve used Antlr for generating the colour information and I’d written (what looked to me like) a complete parser, though I hadn’t used it. So it’s a small step to use the parser to create an AST (Abstract Syntax Tree) suitable for collecting IntelliSense info. And after figuring out Antlr’s tree building language (Antlr really is an extraordinarily flexible system) got an AST.

Then I hit a small problem: to get the information, the parse has to be complete. It’s not terribly useful if it gives up half way through a correct program (syntactically incorrect ones are a different issue). The small problem was that my parser wasn’t as good as I thought. I’ve spent the last two weeks (solid!) as punishment working through every example of Ruby code I could lay my hands on - namely, the two thousand files or so that come with the Ruby distribution plus various gems.

I’ve alternated between sheer disbelief about the really poor quality of some of the code ( being polite - ‘yuk!’) and amazement (‘ I didn’t know you could do that in Ruby!) but the exercise has been worthwhile - my parser now does cope with every Ruby construct (bar one - nested here documents) in 2600 files - and I’ve fixed several colouring bugs in the process. It’s fast too: processing 2600 files took 60 seconds - about 25ms per file.

However, it did strike me that what I was doing was generating a successive approximation to a Ruby parser. The actual C Ruby parser is a combination of Matz’s meanderings and bug fixes over the years - and it’s neither pretty nor elegant. I can get closer and closer to the real thing but the amount of effort required to do that increases. For example, the one construct I won’t deal with at present is nested ‘here’ documents. For various technical reasons, they are hard hard work to implement. And how many people use them, anyway? I’ve only come across two examples so far in all those 2600 files.

I can see why others have used the Ruby parser itself to do syntax analysis. But the downside is that first it’s very, very slow (Antlr is fast!) and second it’s difficult to integrate and expand. The upside is of course that you don’t have to do much work and it’s 100% accurate by definition. But I’ve thought about this quite a bit now, and I’m pretty happy with the direction I’ve chosen: once the parsing is done, adding class browers, navigators and other cross-referencing features become possible. Find All References, anyone?

Ruby IntelliSense

Here’s a small example of a very simple IntelliSense operation. Kudos to Antlr and Terrence Parr!

Bookmark and Share   Keywords:  development
  • You may interested in rubyfront
    13 September 2006

    Creating an antlr based ruby parser from scratch is not easy, it took me half a year to finish(part time although). You may want to take a look of rubyfront at http://seclib.blogspot.com/2006/04/...

  • IntelliSense and Parsing Ruby
    31 July 2006, by Jules

    Hello,

    Hello

    I am a big fan of Steel, I have just spent the last weekend starting to develop an AJAX orientated portal application with VML graphics, using Ruby/Steel environment, its great staying in VS environment.

    I have a question on the next great feature of Intellisence, will it be part of the free edition ? What are the features that distinguish the commercil edition from the free edition ? And what sort of price point are we talking for the commercial edition ?

    Cheers

    Jules

    • IntelliSense and Parsing Ruby
      31 July 2006, by Huw

      Thanks for the message. IntelliSense is one of the features in the commercial (’Developer’) edition only. The other main features that will distinguish the Developer Edition from the free Personal Edition are fast debugging and enhanced ’code browsing’ (to find classes, methods etc. very rapidly). The Personal Edition will have all the features currently available in beta 0.7 including code colouring and collapsing, Rails project management, script-running tools and wizards plus the current debugger. The main difference between this debugger and the one in the Developer edition will be speed.

      We shall announce the price nearer to the date of release. While we haven’t yet decided on an exact price, suffice to say that it is our aim to make this an affordable option for professsional developers.

      best wishes

      Huw

© SapphireSteel Software 2014