Sapphire In Steel

I’ve now got the basic debugger more or less sorted. The next step in to build an ‘expression evaluator’ so that the watch window will work. It doesn’t look too bad, but it’s another half dozen or so COM interfaces to implement. The problem with this stuff is that it’s so low level. But to be honest, I can’t figure out a way Microsoft could have done it and yet retain the flexibility required to cover almost any situation.

For example, breakpoints aren’t just a simple ‘stop here’ sort of thing. You can have many breakpoints coming off one ‘pending’ breakpoint – breakpoints that only fire when a particular condition occurs, say. The these can either be ‘bound’ or in ‘error’. For a simple debugger like my current one, this is overkill, but you still have to abide by the Visual Studio model.

While doing the debugger, I did some research on existing Ruby debuggers. The results were quite interesting. The standard Ruby debugger, debug.rb, apparently executes around 100 times slower than running Ruby code without the debugger. Now that’s simply horrible. I wouldn’t want to debug any serious program with that. Active State have improved on this with their debugger to around 50 times. Better, but still unusable for a large program, I would think. Both of these debuggers use the tracing mechanism built into the Ruby interpreter to get a hook into the evaluator event source. However, it’s a pretty unsatisfactory way to go about things.

The last one I looked at was Arachno Ruby. This seems to have a 50 percent slowdown. That’s more like it! The problem is that Arachno has had to patch the Ruby interpreter to get this to work and patched interpreters are bad news. Not only are they non-standard, but you have to maintain them as well. Arachno currently doesn’t support 1.8.4 for example.

However, I’ve figured out a better way of running the Ruby interpreter in a fast debug mode without patching it. The trouble is, it isn’t at all easy to implement. The current Steel debugger (beta 0.6, not yet released) is just using the original slow debug.rb. The fast debugger will have to wait for the commercial version of Steel.

I had a good look at Arachno. Nice ideas there but there’s something slightly amiss. Say you want to write an IDE for Ruby, Perl, PHP and Python. Ok, ambitious – but doable: Active State have something there along those lines; probably taken them well over 20 man-years at a guess. Now let’s add in the fact that you want to be able to run it on Windows, Linux and Mac OS. Hmm, that’s challenging: making things portable is hard – just look at the work that’s gone into Ruby. So what do you chose to write it in: GNU Eiffel and some open source graphics package?? Now, that’s the heroic school of programming. The result (I’m guessing really) seems to have been that Arachno has spent a fair bit of time getting these wonderful open source things to work properly. That’s great if you have the time, money and inclination (personally I don’t). Still, I wish Arachno well.

Building an IDE of any sort is a big job (far more than one person can reasonably undertake). That’s why I’ve decided to stick firmly to Visual Studio – and that’s difficult enough. It does limit the scope to Ruby on Windows, but I’m happy with that for two reasons. First, I think it’s doable by two people in a reasonable timeframe. And secondly, there doesn’t seem to be any money in Linux products. Over and over again, I’ve seen companies produce stuff for Linux (Borland with Kylix is a notable example) only to withdraw after a short while. It’s true that a number of people have made money out of support for open source products. But I don’t know of many that have made money out of the raw product itself.

It’s taken me a good five days to get my head round the Visual Studio Debugger. For my first attempt, I tried to trap the Set Breakpoint command and go from there. Nope. Not even close. The debugger is a lot more complicated than that. It’s not made any easier by having documentation that’s aimed at a C++ implementation rather than a C# one. After a fruitless day spent trying to hack it in C#, I worked through the TextInterpreter example (which doesn’t work properly in the beta VSIP SDK I’m currently using). Even though it’s in C++, I got the general idea and was able to build a managed version reasonably easily - I’d definitely start with the C++ tutorial if I was doing it again.

The main problem with the debugger is that to build a simple (let alone one that lets you ‘edit and run’, say) you have to implement about ten COM interfaces. Then you have to build some sort of state machine on it’s own thread - plus any worker threads to handle the input and output to the Ruby interpreter itself. Each of the COM interface names starts with ‘IDebug’ so there’s IDebugProcess, IDebugProgram, IDebugProgramNode - and so on. Damned confusing. Then each interface has about ten methods and while they don’t all need coding, they need to be considered. It’s not a small job - and that’s before you get to any Ruby stuff.

One thing that struck me as I was doing the C# bits after the C++ tutorial, was how much faster it is to code in C#. Even with the ATL wizard, it still takes a lot longer than implementing the same interfaces with C#. I have to say that I’m coming round to the idea that Microsoft has done a reasonable job with COM and C#.

In passing, I found a tricky problem with the basic  IDE. The Steel splash icon had vanished (this appears on the Visual Studio startup splash screen along with C#, VB, etc.). The thing was, it had been visible in the past - but no longer! It turns out that the splash icons and menu commands are installed when you do a ‘devenv /setup’ command. But setup works differently from when you just run the IDE. In particular, the DTE automation isn’t initialised. The IDE setup sequence calls the package ‘Initialize’ command before the splash icon is loaded. If you try to access automation in the package Initialise, an exception is thrown and the icon loading is never called! Worse, nothing tells you this interesting fact. You have to turn logging on and look in the log file. The solution is to test for a null DTE in Initialize: if you are running setup, this is null, otherwise it’s set to something sensible.

I’m a big fan of VSIP, but it sure isn’t simple. There’s just a lot of stuff to learn.

I’ve decided to take backups seriously at last. I must be getting old.

So enter one new server from Novatech (a good company to buy from – never had any trouble, the stuff is pretty cheap and they are fast). Then comes the tedious job of installing Windows 2003 server and all the other bits. I’ve decided to try the new Microsoft Team Foundation Server so that I can take advantage of the ‘changeset’ features in the new Visual SourceSafe. I’ve never thought very much of VSS, but the ability to get a set of software files out as a ‘transaction’ looks to get good, so I’ll try it out.

Installation hasn’t gone smoothly: I’m on my second complete re-format of the hard disk. Some of the problems were mine – (RTFM!!), but some seem to be due to the fact that TFS need everything to be just so, or it gives up with inscrutable error messages. The real problem is that it relies on IIS, SQL Server, .NET and SharePoint – all of which have to be configured absolutely correctly with the correct accounts and correct priviledges. Unless you follow the instructions exactly, this is likely to be zero.

Anyway, after a meticulously performed re-installation with all service packs, hot fixes, etc. everything  worked and I’m ready to start. Just goes to show – never go near any software until the first service pack comes out. Also, the licensing restricts the number of users to 5 for the Workgroup edition. It gets expensive after that, but if you’ve got more than five developers then your spending serious money anyway

A further point: Visual Studio goes a lot better with 1GB of memory. I snaffled a ½ GB from the server to see if this would improve my workstation. It did and so I’m going to get a further 1GB for the server. It seems you can’t have too much memory.

On another note, I’ve finally got bracket highlighting & matching to work in Steel. The Ruby def…end type of things now also collapse correctly, though there seems to be a problem with the IDE ‘remembering’ outlines from a previous version. God knows where its getting them.

It took me a day or so hacking through Microsoft’s class library source to fix a couple of bugs that were preventing the outlining working ok. All seems reasonably fine now. Again one of the delights of using freshly released software – though Microsoft’s is quite a bit better than a lot of stuff (including mine).

So – the next job is adding Ruby debugging into the Visual Studio IDE. Should be fun!

I’ve finally got a LL(2) parser built for Ruby. It’s taken me about a month to do this from  knowing zilch about parsing and less about Ruby.

I had a couple of false starts. I started off by using Flex to build a component for Visual Studio colourising, but decided against using Bison and the existing YACC Ruby module since it tied me into using C/C++ for the syntax analysis; that wasn’t a direction I wanted to go.

I then tried using a free C# YACC toolset. That had a number of problems. It didn’t produce the same grammar as Bison and was slow to compile. I also learnt the hard way about the grief you can get when you have to try and debug a YACC style grammar: state tables might be fast, but finding out what’s wrong is a nightmare.

Then I came across Antlr. Magic! Absolute magic!! It produces easy to debug code – you can see what’s going on – and it has intelligible diagnostic messages. I love Terrence Parr’s sense of humour too – “Why program by hand in five days what you can spend five years of your life automating?”. A man after my own heart.

Ruby isn’t easy to fit into an LL(2) scheme, but it can be done - with one exception that I’ve come across so far. I’ve managed to parse nearly all the Ruby test modules so things are looking good. The next step is to bolt it into the Visual Studio/Steel IDE.

Parsing is quite fast for the most part. The main problems come where I’ve had to use predicates to resolve ambiguous Ruby syntax. Syntax-wise, Ruby simply sucks.