Templates with Heist

Not much going on on the JavaScript front for the past couple of days. I’ve been away at a work conference in a swanky hotel, which was nice. Came up with some very constructive things about what we build and how we organise ourselves that makes sense from a technical point of view as well as from an editorial one. Hopefully this will allow us to architect things in a way that makes more sense.

I did a little work on Heist this evening. Heist is a templating system that bears more than a passing resemblance to mason. The nice thing about mason is that it encourages you to build about your templates inside out. You generally start with the common parts of your pages and then the parts that are specific to a page. With other template systems you generally start building a page and split the common parts out into includes. This inversion of control typically results being able to remove practically all duplication within your templates.

Mason works well for static publishing (content production systems) and for dynamic applications when combined with a decent mvc framework. Out of the box, it doesn’t encourage a separation between the url that the user requests and the page that is responded with, or a separation between presentation and business logic. Mason is also Perl specific, which is why I am building a JavaScript version (actually it isn’t a straight port and it will be able to embed different languages within the template markup – including Perl). Heist will generally be used with the mvc pattern to address the url routing and separation between markup and logic issues.

Heist is about mostly written but is going to take a lot of effort to finish (one of those 80/20 things). The parser and code generator (both in JavaScript and Perl) are done. What remains is loading source and compiled templates from files and urls in the browser and on the server-side. The parser is implemented using a combination of a state machine and a collection of regular expressions (one executed for each state during the parse).

I would have liked to use a more serious parsing technology, but working in JavaScript in a web browser limits things somewhat. Using regular expressions in the Perl version will be much more efficient in Perl once I port it to 5.10. At the moment it jumps in and out of the regular expression engine as each state is reached. With 5.10 the grammar can be expressed in a single regular expression without having to continuously run different regular expressions (see this presentation on Perl 5.10 regular expressions). I guess this is much closer to the way parsing will work in Perl 6.

Heist seems pretty fast to compile templates, although I haven’t performed any measurements yet. Compilation results can easily be cached on disk or in memory (as JavaScript functions) so I am not too worried about performance.

If you want to see what Heist templates look like, see the brain dump of Heist syntax.

That’s all for now.

TestJS Documentation

TestJS is a JavaScript unit testing framework that I’m making. I am developing it because I want a test framework that works well outside of a browser (i.e. pure JavaScript, no html) and allows testing of asynchronous code. None of the existing frameworks that I could find did both of these things.

The basic interface is where I want it to be. The main things outstanding are user documentation, making it easy to extend and a few other bits and pieces.

What I’m working on right now is porting the API documentation to JsDoc Toolkit. JsDoc Toolkit is a port of JsDoc to JavaScript running on Rhino, written by a colleague of mine. It seems like a really nice documentation tool and we have recently started using it for $work_project.

I don’t want to expend a lot more effort on TestJS in the short term, as it already does what I set out for it to do – allow me to develop my other JavaScript projects test driven. What I would like to do it get API docs and user docs up on a site and make an initial release, so that if someone is looking for a JavaScript unit tester they might consider using it and contributing to it. The really cool thing about JsDoc Toolkit is that I can create custom templates for how the API docs are rendered. This means I can make a build target that generates wiki pages for Google Code, which I can check directly into the provided subversion repository next to the user docs.

Too Many Projects

The main reason I am writing this blog is because I have a few projects that I think are promising (I’m an optimist), which might be useful to others. I am trying to increase the visibility of these projects and the design decisions behind them so that people might want to use them or contribute to them. These projects are all related to JavaScript, client-side and server-side.

$work are trying to standardise on PHP for front-end development, which is a capable language if not a very aesthetically pleasing one. Perl, Ruby and Python (and probably quite a few others) would all be reasonable choices, but each is has enough issues/quirkiness to make agreement on one very difficult.

Front-end developers at $work need to write an increasing amount of JavaScript for the client-side. Recruiting good front-end developers who know any particular server-side technology is difficult. It seems natural to me to use JavaScript for building user interfaces on the server-side. The current architectural thinking at $work is to separate the server-side logic that handles user experience and presentation (the view and controller if you like) from services that provide storage and business logic (the model). A server-side JavaScript solution would therefore not need to do the more complicated things like interfacing with databases, but would just have to be able to consume services, route urls to controller actions and render resulting pages from templates – fairly simple stuff. An additional benefit to using JavaScript in this manner is that some of the code that does this stuff can be used on both the server-side and the client-side. This lowers the overhead of building accessible interfaces that have a server-side implementation and are progressively enhanced using client-side JavaScript. Finally, JavaScript is a language with a number of mature and efficient implementations that can be used from different technology platforms. If you want to use JavaScript(/JScript) with Java, .net or any language that can link against a C library, then (at least in theory) you can build your server-side code for dealing with user interfaces using JavaScript. While the current direction at $work is to have the code that deals with user interfaces on completely separate servers to the rest of the application logic, it would be nice to have to option of disparate technology stacks using the same reusable user interface components.

There are a few projects out there that are moving in this direction. Steve Yegge (someone who shares a lot of my technology preferences) has done some interesting work creating Rhino on Rails, although this hasn’t been released for public consumption yet. It sounds like this is a faithful port of Rails, including lots of stuff to make JavaScript more like Ruby. While I like Ruby and I like Rails, I want to create something more JavaScriptish, whatever that is. There is also Jaxer, which looks pretty good.

The main reason that I am writing new code, rather than basing it on an existing technology like the ones mentioned above is architectural style. For me software should be architected in small discrete chunks, allowing any part to be swapped out to encourage competing implementations (these are not exactly new ideas). On the decision between library and framework, I like to have my cake and eat it.

This is a lesson that has been hard to learn for me. I have a large Perl MVC framework that I started developing before I had heard of Catalyst or Rails (probably about the same time that they started getting developed). The after literally years of work, I have not yet managed to get it to a state where I am happy to unleash it on the world. It powers a few sites developed by some friends and I fully intend to get it to the point of release one day (it does some pretty cool and unique things and has a nice layered design). If I had the time again, I would have gone down the route that Catalyst has taken and released early and often to gather a community and gain momentum. This is my plan for my new crop of projects and the point of this blog.

The projects that I am working on are:

• TestJS – a JavaScript unit testing library that enables testing of asynchronous code in both the browser and other JavaScript environments.
• JSAL – an asynchronous JavaScript loader and module pattern that handles dependency resolution.
• Heist – a JavaScript templating language heavily inspired by Mason.
• modjavascript – a JavaScript apache module allowing dynamic request processing in Apache using JavaScript.

There will most likely be an extremely lightweight MVC library to accompany all this, once I have these supporting technologies worked out.

The posts that I intend to put on this blog will go into more details on each of these libraries to explain their level of completion, where they are going, what I wanted that wasn’t provided by existing stuff and their general architecture. I expect the order of these posts will be the same as that of the list as the earlier ones are the most complete. If you’re looking for production ready software, it might take a while – depending if anyone shares my views or just thinks I’m bonkers for wanting to put JavaScript on servers (the majority of reactions I have encountered so far). If you think what I am doing is silly or pointless, I fully understand your point of view and don’t need a comment to tell me that you hold it. Otherwise, drop me a line.