update

Again, I have let the blog slip instead of keeping up with it. I have been busy with the development of some tools for a new site. After we finished the 4-week version of the last site, we decided to make some major changes to the way it works and invest a lot more in the architecture of the application.

So I’ve spent about 3 weeks developing a javascript library for the new application. And I decided to, from the beginning, test the js file in a set of browsers: FF2 & 3, IE 6 & 7, Opera 9.5+, Chrome Beta, and Safari 3.1 Win.

It has been a bit of a learning curve to get used to very short development cycles where you have to build one small lego-sized piece of your library and then test it in all the browsers, refining over and over until all of the browsers show the same thing (or at least close enough in all the relevant cases). But as I got into it, I learned how to approach the problem and get a handle on the complexity of all of the different constraints that come from each browser. As I just mentioned, I learned that it worked the best for me to build really small “lego” sized test pages and run them through all the browsers over and over until I understood what each one could and couldn’t do. Really, all it requires is patience until you get the hang of it. So I have one completely separate development environment (project environment) where all of the js development happens, and then I move the javascript over to the mainline source on a semi-regular interval. This allows me to isolate out all of my test pages (and there are lots of them) into the javascript test environment.

One of the things I learned from the process is that of the browsers in my above set, I really like the way javascript debugging works through Firebug/Firefox. I already liked Firebug, but I did not know about the profiler built into Firebug until I worked on this project, and it has made it easy to learn many new things about the performance of different operations in Firefox. Second place goes to Visual Web Developer Express (I’m a PHP developer instead of .Net).

With the profiler in Firebug I was able to take one of my test pages and trim the load time down from 1.2 seconds to about .6 seconds. (This particular page is a unit test for my date/time functions.) It was exciting to be able to quickly see where the performance was going, and really make a few minor changes in structure to speed the test page up.

To test the suite of browsers, I setup a virtual machine on Virtual PC 2007 (microsoft) and downloaded their hard disk images here. Then I installed my test-versions of all the browsers I wanted on the virtual machine. This way I can boot the virtual machine, run any browser tests I want and then close it and “erase the changes” so that the browser cache and all the settings are cleared back to their original state. This makes the development process a lot easier for me – no cache clearing, cookie clearing, etc. And I have all of the versions of browsers I could want to have, available to me when I need them.

Right now, I’m working on a new demo section of our site where we will place some of the better prototypes & mockups to be publicly viewed. Up to this point all of the development has been kept internally because it took too much effort to polish it up for the public. But now we need to start putting our work out there so that people can give us feedback and criticism. And hopefully get a job out of it?

So I’m going to go back to work now. I hope that I will write more often, I think it would be good to discuss some of the things I learn about the different browsers as I go.

version 2 phenomenon

I remember reading several months ago about the version 2 phenomenon, where when you try to take your application into version 2, the scope blows way out of control. Instead of being disciplined like you were in v1, you want to include all of the features you have thought about and wanted to make a part of the application. There are so many things that you can include in an application. So many ways to take it, perspectives on the data, and ways people might approach the problem.

As I plan the next phase of this application, I see it happening to me. It’s interesting, because the first version went so smoothly and quickly. I was hyper sensitive to the amount of time that implementing any single portion of the application was going to take, and cut anything that wasn’t going to be *very* simple. And I had a long list of things in my idea box, but they just stayed there.

Now that I look forward into developing the next iteration, the first one went so smoothly that I want to include all of these things. The ideas become must-haves and the interface becomes an opportunity to “try a new trick”. But this can’t happen. It’s the single biggest project killer. I have to deliver, and I have to deliver on a disciplined time frame.

I have more time to develop this version, and I think that because I do, it’s allowing my scope to expand beyond what my real schedule allows. I haven’t nailed down a schedule, and that’s part of why the application is growing like this. I’m caught between “build an application that solves the problem effectively” and “do it in a reasonable amount of time”. I have to choose what the important features are and then choose a schedule that allows me to get those done and get them done well.

Most of the things I’m thinking about right now are really chaff anyway. Only a small part of the application is really wheat. But which part?

Right now I’m writing and rewriting specification & design documents, done by hand. Start with a blank piece of paper, sketch out the features, think about the user, design the application flow. Add a feature, remove a feature. Crumple it up and start over. Got to get this down to the important essence and make an application that reflects this understanding.

4 week project

I think the scope of the project I am working on just radically changed, so I wanted to post my stats so far. I finished the prototype of my target functionality and solidified the interface. There were a couple of points that I took the “easy way out” so that we would have a working prototype without trying to prematurely optimize the application for the wrong problem.

Here’s the stats:
3 weeks
3,800 lines of php (new) / 5,200 (total, including zx toolkit)
2,100 lines of html & javascript
50 pages
38 webservice functions
11 database tables

The site is not available publicly, but I can make it available to anyone who is interested. I don’t consider the site “done” because there is a lot of testing work that needs to be done to call it finished. But as I said earlier, the scope of the project has just changed significantly. That’s part of why I wanted to put the application together quickly, because as soon as the stakeholder(s) saw what was possible, they wanted more, more, more. So now I have to go back and see just what we think we can accomplish with the application and try to take another run at it.

I wish there were an easier way to get feedback and to communicate, but my experience tells me that this is the most effective way to communicate an idea (as an implementation) and then toss it or do a serious overhaul once the real issues come to the surface.

I have to say that it is terribly frustrating when you walk into that meeting and all at once the thing that you have invested so much time and energy into is blown completely out of the water. I think very holistically, so I see the application as a whole and follow the user(s) in my mind through the entire application. When the scope changes, it sends me back to a blank piece of paper and makes me start all over again. There are things that come more quickly the second time around, but it still sucks. I just want to see that it is the best that it can possibly be.

update

I’m going to be refocusing my efforts for the next several weeks. I’ve enjoyed working on $two, but I need to put it down for a little while. I’m working on a new site and I’ve given myself 4 weeks to finish it. This site will take 100% of my time until it’s finished. I may have the opportunity to improve $two as I work on the site, but it will depend how things go.

There are several small changes that I have made to $two, including a change to strip out HTML comments from the script tags (thanks Mathieu). Those changes, plus some more that I have in mind will be released when I get back. There are good things to come on this project!

My four weeks started this Monday, so I’ve only got 3.4 weeks left!

datagrid demonstration, updated two.js

To demonstrate the capabilities of the native javascript templating library, two, I’ve been hard at work on an all-javascript datagrid control. The demo page is available here.

I built the datagrid out of the desire to see just how well the $two library could handle a sophisticated page and if it would remain manageable from a coding & maintenance standpoint. I’m generally happy with how it has turned out, but it’s still in very early stages. There is a lot more work that needs to go into organizing the code.

Caveats aside, I am quite happy with how the template fragments (what I call assets) worked in this situation. The ability to componentize the HTML made it easy to render the whole page and then render sub-components when only parts of the page change. Since the assets can be treated like a function call (with parameters) it seems easier to keep up with the different parts of the HTML and the arguments that the HTML template uses.

Also, I made a few changes to the library to integrate with jQuery.UI so that I could use their dialog control. The library now has a function to render directly to a dialog and show it.

The datagrid is in early stages of development, so I’m sure there are still bugs in it. But I wanted to go ahead and get it out there “as is” so that you would have the opportunity to comment and make suggestions.

two, v0.1.05 alpha

As a quick follow-up to my last post, I’m releasing the work that I have finished up to this point on the native javascript templating engine as is. This way I can begin gathering feedback to see if this is a problem that other people in addition to myself have.

The general idea is this, instead of trying to keep the html in one block, it is instead broken up into a bunch of little pieces (html template fragments, or what I like to call, assets). A sample asset might look like the following:

<script type="text/html" id="asset_sample1" two:templ_args="param1,param2">
  <span><%=param1%></span>
</script>

The above defines an invisible piece of html in the document that can be used by the $two engine to generate html and insert it into the document. The template tags <% and %> define regions of javascript within the template.

Also, since the assets are compiled into javascript functions, they have a list of parameters to make it easier to remember what variables are available to an asset. These parameters are defined in the "two:templ_args" attribute of the script tag. When $two.render(…) is called, the arguments are passed into the function in the same order as they appear in the .render call.

Assets can also include other assets through a function made available within an asset “include()”. include has one required parameter, the asset name, and any number of optional parameters that will be passed on to the asset as it is processed.

This is a rough sketch of the capabilities of the library. I suggest checking out the examples page. Right click, view source to see how it works.

clouds on the horizon

It’s been several days since I have written, I’ve been working steadily through it. There are some things I need to put down, though. As I dig deeper into PURE (and my project based on PURE, called one), I realize there are some things that I really don’t get about it. Or I should say that PURE/one does not get about the problem that it is solving. So what I want to do is return to my post earlier where I listed 3 points that I like about pure and discuss them again.

One point in particular needs to be revisited. And that is point #1. Part of the promise of PURE is to leave the original HTML alone and just apply the directive/data combination to it – voila – cookie cutter web pages. But there are GOOD REASONS to split up HTML in a page after the original design is done. For the most basic, think about the standard headers and footers on your pages. These elements should be separated from the rest of the content – why would you want to change 45 HTML templates just to restructure something in a header or footer?

Just as a program is assembled out of very small building block functions, called in successively more complex structures and solving successively more complex problems, HTML pages are constructed out of lots of small building blocks. There are elements repeated across pages; some large elements, some small elements repeated on all pages, some elements only repeated on certain pages. But no one wants to have to deal with going through all the source to find every instance where you use “the gold star” so that you can make it “a gold star, that is a clickable hyperlink”. If you componentize the HTML then this change can be made in one place, and that change included into all of the places that actually use it.

I ran into these problems as I tried to make a grid control in native javascript using the pure engine. I realized very quickly that it made sense for me to have “assets” (little fragments of html) hidden in the page that I could pull out for particular pieces of the problem. But these fragments are *my work*, as opposed to the work of a designer. I split the pieces out because of the way the grid is constructed – it has nothing to do with how the grid appears, is colored, or lays in the page. Construction is a programmer problem. See the grid demo page. Warning! The work is incomplete and will not be finished in this form.

As I considered this problem, I realized that PURE does not model this problem the way I see the problem in my head. How do I see the problem in my head? To take my metaphor from earlier a step further, I see the components of an HTML page being just like the building blocks of a program. That is, you can think of a web page being constructed by tree of components included within each other, just like a call tree of an application. Now, if this seems to abstract right now, I’ll try to clarify.

What I’m trying to say is this, that when generating a page, you start with the core template (the “main” function if you will). And “main” calls a series of sub-templates that are contained directly within it. This continues until all of the content has been included. This looks just like a call tree for an application. “main” has a problem to solve and it calls the appropriate functions to generate the correct result.

Now why am I seemingly bending over backwards to explain templating in this way? Because I think we can create a native javascript template engine that uses “assets” in such a way that every template/subtemplate construction acts much like a function call. And that maps perfectly into the way I think about the problem as I construct a page.

I’ve used smarty extensively in the past, and I always found it hard to keep track of what variables were available across “include” commands. And it was hard to see what the actual inclusion structure was when you had to open several documents together and try to follow the logic.

So in a way, I have been thinking about this problem for a long time, and only now did the answer become clear to me – in an accidental way. So, I’ve been experimenting with ways of representing this problem with more clarity. I have implemented a rude tool for generating functions in native javascript that can be used to simply created hierarchies of “calls” which generate html documents. I like the way it works so far. I restarted the grid project that I was working on in PURE, in my new tool, and I like it. I will release what I have soon so that others can provide feedback from their own experience. I’d love to hear what I got right and what I missed.

Edit: Points #2 & #3 from my earlier post still stand as they were stated originally for this new form of native javascript templating.

javascript templating, prerelease 5

update
OK, I’m releasing my up-to-date work. Please see my latest demonstration page. If the page still shows the old demo page, refresh the page (CTRL-F5) to reload it. I set the no-cache headers so that won’t happen in the future.

new features
See the demonstration page, at the bottom, for the new syntax rules. The new feature set allows lots of cool things:
1-append/prepend html into selected nodes
2-using “assets” (hidden fragments of html) to build templates
3-modification of css, modification of attributes, and binding of events is now available
4-more convenient syntax for selection and/or replacement of element(s)
5-new ways to move,clone, or create new elements
6-dynamic names allowed in more places (like attribute names can be dynamic using [@data.attr_name])
7-stock assets are available (lots more to come!) They can be grabbed with names like “input:text”.
8-new “foreach @data.list -> @item else” command
9-new “empty” command to remove children from an element

future features/enhancements
There are a couple of features that are missing:
1-a nice “if, else if, else” construct.
2-some kind of ajax loading of data and/or assets. I haven’t decided what is appropriate. Are there any suggestions?
3-integration with thickbox (or something like it) for creating dialogs?
4-better error messages

It’s interesting, as I work on this, I realize that a future enhancement may be to write the directive as a language instead of placing it within the constraints of the js object notation. I could see making it more flexible and readable by doing that. But right now that’s just a thought in the back of my head. Does anyone have interest in that? The library could compile the “template translation language” into a javascript function that could be stored in a .js file, or interpreted on the fly.

development continues

I’ve been working hard on my adaptation of PURE, new syntax and new capabilities unreleased in my prior version. I read the comments over on Resig’s site and have been trying to find the common ground among the many viewpoints to create a best-of-class implementation. We’ll see how it turns out. Right now I’m exhausted.

I ought to have something up by the end of the week.

Why I think PURE is a big deal

Why am I interested in the PURE project? Why is it important?

I think that PURE is a big deal for several reasons.

1) Simplification of development
On projects that I worked on in the past, an artist/designer would work up a design for each page and when the team was happy with the design, it would be handed over to me to make it “live”. I would take the html and move it into smarty, breaking up the pieces into smaller templates and taking the repetitive portions (header/footer, common grid or table formats, etc) and splitting them out. Then I would write the include logic to properly build a page. That’s all great, until someone decides that they want to change the site’s design. Oh boy. All of a sudden the designer’s work makes for a rework on my end.

The potential I see for PURE, is the ability to create sets of templates that can be designed by the artist/designer and then used just as they are. If significant changes are made in the functionality of the page, then obviously, I get involved, but if things are named consistently and tables/div layouts stay in a consistent form, then the directives (a map that PURE uses to take a JSON object and place it in an HTML document) do not change. I’m excited by this possibility.

2) Single page (or 2 or 3 page apps)
There is also significant potential for PURE to make single page apps more feasible. With the right kind of utilities built into the PURE library it could become a tool that pulls JSON data (via AJAX) from a webserver to process data, but uses hidden html in the document to swap “views” or pull portions of html in that same AJAX call for rendering. The potential is exciting for creating a more “desktop-application-like” environment on the browser.

3) Separation of data and user interface (BIG!!)
This idea is a little bit larger in scope than the others, but I think it’s the one that gets me the most excited. There is potential that PURE could help to split the user interface from the application data. Because the HTML is loaded from one source (HTTP/AJAX) and the JSON is potentially loaded separately, you have an interesting possibility for decoupling them.

Out with crappy microformats. Out with screen scraping code. Now applications can access other applications’ data directly without the interference of user interface coding (or change). If you have a site that wants to use soccer scores to calculate some betting statistics, why pull HTML and regex out the data from the HTML when you could pull the JSON data directly – and use it natively?! The potential for cross-application communication is fantastic.

Also, say I write an application. We could take a wishlist application as an example, like greedyme (A really cool site, BTW). The site lets you login and create lists of things that you want. If the site were developed with PURE, you could replace the user interface of the application. So you design a new interface AND USE THE SAME UNDERLYING JSON CALLS FROM THE ORIGINAL SITE TO HANDLE THE BACKEND. What just happened? The interface was redesigned by someone else, but the backend remains the same. Would this happen? What do you think?

In a way, it’s like developing a user interface in HTML that uses a back-end API. Develop your API in JSON, and use it in your own HTML application.

I do think there is risk in the equation. The risk is that with the data being so easily accessed, that data-stealing just becomes easier. But who knows – it depends how much benefit comes from being able to design the backend entirely separately from the front-end.

What do you think could happen?

I have created a very simple demonstration of capability. Take a look at the source to see how it does what it does.