The DRM free web and innovation

My recent "blue sky" article on my perceptions of the importance I place in the user script field has received skepticism and optimism alike from peers, through varying channels.

As much of what I write, it wasn't a very focused post, and I got right into the deep specifics, rather than giving an overview perspective from 10,000 feet. I thought I'd try for that tech top-down overview in this post, instead.

The web is one of the precious online environments which in itself is still DRM free, and I can't place enough importance in that fact. A lock-in free environment, as open standards and data formats go, allowing HTML, CSS and javascript introspection. You can read the source, you can rewire it, and nobody will hit you on the head for doing either. You are actually encouraged to pick up on the tricks of the trade, and there are flourishing online communities available to help you learn, and get going on your own. The web wants you to participate, and opens its doors to user innovation.

User scripting, and end user innovation in general, is reshaping the web, if still only ever so slowly, a trickle here and there by a small clique of us, the übergeek web populace. (I might even reach large parts of that clique with this blog, at just a few hundred subscribers.) We pioneer a field of choice, comfort and preference, as search engines once pioneered a field of findability. Finding a page by typing its URL is about as user friendly as how well adapted the behaviour of a typical web page is to what you want to do with it, and as we could opt out of the former, we can also opt out of the latter.

To me, user scripting is one of the many things that make the web better than, say, PDF or Flash media. One of the many degrees of freedom that are opening up to end users. A gradual shift of control back towards users. Blurring formerly very strict areas of influence, much like how the past ten years has seen web pages with comments by others (besides the site owner) become commonplace, where previously a web page was strictly uni-directional medium of publication. Where previously we had a one author per page environment, we now have a multi-author environment. Where layout and functionality was previously at the sole discretion of that one author, now the tools and choices are at your disposal, too.

Uncoupling such decisions from authors and focusing on getting the the signal through for you to consume in any fashion you well please is where the web was aiming from its very earliest days, but we are only starting to get there very recently.

We could compare user scripting with the configurability of text editors. Some people take editors for what they are at face value with default settings, at most changing to theme, color or font settings they are more comfortable with. They learn a small feature set that allows them to do what they need, not wasting any time to climb the learning curve any further. Others practically meld with their editor of choice, even making it a well integrated part of their lives; socially and otherwise.

I find myself in this category of people, in a web context. At present, it's mostly new levels of freedom for us hard core geeks, though the tools or shortcuts we invent can be applied with ease by anyone who comes across them in our wake. Sharing this has become a lot easier the past year.

I see the web edging closer to this thriving environment of choice, as browsers have recently started to skew in the same direction. Lowering barriers to get your hands dirty, and actually participate in the look, feel, shape and workings of the web we tread. I love this trend. It is a key trait worth guarding closely and inviting more people to. Some browsers, like Flock, are about remixing the web at their very core, responding directly to use case scenarios like "I want to share my pictures with my friends, and subscribe to theirs", remixing in chrome what user scripts remix in the HTML layer itself.

At present, it's just a few devs and savvy gourmets of the technorati that play with this, but that is natural. While the tech itself is already mature in many respects, the community following around it is still picking up momentum. It's not yet trivial to harness user scripting as a typical J. Random User, but will get easier.

The key thing about user scripting is that we never have to stand in line in any feature request queues just to flex our ideas on how to reshape environments we did not make ourselves. Given some knowledge of the workings of the document object model, javascript and the user script environment, we can adapt any web page, without any help, affiliation or other relation to the page publisher. The power of openness, freedom of choice, and the lack of DRM.

I believe in a strong future for any medium that embraces end-user changes. The web has this trait, and this makes it so much more important and full of possibilities and life than any other media I can think of, save perhaps the physical world around us. That is the kind of heritage I think we should be proud of leaving for those who come after us. A puppy we can keep teaching new tricks, and revel in both doing so and in harvesting the benefits we gain with every new step we take along the way.


Pimp my site

Picture some community site where you hang out socially, talk to people in the open, or by mail, perhaps keep up with personal diaries or play some web game together -- that sort of thing. Any will do, be it MySpace, LiveJournal, The Kingdom of Loathing, Flickr, DeviantArt, some dating site or whatnot.

Most likely, being the savvy netizen you are, aware of the pros and cons of at least a handful of communities all over, there are are a few things you are less than happy about with this particular community, be it the cluttered visual layout, badly picked coloring or graphics, a cumbersome structural layout, enforcing several more clicks to get from A to B than you would prefer, a shortage of omnipresent indicators of new messages, or similar. Maybe you are even so used to the status quo, that it no longer bugs you.

This is today's norm of and on the web, whether you slap on a 2.0 tag on it or not.

Some sites do a really good job of keeping things minimal, readable, accessible, well organized and generally beautiful; 37Signals for instance, usually delivers seriously usable web sites it's a pleasure using. Most community sites are not, however. Some of the biggest are downright hideous beasts. Painful sights! Annoyingly cumbersome. Backwards. The kind of thing that makes you want to rip them apart, strip them to the bone and refurnish the whole shebang from scratch.

And lo and behold, with just the one tool of magic, it's actually possible too: enter the user script.

By now, anyone who has followed this client side javascript applications blog or been running Greasemonkey for a while from the comfort of their Firefox browser, should be aware of just how simple it is to toss up a user script to tweak some little aspect of a web page; add some missing functionality here, drop an ad <div> tag there, fix a favicon, short-circuit some links, and similar. A few lines of code here, a little touch there.

"I want things my way!"

In all, it is a bit like configuration options for the web, as a whole. The user script is all about controlling your environment, having things your way, whichever way that might be, and about being able to easily share it with others. Being in control of their environment makes people happier. Stylish, similarly, does the same thing for user CSS, though without doing quite as much for the easy sharing bit (so many, like Jasper, use user scripts for doing CSS only fixes like that, too).

Scraping off the mold

Most community sites that keep reminding me of their shortcomings, in the face of what other places show up, have not changed much the past five or ten years. Their look and feel remains just what it always was. Dead code and markup, though, as sites, still as vividly full of human life and critical matter as ever, hence keeping its leash on still growing user bases -- despite the flaws that once worked their way in there and were left unfixed, to rot in blissful peace.

As a somewhat impatient client side hacker, I'm growing ever more weary of those functionally, visually and aesthetically challenged web sites I use, where I know all to well that there will hardly ever come a site revamp, lest I give pigs wings or devise ways of adding Thursdays to the week. On the other hand, nothing stops me from making that site revamp on my own. I'm still shaping and exploring the idea.


The recipe is about the same as designing and rendering a community site from scratch, but with a few twists:

  • we have a fix server side feature set,
  • we have a live and fully populated site from day one,
  • we start out with a large user base,
  • and we can perform any changes to the live system without ever affecting any of them
So basically, what's left to us to do boils down to:

  • deciding on a (server side) feature subset of present features,
  • mapping out said subset server side for attaching to it,
  • performing the visual and interaction design for our pimped version,
  • and assembling it, atop the server side plumbing
With Kingdom of Loathing, I did most of the above before the age of the user script, save actually replacing the original plumbing; I stuck to injecting alternate ways of interacting with the server side machinery. What made this viable at all for a bookmarklet in the first place was the site frameset, where you could keep browser state even across pageloads. With a user script, you either:

  • opt to keep client side state by way of provisions like GreaseMonkey's GM_setValue() and GM_getValue(),
  • make the entire site a one-page application, resorting to xmlhttprequest for all site round-trips,
  • add a frameset of your own,
  • or settle for some combination of all of the above
Let's say we end up with a generic, perhaps a bit large, feature set such as this:

  • personal bio:s,
  • public (guest-book style) messaging,
  • private (mail style) messaging,
  • friend lists,
  • publishing (bios, text, art, music),
  • browsing (reading, viewing, listening to),
  • search,
  • subscriptions,
  • and live notification of news (subscribed data and messages)
For practical purposes, I believe the ideal mold would be a mix with a frameset for keeping the live notification page, and then rewriting the rest of the pages on the fly on load time, maintaining the URL namespace (subset) intact, where your design adheres to the same content structuring (but not necessarily visual form) and adding made-up pages of your own in uncharted URL namespace (rendering page not found errors you siulently replace with wanted content) for those that do not.

Alternatively (and probably better), your pages synthesizing (via xmlhttprequest) site bits that were structurally apart on the original site, can be modeled on some small page using the URL fragment (never seen server side) for your needed additional name space. For instance, assuming there was no notification administration page, you might pick the tiny /about/ page (to save the poor server from rendering you a 404 or something potentially lengthy) and render different pages of your own for /about/#subscriptions and /about/#messaging instead of giving them URLs like /subscriptions/ and /messaging/.


This kind of extreme client side orchestration can, if you want to, span and mash up multiple sites with one another, much like how the 43* sites (43things, 43places, 43people...) do, but irrespective of prior affiliation. Theoretically, you could make one mother of all meta site, feeding off multiple community backends, all at once. A community version of a feed reader, aggregating environments instead of feeds.

This is all very blue sky, but on a broader scale, I think it makes sense, and hope it might be what parts of the web are slowly moving towards. The web, as a publishing medium, just isn't as interesting as the web as a services medium, where we introduce loose coupling between service provider and service rendition, affording end users a greater amount of control of their online environment and experience. Monopolizing control over user interfaces leads to the same kind of stagnant environment where the few giants in control of the large amounts of users don't continuously evolve because development costs them money. That is neither good for them nor their user base.

User script technology is a vital force that can help uncouple the web even more than it has been to date, lowering contribution and innovation thresholds for people who know better than the big guys what they actually want, and how they want it served.


A trip to the US

I'll be visiting San Francisco May 30 through June 4, talking to people, seeing Jesse Andrews, perhaps bits of the neighbourhood, and most likely taking part in some local hackery while at it; that kind of thing.

If you are in the region and would want to have a chat with me, this would be a good chance to. It's my first trip across the Atlantic, and I am presently busy picking out some good reading to spend the 14-hour flight plus transit, thinking up things to bring and not to.


Submit forms with Ctrl+Enter

Ever wished commenting something on the web was just a matter of typing some in a text area and hitting Ctrl+Enter to submit it? A few weeks ago (after a period of rather heavy forum and community browsing) I grew wary of the many places where this was not the case, and wrote up a user script to make it so, everywhere, as it's one very nifty feature. Especially when it's become a natural hard-wired browsing behaviour of yours; the surprise when nothing happens is, at best, mildly amusing the first few times, but just annoying in the long run. So, here it is: direct install link and userscripts.org entry, respectively.

At first, I thought I should publish it immediately, but then I thought I would wait for a while and survey how well it worked in practice. After all, scripts written to be run on all pages across the entire web may once in a while end up interfering with things they should not, or mysteriously fail.

It turned out my cautious suspicion was right, and wrong. I have not had it break anything yet, but I have come across at least one poorly written community site where it did not work. You might too, and if it's annoying enough for you on some place you frequent often, you might want to adjust the script to compensate, as I did; the source should be rather straight-forward and readable; just add another case in the on_ctrl_enter function.

In case you do, feel free to drop me a comment with the fix here, and I might try to maintain a catch-all version, mending the web.

What the script does, is to add an event handler to all <textarea> and <input type="text"> tags, which swallows up all Ctrl+Enter events. On encountering one, it runs a potential form onsubmit handler, when in place and does nothing if it said the form was not ready for submission, or submits the form, if it was.

In the vast majority of cases and on all well-behaved, web standards compliant sites, this does just the trick. There are, however, ill-behaved sites where the buttons you believe to be form submit buttons, are in fact not, but rather javascript: links or buttons with javascript onclick handlers that perform some mandated book keeping, updating some required hidden fields in the form, and then performs the submit, which this script will not do, of course. Fortunately, as this kind of bad practice takes additional work to implement (besides requiring a javascript enabled browser to work at all), most sites do not.

The attentive reader might also have realized that the script will not work if you, say, tab away to focus some non-text-entry field prior to hitting Ctrl+Enter or click somewhere in the page first. This is intentional; in part, because my typical use case is "just type away and instantly submit", so it would be a waste listening for events elsewhere, in part, because there might be multiple forms on the page, and there is no other painless straight-forward way of detecting which form has input focus. When listening in on specific text entry widgets, we do know, though.

Share and enjoy!


New edition of Javascript: the Definitive Guide

Good news! David Flanagan recently announced the fifth edition of his book Javascript: the Definitive Guide, something I've been eagerly awaiting for a year or two now, being the only book about the language to date that I have found worth buying. Amazon is accepting pre-orders already, and it seems it will be shipping by early August.

The first part of the book deals with the core language, as described by the ecmascript standard. Personally, I am most of all looking forward to the second part, which focuses on client side applications of javascript -- my own primary interest in the language. David's teaser promises chapters in contexts of SVG, the <canvas> tag, XML processing and scripting HTTP and Flash. If they are even close to the very high standards of the previous (fourth) edition from 2001, it will be an excellent read and all-round reference work to have at hand.


Avoiding javascript leakage

David Baron recently made a good leak detection tool for tracking down leaky extensions, web pages, javascript libraries and the like. Much as I suspected, my own blog template is a sieve -- mostly for using the Google Maps API v1, but the calendar, some small details in my own code, and even in jQuery, which I use to toss in Blogger comment faces on my posts nowadays, leaks references to objects that ought to have been garbage collected.

Avoiding leakage is one of the better marks of differenting excellent programmers from good ones. Or, if you have high standards (you might be Joel Spolsky, for instance, and only afford yourself the very cream of the crop). In a typical browser environment, you rarely notice a leak unless you really look for it, and even then, it has typically been a bit of a hassle, and lots of labour tracking them. Leak Monitor steps in, solving that problem.

Web page code that results in memory leaks in the browser is in a way not really a bug in the web page per se -- non-broken browsers should always catch and dispose of the garbage we generate from user code from the messy steamping pile of dung that is the web. The hallmark of a great browser is polishing up the chaotic and broken web, making it look polished and behave well, giving us a pleasant browsing experience. Which is difficult. Very difficult. A browser that fails to collect the garbage that we strew about is a buggy browser. It is a bit unfortunate that most browsers don't do a very good job about leaky web page code. Typically including our favourites; present day Firefox still has these issues, Internet Explorer too.

So it's worth the effort writing leak free code, registering matching unload handlers to unregister the event listeners you add to a page, for instance. Especially in code meant for others to use, such as libraries like Google Maps, or jQuery, Dojo and so on, where end users are both unlikely to notice them and even less likely to be able to fix or bypass them. It's no coincidence the Google Maps v2 API supplies a method meant to be called on page unload, to unregister all the event handlers the API sets up of its own accord. If you write v2 Maps applications, be sure to use it.

I'm hoping to eventually find some time to shape up my own blog, perhaps rewrite some parts of the calendar code and maybe repackaging a zip file of that with additional bugfixes (as the upstream maintainer seems to have treated their GPL:ed version of the code a non-maintained dead end). I also hope we will see leak fixed versions of the Google Toolbar, the Web Development Toolbar and Flashblock, now that it has become so easy spotting the culprits -- not only do you get to see which objects are leaked, you also get source filenames and line numbers, so it's not really even the matter of tracking down where the bad things happen.

I'd like to share a particularly useful piece of code I picked up on the Greasemonkey list a while ago, which has ended up mostly everywhere I handle events these days. It might need a bit of reshaping to fit an extension context, but the principle is the same. The original class was called EventManager, but I opted for something that would not be even more carpal tunnel syndrome typing than the original DOM API -- so below it's renamed EventMgr:

EventMgr = // avoid leaking event handlers
initialize:function() {
if(this._registry == null) {
this._registry = [];
EventMgr.add(window, "_unload", this.cleanup);
add:function(o, t, fn, uc) {
if(typeof o == "string")
o = document.getElementById(o);
if(o == null || fn == null)
return false;
if(t == "unload") {
// call later when cleanup is called. don't hook up
this._registry.push({obj:o, type:t, fn:fn, useCapture:uc});
return true;
var realType = t=="_unload"?"unload":t;
o.addEventListener(realType, fn, uc);
this._registry.push({obj:o, type:t, fn:fn, useCapture:uc});
return true;
cleanup:function() {
for(var i = 0; i < EventMgr._registry.length; i++)
else {
if(type == "_unload") type = "unload";
EventMgr._registry = null;

Usage is simple: where you previously wrote node.addEventListener( "click", handler, false ) instead write EventMgr.add( node, "click", handler, false ) -- in other words, it's just moving the node to the head of the list of parameters and substituting addEventListener for EventMgr.add.

The class assumes a DOM compliant browser (so it's not IE compatible, for instance), but I have had very good use for it in my user scripts, where the code gets to run on mostly any page I visit, across the entire web -- which can add up to lots of leakage in the long run, if you write sloppy code. But any event handler you add using the above method also gets removed as you leave the page without your lifting a finger.

Even before elder mozillan wizards squash the bugs deep down near the firefox core.


Mine. And yours, too!

All of the code examples, javascript snippets, user scripts, utilities, blog template add-ons and so on I write and post about here are put in the public domain, unless otherwise mentioned, or bound by related lifted-in code with viral licenses. It is my gift back to the online community, encouraging others to join, pick up ideas or good practices, and hopefully share their own ideas and findings, too. Especially including picking up on ideas I strew about, building Something Real on top of them. After all, the value of an idea is close to zero on its own, but follow it through, build and maintain it well, and its value skyrockets.

A free exchange of ideas, knowledge, experiences, practices and tools is what makes the web this great environment to work in, and with, and I want to do my part in fostering that spirit. Occasionally lifting DRY (Don't Repeat Yourself) to Don't Repeat Others Either, when possible; my own time spent at writing something really good, evaluating or researching something, would ideally not be useful to myself alone, but to you too.

Licenses make that more bothersome. A Creative Commons license, while nice in spirit and generally about sharing, still brings law to code, and even if that is just about proper attribution, it raises barriers. Where would you have to show what author how, in the context of your derived work? What constitutes a derived work, in terms of code? And so on, in endless cascade. This code is mine, and it is yours, and we may all do what we like with it. Credit me if and however you feel appropriate and I will be glad, or do not, and I will respect that too. I encourage you to keep docs and pointers around, following and spreading healthy practices, sharing and contributing, but will not force you to do either by power of law. I will never write Greasemonkey scripts that are more license text than code, but feel a bit sorry for those who do.

Share and enjoy!