Wishlist items

I attended Add-on-con's open house meetup at Mozilla headquarters here in Mountain View earlier tonight, airing a wish list item of mine in the trailing discussion about what we want from Firefox, after some talks about what's coming up next, and where Mozilla want to go at the moment. It was a good session meeting friends and new acquaintances, and I even recognized a few faces from Mashup Camp a few weeks ago. Being the season for wishlists, I thought I'd sum up some of mine, on a more interesting scope than those we might share with our friends and family about items, activities and whatnot. Anything goes, high and low -- I even mention the point I brought up earlier during the evening.

This is a rather scattered list of stuff I want, and would find most incredibly practical, useful, powerful extensions of will, mind and capability. Consider it food for thought, if you will, or inspiration to wild, liberating, wonderful hackery, or even for being my Santa this year. If neither, please post your own, to your own blog. I've tagged mine "tech_wishlist" on Del.icio.us, and encourage you to do the same (both tagging your own post with it, and any other person's post that contains items that you want, too). That's a powerful way of connecting brains, folks, sharing and taking inspiration and cooperation to pretty wild levels. We owe it to ourselves and humanity to make the most of the tools and ideas we generate and adopt, and one very important aspect of that is sharing them.

(This also includes music and other media, by the way, but those industries are too busy not understanding how to monetize it right, and doing all they can to tie us up in pre-wired ages where distribution and marketing cost money, to notice. But I digress.)

I want file system access to the web

File systems are often conceived of a way of organizing information on disk, when the important part of them really is a way of accessing information. You'd be surprised how powerful an augmentation tool file systems are, and how easy it is today to write one today, as compared to the old bad days, now that we have great things like FUSE or MacFUSE. File systems let us reach data and resources in a heavily standardized fashion using all the computing tools there is, whether made for the web, desktop, or any other conceivable environment. The entire web is a walled garden not available to all the tooling there is. We should do something about this.

I want XPath file system access to nodes in XML and HTML documents

File system paths don't have to have any posix semantics built into them by any necessity; they are identifiers that discern data storage location, or reference particular information, if you will. The file system is as little bound to following specific conventions about what a path means as a web server is forced to pick any particular semantics to how it interprets path and query segments of its URLs. In practice, this means that a file system path can very well contain an XPath expression digging out information of a web document, serving a smaller document than accessing the document itself would, after filtering out all the junk you were not interested in. Similarly, a file system path can contain SQL queries or poke search words into google. Doing these kinds of operations has historically locked you into particular tool chains, programming languages/environments and whatnot, for no good reason. Computers are here to liberate and augment our will, not force it into constraining walled gardens.

I want file system access to my running browser session

If you've become dependent on and enamoured with any kind of interactive command line / read-eval-print-loop environment like Firebug, favourite programming language and/or the unix shell, and do any kind of Firefox extension development, chances are you've encountered or would benefit from acquainting MozRepl, formerly part of the MozLab extension, which lets you telnet into your running Firefox session (I hear it works in most of the other stuff based the Mozilla toolkit too, but haven't tried myself). This lets you play around in chrome space as you would in the sandboxed content space with Firebug via a simple telnet session, or indeed right from inside of Emacs, if that's your fancy.

Well, I think browsers have been divorced from the unix and the rest of the tool chain for too long, and I want to operate on and work with the files I have in my brower session without any silly fuss about having me store dead documents on disk, picking a location for them, directing my tool of choice to it, and losing all the benefits of live computing and convenience of access that entails in one blow. The web is mine, and I want to interact with it in my fashion, whichever that might be at the time. I like some aspects of accessing it through the glossy chrome that is Firefox, or Google Chrome, or even the iPhone, but I don't want to be walled in there either, and getting file system access to my live session's tabs and documents means getting access to a specific view pointed at me with the cookies and other me-centric data that some random tool like wget wouldn't see without a lot of laboursome work.

And it would let me into the same walled garden that my current web browser session has already opened the doors to, when I first logged in to whichever site I am on, to do whatever I do there; socializing, shopping, blogging, banking, researching, mailing, playing, working, or whatnot. Cd:ing from window to window, tab to tab, and reaching the documents, images, audio, video and linked documents other referenced from there to do whatever my bidding. Scores of operations you wouldn't even come to think of as possible become downright simple, and useful, in an environment like this. Again, it breaks down silly walls that block our view and perception today.

I'll wager our kids and grandchildren will find the web and computing environments of the early century very backwards indeed, scoffing at the insane amounts of effort spent theming, skinning and reshaping all the walls we locked ourselves up behind, with stuff that did not evolve together as an ecosystem, but separated, and screened off from one another in different dimensions pasted together only very lightly via human-bound I/O, and the monitor/speaker-mouse-keyboard read-eval-print loop. Silly, really. But I guess we really like these walls. It's so easy thinking inside the boxes we grew up with.

Neither of these ideas are new, by the way; there have been wonderful pearls like Plan 9 in the past, but they were, you guessed it, walled away from the other stuff that was evolving at the time, and died out in niches few humans ever colonized and made their own. It's not enough making the best stuff humanity has ever seen, if humanity never sees it, for anyone but yourself, and your legacy and contribution back to the world is measured by how you both execute it, evangelize and show and tell it to others so that they, too, can benefit from it and keep evolving it, adapt and repurpose it, bring it across other walls you didn't care to cross, came up with the idea to, or even figured possible. Note also, how they tend to be orthogonal to one another, and how well they stack up when combined together again, building atop one another.

Combine with all the tooling you had which was previously prevented from being part of the ecosystem, and not even the sky is your limit. Which begs my next item:

I want file system accessible cloud storage

This already exists, but not for long enough to have been commoditized yet, the way web mail is; Amazon S3, for instance, can already be combined with S3FS (free, besides applicable S3 fees) and FUSE or MacFUSE, or SubCloud (expensive enterprise software).

I want browser standardized access to commodity cloud storage

In our walled-in browser reality of today, it would already be a big win having a user provided bucket in the sky to tie browser data to, rather than sticking the goo on local disk, and figuring out complex syncing schemes to connect one computer's local disk data with another one's. Just access the web with any device and have them all phone home to your bucket in the sky to pick up the you bound data.

Note that it should be your bucket, which you handle with ease and can transfer to some other bucket cupboard provider, should you for some reason want to; we want to avoid lock-in. Even though I kind of like the shape and sound you get from Amazon's buckets. Everybody won't. Heck, Google might give you a free bucket, some day, or at least charge you fewer bucks. Or maybe Mozilla would find some way of remaining a nonprofit by routing bucks their search engine integration, or a Mozilla held app store or market for extensions would give them, from your usage of said search engines, or a percentage of the profits of commercial apps sold through such an entity -- into the payment of your Mozilla sponsored bucket, for browsing with Firefox, when browsing the web through their particular chrome.

Which reminds me I promised to share my tech-wishlist item from earlier tonight:

I want Firefox to remember the URLs it installed extensions from

That way, us extension authors (I speak as one of the maintainers and developers of Greasemonkey and MashLogic alike) can pick up configuration data passed in, for instance, the query and/or fragment parts of that URL.

This is primarily useful for improving and streamlining a most seamless installation experience for users, so that -- in the case of Greasemonkey for instance -- a user script link, when clicked by a user that has Greasemonkey installed, installs the script, and when clicked by a user not having Greasemonkey installed, prompts with the Greasemonkey extension installation sequence, which installs Greasemonkey, which then proceeds to install the script.

Via the same link, which points to the Greasemonkey xpi parametrized with the URL for the user script to install. The same kind of streamlining and on-web-sharing of initial configuration data applies very much to other extensions too, perhaps particularly so commercial grade extensions where business models benefit from shaving down the install and setup threshold for users as much as conceivably possible, and what better way to do it than to allow the prospective user to do it on the familiar web rather than to mandate doing it within some walled-in confines of a XUL garden?

I know; I wouldn't want to have to do it on the web, but that's a familiar setting for web users, and it allows people to help one another on the web with doing actual configuration presets, rather than describing in words and pictures how to do manual labour in often contrived and messy user interfaces, which makes non-expert users feel it is way too difficult and well beyond their understanding. But clicking an install link after reading a TechCrunch or LifeHacker post about a specific use case and setup for it, shared in same post, is available to everybody. This is the kind of direction at least we at MashLogic want the web to take, but it's presently awful messy doing that kind of thing. (Not that we'll let that discourage us, but it's an unnecessary shortcoming of the platform forcing you to bring your own duct tape to do it.)

A few of these ideas probably constitute quantum leaps in computing, if or when pulled out of idea space and reshaped as implemented functionality, given more detailed shape and form, convention, and figuring out idealized notions of how to come across the many mismatched and yet unmet problems that will come along the way, as we evolve the tools of the past into the tools of the future (what we always do as wizards and programmers, somewhat romanticized, and sometimes with more or less merit than otherwise). Figuring out ideal solutions is not important upfront; getting started and making incremental improvement over the present is. Doing even a subset of even one or a few of these will enable us to do many things previously much more difficult or undoable, and you, and/or others can keep polishing it up later. That's some of what open source and publication does best.

No-interface interactive maps

I've wanted to do this kind of hack for ages, since social software became commonplace on the web. Being both very bad at geography (even worse so, US geography, having moved here only just recently), and rather well versed with mucking about with other web developers' user interfaces, I finally took to it yesterday night (I've admittedly done a similar Greasemonkey hack for Facebook at some time, but it wasn't nearly as fun, and has most likely rotted since then, anyway).

The idea is basic: you tend to see location names (cities, addresses and the like) associated with users or events strewn out all over the place. Those don't tell me a whole lot unless I've been there, live near them, or similar. Maps do, though -- at least if I have some reference point I know to relate them to. For a site like OkCupid which is all about getting to know people (or yourself, but that's a different story), the natural point is of course home. And these days, Google Maps offers some really nifty geocoding and travel directions APIs. Add some Greasemonkey magic to the mix, and voilà -- a context sensitive map section which shows you where people are, in relation to yourself, as you hover a location name. It's quite snappy, and completely effortless from the user's point of view; no typing, clicking, dragging or otherwise involved:

This really is my favourite kind of interface: it doesn't have a learning curve, takes no clicking about, and has intuitive discovery properties; hover a place, and the same place shows up, name and all, in the map view.

It can of course use some additional polish like drag-a-corner-to-resize, a get-out-of-my-face icon to turn it off and the like, but suddenly it's a whole little software project rather than a one-nighter. That said, I think I like it well enough to want it across the entire web. Add those, a little toolbar to drag it around with and whip up geo and adr micro formats detection for it, and all of a sudden it's a neat all-round tool.

To finish off, closing the circle of the original OkCupid hack, replace that one with one that just sweeps OkCupids pages and converts its place name HTML to the adr micro format -- and the two tools snap together forming a stigmergy of cooperating Greasemonkey scripts. (It's a development pattern I really enjoy, for the unanticipated emergent bonus effects it tends to bring.)

Anyway, I thought I'd shine some light in the rest of the post on how I put it together, as it's painful doing things with Google Maps from Greasemonkey (can't inject the Maps API comfily, as its loader mucks about with document.write, which only works as intended for inline script tags, as opposed to dynamically injected code), and since my technique is rather easy to borrow.

The map area and geocoding is all done in a little iframe I load from here, starting it off with some URL encoded query parameters like at=Mountain View, California, United States (to center the initial map and put a little blob there), country=United States (which helps the geocoder, IIRC), nav=2 (if we want a medium-size zoom/pan control) and z=9 for a useful default zoom level. (The latter adapts to an appropriate scale when we ask for routes.) It will fill the whole page, or iframe, so we don't need to state any pixel dimensions.

So how do we tell it to look up places for us? This is where it's getting fun. Using window.postMessage on the iframe's window object, we send the geocoder its instructions, and using window.addEventListener("message", listener, false), we listen to whatever it figures out for us (Firefox 3+, and other modern web browsers).

For convenience, I went for a rather minimal messaging API; send it a newline-separated list of locations, optionally preceded by a "focus" (to move the view and blob there) or "route" line (to try drawing a route between the first and last address). All places in the list get geocoded, and the return value passed is a JSON structure {"coords":{"place name":[long, lat], "other place name":[long, lat], ...}, "took":t, "type":"geocode"} where t is the lookup time in milliseconds. Route requests are in addition populated with "distance" and "time" properties, either containing a "html" property (for a human readable form), and a "meters" or "seconds" property respectively.

And that's really all you need to use it.

Wanting it somewhat speedy (as the iframe only has page longevity, and thus won't learn and retain address to lat/long mappings over time), I opted to add a little local cache of places already looked up in the past with globalStorage (Firefox 2+, non-standard). Quick and dirty, but functional, it puts places in a little JSON tree indexed on country, region, city, and so on, however deeply the hierarchy goes. As OkCupid's place names are all on a tidy machine-readable comma-separated list format with consistent capitalization this is a rather small representation; place names in the wild might find it less useful, if still not worthless. As your web browser's IP might get blocked from using the Google Maps geocoder if you use it too recklessly (>15,000 requests per day -- but still), the persistence might be good to have there, just in case.

As usual, feel free to adopt, embrace, extend and hack about madly with it; code wants love too. The full source (and script, docs, et cetera) of the Greasemonkey script is on userscripts.org, and the Google Maps part is linked above. Happy hacking!

MashLogic

I'm taking a few moments' refuge from the growing absence of furniture in my echoing soon-to-be former home in Sweden (as my household and me are finally moving to the California Bay Area now -- after about two years more or less stuck in that ghastly red tape/inertia land), to join the little crowd of my coworkers at MashLogic, that are just cheerfully lifting the veil off our Firefox/Flock extension of the same name: MashLogic sets sail!

Update: TechCrunch liked us too. Yay! :-)

A close sibling of Greasemonkey's, we try to shift how the web works, from the walled gardens we have today (where a feature you like would stay within the confines of a particular web site, or, at best, group of web sites) -- towards working more like how communicating vessels or a sea of information would: add a feature you like once, and have it propagate across the entire sea, wherever you may roam.

Let's say you love how some web site offers explanations of the difficult words it uses, or even links advanced concepts to their Wikipedia article counterparts. While that is a neat feature, the rest of the web remains stuck on its now annoying lower evolutionary notch, devoid of it.

We think such features should not be implemented in server side logic, once per site, to give you the benefit of it only on a small fraction of the web: they should be part of your browser's toolbox and normal web browsing experience, and subject to your own control and choices.

That is what we have browser extensions for in the free software world. The extension user experience in Firefox isn't without its flaws, though; easy toggling of functions you might want or not is not is always subject to whether the extension implements its own such toggle, and if they do, they all end up having Yet Another User Interface, yet another icon or menu entry somewhere, and so on. And while you can disable any extension, you have to restart the browser to be rid of it, and for reversing the operation, restart yet again. There is a very distinct lack of convenience there.

Greasemonkey improved that picture a lot -- Greasemonkey user scripts all show up the same way in the monkey menu, and you have both an effective off switch for them all in one go, and each individually by checking or unchecking them, at the less insane cost of just reloading the page. MashLogic lets you toggle them and have them apply or undo their effects in real time instead, thus letting you play even comfier still, without even reloading the page, and much less restarting the browser.


It is some rather boring menial labour doing much of the text crunching that goes into a user script or extension that scurries through document node trees too, if you were to make one doing things like the above -- or perhaps an automated currency converter that translates weird foreign price tags, or articles or blog posts naming amounts in foreign currencies, into your own currency of choice.

The lion's share of the work isn't in mashing up that currency conversion web API you bookmarked the other day with data for it, or even in the challenge of recognizing free form mentions of amount + currency, but in hacking up a little document crawler that finds the stuff and then changes it (without completely locking up the browser). And by the time you realize that, it's not very fun hackery any longer.

We sort of wanted to address that too, making it more like plugging in your regexp, or a little dictionary, in a polished crawler that already does that boring job for you, so you don't end up reinventing those icky wheels over and over again client side too.


The particular examples above are, unsurprisingly, stuff we already provide, among a handful of other features. There is plenty of room to grow in still, even in features available to Greasemonkey users already (for instance, we lack the provisions for installing new mashes, which is what we call the separate MashLogic features), but we'll mature over time, as any tech worth having does.


I'm eager to get back to the hackery again soon, but you would not believe how long this whole real-worldly stuff of moving from one country to another is taking me; it's almost as if this world of physical matter of ours is immune to my wizardry skills. It just does not compute. I think it's broken somewhere. Where do we file tickets on it?

Fun with Greasemonkey: @require

After the release of Greasemonkey 0.8, the door is open for @require helper libraries, the user script take on Ruby's require, or the C family's #include pre-processor directive. For security reasons, the required library is downloaded a single time at script installation time, and from a functional perspective, is equivalent to pasting the script or scripts referenced at the top of your script, in the order of the @require lines listed. (==UserScript== headers in the referenced files are presently ignored, so a library may not yet use @require or @resource lines of its own.)

@require in practice makes user scripting scale down better, which is essentially to lower the threshold towards getting a job done, to a level where implementation cost to scratch an itch is so low that rather than feeling that mild annoyance with something twice, you address the issue once, and never see it happen again -- on that site, or the web at large. We are still far from there (in as much as "getting there" would ever happen -- that threshold can always be chipped off further, I assure you) -- but @require is healthy progress.

This is what the feature can look like in an example script of mine which adds image captions to online comics, by default Dinosaur Comics and xkcd (expect another upcoming post about $x and $X):

// ==UserScript==
// @name      Image title captions
// @namespace http://code.google.com/p/ecmanaut/
// @require   http://ecmanaut.googlecode.com/svn/trunk/lib/gm/$x$X.js
// ==/UserScript==

I'll be sharing some of my own helpers here, in a little series of "Fun with Greasemonkey" posts. I encourage you to write, and share, your own, on your blog, as well. User script authorship is a very social form of software, especially when you share your results on userscripts.org, and your best practices, for instance by way of blogging.

Javascript books

I just received two javascript books I recently ordered from Amazon (the US branch, for once, since the dirt cheap dollar made shipping across the Atlantic notably favourable to shipping across the English Channel): John Resig's Pro javascript techniques and Douglas Crockford's Javascript: the good parts.

I had for the longest time (since the nineties) been of the opinion that the only javascript book worth getting was David Flanagan's Javascript: the definitive guide, which I at the time found excellent in it's third edition and still good (and much updated) in its fourth (by my cursory looks and used for reference, at least), but as two of my favourite authors on the subject of javascript (the last being Dean Edwards, whose primarily means of expression on the subject, so far, at least, is terse, solid code) have been published since then, it was time to challenge that assumption. And I'm quite glad to say that both were up to par.

Pro javascript techniques really shows Resig's experience in library writing, and shares lots of useful input and practices, and provides an enjoyable and rather useful walk-through of the popular javascript libraries of its day, which manages to still be fairly relevant, in spite of the evolution they have seen since its publication. Much the same applies to the bits and details it shares on browsers and the tool-chain, but that is to be expected. (I am really looking forward to its successor, Secrets of the javascript ninja, much of which can already be read online.) Particularly enjoyably and usefully, he shows off strengths of each library without much fuss.

As for Crockford's Javascript: the good parts, it follows his usual seasoned narrative which has earned him nicknames such as "the Yoda of javascript", and keeps a close focus on the more timeless parts of the core language itself, devoid of all the clutter and temporally challenged aspects of the document and browser object models. It follows a clean-slate approach to describing the beautiful parts of the language most worth learning, while warning about the pitfalls of some of its worst warts, and is just as enlightening as the rest of Doug's work. It's something of a feel-good book for javascript buffs wanting to meditate over their use of the language -- or at least it is for myself.

Great companions for a season where the weather merits some good outdoors rather than indoors geekery. ;-)

Expensive, yet responsive, live interaction

Let's say you have a slider widget that users can drag to filter a largeish table so it only shows rows where some column is greater than the a threshold, defined by the slider. That's an application with rather strict real-time constraints, but filtering the table is expensive; a naïve solution that refilters the table with every drag event you get, or even just every event that changes your threshold value, will still be too slow for the interface to feel responsive.

I just came up with a rather elegant solution for that problem, which makes sure that your expensive code won't make the interface feel sluggish, by postponing calls to it that come in a more rapid succession than the time it would have taken for it to run. Or, to be precise, the time it took last time it was called. It's a simple wrapper function you can reuse anywhere:

// returns a function that only runs expensive function
// fn after no call to it has been made for n ms, or
// 100, if not given, or the time fn took last time, if
// it has been run at least once and no n was given.
function expensive(fn, n) {
function run() {
fn.timeout = null;
var timer = n || new Date;
fn();
if (!n) duration = (new Date) - timer;
}
var duration;
return function postpone() {
if (fn.timeout)
clearTimeout(fn.timeout);
fn.timeout = setTimeout(run, n || duration || 100);
};
}

You might of course want to add some argument and this re-binding to it, for cases where the callback isn't self-contained, but then it gets a bit more cluttered, and I really just wanted to show the principle here. Typical usage is replacing your node.addEventListener("mousemove", callback, false) with the wrapped node.addEventListener("mousemove", expensive(callback), false) version, and suddenly, the UI is not only bearable, but probably even rather pleasant to use.

Google site index bookmarklet

The bookmarklet first, for anyone who wants to opt out of the verbiage: I call mine @site, which fits snugly in my bookmarks toolbar. It searches Google for any page Google has indexed on the site you are presently visiting, and presents them in Google's usual interface which you learn how to navigate once, and can then use for the rest of your life, pretty much without ever thinking about it.

I am well aware that one of the things that made the web so hugely popular with content producers, is its very anarchic non-standardization of the user facing interface. Yes, your graphics and interaction designers are at full liberty to devise pretty much any user interface and interaction scheme they damn well want to, and they happily do so, too. The same applies to DVD menus too, though, thankfully, the dvd consortium at least recognized that their medium was one of entertainment primarily (or, through dumb luck) and somewhat constrained the technical possibilities a little.

Web sites are fairly unconstrained about their navigation (im)possibilities, though, and as every web site ever made is a country of its own with conventions of their own (most -- thankfully -- even shared with or at least inspired by others, modeled from similar components, and so on), so lots of features you might want the medium to provide, like "show me the most popular pages of this site" have to be found, when they exist, identified as such, and learned, before you can use them, once per site, or even once per content producer and site, when shared between several.

That most of the web imitates most of the rest doesn't change the fact that visitors still have a level of interface burden that is not associated with media like books, where every language has a common convention for which order you flip the pages, where to start reading, and so on, and, even across languages, most agree on left to right, top to bottom. The web could have conventions like these, but has not evolved to that stage yet (and might never do). There is some consolation in that browsers have a set of basic functionality for finding text within a page, scrolling it and the like, which is rarely intruded upon by web pages (but little prevents them from doing so).

While I don't know the particular sort criteria Google uses for open searches like the one this bookmarklet does, it often does seem to list the more popular pages first, at least on web sites like Alertbox, some of the first hits are the basic sections of the site. Reading Jacob Nielsen's top-10 application-design mistakes, how many perceive the web as one application? I do. I think most naïve, non-technical users who could not care less about the detail inner life workings of the web, do too.

And I think you should, too, despite knowing a lot more about it than that. Your web sites and design thoughts will surely benefit from it. I am rarely happier than when taming the web to behave more like one single, internally consistent, application, that responds to my actions in the same way, regardless of whichever of its zillion dialogs I happen to interact with. Try not to waste their brain power on working your part of that application, if you can. Yes, it is more fun for your own ego to have them adapt to your whims than you to their expectations and there are counter-examples to every rule, but those account for miniature fractions of a percentage, and it is pretty safe to assume that your application is not one of them.

That said, most of my web sites and hacks venture far into the territory of doing things backward, one way or another. I can assure you that the visitor appeal of said hacks, measured in raw numbers, are inversely proportional to their obscurity. The same is true of my obscure writing.

JSEnv live XPath visualizer

I've had some productive hack weekends recently -- last weekend, among other things, I took a plunge into the Firebug 1.1 beta, both to re-apply a rather useful feature patch I submitted upstreams a year ago (which was lost, and found, and lost and found again -- might make 1.2) and to add some handy keyboard shortcuts to the HTML view (deleting nodes with delete and backspace, and both folding/unfolding and stepping up/down the DOM tree with arrow left and right).

Anyway, I ended up excavating Firebug's highlight-a-node code to turn it into a tiny lib and then use that lib for beefing up Jesse Ruderman's jsenv bookmarklet with a live edit XPath field highlighting all matches in the page as you type:



It tells you what the expression matches in the title bar of the jsenv window, and signals some additional colour coded info via the XPath input field's border:

• red for illegal expressions,
• green for matching nodes,
• gray for 0 matched nodes,
• yellow for a string,
• blue for a number and
• magenta for a boolean value.

Ah, one more feature -- you move the focus node you see with arrow up and down in the input field, or scroll wheel up/down when hovering it.

The ideal spot for this feature would probably be as an extra feature to toggle on and off in Firebug's HTML view -- to filter the view by, for instance. The more I think of it, the more useful this feature would be in that shape. I hope the Firebug Working Group is as open to additional community developers as it seems to aim for.

Anyway, here is the XPath jsenv; enjoy!

Google safing passwords

Apparently, lots of places on the net, prominently including all WordPress blogs, use unsalted hashes for password verification, i e you get to log in if hash(given password) == hash stored in the password database; commonly using cryptographic hash functions like MD5 or SHA1. If that password database leaks, people can fairly effortlessly use Google as an O(1) lookup function from hash to plaintext, for most weak passwords.

The problem is easily avoided by using salted passwords -- i e hashing the password concatenated with a random component, say 32 bits long, and storing this salt next to the hash, in plaintext. Net effect: your password database, if compromised, will not be susceptible to the same attack, as the search space is 4,294,967,296 times as large, making it less likely that Google knows how to reverse map the hash for you.

I whipped up a quick tool to see whether a password is susceptible to the Google lookup or not (for MD5 and SHA1, and optionally, whether the plaintext password itself is found on the web too, though I discourage using it for reasons of potential snooping of your traffic). The hack uses some neat client side tools to query Google's Searchmash, using Yahoo! Pipes to decorate its JSON result with the JSONP callback they apparently lack (Google unfortunately still rarely gets that right), and some micro-libraries I've started accumulating recently:

Sample Google for a password dictionary attack on:

Check the checkbox to query google for the plaintext version of your prospective password, and not just cryptographic hashes of it. (Anyone snooping your, Google's or Yahoo's network traffic, will see it.)

User scripting new GMail messages

With the recent change in how GMail works, lots of prior user scripts improving the service, like a small hack that removes <font size="+1"> tags from mails broke, in the wake of the DHTML improvements (which make the site more difficult to augment via user scripting).

The GMail developers, generously, were not very far behind to release a smallish Greasemonkey API, somewhat alleviating the problem -- but it still is less than trivial to target mail bodies with your user scripts.

I took a stab at it the other night, to figure out what would go into it, and managed to come up with a new font manhandler (it targets a rather local nuisance on the Greasemonkey mailing lists). As I have been trying out MailPlane for a while for reading my mail I decided to see if I could make the script run both in Firefox on Greasemonkey and Safari/WebKit/MailPlane on the GreaseKit user script manager. That turned out quite doable. The script, in its entirety:

// ==UserScript==
// @name           Gmail - deBill:ifier
// @namespace      http://code.google.com/p/ecmanaut/
// @description    Manhandles all font size tags to stop all the yellin'
// @include        https://mail.google.com/*
// @include        http://mail.google.com/*
// ==/UserScript==

window.addEventListener("load", loader, false);

function loader() {
  var api = typeof unsafeWindow != "undefined" && unsafeWindow.gmonkey ||
    (frames.js ? frames.js.gmonkey : null);
  if (api) api.load("1.0", init);
}

function init(gmail) {
  function viewChanged() {
    var view = gmail.getActiveViewType();
    if ("cv" == view) {
      var div = gmail.getActiveViewElement();
      div.addEventListener("DOMNodeInserted", deBill, false);
    }
  }
  gmail.registerViewChangeCallback(viewChanged);
}

function deBill(event) {
  var font = event.target.getElementsByTagName("font");
  for (var i = 0; i < font.length; i++)
    font[i].removeAttribute("size");
}

Feel very free to base your own hacks on the recipe. I recommend starting out with an action function that does something basic like setting event.target.outline = "1px solid red", to see where it does what it does -- since the callback runs for every conversation view change, including updates to the ad pane on the right, not only when expanding mails you read.

Cooperating Greasemonkey scripts

I should apologize in advance that most of the scripts I link here use a not-yet-released @require feature of the svn development version of Greasemonkey (the equivalent of #include in C/C++, so if you exert yourself, you can paste together the scripts by hand by inlining the code at the top of the script). Pardon the nuisance; at least it won't ruin this post for you.

I have taken another stab at stigmergy computing, or writing scripts that communicate by changing common environment, here the web page they execute in.

Anyway, last time, I wanted a decent portable web photo album browser; functionality that would follow me around irregardless of where I roam on the web. Something to flip through images with the arrow keys, perhaps with some additional handy features too.

The album browser would consume the very basic microformat <a href="*.image extension">, and turn it into a gallery. This meant that I could use my familiar album browser mostly everywhere without any work at all. It also meant that, when encountering some photo site working differently, I could adapt not my album to the site, but the site to my album. A minimal script that changes a single site only very slightly, is trivial -- adapting old code rarely is. So this little ecology convened around a minimal ad-hoc image album microformat.

This time I have several microformat consumers convening around a microformat I came up with for destructuring pagination into what is popularly (if somewhat incorrectly) called endless scrolling; a page that, when you approach the end of it, loads another pageful or so by means of XMLHttpRequest. Usually an abomination, when some site designer forces that browsing mode for you, but ever so handy at times, if you only got to choose for yourself.

Anyway, I started off basing it on the old <link rel="next"> tag, which is actually standardized, hoping that I would, at least once in a while, not need to produce this bit of the microformat myself, but could ride on data in the web page itself. So far, that has not saved me any work but cluttered the microformat into picking next links in two orthogonal ways, depending on what is available, so I'm inclined to write that off as a bad, yet well-intended, idea (see my first reply below for the messy details of how standards bloat like this typically come about).

I needed three data items: an url to the next page in the sequence, an XPath expression that matches the items on every page, and an (optional) XPath expression that identifies a pagination pane to update with the last page loaded so far. I came up with three meta tags, named "next-xpath", "items-xpath" and "pagination-container". If the web page gave me those, my pagination script could deal only with undoing pagination, and care less about whether the data it actually handles is Flickr photos, vBullentin forum posts, Google or Yahoo search results, or something else entirely.

After doing some crazy limbo (which deserves a post of its own) loading the next page into a hidden iframe (both for letting Greasemonkey scripts producing the microformat run, and to get a correct DOM for the page to run the XPaths -- as it turns out that, amazing as it may sound, browsers can't turn an html string into a DOM without ruining the html, head and body tags; at least not Firefox 2 and Opera 9), the remainder of the script was a walk in the park.

Having written up a few microformat producers for various sites to exercise the microformat and see if it addresses the problems that show up well enough, found and straightened out some minor quirks, I felt the urge to do more with this smallish, but oh, so useful, microformat, telling apart content from non-content in web pages, so I decided to hook up my custom keybindings script with this hack, making it assign "p" and "m" to scrolling up and down among the items of these pages. That came down to a whopping ten new lines of code, to make a dozen, or, factoring in future microformat producers, an unbounded number of sites scrollable in that way.

And -- best saved for last -- both microformat consumer scripts coexist happily, independently and work orthogonally; the unpaginated whatevers that get added to the end of the page are browsable with the new keyboard bindings, so you can work mostly any web page as comfortably as ever your feed, mail or whatnot reader, right there.

If you craft any microformat producer or microformat consumer user scripts, please tag them as such on userscripts.org, and also tag them with the name or names of the microformat(s) they handle too (I picked, somewhat arbitrarily, photo album microformat and pagination microformat for mine), so they form findable microformat ecologies for other users. Any site your microformat gets a producer for, improves the value of all the microformat consumers of that format, and any microformat consumer using the format in some useful way, improves the value of all the producers.

It is very rewarding doing stigmergy / ecology programming; the sense of organics and growth is very palpable. There are some sharp corners to overcome, and Greasemonkey could help out more, but I'll have to get back to that at some later time.

High cpu load web page diagnosis

For web development, a laptop with a case fan that starts screaming like a jet engine when your cpu load skyrockets is sometimes a Really Good Thing, since it gives you cues to what happens in your browser, whether you set out do do profiling or not when, bad enough things happen. (A bit like how you could debug your programs on the C64 from the overtones that would leak from cpu to audio circuitry, giving distinct audio signatures to the occasional choke point in your code.)

Anyway, a Swedish social site I occasionally frequent used to have this effect for me, and get worse the more tabs I had open to it (and it was bad already with a modest three). I used to think it was a bugged Flash app (which they use for ads), but when I realized I had killed them all with AdBlock and FlashBlock, it occurred to me that it might be run-amok javascript.

Only that javascript that runs amok usually gives you the annoying modal dialog asking whether you want to keep running the cpu hogging script. That, however, only happens when page javascript hogs the cpu from the UI, so the single UI thread gets starved, and you, the user, get annoyed with an unresponsive UI. That can be avoided by running smaller loop fragments that call one another with setTimeout calls, leaving some breathing room for the UI thread in-between.

So how do you debug if that is what is happening? Well, for one thing, at least in Firefox 2, you can call setTimeout yourself two times in a row, some second or so apart, and compare the timeout id:s you get back, because consecutive calls get consecutive id:s. So if the difference is one, no setTimeout call was made in-between.

I crafted a bookmarklet to count setTimeout() calls per second (well, I honestly started with something smaller in the Firebug console, but figured it was useful enough to save, share and reuse later), and sure enough, that site was calling setTimeout 178 times per second. (It's not very interesting to run it on this site at present, but you can always race with yourself and the UI responsivity of your browser, attack clicking it to see how many setTimeout calls you can fire off in a second by hand. I believe it is not one of the performance tests we will see in any side by side browser shootouts any time soon. My personal highscore with Firefox 2 is 3. :-)

To do what? Well, time for another bookmarklet that shows which arguments setTimeout gets called with (it won't run on Internet Explorer; sorry: kindly advise Microsoft to implement the toSource prototype for all javascript native datatypes -- I would want to say it would not be a decade too late, but that would make me a liar :-), and lets you restore the original setTimeout implementation when you are done peeking.

It turned out the site was trying to fetch a node with an advertisement I had AdBlocked away. Over and over again. A tiny Greasemonkey script later, and the laptop fan was happily put to sleep, once again. A toast to audial debugging!