I reread Paul Graham's essay Programming Bottom-Up yesternight, and mused for a while about a set of ideas Olivier Cornu launched on Greasemonkey-dev around Christmas last year, about offering user scripts some functionality to cooperate with each other. It is, I realize now, the same territory.
That initial message quickly grew to astronomous proportions as we developed the territory (him and I sharing similar experiences of wanting user scripts to be able to share functionality with each other), trying to come up with how, while also selling the idea to remaining Greasemonkey maintainers Aaron and Anthony, neither of which wrote huge site specific user script applications. I will not attempt to sum up much of the proposals or their implementation, but Olivier's fork of the project has the latter, in what state it reached at the time, plus a healthy bit of refactoring and polishing of corners here and there, in case someone is curious.
What I will do in this post, however, is to transplant Graham's post to the domain of web development in general, and user scripting in particular -- moving it into the language domain of Javascript and the implementation domain of web browsers in general, and, lastly, user script programming under Greasemonkey in particular. A few steps away from Lisp and web server applications, but where all the same principles still apply. (The remainder of this post will not make as much sense if you haven't read Graham's Programming Bottom-Up.)
In client side web development, we face diverse host environments where most specified APIs work (and break) subtly different, dependent on the user's browser. The largely adopted solution to that has been the breadth of ajax libraries such as jQuery, YUI, Dojo, Prototype, Base2 and others. To the client side web world these are the first step towards changing the language to suit the problem: bridging the (largely unwebby) DOM API into something more tractable.
Thus your javascript programs become about implementing sought functionality, more than about jumping the hoops of the browser environment.
User scripting, by comparison, is a much smaller domain in terms of body of programmers that do much of their work there, and in terms of ajax library coverage. Greasemonkey, probably still its biggest sub class, in addition to the hoops of the web, adds all the hoops of the Mozilla javascript sandbox, which is an even more hostile environment than the web: there are XPCNativeWrappers around your DOM nodes and lots of other gotchas to watch out for.
And due to that added baggage on top, you will likely find that most or all of the ajax libraries subtly break here and there, if invoked as you would any other code under Greasemonkey, and/or need slight changes here and there in order to work as intended. So writing user scripts for this environment, becomes even more in need of libraries if you aim to do anything beyond very tiny hacks, or you waste dev time jumping the hoops.
Greasemonkey got @require support last year, to give some syntactic sugar for inlining code from a remote URL at installation time, letting us at least hide away them from the main body of code, which was a slight blessing. But that only does install-time composition.
This is where Olivier's work comes in, letting scripts export API methods, usable from other user scripts -- effectively acting as a middleware between a web page or site and the other user script. This is yet another step towards changing the language to suit the problem, or more precisely still, changing the provisions of the host environment to suit the problem.
Because not only do Greasemonkey user scripts suffer the penalties of normal client side development and the traction penalties of the Mozilla javascript sandbox, but also the constraints of working with whole web sites (where data or functionality needed by a user script could be spread across different urls) through the eyes of one single web page at a time. When you develop a browser helper user script for GMail, a dating site, some web forum, game or similar, some of the functionality you want to add on one page might require keeping track of data only found at other pages.
That quickly becomes a great endeavour keeping track of in a user script, meaning that you will spend more time jumping through hoops and wasting more code in your script on building and maintaining site interfacing code than the function of the script itself. The host environment starts invading your core logic, again, leading you off what your script is really about.
Attempts at bridging that with @require will force you (read: your script's user base) to reinstall every script you use that library in every time something on the site has changed that requires a fix in the library. Slight improvement, but we could do better.
What Olivier and I wanted (and evolved through much discussion, together with all the other smart voices on the dev list) was a system that would let a user script abstract away that functionality for other scripts, maintaining rules for which pages to run on, keeping local storage for whatever state info it see fit and so on, exporting some abstract API to third party scripts that in turn want to focus solely on site related functionality, and not on site plumbing.
It proved staggeringly difficult to reach consensus about the soundness of these ideas on-list, though two or three of four or five had a very good grasp of what value they would bring to user script development. I think we may soon see a somewhat different take on how the Greasemonkey project evolves and is maintained. I hope we will keep about the same track record of stability and security in what we release to you at addons.mozilla.org.
I also hope we might make it easier to bring about new tooling like this in the future.
