Thought leadership that does not translate to leverage through exclusive ownership is an excellent paradigm for success and creating value.
Creating environments where as many as possible win as much as possible, is a more general and even more widely applicable truth super set of the above one, but it offers less useful and easy to apply cues, clues and information.
Google does a great job of that in the field of web search, marrying users seeking information on topic X and commercial entities selling topic X centric information and solutions.
The web does a great job of creating a publicly accessible place to convey, create and consume information and many aspects and sorts of culture and entertainment, largely without middle-men and gate- keepers, at least for the technically minded portion of the creative class.
Palm seem to be making a good attempt at bringing this to the mobile industry, that has been very far behind the game in those respects, so far.
Apple has traditionally been making good efforts at creating an environment where the user, the developer, and Apple, win, in their computer and to some extent music industry ma ouvers. They are doing a terrible job with their mobile app store.
Mozilla.org has made a valiant attempt at making web users and web developers win, by opening up the web browser as a platform subject to evolution without central governace and vested interest gate-keeping, but have to date done a similarly bad job with addons.mozilla.org, which has a loud primary imperative of making mozilla.org not lose, likely in some US law centric contorted version of liability. This very likely applies equally to the Apple app store.
User scripts (via Greasemonkey in Firefox and other Mozilla derivates, user javascript in Opera and Google Chrome, Greasekit in Safari and other Webkit derivates, IE7 pro, Trixie and others in Internet Explorer), aim for and to varying degrees manage to open up web content to being improved, recontextualized and made more available and easy to access to users, where formerly gate-kept by web site owners.
This post would be better contained by ending here, but I feel I want to diverge a bit into a side track about a project close to heart to me -- Greasemonkey -- and how its politics relate to the above, and what has been going on there, recently; feel most encouraged to break off reading here, if disinterested.
As a Greasemonkey maintainer, I effectively in part act a gate-keeper for what goes into Greasemonkey and (somewhat stream-linedly) all web browsers of those running it in their web browser. That does not mean I want to remove from user scripting the element of being subject to the un-gate-kept evolution, but it does mean that I have an agenda about to what extent the project and product named Greasemonkey subjects its users to security hazards, complexity of usage and opportunities for sabotage (whether through any ill intent, or merely unfortunate accidents made possible through ill-thought-through or ill-conveyed design) and irrevocable breakage.
The actual codebase of Greasemonkey explicitly does not come with that agenda, though (it is MIT licensed, which I'd say roughly translates to "take it apart, borrow or evolve parts or all of it pretty much any which way you choose, with or without attribution, but don't pass it on to others under the guise of carrying the same guarantees or governance Greasemonkey did -- such as by attempting to claim and market and redistribute it under the same name, logo and the like"), and I know I speak for both myself and Anthony when I note that we put it up on Github with the express purpose of making it easy to fork off and improve on with as little friction and gate-keeping as possible.
Github makes it very easy to do so, which is good. It is a fantastic place for rapid evolution where everybody are naturally cooperating with each other.
We're a bit sad that it unfortunately (so far, at least) does not come with enough controls and measures to bound ill effects of people that do neither cooperate nor take constructive input, but does whatever they themselves decide best. The codebase we host is protected from ill effects like that, but the wiki is not, which makes it very easy for anyone to poison projects on Github, through poking about in a project's wiki pages, under stealth guise and semblance of project officiality. This wiki can be hidden, but not turned off, and anyone following the project will get news of the latest stream of edits and spammage.
We have one or two (might be the same person, might not -- it's not important) self-appointed wiki editors doing this, disregarding from input from us maintainers, wreaking some havoc with taking the official Greasemonkey wiki's content (after having been banned there, for the same reason), reworking it and adding disinformation here and there. Very likely just from failing to realize that it is wrong, but the end effect to readerships is bad, whatever the reason.
All the good annotation support (the wiki, of commits, in tickets, on lines of source code in the repository itself, and so on) thus becomes unmanageable, when a busy agitator spams it with venomous attitude and tone, setting a example for collaboration by poisoning the well for us. The lesson in it for us may be that if we want to set a friendly tone and maintain the cooperative but guided evolution we prefer, we need to self host the repository and all of the channels we operate, though the only thing we really care about is the project itself and how our efforts in that direction don't get wasted on administrativia and baby-sitting unlistening back seat drivers.
I think this is a plea to Github to make more people win, by letting poisonous people lose.
2009-11-19
2009-10-15
Plan B 4.0
It's Blog Action Day, and 2009's topic is climate change. With the most enlightening read on my 2008 list being Earth Policy Institute's Plan B 3.0, I was really glad to see Plan B 4.0 released a while ago -- it is available in its entirety in both HTML and PDF formats online, which happily means we can augment the material using tools like Greasemonkey and MashLogic (I am one of the developers and maintainers of both extensions).
Both extensions, in different ways, allow us to easily improve our reading experience by running code that someone else already wrote once, for material that was published by yet another unrelated person or organization, without any sort of organized cooperation among either of us. I find that notion incredibly powerful, and an uplifting thought about where decentralized cooperation is taking us online.
When I read Plan B 3.0 (the paper copy I bought at the release party of the Swedish translation, as it were), I found myself wishing I could more effortlessly look up the many high quality source references that are strewn about the book for the immense amount of facts about recent macro events affecting our planet, on all sorts of levels, instead of flicking back and forth as soon as I got curious. Footnotes work really well in hypertext. The online versions actually don't yet do that for footnotes.
So I wrote a handy user script that not only adds bidirectional footnote hyperlinks, but also pulls them into the end of the paragraph that references them, the first time you click them. Try it (direct install link); it's a very comfortable reading style.
If you run MashLogic too, you get essentially the same feature for bringing in content from Wikipedia (and sometimes references to people from LinkedIn), without losing more focus on the material than you find you want to, getting up to date about relevant data about, say, the World Food Programme's operations, or where the Xinjiang Province is. I find it a rather rich and low-effort way of broadening your general knowledge about everything.
It would be delightful if Google Books would export an index of their entire repository of books, and provide a linking scheme so we could turn those book and page references in the footnotes into actual hyperlinks too, the way we currently do with Wikipedia article names. Take the first reference in the introductory chapter, for instance (full precision of the deep-linkage only available after you install the above mentioned Greasemonkey script :-) -- with a catalog of all book titles to book id:s available, we could turn that into a link like "[1] Sandra Postel, Pillar of Sand (New York: W. W. Norton & Company, 1999), pp. 13–21.]" -- just like that.
The web is a very exciting medium, only just starting to show its potential. And as for Plan B 4.0, I can warmly recommend sinking into it, letting yourself fascinate with richer understanding of where we're heading, and all the fascinatingly intricate ways our global economy and ecology connect, affect each other, consequences and solutions.
There's even a good number of tips for ways we can all join in making the world a better place on the Earth Policy Institute's site. Happy Blog Action Day, and do spread the word!
Both extensions, in different ways, allow us to easily improve our reading experience by running code that someone else already wrote once, for material that was published by yet another unrelated person or organization, without any sort of organized cooperation among either of us. I find that notion incredibly powerful, and an uplifting thought about where decentralized cooperation is taking us online.
When I read Plan B 3.0 (the paper copy I bought at the release party of the Swedish translation, as it were), I found myself wishing I could more effortlessly look up the many high quality source references that are strewn about the book for the immense amount of facts about recent macro events affecting our planet, on all sorts of levels, instead of flicking back and forth as soon as I got curious. Footnotes work really well in hypertext. The online versions actually don't yet do that for footnotes.
So I wrote a handy user script that not only adds bidirectional footnote hyperlinks, but also pulls them into the end of the paragraph that references them, the first time you click them. Try it (direct install link); it's a very comfortable reading style.
If you run MashLogic too, you get essentially the same feature for bringing in content from Wikipedia (and sometimes references to people from LinkedIn), without losing more focus on the material than you find you want to, getting up to date about relevant data about, say, the World Food Programme's operations, or where the Xinjiang Province is. I find it a rather rich and low-effort way of broadening your general knowledge about everything.
It would be delightful if Google Books would export an index of their entire repository of books, and provide a linking scheme so we could turn those book and page references in the footnotes into actual hyperlinks too, the way we currently do with Wikipedia article names. Take the first reference in the introductory chapter, for instance (full precision of the deep-linkage only available after you install the above mentioned Greasemonkey script :-) -- with a catalog of all book titles to book id:s available, we could turn that into a link like "[1] Sandra Postel, Pillar of Sand (New York: W. W. Norton & Company, 1999), pp. 13–21.]" -- just like that.
The web is a very exciting medium, only just starting to show its potential. And as for Plan B 4.0, I can warmly recommend sinking into it, letting yourself fascinate with richer understanding of where we're heading, and all the fascinatingly intricate ways our global economy and ecology connect, affect each other, consequences and solutions.
There's even a good number of tips for ways we can all join in making the world a better place on the Earth Policy Institute's site. Happy Blog Action Day, and do spread the word!
2009-09-04
How to convert a FAT32 disk to NTFS
I had a near-Windows experience today.
I have had an old 500GB USB disk sitting around, that I used for manual backups in pre-Time Machine times, back when I deemed FAT32 to be the comfy file system choice for hooking up a disk with any computer I might want to reach it from. Today, I think that sweet spot is NTFS.
Especially as Time Machine doesn't want anything to do with FAT32 devices as backup targets.
I figured the easiest way to convert the filesystem in place (as I didn't have any other disk around) was to boot up a Windows XP virtual machine in Parallels and let Windows do it.
That procedure was perilous. If you ever embark on doing it yourself, here is how you probably want to do it:
If you skip that first step, you are wise to remember that a Windows machine needs baby sitting not to do bad things to itself. Like deciding it has found and applied security updates that it really wants to restart to complete in ten, nine, eight, ...all the while that long file system conversion pass is humming about in the background.
I have had an old 500GB USB disk sitting around, that I used for manual backups in pre-Time Machine times, back when I deemed FAT32 to be the comfy file system choice for hooking up a disk with any computer I might want to reach it from. Today, I think that sweet spot is NTFS.
Especially as Time Machine doesn't want anything to do with FAT32 devices as backup targets.
I figured the easiest way to convert the filesystem in place (as I didn't have any other disk around) was to boot up a Windows XP virtual machine in Parallels and let Windows do it.
That procedure was perilous. If you ever embark on doing it yourself, here is how you probably want to do it:
> net stop wuauserv
The Automatic Updates service is stopping....
The Automatic Updates service was stopped successfully.
> convert E: /fs:ntfs
The type of the file system is FAT32.
Enter current volume label for drive E: [...]
> net start wuauserv
The Automatic Updates service is starting.
The Automatic Updates service was started successfully.
If you skip that first step, you are wise to remember that a Windows machine needs baby sitting not to do bad things to itself. Like deciding it has found and applied security updates that it really wants to restart to complete in ten, nine, eight, ...all the while that long file system conversion pass is humming about in the background.
2009-08-07
Fixing $(document).ready() to work for any document
The first time I saw jQuery's
The first time I used it in the latter capacity I was a bit saddened to find that it was not quite that good yet; it always tests the same document that jQuery itself was loaded into. (At present time, meaning jQuery 1.3.2 and earlier.) The document object you pass isn't actually even fed to all the code that together implements the ready function.
As we needed it to do that at MashLogic though, my colleague Dave took to fixing it so it does indeed work that way, as per the jQuery ticket that has been sitting around for two years (from someone else that apparently wanted and expected the same thing), and I submitted a patch upstream. I'm hopeful it gets accepted for next jQuery release.
We even prepared a little test case that loads jQuery in an iframe and puts document ready callbacks on both itself and its parent window, which seems to work fine in all browsers and OSes we have tested it in so far.
I think the original ticket may have gotten closed over it often being a bit tricky to get a reference to the document object of an arbitrary frame or iframe for some frame you want to track -- the javascript security model forbidding DOM access to content in documents loaded from a different domain, for instance, and timing issues with grabbing a reference to the document object of a frame that is inlined in a document's source -- before we can get a handle to the document, it is a little difficult to have that parameter to pass to jQuery for listening in the first place, after all. But I hope such issues don't prevent the fix from landing, for the cases when we do have one.
$(document).ready(callback) idiom, I thought it was a great idea: not only is it readable as plain English, grep(1)able (it is available as $(callback) too, but I advise against using that in larger code bases, where you might need to dig up all references to it, at some time) and intuitive, it also lets you provide a different document object you are waiting for.The first time I used it in the latter capacity I was a bit saddened to find that it was not quite that good yet; it always tests the same document that jQuery itself was loaded into. (At present time, meaning jQuery 1.3.2 and earlier.) The document object you pass isn't actually even fed to all the code that together implements the ready function.
As we needed it to do that at MashLogic though, my colleague Dave took to fixing it so it does indeed work that way, as per the jQuery ticket that has been sitting around for two years (from someone else that apparently wanted and expected the same thing), and I submitted a patch upstream. I'm hopeful it gets accepted for next jQuery release.
We even prepared a little test case that loads jQuery in an iframe and puts document ready callbacks on both itself and its parent window, which seems to work fine in all browsers and OSes we have tested it in so far.
I think the original ticket may have gotten closed over it often being a bit tricky to get a reference to the document object of an arbitrary frame or iframe for some frame you want to track -- the javascript security model forbidding DOM access to content in documents loaded from a different domain, for instance, and timing issues with grabbing a reference to the document object of a frame that is inlined in a document's source -- before we can get a handle to the document, it is a little difficult to have that parameter to pass to jQuery for listening in the first place, after all. But I hope such issues don't prevent the fix from landing, for the cases when we do have one.
2009-07-22
Tracking multiple svn branches in git
I've been using git-svn for a while, to track projects still trapped in the subversion age, with git. It's not too messy to set up, though in order to get nice names addresses you need to map all the subversion identities to their corresponding name and email address. I recommend taking the time to suck out the entire repository history -- it's a one-time operation, and you get free offline history browsing and inspection.
If you have the names and email addresses of all svn committers, list them in a newline-separated file, each line reading
The docs for git-svn are not the best, and didn't mention how to import news into your local git repository for more than the branch you have checked out at the moment. The trick, which might be obvious for some but was not for me (thanks, singpolyma!), is to run
And Bob's your uncle.
If you have the names and email addresses of all svn committers, list them in a newline-separated file, each line reading
svn-id = Person's Name <email@address> -- I kind of started tracking a list of jQuery committers, but got bored and/or side-tracked before finding the last ones. Anyway, if you go through the trouble, pass along this file on the command line (git svn clone -A committers.txt ...) when you create your local copy.The docs for git-svn are not the best, and didn't mention how to import news into your local git repository for more than the branch you have checked out at the moment. The trick, which might be obvious for some but was not for me (thanks, singpolyma!), is to run
git svn fetch --all to update your local repository with data for all branches, instead of (well, or besides) git svn rebase that just updates your currently checked-out branch.And Bob's your uncle.
2009-06-19
Updating Firefox extensions from 3.0 to 3.5
When I started writing this post, two weeks ago, 46% of the top 95% most used Firefox extensions on AMO (according to Firefox Add-on Compatibility Report) still did not manifest 3.5 support. The Mozilla team has been checking in with at least the topmost few of those projects (being one of the Greasemonkey maintainers, I got one of those pings, a few weeks ago), about whether we were having any issues upping the max-version of their extension to 3.5. We had not tried, at the time, but (happily) did not see any big issues, so we just let the release already out claim 3.5.* working.
For other extensions, like MashLogic which I hack on at work, that transition was not as trivial. Our integration surface towards Mozilla being larger than Greasemonkey's, more things changing underfoot get a shot at breaking things, and did.
The information about backwards incompatible changes available at MDC was and is still a bit sketchy, and did not cover issues we were hitting, at the time -- it seemed XUL documents were getting XPCNativeWrapper treatment in 3.5, that they did not have in 3.0, so we had to start adopting Greasemonkey-like roundabout practices to read and write properties of the XUL DOM. That's the kind of information that is good to be able to read about -- and there, specifically.
It's a wiki, though, so if you find something that isn't already there, while you are scratching your head about why things are not fine and dandy in 3.5 (due rather soon), please add your findings too. I did a brief writeup on my own findings.
For MashLogic, we actually ended up removing the xul page we had used for configuration to do the same thing in HTML instead; work we had already mostly done for our IE port, anyway. Other tidbits remain here and there, though, before we can claim having fully working functionality under 3.5.
On top of that, there is the not so enjoyable multi-week delay at AMO, getting our latest release through their review system. Our current release is thus so far only available if you look at their see all versions page, and grab the so-called "experimental" build on top. User reviews seem to help get the ball rolling a little, though, so if you try it out, we're really glad if you share your opinion with other users there by writing a review.
Whether you are a user or extension developer, sharing your experience like this really helps others. Especially if you do it where others go looking for it. MDC and AMO are rather good funnels for developer and user centric Firefox extension related things, and the more we use both, the better they get. Thanks for your help!
For other extensions, like MashLogic which I hack on at work, that transition was not as trivial. Our integration surface towards Mozilla being larger than Greasemonkey's, more things changing underfoot get a shot at breaking things, and did.
The information about backwards incompatible changes available at MDC was and is still a bit sketchy, and did not cover issues we were hitting, at the time -- it seemed XUL documents were getting XPCNativeWrapper treatment in 3.5, that they did not have in 3.0, so we had to start adopting Greasemonkey-like roundabout practices to read and write properties of the XUL DOM. That's the kind of information that is good to be able to read about -- and there, specifically.
It's a wiki, though, so if you find something that isn't already there, while you are scratching your head about why things are not fine and dandy in 3.5 (due rather soon), please add your findings too. I did a brief writeup on my own findings.
For MashLogic, we actually ended up removing the xul page we had used for configuration to do the same thing in HTML instead; work we had already mostly done for our IE port, anyway. Other tidbits remain here and there, though, before we can claim having fully working functionality under 3.5.
On top of that, there is the not so enjoyable multi-week delay at AMO, getting our latest release through their review system. Our current release is thus so far only available if you look at their see all versions page, and grab the so-called "experimental" build on top. User reviews seem to help get the ball rolling a little, though, so if you try it out, we're really glad if you share your opinion with other users there by writing a review.
Whether you are a user or extension developer, sharing your experience like this really helps others. Especially if you do it where others go looking for it. MDC and AMO are rather good funnels for developer and user centric Firefox extension related things, and the more we use both, the better they get. Thanks for your help!
2009-06-06
Greasemonkey and bottom-up user scripting
I reread Paul Graham's essay Programming Bottom-Up yesternight, and mused for a while about a set of ideas Olivier Cornu launched on Greasemonkey-dev around Christmas last year, about offering user scripts some functionality to cooperate with each other. It is, I realize now, the same territory.
That initial message quickly grew to astronomous proportions as we developed the territory (him and I sharing similar experiences of wanting user scripts to be able to share functionality with each other), trying to come up with how, while also selling the idea to remaining Greasemonkey maintainers Aaron and Anthony, neither of which wrote huge site specific user script applications. I will not attempt to sum up much of the proposals or their implementation, but Olivier's fork of the project has the latter, in what state it reached at the time, plus a healthy bit of refactoring and polishing of corners here and there, in case someone is curious.
What I will do in this post, however, is to transplant Graham's post to the domain of web development in general, and user scripting in particular -- moving it into the language domain of Javascript and the implementation domain of web browsers in general, and, lastly, user script programming under Greasemonkey in particular. A few steps away from Lisp and web server applications, but where all the same principles still apply. (The remainder of this post will not make as much sense if you haven't read Graham's Programming Bottom-Up.)
In client side web development, we face diverse host environments where most specified APIs work (and break) subtly different, dependent on the user's browser. The largely adopted solution to that has been the breadth of ajax libraries such as jQuery, YUI, Dojo, Prototype, Base2 and others. To the client side web world these are the first step towards changing the language to suit the problem: bridging the (largely unwebby) DOM API into something more tractable.
Thus your javascript programs become about implementing sought functionality, more than about jumping the hoops of the browser environment.
User scripting, by comparison, is a much smaller domain in terms of body of programmers that do much of their work there, and in terms of ajax library coverage. Greasemonkey, probably still its biggest sub class, in addition to the hoops of the web, adds all the hoops of the Mozilla javascript sandbox, which is an even more hostile environment than the web: there are XPCNativeWrappers around your DOM nodes and lots of other gotchas to watch out for.
And due to that added baggage on top, you will likely find that most or all of the ajax libraries subtly break here and there, if invoked as you would any other code under Greasemonkey, and/or need slight changes here and there in order to work as intended. So writing user scripts for this environment, becomes even more in need of libraries if you aim to do anything beyond very tiny hacks, or you waste dev time jumping the hoops.
Greasemonkey got @require support last year, to give some syntactic sugar for inlining code from a remote URL at installation time, letting us at least hide away them from the main body of code, which was a slight blessing. But that only does install-time composition.
This is where Olivier's work comes in, letting scripts export API methods, usable from other user scripts -- effectively acting as a middleware between a web page or site and the other user script. This is yet another step towards changing the language to suit the problem, or more precisely still, changing the provisions of the host environment to suit the problem.
Because not only do Greasemonkey user scripts suffer the penalties of normal client side development and the traction penalties of the Mozilla javascript sandbox, but also the constraints of working with whole web sites (where data or functionality needed by a user script could be spread across different urls) through the eyes of one single web page at a time. When you develop a browser helper user script for GMail, a dating site, some web forum, game or similar, some of the functionality you want to add on one page might require keeping track of data only found at other pages.
That quickly becomes a great endeavour keeping track of in a user script, meaning that you will spend more time jumping through hoops and wasting more code in your script on building and maintaining site interfacing code than the function of the script itself. The host environment starts invading your core logic, again, leading you off what your script is really about.
Attempts at bridging that with @require will force you (read: your script's user base) to reinstall every script you use that library in every time something on the site has changed that requires a fix in the library. Slight improvement, but we could do better.
What Olivier and I wanted (and evolved through much discussion, together with all the other smart voices on the dev list) was a system that would let a user script abstract away that functionality for other scripts, maintaining rules for which pages to run on, keeping local storage for whatever state info it see fit and so on, exporting some abstract API to third party scripts that in turn want to focus solely on site related functionality, and not on site plumbing.
It proved staggeringly difficult to reach consensus about the soundness of these ideas on-list, though two or three of four or five had a very good grasp of what value they would bring to user script development. I think we may soon see a somewhat different take on how the Greasemonkey project evolves and is maintained. I hope we will keep about the same track record of stability and security in what we release to you at addons.mozilla.org.
I also hope we might make it easier to bring about new tooling like this in the future.
That initial message quickly grew to astronomous proportions as we developed the territory (him and I sharing similar experiences of wanting user scripts to be able to share functionality with each other), trying to come up with how, while also selling the idea to remaining Greasemonkey maintainers Aaron and Anthony, neither of which wrote huge site specific user script applications. I will not attempt to sum up much of the proposals or their implementation, but Olivier's fork of the project has the latter, in what state it reached at the time, plus a healthy bit of refactoring and polishing of corners here and there, in case someone is curious.
What I will do in this post, however, is to transplant Graham's post to the domain of web development in general, and user scripting in particular -- moving it into the language domain of Javascript and the implementation domain of web browsers in general, and, lastly, user script programming under Greasemonkey in particular. A few steps away from Lisp and web server applications, but where all the same principles still apply. (The remainder of this post will not make as much sense if you haven't read Graham's Programming Bottom-Up.)
In client side web development, we face diverse host environments where most specified APIs work (and break) subtly different, dependent on the user's browser. The largely adopted solution to that has been the breadth of ajax libraries such as jQuery, YUI, Dojo, Prototype, Base2 and others. To the client side web world these are the first step towards changing the language to suit the problem: bridging the (largely unwebby) DOM API into something more tractable.
Thus your javascript programs become about implementing sought functionality, more than about jumping the hoops of the browser environment.
User scripting, by comparison, is a much smaller domain in terms of body of programmers that do much of their work there, and in terms of ajax library coverage. Greasemonkey, probably still its biggest sub class, in addition to the hoops of the web, adds all the hoops of the Mozilla javascript sandbox, which is an even more hostile environment than the web: there are XPCNativeWrappers around your DOM nodes and lots of other gotchas to watch out for.
And due to that added baggage on top, you will likely find that most or all of the ajax libraries subtly break here and there, if invoked as you would any other code under Greasemonkey, and/or need slight changes here and there in order to work as intended. So writing user scripts for this environment, becomes even more in need of libraries if you aim to do anything beyond very tiny hacks, or you waste dev time jumping the hoops.
Greasemonkey got @require support last year, to give some syntactic sugar for inlining code from a remote URL at installation time, letting us at least hide away them from the main body of code, which was a slight blessing. But that only does install-time composition.
This is where Olivier's work comes in, letting scripts export API methods, usable from other user scripts -- effectively acting as a middleware between a web page or site and the other user script. This is yet another step towards changing the language to suit the problem, or more precisely still, changing the provisions of the host environment to suit the problem.
Because not only do Greasemonkey user scripts suffer the penalties of normal client side development and the traction penalties of the Mozilla javascript sandbox, but also the constraints of working with whole web sites (where data or functionality needed by a user script could be spread across different urls) through the eyes of one single web page at a time. When you develop a browser helper user script for GMail, a dating site, some web forum, game or similar, some of the functionality you want to add on one page might require keeping track of data only found at other pages.
That quickly becomes a great endeavour keeping track of in a user script, meaning that you will spend more time jumping through hoops and wasting more code in your script on building and maintaining site interfacing code than the function of the script itself. The host environment starts invading your core logic, again, leading you off what your script is really about.
Attempts at bridging that with @require will force you (read: your script's user base) to reinstall every script you use that library in every time something on the site has changed that requires a fix in the library. Slight improvement, but we could do better.
What Olivier and I wanted (and evolved through much discussion, together with all the other smart voices on the dev list) was a system that would let a user script abstract away that functionality for other scripts, maintaining rules for which pages to run on, keeping local storage for whatever state info it see fit and so on, exporting some abstract API to third party scripts that in turn want to focus solely on site related functionality, and not on site plumbing.
It proved staggeringly difficult to reach consensus about the soundness of these ideas on-list, though two or three of four or five had a very good grasp of what value they would bring to user script development. I think we may soon see a somewhat different take on how the Greasemonkey project evolves and is maintained. I hope we will keep about the same track record of stability and security in what we release to you at addons.mozilla.org.
I also hope we might make it easier to bring about new tooling like this in the future.
2009-06-01
XPath bookmarks
A few years ago, annoyed with web pages that don't offer in-page permalinks where they ought to, I wrote a tiny XPath bookmark user script to solve half of the problem -- scrolling to the location in the document to which a URL fragment formatted as
I didn't end up using those URLs much in practice, largely because it was a bit of a hassle crafting those XPath expressions, and to some extent because they wouldn't do much for people that did not run that user script too.
At some later point in time, I seem to have realized that I didn't, and that my custom keyboard bindings script, which I mainly use to scroll search results and other (typically paginated) similarly item-oriented pages, one item at a time (via
So I apparently implemented that feature, called it
Today, I realized that my (forever growing) fold comments script knows a thing or two about XPath expressions to stuff in a page (visitor comments, typically) and ought to expose that to this magic bookmark creation script. Especially since it is typically bookmarking some specific comment I wanted, anyway. It at the same time dawned on me that if I just let it share this XPath expression referencing comments (that it would fold) with the
I was quite bemused finding that I had implemented the
Not all XPath expressions make good bookmarks, of course, on web pages that change over time, but very many do, and for static content especially, it's pretty much a guarantee -- even in the face of stuff like CSS layout changes, just as permalinks do.
[1] Zipping through template-formatted items one item at a time with some keyboard command, so your eyes do not have to shift focal point, and can focus on just the changing content (minus static template administrativia, which suddenly cognitively completely disappears from view), is a positively huge UI improvement over either of the normal practices of reading a little, scrolling a little, and reading a little more, or skimming a page, hitting page down, reorienting yourself, and repeating the procedure.
It gets even worse with paginated web pages like search results, where, in addition to that, you have to click "Next" and wait for a moment somewhere in the work flow. Fixing this problem by pushing "endless pages" (that automatically load another pageful of content once you scroll to the end of the page) onto all your visitors, is the wrong way; many will hate you for it. As for my own browsing, I sooner or later tend to end up adding a pagination microformat producer script (if a site already has those meta tags present with correct XPath expressions in them, I of course would not need to do so myself) for my unpaginate pagination microformated web pages script.
Those scripts work well in unison, so I can key down an item at a time for pages upon pages, at worst having to wait for a moment here or there while next page is loading.
#xpath:some-xpath-expression was pointing, when visiting any such URL.I didn't end up using those URLs much in practice, largely because it was a bit of a hassle crafting those XPath expressions, and to some extent because they wouldn't do much for people that did not run that user script too.
At some later point in time, I seem to have realized that I didn't, and that my custom keyboard bindings script, which I mainly use to scroll search results and other (typically paginated) similarly item-oriented pages, one item at a time (via
m/p -- don't ask :-), knowing about XPath expressions for the stuff it was navigating, could easily offer the feature of creating those XPath bookmarks for me. [1]So I apparently implemented that feature, called it
? and promptly forgot all about it. Needless to say, I still didn't use them any more than I used to before.Today, I realized that my (forever growing) fold comments script knows a thing or two about XPath expressions to stuff in a page (visitor comments, typically) and ought to expose that to this magic bookmark creation script. Especially since it is typically bookmarking some specific comment I wanted, anyway. It at the same time dawned on me that if I just let it share this XPath expression referencing comments (that it would fold) with the
items-xpath meta tag of the pagination microformat that the above keyboard bindings script groks, and that script, in turn, always updated the URL when scrolling to a new place in the document, my primary use case would be covered for most of the sites I frequent a lot, at least.I was quite bemused finding that I had implemented the
? thing, and in short work updated it to do that automatically. So, at long last, setting XPath bookmarks to in-page items of opaque, permalink deprived web pages, is a breeze. I wonder how much they will end up being used. As I never even blogged about the XPath bookmark resolver script in the first place, it hasn't picked up any mentionable user base (245 installs at present time of writing). I think it is a feature it would do the web a whole lot of good, if something like it gained traction and widespread adoption.Not all XPath expressions make good bookmarks, of course, on web pages that change over time, but very many do, and for static content especially, it's pretty much a guarantee -- even in the face of stuff like CSS layout changes, just as permalinks do.
[1] Zipping through template-formatted items one item at a time with some keyboard command, so your eyes do not have to shift focal point, and can focus on just the changing content (minus static template administrativia, which suddenly cognitively completely disappears from view), is a positively huge UI improvement over either of the normal practices of reading a little, scrolling a little, and reading a little more, or skimming a page, hitting page down, reorienting yourself, and repeating the procedure.
It gets even worse with paginated web pages like search results, where, in addition to that, you have to click "Next" and wait for a moment somewhere in the work flow. Fixing this problem by pushing "endless pages" (that automatically load another pageful of content once you scroll to the end of the page) onto all your visitors, is the wrong way; many will hate you for it. As for my own browsing, I sooner or later tend to end up adding a pagination microformat producer script (if a site already has those meta tags present with correct XPath expressions in them, I of course would not need to do so myself) for my unpaginate pagination microformated web pages script.
Those scripts work well in unison, so I can key down an item at a time for pages upon pages, at worst having to wait for a moment here or there while next page is loading.
2009-05-15
Resolving IDNA urls in browser javascript
A handy feature I discovered a while ago: resolving (for want of a better word -- I can't make up my mind whether to call turning Unicode to ASCII encoding or decoding in this context :-) IDNA URLs (RFC 3490: International Domain Names in Applications) is a breeze:
It doesn't seem to work in all browsers yet, though; of those I've tested, Firefox 3 and Safari 3 both support it, whereas Opera 10.4.11 and Google Chrome 1.0.154.65 do not.
Update: Both as my google memory didn't find this when I sought it out (decoding a non-ASCII, non-IDNA-encoded hostname to its punycode/IDNA-encoded ASCII counterpart), and for having found a use for this in the wild, here is the altered function that just takes a unicode hostname and turns it into its punycode/IDNA form:
function resolveIDNA(url) {
var a = document.createElement("a");
a.href = url;
return a.href;
}It doesn't seem to work in all browsers yet, though; of those I've tested, Firefox 3 and Safari 3 both support it, whereas Opera 10.4.11 and Google Chrome 1.0.154.65 do not.
Update: Both as my google memory didn't find this when I sought it out (decoding a non-ASCII, non-IDNA-encoded hostname to its punycode/IDNA-encoded ASCII counterpart), and for having found a use for this in the wild, here is the altered function that just takes a unicode hostname and turns it into its punycode/IDNA form:
function IDNAtoASCII(hostname) {
var a = document.createElement("a");
a.href = "http://" + hostname + "/";
return a.hostname;
}2009-05-10
Little projects I'm warming up to
I've been cooped up a lot between real life, work and a low energy budget for a while now, but think I'm getting a hankering for some ideas I'd like to develop or see developed, again. Here's a little off-the-top-of-my-head list of a few of them, small and large:
Taking down that specification in text instead of code seems indicative I'm not really in a coding mood today and ought to go have lunch. See you around!
- Data SVGs: Craft a few examples of how to encode data in its original raw form in SVG graph(ic)s and have the SVG render them right via coordinate transforms. Forward goal: get Wikipedia to incorporate this principle in their guidelines for how to represent data in vector graphics. Erik Möller (Deputy Director at Wikimedia Foundation) and Brion Vibber (CTO) both found this a cool idea. I ought to at least blog it if I'm not going to be the main advocate rallying people around the idea.
- Migrate Kronos Utils (user script here), which I'm not maintaining or developing as much as I used to, and other ikariam related hackery I've done at some time or others, to github, so others that want to hack at it have an easier time forking it. I apparently started doing that, at some time or others, in an ikariam repository, and then promptly forgot about it. *pushes recent changes*
So I guess this pretty much boils down to migrating my updates off of Google Code and onto githubg. And figuring out how to have my pushes and/or commits trigger a tweet to @ikariam_tools, which apparently people have already spotted and started following, despite its severe lack of content. :-) Does anyone remember how that is done? I seem to recall some setting somewhere for it at github, but wasn't successful finding it in all of a minute's scouting around.
Oh, and figuring out how to have vc-mode in Emacs do git things. - Run the jQuery test suite from a Greasemonkey script, to figure out how much breaks under the Firefox javascript sandbox. It'll be nice to have fine grained data on which parts that remain useable and which don't, so maybe someone would even care to fix the latter.
- Revive the live Exhibit web scraper thing I crafted some year ago and never got to the point of happiness with where I ever blogged about it. With recent development in Ext JS, it seems it could get a nice UI indeed, this time in a pop-open window, so I don't have the CSS invasion problem, and don't invade the DOM we're trying to scrape.
- Don't fold comments on pages we arrived at through a permalink with a fragment pointing to something in the foldable section (at that point, we know we foremost wanted to read that specific comment, and not the article, so don't tuck it away). Edit: Done!
Taking down that specification in text instead of code seems indicative I'm not really in a coding mood today and ought to go have lunch. See you around!
2009-04-02
A Mozilla bug
CSS
Nostalgia to be had! Try this bookmarklet, say, on this page and play around with the mock window at your heart's content.
rgba(...) colours are fun. Firefox 3 doesn't handle them quite perfectly when applied to the <html> tag, though. I haven't seen this sort of reflow bug (leaving shadows behind dragged objects) much since I, pretty much, ceased using Windows machines a few years ago.Nostalgia to be had! Try this bookmarklet, say, on this page and play around with the mock window at your heart's content.
Subscribe to:
Posts (Atom)