aaronboodman.com

Life: Updated instructions

2010-11-17T11:59:00.001-08:00

Wow, there was tons of response to my last post. I probably got 40 pings on IRC over the last week or so, and already have a few code reviews in progress by new contributors.

I want as many people as possible to get involved in programming, and Chromium in particular, but I can't help people individually at that rate. So I've updated the instructions so that they require less manual steps by me.

I'm still more than happy to help people with questions once they get started.

Good luck!

C++ version of JSON Schema

2010-11-11T17:28:00.000-08:00

Way back when we were starting the extensions project, I mentioned that we were using JSON schema for API validation.

This worked out pretty well, but the downside is that it was implemented in JavaScript, so it couldn't really be used effectively in Chrome's browser process (for somewhat obscure technical reasons it's complicated to run JavaScript in Chrome's main process -- we usually only run it in child processes).

So for a really long time, I've been wanting to re-implement it in C++ so that I could use it for work in the browser process too. Finally got around to it:

json_schema_validator.h

This only relies on Chromium's "base" library (which is really quite nice, btw), so it could conceivably be re-used by other projects. For example, I'm hoping that we can use it on the server side, in Chrome's web store.

Figured it might be useful to someone else someday, too.

Wherein I help you get a good job

2010-10-20T00:38:00.001-07:00

Edit: The response to this post was kinda overwhelming. I've modified some of the steps below to require less manual work by me.

From a recent post on Hacker News:

Friends of mine are in the same position as this guy. They recently graduated with good degrees from a decent university with no illusions that finding a job was going to be anything but difficult. Rather than feel like they're owed a job the ones I know have spent hours _every_ day applying for jobs for the past (up to) 2 years with no success. Most of the time they get no reply from these companies (fair enough), but the few times they do get an interview someone else with more experience gets the job.

This sucks. There has been an education bubble over the last few decades that has told kids that all they need to do to guarantee success is go to a good school, make good grades, and get a degree. The result is that in many fields, there is vastly more supply than demand.

But there are a few fields that are hiring like crazy. One of them is mine: software engineering. Major tech firms cannot hire fast enough. And the barrier is low enough that you can start learning today, for free. If you work hard, you can know enough in a year to get a good starter job. It's been this way for more than a decade, and I suspect another decade of growth at least.

That's the good news. Here's the bad news: It's going to be hard, and you'll need to give up on whatever you went to school for and spend a year learning something new.

But remember: the time you spent in school is a sunk cost. You can't get it back. Doing nothing just makes matters worse. I'm offering you a way to make progress today.

Ok, still here? Here's how to get a good job programming in six easy steps:

1. Download and build Google Chrome

There are also build instructions for Windows and Linux if you prefer
If you don't like Chrome, you can also work on Firefox or Safari, but the rest of these instructions are optimized for Chrome since that's what I'm familiar with.

2. Browse the bug database to find something to work on. Here are some useful queries:

I can personally review changes in these areas, and I try and keep these lists well-stocked with good starter bugs. You can also work in other areas, but you may have to find someone else to review your work.

If a bug has an owner assigned, but there has been no activity for a week or so, comment on the bug and ask whether you can take it over.

3. Once you have picked a bug, comment on it saying you're going to work on it. If it is in feature:extensions,apps, I will get notified of your comment, and will mark the bug as owned by me so nobody else grabs it. I will probably also add some comments about where to start.

4. Study the code, referring to a C++ tutorial as needed. You may also want to buy an actual paper book on the subject. Make the changes required to fix the bug. Once you have it working, send me (aa-at-chromium-dot-org) a code review with your changes.

5. Repeat steps 2 and 3 about 50 times.

6. Update your resume to say that you are a junior software engineer and you have contributed significant code to Google Chrome (or Firefox or whatever), and provide a link showing your work.

Ok, so these steps aren't really "easy". Some of them will take months. Just getting Chromium compiling is going to take awhile. I've purposely been vague on the details so you can figure it out on your own. You'll need ingenuity, a search engine, and a lot of patience.

But I guarantee, 100%, for-sure, that if you complete my program, you will be able to easily find a good starting job programming.

Hiring programmers is super hard. It's nearly impossible to evaluate whether someone is good without examples of their work. By working on open source projects like the ones I've listed, you make it easy for hiring managers. They can see what you can do and how you work with others.

There's nothing stopping you from making this happen. It's going to be hard, but you can do it. And there's lots of people that are willing to help once you take the first few steps.

Academic paper on Chromium extensions

2010-01-28T00:12:00.000-08:00

Protecting Browsers from Extension Vulnerabilities is a paper that covers some of the interesting security features of the Chromium extension system.

The isolated worlds feature that I wrote about earlier is described toward the end. Isolated worlds separate each JavaScript program that has access to a web page's DOM. Each program can modify the DOM and see changes made by other programs, but programs cannot exchange JavaScript references. This setup prevents privileges from accidentally leaking between programs. Isolated worlds are now implemented directly in WebKit (thanks to Adam Barth), so they could show up in other WebKit applications in the future.

My other favorite feature is that an extension's unique ID is a public key. The extension is signed with the corresponding private key, which means it is impossible to have ID collisions. Even if a developer copies an existing extension to get started, he won't be able to copy the extension's ID because he would need the private key in order to sign it.

Hey I used that license, too

2009-12-24T14:35:00.000-08:00

Was amused by Evan's take on JSMin's software license. Apparently all of Crockford's work includes the clause "The software shall be used for Good, not Evil" in addition the standard MIT license text.

I think at one point I saw this, internalized it, and spat it back out as "This code is public domain. Please use it for good, not evil."

I know that this stuff is really serious business to some people, and they get frustrated by these random acts of frivolity. "Don't you know," they say with a stern face, "software licenses are no place for fun and games".

Pfffft. The license is awesome. I appreciated the joke when I first saw it, and am appreciating the ongoing humor it is providing.

I do see how it would prevent some organizations from using the code. Mostly those that have become large, risk-averse, and so disconnected from the community that they are actually, honestly afraid that Crockford is going to come after them for using JSMin for evil.

Sadly, Google apparently now falls into that bucket.

Bundling multiple versions of binary XPCOM components

2009-07-17T00:55:00.001-07:00

I happened to come up with what I think is a clever hack for getting around a sticky compatibility problem in Firefox extensions.

The problem is that if your extension includes a binary XPCOM component that uses unfrozen interfaces (and it is hard not to, lots of things are unfrozen), then it is highly likely that you will have to recompile your component for each new version of the Gecko SDK (and therefore most new major versions of Fierfox).

The Mozilla wiki proposed the idea of using a "stub component" to sniff the environment version and load the right XPCOM component, but this is pretty complex to implement.

I realized that you can just use JavaScript to do the same, and it is trivial. Hope this helps someone else out (I've also updated the wiki).

The designs of dustincurtis.com

2009-06-26T09:52:00.000-07:00

Are beautiful.

I say designs because I just realized that each article has its own unique design.

I read most of these when they were posted, but I think the fact that they were posted relatively far apart made me miss the fact that each one has a different look.

Some of the designs are very intricate and clearly were a lot of work. For example, check out about.html. Tying things together is a common header and footer that brings continuity and allows you to navigate between articles.

I attempted to do something like this once for my own blog, but found that I simply did not have the time and energy to come up with a good design for every single post.

I'm glad that someone else did, and I'm looking forward to the next article.

JSON Schema, part 2

2009-04-15T01:00:00.000-07:00

So the JSON Schema stuff worked out:

chromium.tabs.createTab = function(tab, callback) {
  validate(arguments, arguments.callee.params);
  sendRequest(CreateTab, tab, callback);
};
chromium.tabs.createTab.params = [
  {
    type: "object",
    properties: {
      windowId: chromium.types.optPInt,
      url: chromium.types.optStr,
      selected: chromium.types.optBool
    },
    additionalProperties: false
  },
  chromium.types.optFun
];

--http://codereview.chromium.org/66006/diff/1087/1101

I'm really happy with how this came out because it means that all our APIs will get great error messages for free; it's not something that we'll have to think about (and therefore get wrong) on a case-by-case basis.

I tend to obsess about errors messages because they are basically the first-run experience for any library. No matter how well designed, people are going to typically call your API wrong the first time, especially in a loosely-typed language like JavaScript. What happens in that case? Does the API stare blankly back at them, leaving them to wonder if their code is even running? Or does it helpfully tell them what they did wrong?

Chromium APIs will give errors. Specifically, they will retort something like:

Invalid value for parameter 0. Property windowId: expected integer, got string.

Content Scripts in Chromium

2009-04-13T02:01:00.000-07:00

Here's an interesting factoid about browser extensions: lots of them are not about extending the browser at all. By my count, about 75% of the this week's top 20 Firefox extensions are more about extending the web content rendered by the browser than extending the browser itself. Similar trends exist in other browser extension systems.

Chromium extensions will be able to interact with web content too, using a feature we're calling content scripts (we've gone around and around on the name, this may not be final). The code for this is at a pretty good stopping point now, so I wanted to pause and write down what we did, why we did it, and some ideas I have for future improvements.

If you want to try it out, you can check out the beginnings of our Extension Tutorial, which covers most of what I'll talk about here.

First, some background on the feature...

Content scripts are basically the same thing as Greasemonkey scripts, with some important improvements.

You register your content scripts declaratively in your extension's manifest, like this:

{
  "name": "My first extension",
  "description": "The first extension that I made",
  "version": "1.0",
  "content_scripts": [
    {
      "matches": ["http://www.google.com/*", "http://mail.google.com/"],
      "css": ["foo.css", "bar.css"],
      "js": ["hot.js", "dog.js"],
      "run_at": "document_start"
    }
  ]
}

The syntax for matching URLs is slightly different than in Greasemonkey. The reason for this is that we wanted to eliminate a common bug in Greasemonkey scripts, where people accidentally match URLs more loosely than they intend. A classic example is the common Greasemonkey pattern @include *.google.com*, which matches every domain, not just google.com and its subdomains.

The matching syntax used in content scripts separates the domain portion of the pattern from the path portion, making it more explicit which sites a script will run on. One way we could use this is to someday do UI like this:

==============================================
Install 'My extension'?
----------------------------------------------
This extension will be able to interact with
web pages on:

www.google.com
mail.google.com

                            [ok] [cancel]
==============================================

Other minor feature differences:

A content script can consist of multiple physical JavaScript files or CSS files, and it can also reference images or other resources included in the extension by URL.
Content scripts support "early injection", which allows them to request being injected before any nodes have been added to the document by using the optional "run_at" key.

Execution Environment

To understand the execution environment for content scripts, it helps to first understand the execution environment of normal web page JavaScript.

All JavaScript is defined in a context. Each DOM window gets its own context, one purpose of which is to hold the prototypes of all the global objects (Object, Array, String, and so-on). This is why when you extend Array.prototype in one frame, it doesn't affect Arrays created in other frames.

Importantly, you can call functions and access objects across contexts. This happens normally when you do something like window.frames['otherframe'].someFunction().

Here's a diagram that explains the relationship between the various objects in pretty picture form (thanks, Gliffy!):

Each context also has a single global object. When you access global variables in a JavaScript program, you are really interacting with the properties of this global object. In HTML, the global object is of course the Window object.

To make property hiding work, in Chromium's implementation, the global object is not actually the same JavaScript object that represents ("wraps") the C++ DOMWindow. There is actually a separate JavaScript object whose __proto__ points to that object. When you define global variables, it is this object where the properties are actually defined.

Ok, so how do content scripts fit into this?

Content scripts run in a very similar-looking environment. They run in a separate context, and have a separate global object. But that global object's __proto__ points at the same JS object that represents the Window.

So content scripts get their own global scope and their own set of prototypes. Variables defined in the web page won't be "visible" by default in content scripts, and the same is true in reverse. Other than that, the environment for content scripts is exactly the same as for normal JavaScript running in web pages. Writing content scripts should be exactly the same as writing JavaScript for web pages.

Sometimes it is useful to access the page's global variables. For example, in Gmail there is an API that allows Greasemonkey scripts to drive some parts of the UI. To allow this kind of functionality, the content script envionment has a special contentWindow global variable defined that can be used to access the global scope of the page's JavaScript.

Permissions

Another difference from Greasemonkey is the model for accessing privileged APIs. Greasemonkey scripts have direct access to some privileged APIs. The most popular of these is GM_xmlhttpRequest, which provides access to origins other than the one for the current document. These APIs are very useful, but there have been bugs where they leaked into web content, which was bad.

In order to prevent this from being possible, Chromium extensions are split into two main pieces: a privileged part (I'll call it just 'the extension' from now on) that has access to special powerful APIs, and an unprivileged part (the content script) that runs in the renderer and has no special APIs.

The two parts cannot interact directly. In fact, they run in separate OS processes, so direct interaction is impossible. The only way they can communicate is via message passing APIs, similar to postMessage().

(NOTE: The implementation of content script messaging is still in progress and is incomplete in current trunk and dev builds)

It is the extension developer's responsibility to send only specific messages to the extension process from the renderer, and to validate those messages carefully. Extension developers need to be aware that malicious web pages could send them messages exactly the same way their content scripts can.

This design is modeled after the way Chromium itself works, where the renderers are untrusted and have to send messages to the browser process to get interesting work done.

Future Directions

I have a couple ideas for where I'd like to take this next...

Idea 1: Completely separate content scripts and page JavaScript

Right now, the way that JavaScript access to the DOM is implemented, there is essentially a global table of JavaScript wrappers for each C++ DOM object. Whenever code needs to find the JS object for a given C++ object, it consults this table:

This single table creates a bridge between any two JavaScript contexts that have access to the same DOM nodes. For example if page JavaScript does something like document.body.onclick = function() { ... }, any other code that has access to document.body will also have access to the onclick function handler that the page JavaScript defined .

This makes sense for web pages, where you want frames in the same origin to see the same sets of JavaScript variables. But for content scripts, it would be nice to wall these two worlds off from each other. It is relatively infrequent for content scripts to need to see the JavaScript enironment fo pages. It is more typical to only need access to the DOM.

In order to isolate content scripts from page JavaScript, we'd have to have separate mapping tables: one for the page JavaScript, and one for each content script. A C++ DOM node could have multiple wrappers, one for each of these "worlds". Then, when we needed to get a JavaScript object for a particular C++ object, we'd decide which table to look in based on which context the calling code was running in. Every context could only be in one "world".

We could even add assertions to the JavaScript engine that worlds are never bridged. That way if we ever had a bug, in the worst case we'd crash the renderer, not have a security problem.

If we can wall these worlds off from each other, then we can offer some increased privileges to content scripts directly, because we'd be confident that they couldn't leak to web content. You'd no longer have to go to the extension process to get cross-origin XHR, for example. This would also have the advantage of not requiring extension developers to carefully validate their messages, since we would know that page JavaScript could not send extensions messages.

We'd still probably need content scripts as they exist today if you want to interact with the JS defined by the page (for example for the Gmail API). But lots of use cases don't need that, and this idea would decrease complexity for those cases.

Idea 2: DOM Access from Extension Processes

Another idea is to offer some form of DOM access directly to extension processes. There is a team in Chromium working on an out-of-process version of the web inspector. This will clearly need some form of DOM access to work, so we can probably reuse what they do to give extension developers the ability interact with page DOM directly from their extension process.

I can imagine something simple based on querySelectorAll(). You ask for some nodes based on a CSS expression, get back a snapshot, and then send some updates. Of course, there are problems with races: the nodes might be gone by the time you send the update. But I think in most cases this would work pretty nicely I think. Again, I think we'd want to keep content scripts as they are today for more complex needs.

Yawn... Greasemonkey is great, but when do we get real extensions?

I know, I know. These aren't "real" extensions. You want to know when you'll be able to put things in the Chrome UI. Good news: that is well underway. Hopefully my next blog post will be about how to add "toolstrips" to Chromium.

Until then, have a look at content scripts and let us know what you think.

Yo dawg...

2009-04-05T20:41:00.000-07:00

So on the extensions project, I've been working on what I think is the somewhat novel idea of API non-design.

To be more specific, I'm using the CRUD pattern as a starting point for all the major sub-systems' APIs. My hope is that this will have a number of positive effects:

Minimize API design hand-wringing
Provide a large base of functionality quickly
Make it easy for Chromium developers to add new APIs
Make it easy for extension developers to learn new APIs

We decided to use JSON heavily in the implementation. For example, the createTab() API looks like this:

chromium.tabs.createTab({
 "url": "http://www.google.com/",
 "selected": true,
 "tabIndex": 3
});

So I got this all working for a few methods, and then I got to writing the validation code. I could write the code by hand, but that's so much work. And why bother when somebody has gone and invented JSON Schema.

That's right, it's a schema language for JSON. And of course it has a schema, written in JSON schema. Whee!

So we should be able to just declare the expected structure for our API parameters and push the validate() button. Probably there will have to be extra stuff around the edges, but this should get rid of a majority of the grunt work.

Not quite there yet

2009-03-18T23:00:00.000-07:00

Matt jumped the gun on announcing extensions.

There's not much to see yet. You can --enable-extensions in developer builds, but all that gets you so far is Greasemonkey-type functionality, which was already more or less there via --enable-user-scripts.

We're working on more toys, and you can bet we'll start blogging when there's something to play with. But not... quite... yet.

If there ever were something w00t-worthy...

2009-02-24T01:36:00.001-08:00

I forgot to post this back when I noticed it. Alas, they don't appear to be using it for anything.

I tracked down the element that is intended to be draggable, but I can't find any code that ever makes it visible.

Perhaps it is a feature that got killed, but not completely cleaned up. Or maybe it is used on the CMS side of the site, but is leaking a bit into the public.

I sent an email to webmaster@ and through their contact form, but got no reply :).

Update Fail

2009-01-29T15:27:00.001-08:00

I'm a big believer in silent autoupdates.

I know that it's a touchy issue, but I really believe that silent updates are the right thing for most users and for most software. Especially for consumer software connected to the internet, where security is especially important.

This dialog is an example of why. Today, Firefox showed me this, but since I was right in the middle of something at the time, I pressed "Later", without even really reading it.

(Note: I don't mean to pick on Firefox specifically. There is lots of software that works this way and it all irritates me equally. This just happened to be the occasion that finally pushed me to blog. Also, it's harder for Firefox to autoupdate due to the huge number of add-ons that are difficult to keep working across upgrades.)

Usually when I'm using my computer, I'm doing something, so I typically dismiss dialogs like this. I know it isn't the brightest thing to do, but my experience is that these things often take a long time, force me to restart, and result in incompatibilities.

I was talking about this yesterday with a friend and he put it well:

It's basically saying, "Dear Mr. Boodman: Would you like to deal with our problems right now, or with your problems?"

Like most people, I'm usually going to go with my problems.

I think it's the responsibility of the software vendors to keep their own code up to date. This is hard work, but the vendor is certainly better equipped to do it than I am.

The two places I feel that autoupdate might not make sense are Linux and corporate environments. I'm not hugely familiar with how Linux packages are kept up to date; I get the sense this still a largely manual process. But it seems like the distro might be better able to do autoupdates than the vendor, since the distro knows how to create binaries that will work best with the rest of the packages on the machine. In corporate environments, there is often an IT department whose job it is to keep your software up-to-date and functioning correctly.

Many people fear that software vendors would use autoupdate to install additional unwanted products on their machine, or to update to a new buggy version. This is certainly possible, but I think it comes down to an issue of trust. When you install software, you are making a conscious decision to trust its creator to treat your privacy, data, and machine with respect. This is just the way client software works. As long as your are already trusting the software in all these ways, why not also trust it to keep itself up to date? If the vendor abuses your trust -- by violating your privacy, stealing your data, hogging resources on your machine, or installing new buggy software -- then fine, be upset about that. But when implemented well, I think that autoupdate is a Good Thing.

Introducing nerdsz

2008-12-07T15:44:00.000-08:00

For awhile now, I've been wanting something that could answer the question: "What are my friends doing right now?"

There are probably already approximately 200 apps that do this, but for some reason I couldn't find one that worked the way I wanted. So instead I took the evening and hacked together nerdsz (pronounced "nerds-zee").

To use it, enter the twitter usernames of a group of people you're interested in. For example, http://aaronboodman.com/nerdsz?aboodman,ev,biz,goldman.

PS: You may be wondering about the name. For years now, I have been calling the group of friends I hang out with the nerds. The z is an inside joke that you probably have to work for Google to get. Sorry about that.

PPS: This hack made possible by twitter's awesome, crazy-easy-to-use API. Thanks!

PPPS: I also borrowed John Resig's PrettyDate.

Extensions in Chromium

2008-11-29T16:03:00.000-08:00

For the past few weeks, myself and a few others have been looking into how extensions might work in Chromium. Last Wednesday, we published a design doc with the beginning of our plan.

If you're an extension developer or user, we'd love to hear your feedback. You can reach us at #chromium on irc.freenode.org or chromium-dev@googlegroups.com.

Gmail in Chromium "Test Shell" on Linux

2008-11-11T21:26:00.000-08:00

The other three guys in my office all work on Chromium for Linux. Right now, they are working on bringing up the "test shell". The test shell is a very simple browser that the Chromium development team uses for testing our integration with WebKit. It is the first step of porting Chromium to a new platform.

Today, one of the linux guys sent a message from Gmail in the linux test shell for the first time:

From: "Dan Kegel" 
To: chromium-dev@googlegroups.com
Subject: sent from test shell on Linux!

Dude, gmail works in the test shell on Linux!

The team is still a long ways from even getting the web to render correctly, let alone building the real browser UI. But it's exciting to see things falling a little more into place each day.

New work blog

2008-11-01T21:42:00.001-07:00

This is my new blog about work stuff. Cat pictures and vacation stories can be found over yonder.

Greasemonkey in Chromium

2008-11-01T20:26:00.000-07:00

Evan and I have been hacking on Greasemonkey support for Chromium in our 20% time. It's been fun. Actually, it was Evan who started the work... after months of pestering me, he got frustrated and started hacking.

Implementation is more interesting than in Firefox because of Chromium's sandbox. The renderer processes are where scripts must be executed, but the renderers are not allowed to read the scripts from disk. So we set it up so that the privileged "browser" process reads the scripts into shared memory and broadcasts them to the renderers. It also watches for changes on disk and rebroadcasts when changes occur.

Note that like everything else in Chromium nightlies, Greasemonkey support is considered experimental and may change, break, or disappear from build to build. That said, this is something a lot of us on Chromium are excited about, and look forward to continuing to improve.

WorkerPool destruction cleanup done, now on to actual features

2007-07-12T10:53:00.000-07:00

The main things that I have volunteered to do for the next Gears milestone are:

Implement better error-handling support for WorkerPools
Make the onmessage callback optional

I'm diving into the first of these two today. The first thing to do is figure out how to throw errors in each of the JavaScript engines globally. That is, if an error occurs in a WorkerPool and isn't handled by WorkerPool.onerror, then the error should be thrown in the global context of the parent page. Basically, it should do the same thing as what happens when you run this JavaScript code:

window.setTimeout("throw new Error('boo!')", 100);

Spidermonkey (the JavaScript engine inside Firefox) has an API that I believe we can use for this purpose: JS_ReportError. I'm not sure if there is an equivalent for IE, or if I will have to basically do something like: eval("throw new Error('boo!')"). I would like to avoid using eval if possible because it seems hacky and there are more security considerations.

Gearpad improvements for high-latency connections

2007-07-07T19:10:00.000-07:00

I made some changes to Gearpad to make it work better on high-latency connections, like the one I use on my daily commute.

The problem was in my implementation of optimistic concurrency. This is what I used to store on the server for each note:

Id
Version
Content

At client startup, Gearpad fetches the current content and version of the note from the server. Then, when an edit occurs, it sends the version it has, plus the new content to the server. The server checks to see that the version sent matches the one that it has. If it does, it makes the edit, increments the version atomically, and returns the new content to the client. If the versions don't match, then a different client must have updated the note and this is a conflict. Right?

Wrong. What happens if an edit occurs before a previous edit from the same client returns? Here are a couple options:

Wait for the first edit to return, then send the second edit. This will make things much slower, and more complex since the editor state and server state start diverging more and more when the user is faster than the connection to the server.
Cancel the first edit and send the second one. This is what Gearpad does. But this means that sometimes the server gets the update and increments the version, but the new version doesn't make it back to the client. Then the second edit is sent and the server things a conflict has occurred.

Solution: the server needs to also store the last client that talked to it. It's only a conflict if the version is out of date *and* the update is from a new client. New server schema:

Id
Version
Content
LastClientId
NextClientId

This change also involved migrating Gears database schemas, which can be tricky. I'll talk about that next time.