I don't understand the need for the payload server from the slides. That makes me wonder - why not just use a HTTP server to serve the static files (e.g. nginx)? I'm sure I'm missing the obvious, but I'm probably not the only person wondering it.
It appears from slides 14, 21, and 24, that copying all of the static files that needed be served onto a single machine could take over 12 hours. Another way to say that is that the totality of content that is served from that single host is too large to put on a single machine. The "groupcache" abstraction lets the http-serving machines dynamically serve the hot subsets of the larger file set (represented by "storage svc" on slide 55).
But nginx can work as cache, acting as reverse proxy on misses. Minor amount of Lua scripting or C (or, better, whatever fancies one's taste if that could be called from C) hacking can teach it any reasonably complex semantics (fancy ACLs, not-before dates etc.). Many modules (dynamic archive generation, GeoIP, various cache backends) are already there, well tested in production environments.
Surely, one would probably need a separate external tool that would pre-fill (nginx wouldn't know that some file's pending before it's requested) and clean up caches (provided that rules are complex than trivial LRU removal on some threshold), as one probably wouldn't like webserver doing this unsuitable job, but that should be another story.
From the looks of it, they wanted support for putting files in place before a release date and easy per-file header/caching/access control. Add in a few other miscellaneous features and make it available to everyone and you're at a point where an HTTP server probably won't cut it.
Files also apparently come from some central repository and need to be synced.
I think the biggest reason would be that almost any off-the-shelf server software would struggle in a Google server environment. They have solved scale at the machine level, and he alludes to this on one slide:
... why aren't you using the cluster file systems like everybody else?
... cluster file systems own disk time on your machine, not you.
Slide 62 mentions the proprietary bits: ACL policies and RPC storage access. Does an off-the-shelf httpd support ACLs? How easy would it be to make them support google storage instead of a file system?
(I wrote a rough equivalent of "payload_server" in Go at my current employer to solve authentication, access control, and some other business logic issues.)
> Does an off-the-shelf httpd support ACLs?
Not really. You inevitably end up writing custom code to conform to your particular requirements and/or existing systems. If you want high-performance, you end up writing it in C as a module for Apache/nginx/whatever.
> How easy would it be to make them support google storage instead of a file system?
Unless said storage system is presented to userspace through ordinary file interfaces, same as above. There's no general turn-key solution built into webservers. The problem space is too wide.
Using Go in this way gets you good performance, simple architecture, maintainability, and easy deployment with total flexibility to do whatever you need in order to solve your version of the problem. There are no straightjackets, you don't have to conform to (or find ways around) anyone else's conception of the problem space.
It appears to be a caching proxy that has some very specialized features like dynamic zip files and what appears to be origin server notifications which you don't normally see in off the shelf caching proxies like say, Squid.
Then again, I'm not sure why the caching wasn't better handled in the CDN that lives in front of it (cache hierarchies work well) leaving this server to simply serve only the very first request.
I wondered the same thing. Why not handling the connexion and serving the files through a regular event-based server like nginx, and handle the specific business logic in java or any other popular language inside Google.
I guess that some details on the slides would answer those questions, but if anybody here know the answer (some slides were pretty obscure if you're not already familiar with file serving and/or google architecture).
You have two options here, assuming you ignore option 0:
0) copy all the files from the DFS to the local storage.
1) Attempt to make a distributed filesystem available as a mount point (for example, Hadoop FUSE) on the nginx server so it can serve the data.
2) Make the nginx server know the distributed filesystem API and talk to it via user space directly.
The former leads to insanity (writing a really good, fast FUSE implementation is devlishly hard). The latter can work; I've seen a number of open source codes adapted to distributed filesystems.
This is made easiest if the underlying includes an IO abstraction layer. From a quick skim, it seems nginx has a OS abstraction layer (win32 and unix implementations):, which is more than sufficient.: http://trac.nginx.org/nginx/browser/nginx/src/os/{win32,unix.... From that, it wouldn't be hard to write a "distributed FS variant" (although IO and OS are not orthogonal concepts).
The next problem you're going to have is performance. nginx and other systems really are written with the expectation that IO is local, high throughput, and low-latency. Opening a remote file often takes some time (hundreds of milliseconds), and serial roundtrips cause additional latency, especially if DFS and server are not within 5ms RTT.
The next logical step is to write a readahead layer (if you're serving files larger than a single read block to the DFS) to get better throughput.
A combination of the above techniques, with a bit of tuning and elbow grease, is a good foundation for scalable (in terms of file size, total # of files, etC) serving.
