Thrift IDL (protocol)

After the horrible experience with Avro, I considered using Protocol Buffer and Thrift for the company. Protocol Buffer's strongest point is that it is stable (not much has changed in the past few years). It is used in every single possible service in Google, it has gone through a very stringent code-review process, it has been written by the world's most seasoned and anal engineers, and thus has been well battle tested. However, I consciously passed over the opportunity to suggest Protocol Buffer to use for the company partly because I'm considered a bias party, and to suggest it will simply reinforce the idea that "Kevin is a Googler so he's obviously biased. He thinks everything coming out of Google is amazing." To be fair, I really think that Google cranks out shit end-user products most of the time (Wave, Buzz, G+, Location, Google Base, Android, etc etc...). Sometimes Google happens to make good end-user products only because Google throws a billion darts in the dark and occasionally one of the darts hits the bullseye. That's all.

I tested Thrift, and it is acceptable. In terms of feature, it is very similar to Protocol Buffer. The first thing I tested was message backward and forward compatibility. There was no problem in either case. Whereas Avro returns an error saying that message format is different, Thrift server gracefully (and correctly) disregards new message types or ignores old messages.

In Java Thrift, you can set your Thrift objects using getters and setters, which is great because if the message type changes (name or type), the Java compiler will give you an error immediately. In Java Python, you can also set your Thrift objects using the constructor and the runtime system will catch name errors. In contrast, Avro does not do any of this, so your program will just run along happily even though you're setting my_integer="Not an integer" and somewhere down the line your program crashes and you're scratching your head.

One last thing I love about Thrift: there is an asynchronous transport!!! This is exactly what powers AdSense, and allows people to easily prototype distributed computation architectures.

http://blog.rapleaf.com/dev/2010/06/23/fully-async-thrift-client-in-java/

There are a few Thrift "bugs" that should be fixed. For example, suppose you set the following as message definition:

2: string lastname = "last_default",

7: string lastname = "HO",

...

The above should signal a compiler error (e.g. "Same type name not allowed."). There are many other errors that should have signaled an error, but are not. I guess either they are too busy, too lazy, or just expect the compiler (either C or Java) to catch the error.

One other minor difference between Protocol Buffer and Thrift: In Thrift, there is no deprecation keyword. In Protocol Buffer, deprecation field compiles into Java, and the compiler will tell you the field is deprecated to allow programmers to update. It's not a big deal, but it may be a big deal for companies that keep updating contracts between two services.

In the end, my take on Avro vs. Thrift is like this. Avro is like Microsoft Zune. Zune has all the bells and whistles-- AM radio, recorder, more buttons, higher display resolution, external HD, blah blah blah. The iPod on the other hand, just does one thing. On paper, Zune is superior over iPod. On paper, Avro is superior over Thrift. But in the end, Avro just doesn't work well (no forward/backward compatibility, buggy buggy buggy and the developers don't even respond to my bug report). What looks good on paper, isn't necessarily good in practice. You can't trust everything you read. You have to play with it.

Avro, what a complete waste of time

I'm responsible for evaluating the different IDLs (Protocol Buffer, Avro, and Thrift) as a unified form of communication between different services in the company. A key feature of today's IDL is backward and forward message compatibility. For example, if the client adds one more field to a message, the server should be able to take the new message and process it (while ignoring the new field). The opposite is true, where the server takes in additional fields in the message while the client does not, and the server should just assume that the field is empty.

I started with Avro because I had high hopes for Avro. It had great features that neither PB nor Thrift had (no need for field deprecation, no need for deprecation, no need to get an IDL compiler), and because it's built in to Hadoop's MapReduce. My experience with Avro began with downloading the package (version 1.6.1, the latest). I tried out an example code (phunt-avro-rpc-quickstart-avro-release-1.2.0-9-gce46e91.zip) with included two small Python codes, start_server.py and send_message.py (client). Both of them used the same IDL (mail.avpr). I got the client to send a message to the server with ease. Then, I tried the most important aspects of IDLs-- forward and backward message compatibility. I expected the server to gracefully accept old and new messages, but instead got something completely unexpected:

PATH=~/code/avro-example/avro-1.6.1/src ./send_message.py AA BB MSG
Traceback (most recent call last):
File "./send_message.py", line 56, in
print("Result: " + requestor.request("myecho", {"mymessage": message}))
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 145, in request
return self.issue_request(call_request, message_name, request_datum)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 260, in issue_request
call_response_exists = self.read_handshake_response(buffer_decoder)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 204, in read_handshake_response
handshake_response.get('serverProtocol'))
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 120, in set_remote_protocol
REMOTE_PROTOCOLS[self.transceiver.remote_name] = self.remote_protocol
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 475, in
remote_name = property(lambda self: self.sock.getsockname())
AttributeError: 'NoneType' object has no attribute 'getsockname'

A well designed IDL should at least show a warning message indicating that the field is unknown (or new, etc). Nope! Avro returns with a weird socket-related error. Upon looking at the Avro library (avro-1.6.1/src/avro/ipc.py), line ~474 yields:

# read-only properties
sock = property(lambda self: self.conn.sock)
remote_name = property(lambda self: self.sock.getsockname())

So, I'm no Python expert but it's clear that self.sock does not exist, so I manually set remote_name in the constructor __init__ (meaning it's not a readonly variable anymore, but who cares) and viola, it works! Who the heck checked in this code anyways? My next attempt was the reverse: the server takes in a newer message and the client sends an older message and here's my very useful Avro message:

/code/avro-example/avro-1.6.1/src ./send_message.py AA BB MSG
Traceback (most recent call last):
File "./send_message.py", line 56, in
print("Result: " + requestor.request("myecho", {"mymessage": message}))
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 145, in request
return self.issue_request(call_request, message_name, request_datum)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 264, in issue_request
return self.request(message_name, request_datum)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 145, in request
return self.issue_request(call_request, message_name, request_datum)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 262, in issue_request
return self.read_call_response(message_name, buffer_decoder)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/ipc.py", line 222, in read_call_response
response_metadata = META_READER.read(decoder)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/io.py", line 445, in read
return self.read_data(self.writers_schema, self.readers_schema, decoder)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/io.py", line 486, in read_data
return self.read_map(writers_schema, readers_schema, decoder)
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/io.py", line 615, in read_map
block_count = decoder.read_long()
File "/home/vm42/code/avro-example/avro-1.6.1/src/avro/io.py", line 184, in read_long
b = ord(self.read(1))
TypeError: ord() expected a character, but string of length 0 found

Alright, I've had enough. What I just attempted, was a very very common test case and if there's any decent amount of unit tests, this problem would never have existed. My guess now is that there is no unit test whatsoever, and there isn't much user base because I can't find this complaint anywhere via Googling (and I can't find much Avro documentation in the first place)! I sent an email to the Avro developer team this weekend and I've yet to receive a response. I am most utterly not impressed so far.

Conclusion:

1) Save yourself some time by using something else that is battle tested. Bleeding edge (in this case) is a waste of time.

2) The only thing that matters is your hands-on experience. Marketing and bias makes Avro look amazing (dynamic features, flexibility, maintenance free, language support, ...), but it doesn't matter if it does not work TODAY.

Recommending a fellow Googler to a startup

Once in a while I get this question: "Hey Kevin, thank you for recommending a fellow Googler. What do you think about X's technical skills?"

My long response is this:
The Google technical interview process is one of the most challenging interviews one can get. There's the resume screening (only one out of 1000 resumes pass through), then email screening, then phone screening, possible secondary phone screening, on-site screening, and finally the hiring committee (from Mountain View) reviews long and very detailed written feedback from 7-10 interviewers. If someone makes it in as an engineer, you are sure that person is way above average over the millions and millions of people who send in their resumes to Google each year. Think about this: if you're an engineer at less-than-stellar company that don't value core engineering (Fox, Yahoo, Citysearch, AT&T Interactive, MySpace) and you think you can do better, you have already at some point in your career applied to Google. It's only human to want to do better. Look. Chances are, an engineer you already know (who never went to Google) already applied and chances are he/she failed. I realize what I'm saying is really harsh, but this is harsh reality. For this reason, even a really bad Googler is still above tech industry average (e.g. especially from what I see in Los Angeles). Secondly, if a person survives the Google culture for a few years, you're sure that person is at least average amongst Googlers because the below average Googlers get kicked out very very fast; 2 to 3 quarters and you're out. I personally know a few that don't survive a year-- usually they're super smart but unmotivated and/or had other issues.

Having that said, technically, almost everyone I know at Google can kick the industry average programmer's ass. Googlers tend to come from top-tier schools or top-tier companies. They made it into the system. They are hardcore, trained under the stringent Google Code Readability process. People strive to get badges on their Moma page by being Googly-- being technically good. I am not exaggerating or bragging, I'm just saying this after observing different people from different backgrounds, and relative to Google, the average tech standard is a pathetically low bar.

Anyways, if I vouch for someone from Google, then that person is almost more than technically adequate. But then again, so is 90% of the other Googlers. There are of course distinctions amongst the group of the Special Force. Some people are slow but precise (they like to work on mission critical code). Some people are fast but sloppy (they like to work on social networking sites). Some like Java. Some like Python. Some like Javascript. Some like C++. Some people are smart, and some people are simply mind blowing brilliant. The Google gene-pool isn't all homogenous.

In the end, you should not have to worry about a Googler's technical skills. You may however, have to worry about many other things, like being able to give them challenging enough of a task, making them feel like they're making a big impact to the world, and providing enough incentives and rewards for keeping them; believe me, everyone is getting poached here and there these days with ridiculous packages. Keep in mind, there's a reason why Google managers tend to come and go very fast-- an x-manager once commented to me that it's really really hard to motivate and manage someone who is clearly much smarter than you are. I wasn't a manager at Google but I can understand why. Some of the smartest people I've met in the world are people I met in Google, and a few are a total pain in the ass to work with.

Product Managers

My startup is hiring a product manager because none of the engineers want to spend day after day doing product research, testing/trying out competitors' products, meeting solution providers and clients, blogging, product evangelist, PR, writing white-paper, doing patent research, and other things that engineers feel are too un-intellectual to do. While I have a nice Google system for interviewing elite engineers, I don't have a good system for hiring a good PM. In the end, I think there is so much variability in PMs that I don't think you can actually design a process for it. I do however think the most important thing is that the PM's style should match well with the overall vision of the company. For example, if the company is creating a business product, then the PM should be of the "spec hunting" type (more on this later). If the company is creating a user-end product, then the PM should be of the Steve Jobs type. In a hypothetical world were there are only two extreme types of PMs, then they are described as follows:

1) The most common PMs are the "spec hunters." They will write down specs that their competitors have, and try to compete on specs. They will list out a long matrix of features and check them off one by one. Case in point a few years ago the Microsoft Zune on paper is much more feature rich than anything out there. It's got a voice recorder, FM radio, more storage, Oled, 720p, 33 hr play time, so on so forth. The iPod does not have a voice recorder, does not have a FM radio, smaller storage, older/less resolution display, 30 hr play time. It doesn't do much. Almost all of Microsoft products are spec'ed products, designed by a committee with a long feature list that each committee member checks off. On paper, the Zune is clearly superior to the iPod. However, hiring a "spec hunting" PM is not a good match for consumer products; we all know the story with Zune vs. iPod today. Zune is dead.

Other examples of "spec hunting" PMs: America Online and Y! [homepage] are committee designed -- the idea with those products is that the more stuff you slap on a page, the happier the committee. It is no surprised that AOL looks like a mess. Ditto with Yahoo. Dell is yet another example. The Dell laptop on paper is superior to the Mac-- brighter screen, more HD, faster processor, bigger capacity battery, 1/2 the price. The list goes on and on and on.


2) The less common PMs are the minimalists. One example-- Steve Jobs is one such PM who designs with minimal features. Other example include Porsche, Ferrari, and other Italian designed products; all these cars have minimal features that simply run fast and look nice, and none have fancy OnStar or XM radio or GPS display or voice activated commands built in. Dropbox is in this spectrum too... it just does one thing-- let people drop files into the file system. Google search is another example of a minimalist page; you don't do anything on the main page except to search. http://www.google.com/corporate/tenthings.html Read point #2. Here the classic example: the Apple iPod only does one thing (whereas Zune and competitors do 100 other things), yet it still slaughtered competition. Zune is dead and the spirit of iPod lives on in the form of iPhone and iPad.

The minimalist PMs understand the power of looks, feel, navigation, intuitiveness, cohesiveness, and consistency. Whereas the "spec hunters" think they have taste, the minimalist PMs actually have taste.


In the end, I see that most companies are "run by committees" ruled by "spec hunting" PMs. I think the reason is clear-- minimalists who have a sense of taste are as rare as diamonds. Of course, there's nothing wrong with that. Despite Microsoft's extremely tasteless designs (case in point Microsoft Bob aka The Useless Paperclip Helper), it is still one of the most successful companies in the world.

Koobface, a highly organized botnet

http://www.infowar-monitor.net/reports/iwm-koobface.pdf

Not worried about botnet affecting the online ads industry? Read about the sophistication of Koobface. Here's an example: "...The operators of Koobface have been able to setup a stable botnet infrastructure that allows them to maintain tens of thousands of compromised computers and profit immensely from PPC and PPI, earning a total of $2,067,682.69 between June 23, 2009 and June 10, 2010."

My favorite part is faking CAPTCHA to users to create new spam accounts. This is pure evil and brilliant at the same time!

Archiving Public Materials vs. First Amendment

News: Archive of Geocities will be released as a ~1TB torrent.
http://www.techdirt.com/articles/20101029/03055711647/archive-of-geocities-released-as-a-1tb-torrent.shtml
While this is great news to archivists and web historians, for the rest of the people who contributed to Geocities, their content is now exposed to the public. What if some college kid posted dark secrets that could now jeopardize his life? For example, what if a college kid posted a picture of him smoking pot 15 years ago, but is now running for a seat in Congress?

Archiving is full of controversies. Over 15 years ago, I thought it would be really cool to create a Time Capsule by archiving a bunch of my friends' web sites. At the time, I thought it would be interesting to see how people change over a decade, and if we ever have a reunion we'd be able to look back at the Time Capsule and reminisce about the good 'ol days. At the time, I was naive and thought archiving content would be like taking pictures-- people usually thank photographers for capturing precious moments that would have been lost without cameras+films. However, I've come to the conclusion that archiving content is not the same as taking pictures of precious moments. The reason is that most people seem to be embarrassed by the very same content they uploaded many years ago! I know this because in the past few years I've been getting requests after requests by these same friends to take down their sites. I can't really talk about the content of those sites publicly, but needless to say, the contents usually contain materials that are potentially harmful to their professional and/or personal lives.

In almost all cases I'd take down an archived content when asked to do so because I want to be nice. Now, let's suppose I don't play nice. What if I leave the archived content and somehow, the content becomes harmful to those people (e.g. pictures of them smoking weed when they were young, or blog exposing their weird fetishes/political party/admission of crime, and such)? Will the person who repost the content be liable for ruining their professional and personal lives? How about Google, Yahoo, Ask, Bing, Facebook, Twitter, Yelp, and a zillion other services that archive content... can these sites be held liable for archiving potentially damaging materials to the uploader? How does the First Amendment protect from reposting archived public domain content? How much legality is there to leave the content? How much obligation is there to take down the content?

As you can see, arguments can go in all directions. So I asked my friend D. Silverstein who is a lot more familiar with law than I am, and here is his response:

======================================
Kevin,

Much of what I say is based on seeing the EFF's copyright/patent/trademark for programmers. First of all, the copyright notice doesn't matter. Under modern IP law (as agreed to by most developed countries), things are automatically copyrighted the moment they pop into existence. Technically, when you send an email to someone it has a copyright. But, given how email is used, it's very questionable that you could enforce that copyright, i.e. sue or collect a fee from someone who forwarded your email without your permission. Web pages are a little different. They have some more protection, but still, if it's a web page on the public internet, it's probably fair game. But there are some complications. Is the web page protected by a robots.txt file? Not all indexers respect robots.txt, but let's assume the ones in question did... the first thing that will reasonably happen before people actually try to take legal action is they will send you a cease and desist (C&D) letter. I take the stance i.e. if you're not hosting something that has legal attack dogs tied to it (i.e. copyrighted music or movies), just put it up, and don't worry about it until you get a C&D. If you get a C&D you can, at that point, decide if you want to fight or take it down. Also, since you're archiving it... you may qualify as a third party for DMCA safe harbor purposes, assuming someone gives you the content to host, or ask you to host the content. If you receive a takedown notice and honor it, you're not liable. If you simply copied it, then you probably don't qualify as a third party provider for DMCA safe harbor. Just to give an example, third party provider or ISP for DMCA safe harbor purposes is, e.g. YouTube. They host content that other people post. Either way, don't worry about it unless you get a C&D. Keep in mind that C&D's are basically lawyer nastygrams. They cost nothing to send, and they rack up a couple billable hours. So usually that's the first step someone takes before taking more serious action, i.e. trying to sue you. In short, the copyright is the one that has the most teeth, but these aren't secrets and this isn't content that he's trying to profit from. There might also be an angle where he could go after you for defamation... but if it's content that he wrote or collected... then I would think it would be hard to make a defamation case. And, in general, litigation is expensive.

Lastly, if this person is famous (e.g. running for public office/people who are highly public, such as celebrities, getting a promotion at work, etc), he/she knowingly give up a reasonable expectation of privacy that "private" citizens are entitled to. We actually owe that to Larry Flynt, publisher of Hustler Magazine. :) They published a satirical article that suggested Jerry Falwell committed incest, and it went to the supreme court. The First Amendment's free-speech guarantee prohibits awarding damages to public figures to compensate for emotional distress inflicted upon them. Thus, private citizens are entitled to more privacy than celebrities! There are of course exceptions to libel/slander and copyright laws for parody. Obviously if you publish malicious falsehoods about a celebrity, you may still be liable. That would be considered lying, and that is a totally different can of worms.

D. Silverstein

=======================================================

So, that is his response, and it makes sense, but it makes me wonder what the world will be like 15 years from today.

In the mid 1990s, only geeks logged on the internet and created their own web sites filled with ugly HTML/blink/bold content. Fast forward to the 21st century and you see that almost everyone has contributed some content to the public web, be it a blog, Facebook messages, Flickr pictures, Twitter messages, Yelp review, Amazon votes, so on so forth. I can't help it wonder what the ramifications of archive be in yet another 15 years? Facebook (and Google and Twitter and all the other sites) are archiving EVERYTHING people contribute today. There's MORE content than ever, and much of the data will be potentially damning to people's lives in the future. What do you think could happen 15 years from today? Will there be lawsuits from people to take down archives? How much leg will they have? Should the law protect common people from embarrassing themselves, or should the law protect archivers?


-Kevin

=======================================================

Follow-up:
http://techcrunch.com/2011/01/01/california-bill-criminalizing-online-impersonations-in-effect-starting-today/
"California’s SB 1411, which adds a layer of criminal and civil penalties for certain online impersonations, goes into effect starting today." Is archiving considered impersonation?

Hiring, hiring, hiring... and... NO GO.

My company is cash flow positive. The VC is getting ambitious and wants to expand expand expand. The problem is, I haven't met a single candidate that blows my mind. 90% of the resumes look horrible, and 98% of the candidates I interviewed are utterly awful, and mediocre at best. The founding eng is a hacky band-aid hacker-- non computer science. I'm a second eng. The third hire is pretty good, but only after scouring 100s of resumes. I've been trying very hard to bring good Googly culture in -- like HIRE ONLY THE BEST so that you don't have to manage, and so there will be no need for silly hierarchy and perf and promotion committee. Some take-away points in my startup so far:

1) I use all standard Google interview process. That screens out 98% of the people who call themselves "developers." Case in point: "Cal State LA doing B2B in J2EE asking for 120K salary." -- this guy can't even do recursion or order 4 functions in the increasing order as n approaches infinity. Or, some can't even do a Venn diagram to explain some hypothetical question! A large # of these guys will give you a dumbass linear scan answer to an question that can be solved in O(1)! Most are just mind-boggling stupid!!!
2) I can't believe how high they're asking for their salaries (doing banking, defense, bullshit B2B or other BS web site). Something doesn't sound right. These companies (in Southern Cal) that pay them that type of salaries are probably stupid, desperate, or both.
3) In my entire life, I'm used to being with people like me (similar background). I've been in the ivory tower and a corporation [that can be best described as a bubble], but I now realize that living in an academically driven bubble is not normal. What is normal in life, is having to deal with a bunch of "normal people" -- loud talking ones. Instead of doing work, I now need to spend much more time explaining, teaching, guiding, and even delegating! UGH!!! Ultimately, the sudden realization that I stepped outside of the ivory tower/bubble/whatever you call it, and having to deal with "normal" people makes me feel-- very very VERY lonely at times.
4) Hiring people who are anywhere close to the average caliber in Google (which unfortunately, doesn't even say that much these days) is near impossible.
5) The VC pressure to do more things, faster, expand, is killing my ability to do just that.

You know anyone who have/had similar frustrations? What's the best way to go about this?