« Why Outsourcing Sucks Yet another new web browser – Google Chrome »

Google Protocol Buffers

Posted August 26th, 2008 by Nazmul

Comprehensive Real-World BlackBerry Training Courses by developerlife.com

Want to learn from the Masters? We’ve created groundbreaking BlackBerry software (we are finalists in the 2009 BlackBerry Developer Challenge). We have a complete set of instructor-led training courses available for BlackBerry development that will turbo-charge your BlackBerry development efforts. Whether you are a newbie or an experienced developer, we have modules that will fit your needs. Learn more about our courses & schedule.

I’ve been looking for a good solution to serializing object/data streams from mobile environments to services. Object serialization doesn’t really work well here since these mobile environments all have different flavors of Java… some run Android, and others JavaME. On Android, object serialization between the DVM and a JavaSE VM works with lots of caveats, so that’s not really usable.

I came across Google’s Protocol Buffer API recently. Protocol Buffers are similar to XML schemas, that you can compile from a language neutral schema definition into code (in different languages like Java, C++, and Python). However, there’s no XML involved here, it’s a lightweight binary encoding/decoding mechanism. You create your schema definitions in a .proto file and you compile that into Java code that you include with your applications (services, and mobile apps). Google extensively uses this technology internally, throughout their apps, services, and infrastructure, so it’s vetted, and well supported.

You can get started using this technology with Java by reading this tutorial.

Here’s some interesting history that prompted Google to create this technology:

Protocol buffers were initially developed at Google to deal with an index server request/response protocol. Prior to protocol buffers, there was a format for requests and responses that used hand marshalling/unmarshalling of requests and responses, and that supported a number of versions of the protocol. This resulted in some very ugly code, like: 

 if (version == 3) {
   ...
 } else if (version > 4) {
   if (version == 5) {
     ...
   }
   ...
 }

Explicitly formatted protocols also complicated the rollout of new protocol versions, because developers had to make sure that all servers between the originator of the request and the actual server handling the request understood the new protocol before they could flip a switch to start using the new protocol.

Protocol buffers were designed to solve many of these problems:

  • New fields could be easily introduced, and intermediate servers that didn’t need to inspect the data could simply parse it and pass through the data without needing to know about all the fields.
  • Formats were more self-describing, and could be dealt with from a variety of languages (C++, Java, etc.)

However, users still needed to hand-write their own parsing code.

As the system evolved, it acquired a number of other features and uses:

  • Automatically-generated serialization and deserialization code avoided the need for hand parsing.
  • In addition to being used for short-lived RPC (Remote Procedure Call) requests, people started to use protocol buffers as a handy self-describing format for storing data persistently (for example, in Bigtable).
  • Server RPC interfaces started to be declared as part of protocol files, with the protocol compiler generating stub classes that users could override with actual implementations of the server’s interface.

Protocol buffers are now Google’s lingua franca for data – at time of writing, there are 48,162 different message types defined in the Google code tree across 12,183 .proto files. They’re used both in RPC systems and for persistent storage of data in a variety of storage systems.

 

Comprehensive Real-World BlackBerry Training Courses by developerlife.com

Want to learn from the Masters? We’ve created groundbreaking BlackBerry software (we are finalists in the 2009 BlackBerry Developer Challenge). We have a complete set of instructor-led training courses available for BlackBerry development that will turbo-charge your BlackBerry development efforts. Whether you are a newbie or an experienced developer, we have modules that will fit your needs. Learn more about our courses & schedule.

Leave a Reply

We use Thank Me Later.