Dealing with the missing rawBody

If you’ve been following development trends in the past couple of years you have surely heard of Node.js by now. Node.js has been getting all sorts of attention lately, some of it has been positive and some of it has been, shall we say… critical. I think the reason many people seem to love Node.js is because it seems to offer a way for backend/server-side developers and front end developers to potentially work in the same language: the easy to learn, ubiquitous and well understood Javascript.

One of the first things a developer wishes to do when trying Node.js for the first time is to see how easy it might be to create a web server. In fact, many tutorials, documentation pages and blog posts provide instructions on how to code a simple “Hello World” web server to demostrate just how easy it is to create a fast webserver with Node.js. Building on this, one can quickly arrive at a very popular library used for creating web applications: Express.js. I installed it with the use of the fantastic npm tool, a package manager for Node.js (which was recently bundled with the Node.js installation itself). After reading the documentation for a bit, I was quickly able to set up some routes for my webserver to be able to respond to various different types of requests. Express also allows for all sorts of configurability. It is able to integrate with HTML templating tools such as , allows for custom “middleware” to modify or inspect the HTTP request object, logging, and all sorts of other powerful features. I really encourage you to have a look at Express.js. It’s wonderful. Really.

So, fast forward a few weeks and I begin working on a RESTful API and have decided to go with Node.js and Express.js to power the implementation. With Express’s route handling, nothing could be simpler for assigning URLs to functions to implement the API. Since the REST service was going to be dealing with JSON exclusively, I needed a way to easily grab any JSON documents that would be PUT or POSTed to the various URLs that I desired. Express has a mechanism that allows for the extraction of that sort of content from HTTP requests called bodyParser. With the use of bodyParser (an Express.js middleware), the request object will be equipped with a new property called rawBody, which is perfect for obtaining POSTed JSON with minimal hassle.

Unfortunately, in a recent upgrade of Express.js, the rawBody property in the request object was silently removed. I think it’s telling that something as important as this was removed so casually and with very little documentation. Personally, I think it should have waited for a major release or, perhaps, been much better publicized as a “heads up” to the user community. This is the sort of thing I think folks getting serious with Node are going to have to learn to deal with for a while until the project stabilizes in the years ahead.

Needless to say, the loss of rawBody had a catastrophic effect on the implementation I was working on and I quickly reverted the version of the libraries I was using backwards so that rawBody was restored. I resolved to look into it further at a later date.

Here’s how to add rawBody back again if you need it like I did: make your own custom Express.js middleware

     // For Express 3 (won't work with express 2.x)
     var app = express();
     app.use (function(req, res, next) {
         var data = '';
         req.setEncoding('utf8');
         req.on('data', function(chunk) { 
            data += chunk;
         });
         req.on('end', function() {
             req.rawBody = data;
             next()
         });
     });

     // Now your routes from here down
     app.get('/something/:id', something_handler.get_something);

What’s happening is that we have injected a custom middleware function that sets the encoding to UTF-8, then sets up callbacks that accumulate any sent JSON data into the “data” variable. When the request is finished, the contents of “data” are complete and req.rawBody is defined before next() is called, which passes control to any subsequent middleware functions.

The critical thing here is that this code must be added BEFORE your routes are setup. If you use the middleware afterwards, it won’t work properly…