Python 2.7.3 serializer speed comparisons

In a recent python project where I was sending multiple messages per second of data over a basic socket, I had initially just grabbed the cPickle module to get the prototype proof-of-concept functioning properly. cPickle is awesome for easily serializing more complex python objects like custom classes, even though in my case I am only sending basic types.

My messages were dicts with some nested dicts, lists, floats, and string values. Roughly 500-1000 bytes. cPickle was doing just fine, but there came a point where I wanted to investigate the areas that could be tightened up. The first thing I realized was that I had forgotten to encode cPickle in the binary format (the default is ascii). That saved me quite a bit of time. But then I casually searched online to see if any json options might be better since my data is pretty primitive anyways.

I found UltraJSON, which is a pure C json parsing library for python, and ran some tests. There are benchmarks on the project page for ujson, as well as other articles on the internet, but I just wanted to post up my own results using a mixed type data container. ujson came out extremely fast (faster than cPickle in binary)

This test included the following:

Here is my IPython test script using timeit for each encode and decode step. I’m using IPython 0.13 with python 2.7.3.

import pickle import cPickle import json import simplejson import cjson import ujson import yajl d = { 'words': """ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris adipiscing adipiscing placerat. Vestibulum augue augue, pellentesque quis sollicitudin id, adipiscing. """, 'list': range(50), 'dict': dict((str(i),'a') for i in xrange(50)), 'int': 100, 'float': 100.123456 } print "*Encoding*" print "pickle.dumps" %timeit _ = pickle.dumps(d) print "pickle.dumps (binary)" %timeit _ = pickle.dumps(d, 2) print "cPickle.dumps" %timeit _ = cPickle.dumps(d) print "cPickle.dumps (binary)" %timeit _ = cPickle.dumps(d, 2) print "json.dumps" %timeit _ = json.dumps(d) print "simplejson.dumps" %timeit _ = simplejson.dumps(d) print "cjson.encode" %timeit _ = cjson.encode(d) print "ujson.dumps" %timeit _ = ujson.dumps(d) print "yajl.dumps" %timeit _ = yajl.dumps(d) pickle_src = cPickle.dumps(d) pickle_src2 = cPickle.dumps(d, 2) json_src = json.dumps(d) print "*Decoding*" print "pickle.loads" %timeit _ = pickle.loads(pickle_src) print "pickle.loads (binary)" %timeit _ = pickle.loads(pickle_src2) print "cPickle.loads" %timeit _ = cPickle.loads(pickle_src) print "cPickle.loads (binary)" %timeit _ = cPickle.loads(pickle_src2) print "json.loads" %timeit _ = json.loads(json_src) print "simplejson.loads" %timeit _ = simplejson.loads(json_src) print "cjson.decode" %timeit _ = cjson.decode(json_src) print "ujson.loads" %timeit _ = ujson.loads(json_src) print "yajl.loads" %timeit _ = yajl.loads(json_src) """ *RESULTS* Python 2.73 IPython 0.13 *Encoding* pickle.dumps 1000 loops, best of 3: 527 us per loop pickle.dumps (binary) 1000 loops, best of 3: 576 us per loop cPickle.dumps 10000 loops, best of 3: 56.8 us per loop cPickle.dumps (binary) 100000 loops, best of 3: 11.3 us per loop json.dumps 10000 loops, best of 3: 30.9 us per loop simplejson.dumps 10000 loops, best of 3: 36.2 us per loop cjson.encode 10000 loops, best of 3: 22.9 us per loop ujson.dumps 100000 loops, best of 3: 7.03 us per loop <== FASTEST yajl.dumps 10000 loops, best of 3: 23.2 us per loop *Decoding* pickle.loads 1000 loops, best of 3: 652 us per loop pickle.loads (binary) 1000 loops, best of 3: 234 us per loop cPickle.loads 10000 loops, best of 3: 43.9 us per loop cPickle.loads (binary) 100000 loops, best of 3: 13.7 us per loop json.loads 10000 loops, best of 3: 63.2 us per loop simplejson.loads 10000 loops, best of 3: 33.8 us per loop cjson.decode 10000 loops, best of 3: 36 us per loop ujson.loads 100000 loops, best of 3: 12.8 us per loop <== FASTEST yajl.loads 10000 loops, best of 3: 27.4 us per loop """