Etherpad-lite: Migrate data from dirtydb to mysql

Recently I've been playing with etherpad-lite, which is a port of Etherpad (a collaborative editor a'la Google Wave acquired by Google and then open-sourced) to node.js platform.

As my node.js project is getting ready for production use, I have decided to migrate from the default dirtydb to mysql.

Dirtydb is a fun little database written in JavaScript that just appends key/value pairs to a json-formatted text files and then basically reads in backwards to find a matching key if you need to query it. Super-simple and works nicely. The author claims it can handle up to 1M records.

I would happily stay with dirtydb, but etherpad-lite adds new revisions of the pad on every character typed, so 1M records can be reached pretty quickly with it, so I have decided to move to mysql as a backend.

Etherpad-lite uses a DB abstraction library ueberDB (the name probably should have been überDB, but the author didn't have the 'ü' on his keyboard? ;-)), which basically mimics the dirtydb interface, but can use other backends, like mysql or sqlite for actual storage.

You configure etherpad-lite/ueberDB to use mysql in the settings.json file:

   "dbType" : "mysql",
   "dbSettings" : {
      "user": "etherpad",  "host": "localhost",
      "password": "etherpad", "database": "etherpad"
   },

This already works provided you have created the necessary mysql database and user:

• mysqladmin create etherpad -u root -p
• grant all privileges on etherpad.* to etherpad identified by 'etherpad';

But if you have been using the app with dirtydb for some time, you probably want to migrate the data from dirtydb to mysql, here is a simple script to do just that:

  var dirty = require("dirty")('../var/dirty.db');

  var db = require("./db/DB");

  db.init(function() {

      db = db.db;

      dirty.on("load", function() {

          dirty.forEach(function(key, value) {

              db.set(key, value);

          });

      });

  });

Save it as {etherpad-root}/node/dirty2mysql.js, then

• cd node
• node dirty2mysql.js

If it doesn't do anything, try installing the dirtydb module explicitly (not as a dependency of ueberDB):

• npm install dirty

And you're done!