Writing Code Like There Was No Tomorrow

There has been a lot of engineering activity during recent weeks. That has kept me busy writing code and I totally slipped blogging effort. Writing code and blogging seem happen in different planets.

Anyway, here is a short summary of recent engineering activities:

• Galera MySQL integration has been merged with 5.1.28rc release. This was actually quite simple operation. Our footprint inside MySQL code is either not too large and/or we have been so lucky to not get too serious conflicts. Actually most merges have been as fluent as this one so far. One exception was the merge between 5.1.22 -> 5.1.23rc, which took one full week to resolve.
  Our short term plan is to ship our MySQL integration code in MySQL launchpad bazaar repository in the near future. It just takes some code cleanup before we dare to publish.
• Loading large MySQL dumps caused unnecessary memory image growth and we needed to implement new memory management model to better control how Galera uses memory resources. Our final target is to define strict memory limits where Galera must be able to operate. Currently we allow ~100M for Galera processing and it seems more or less to fit in that space.
• DDL support has been extended to allow concurrent DDL and DML processing. We provide DDL processing by a method we call "Total Order (TO) isolation". The DDL statement is replicated to each node up front, and they wait for their turn in TO queue together with all other (DML) statements. The processing of DDL statement happens while keeping TO monitor locked, no new certified transactions can intervene, while DDL is processing.

  The nuisance with this implementation is that local state transactions are still free to process and can conflict with the DDL. I resolved this by providing prioritized abort mechanism for MySQL table level lock manager (thr_lock.c). I tried to avoid messing up with MySQL lock manager, it is by no means a joy ride you would take on voluntary basis.

• Initial version of cluster state exchange has been implemented. Now new nodes can now join the running cluster at will. This will eventually make it possible to provide full fail over feature with state catch up .e.g. by database snapshot transfer. Just wait and see, so do we...

Testing is difficult - osdb is busted

I tried benchmarking with osdb test, but got disappointed by the test methodology. osdb measures performance through a single connection (although having several other connections to the database). For database cluster testing, this does not make much sense. A cluster cannot speed up the processing of single connection in any way. On the contrary, clustering adds some overhead to each query processing, the penalty of synchronizing with the cluster. The performance of database cluster comes from the concurrency of multiple connections. As Jamie and Adam would put it: "osdb is busted", we need to look for something else.

At the moment, I'm setting up sysbench test to see how it behaves. sysbench has nice multi host property, which does load sharing directly in the application. (well, just like our sqlgen tester does). Also, the sysbench oltp test seems quite good quality at first glance.

All in all, we are in a situation where it is difficult to find any more suitable benchmarks or publicly available test applications. At this point, it would be good to test Galera/innodb with some real applications. If you happen to have interest to try Galera with your application, just contact us and we'll see how you could contribute.