The State of MySQL

Even if you’ve only been paying marginal attention to the MySQL world in the last few months, you probably know that there’s been a lot going on. But putting the Sun and Oracle headlines aside, there’s even more happening in the MySQL community–both from a technology and organizational point of view.

Coming back from the MySQL Conference a few months ago, one thing was very clear: The ecosystem I’d long expected to develop around MySQL has really come of age. The expo hall was packed with hardware vendors hawking their MySQL appliances, software vendors who work with MySQL, consultants, trainers, and various related Open Source projects.

However, nobody could escape the news about Oracle’s proposed acquisition of Sun. Heck, some of us are still getting used to the idea of MySQL being part of a company as large as Sun. See Bryan Richard’s coverage for more on the Oracle story.

Rough Seas

It has not been the smoothest of sailing recently. MySQL was having problems even before the Sun acquistion shook the community. In it’s effort to help the product “grow up” and cater both to enterprise users (which means adding more functionality) and newer, less sophisticated users (which means making things more friendly and approachable–and building more tools), the team seemed to lose direction and ended up being spread too thin.

The large gap of time between MySQL 5.0 and 5.1 is partly to blame on management and priorities. But it’s also largely the result a codebase badly in need of a good cleanup and refactoring–something that has not happened to this day (though there are signs of that changing). The unnecessary complexity and lack of clear standards not only slowed down existing developers but was surely a barrier for new developers as well.

These problems continued and people began to depart Sun not long after the acquisition. Members of the senior management team as well as some of the most experienced and talented developers are no longer there to guide the development. A subset of those who’ve left are actually contributing to the MySQL community in other ways–more on that later.

Another surprising discovery at the conference was slow adoption rate of 5.1 in larger shops. I personally talked with my counterparts at some of the largest Internet companies was surprised to find how few of them had jumped from the 5.0 series to 5.1. In many cases it just wasn’t a compelling upgrade yet. But that’s just another way of saying that even the latest and greatest MySQL 5.1 didn’t do much to solve problems they are actually battling.

In retrospect, I shouldn’t have been that surprised. Their experiences mirrored my own and played into a much larger trend that’s been developing around MySQL in the last year…

Outside Innovation

A striking amount of new development has been happening outside of Sun/MySQL–both in terms of features and scaling/performance. And it’s a trend that seems to be accelerating. Probably the most well known example of this is InnoDB, the storage engine of choice for most MySQL deployments. Innobase Oy, the company that developed and supported InnoDB, was bought by Oracle a couple of years back.

In the last year they’ve released the InnoDB plug-in with a new file format, data compression support, and many performance and scalability improvements. Unfortunately, a lot of that development has happened behind closed doors despite being open source. This has often left the community wondering what their roadmap and timelines look like.

For years there have been outside consulting companies that specialized in MySQL troubleshooting, tuning, and feature development. Several of them were even started by former MySQL employees. One of them in particular, Percona, has put a lot of effort into providing patches for MySQL and InnoDB. Recently they’ve gone even farther and created an enhanced version–a fork–of InnoDB called XtraDB. They’ve also developed an open source hot backup utility called xtrabackup. Previous to this the only option was to pay Oracle for the commercial InnoDB hot backup utility.

Percona’s work is getting traction too. The high profile photo sharing site SmugMug has been an early adopter of their work. Founder Don MacAskill has written quite a bit about his work with MySQL, XtraDB, Solaris and ZFS on his blog. We’ve begun testing it at Craigslist as well.

In true open source form, Percona is standing on the shoulders of giants–some really big giants, in fact. Some of the enhancements found in XtraDB were either inspired by or directly copied from work done by a small team at Google that focuses on supporting MySQL and adding features that Google needs. Led by Mark Callaghan, who also blogs about this work, they’ve enhanced replication, created numerous performance and scalability patches, and added useful statistics to both MySQL and InnoDB. Most of their work to date has been in the MySQL 5.0 tree, since that’s what they’re using, but some features have been ported to MySQL 5.1 as well.

Monty Returns

Not long aog, Monty Widenius (the co-creator and “father” of MySQL) left Sun after a year of frustration trying to influence the development of MySQL. He took several other Sun/MySQL employees with him and has formed a new company called Monty Program AB to work on a truly open and community centric fork of MySQL that he’s calling MariaDB which will include the Maria storage engine (a replacement for the venerable MyISAM). This new company will do consulting as well as MySQL/MariaDB development for hire and work on community features as well. Monty’s blog is a good source for keeping track of what he’s working on.

Monty is being very open how the how his new company operates. He adopting and enhancing the philosophy that guided MySQL AB while it was independant. You might think that he is directly challenging Sun/MySQL (or Oracle/Sun/MySQL soon)–and on some level he is–but the reality is that the MySQL ecosystem is much larger than most of us realize. I believe there’s room for the Oracle/Sun/MySQL juggernaut as well as many other companies and customized or forked distributions.

This is not unlike Linux. We all accept that Red Hat, Ubuntu, and SuSe are viable distributions of “Linux.” But the reality is that they’re all different in often subtle ways that only matter to a small percentage of the userbase. Regardless, they and other vendors continue to thrive. I believe the MySQL community is witnessing a similar evolution right now.

And that’s not all…

Believe it or not, that’s just the beginning. There’s a lot more interesting development going on in the MySQL universe these days. From new storage engines (Falcon, PBXT, Maria, Tokutek) to full-blown appliances from half a dozen vendors, and an extensive re-write of the MySQL database kernel. I’ll be covering several of those here in the coming months.

What’s most surprising of all is the fact that MySQL 5.4 was unveiled at the conference. It’s not a stable release yet, but it represents a compromise between 5.1 and waiting another year or more for stable builds of MySQL 6.0. Realizing that development was going more slowly that they’d like, the MySQL team picked some of the enhancements planned and coded for 6.0 and moved them into the 5.1 codebase, thus making MySQL 5.4. (They apparently wanted to skip 5.2 and 5.3.) Martin Streicher covered this earlier.

This is the first of several recent signs that the MySQL organization has begun to work toward improving some of the things that bothered Monty so much. That, to me, is encouraging. I expect and hope that it continues.

Finally, I’d like to say it’s good to be back working with linuxdlsazine. Having written several feature articles and many columns years ago and then taken a few years off, I can only say that it’s good to be back. linuxdlsazine has a long history of providing unique and intelligent coverage and analysis of Open Source technologies and businesses. I’m glad to be part of that on a regular basis again.