John Melesky

I was recently asked what my predictions were for trends in open source over the course of the next couple years. I wanted to restate and expand on that answer.

“Web frameworks are plateauing.”

¹ Not to detract from the rest of the pack. Many of them are just as capable as the frontrunners. That doesn’t change who the frontrunners are, though.

A couple years back, we saw the MVC web frameworks start to appear. We saw dozens of them. Some were full-stack, some patchwork. Some opinionated, some highly configurable. Eventually we saw Rails and Django rise up in popularity over the others¹. And along the way we learned some things about web development.

And now we’re evening out. The number of new viable web frameworks will drop in the coming year, because we’ve found and refined the dominant web app paradigm. Innovation will happen, but it will wait a couple years, the same way we waited after apache’s language mods (mod_perl, mod_python, etc) took over. Web frameworks are plateauing.

I’m not sure yet what the next step in web frameworks will be. Indeed, at this point it’s tough to imagine what might come next. We need the upcoming years to find the limits of AJAX RESTful unobtrusive MVC apps over sharded databases sitting on virtual on-demand servers. But we’ll run into the limits, then hack around them enough times to really get to know the problems. A new pattern (or five) will emerge, and a new generation of frameworks will pop up.

“The important fact is that these criticisms of RDBMSes are out there and becoming more popular.”

That’s great, but what’s the real next big thing? I took that web framework digression for a reason. We haven’t found the problems with them yet, but we have started to run into limits elsewhere. Specifically, i’m talking about our current crop of RDBMSes.

Let’s look at the criticisms that have been cropping up recently:

² Who created his 12 Rules specifically to counter the notion that DB2, Oracle, etc. were truly relational.

• SQL isn’t truly relational. This is an old criticism, one put forth by the creator of the relational calculus, Edgar Codd². It’s not clear how relevant this criticism is, but it is clear that the notion itself is gaining traction in the programming world.
• There is an object-relational impedance mismatch. This is one of the points where OO and the MVC/RDBMS models run into issues.
• General-use RDBMSes are slow and inefficient compared to specialized solutions. This notion is not just in industry, but in academia (PDF) as well.
• relational databases are some kind of sinister death cult. Hmm.

I’m not interested in debating whether these criticisms are valid (at least not in this article). The important fact is that these criticisms of RDBMSes are out there and becoming more popular.

³ It’s certainly debatable that many of these approaches are “new”. Again, the important part is that it’s becoming popular. After all, MVC was hardly new when Rails popularized it.

On the flip side, we’re starting to see alternate DB architectures take the stage³. Column-oriented databases are gaining steam for Data Warehousing applications. Object stores like Mnesia are becoming more popular as simple OO persistence backends. CouchDB is getting a huge amount of buzz for something as simple as a document store. And Google’s BigTable is causing tech-envy in the open source sector.

What’s this all mean? Well, it means the next big thing.

“The next big thing in open source will be database engines.”

The next big thing in open source will be database engines. Specifically, the next big thing will be a crop of database engines that implement alternate (i.e. non-SQL-based) solutions to storage persistence. We’ll see a bunch of object stores. We’ll see neo-relational DBs, possibly based on Tutorial D. We’ll see highly distributed stores. We’ll see people tackling the problem of distributed transactions. We’ll see databases that perform type inference on whole relations.

Not all of these will be built from scratch. I expect we’ll see memcached, the current darling of distributed caches, repurposed to a distributed object store. I also expect the venerable BerkeleyDB will continue to be leveraged by new projects.

Eventually, though, we’ll see a couple projects rise up to the top. And the big iron DBs will cherry-pick projects and features and attempt to incorporate them into their existing products. And we’ll be ready for the next big thing.

That’s my prediction. And you can quote me on that.

“This last big thing (web frameworks) didn’t see a great deal of competition from commercial software producers.”

The interesting piece of this, for me, is the commercial aspect. This last big thing (web frameworks) didn’t see a great deal of competition from commercial software producers. The medium was centered on dynamic languages like Perl, Ruby, and Python—stalwart languages of the open-source world. Databases are a different beast, and will face strict competition from the existing solutions, as well as higher demands on performance and reliability. Inefficient, buggy databases are of far less use than inefficient, buggy web frameworks. Who knows whether the open source community will be up to the challenge?

This will be an interesting couple of years.