Adam Wiggins says that SQL Databases Don’t Scale.   Some of the comments there mention Oracle RAC being able to scale quite well, but I do not know if RAC still has a scaling limit, albeit higher.   Maybe Oracle’s RAC limit is effectively infinity (much like no one will need more than 640K ram), in which case RAC solves our problems, and the discussion is complete.

But, let’s assume SQL databases do not scale now, I believe they can be scaled without changes to the application logic.

The solution can be found in Sharding; which means partitioning the data between servers according to access patterns.  I propose automatic sharding which will take the database requests, at the client end, and redirect those requests to the machine(s) with the required shard of data.

Automatic sharding should be completely possible:  Database constraints reveal the strongly connected data, but also reveal the natural break lines in that data.  Application access patterns (from profiling) can provide evidence of about what tables do not change often, and what data is ripe for replication.

For example, the automatic sharder should “see” that partition by user id is effective because the data dependencies between users is quite small.   Furthermore, the mutually dependent portion of the database will consist of lookup tables, and other rarely changed data; which can be replicated given the few times it changes.

The relational database was designed around the independence of rows.  This row independence is necessary for highly parallel operations, which is exactly what sharding needs.   If it is true that the database community has been “trying to solve a problem for twenty years and still haven’t managed to come up with an obvious solution”, then I am dismayed:  After sharding a database or two, it should be obvious how to automate the sharding.