Many folks would say the database is where scaling problems belong.
This makes sense but I've found the db to be a rather large bottleneck. On one hand, it's nice to store everything in a single blackbox and the apps don't have to care about data. On the other, databases can be slow and I figure some kind of caching is involved between them.
I could see that a site would simply use the db to generate static pages which would then be cached, but as you said, multicast messaging with a database backend seems like a tricky combination.
Edit: I realize I've kinda thought outloud here. I've summed up the general idea with the question below.
I take the problem to be that going to the actual database is necessary. What are the possible solutions for something like this?
Horizontal partitioning seems like the obvious answer. Is it that easy, though?
Horizontal partitioning depends on your data. Do you have lots of joins and interdependent data? Partitioning will be tricky. Do you just need to store a lump of data relevant to one customer, many times over? Horizontal partitioning is easy.
Relational databases are really cool. They're not the right technology for all apps though. The big wave-makers right now are things like Google's BigTable and Amazon's SimpleDB. They're implicitly horizontal but with reduced querying abilities compared to traditional SQL.
It's a hard problem to scale a traditional relational DB across multiple independent commodity boxes like we see at Google or with AWS. The "stuff everything in a giant hashtable" approach scales nicely across a bunch of cheap boxes.
This makes sense but I've found the db to be a rather large bottleneck. On one hand, it's nice to store everything in a single blackbox and the apps don't have to care about data. On the other, databases can be slow and I figure some kind of caching is involved between them.
I could see that a site would simply use the db to generate static pages which would then be cached, but as you said, multicast messaging with a database backend seems like a tricky combination.
Edit: I realize I've kinda thought outloud here. I've summed up the general idea with the question below.
I take the problem to be that going to the actual database is necessary. What are the possible solutions for something like this?
Horizontal partitioning seems like the obvious answer. Is it that easy, though?