I’m eventually going to write a more formal piece on my company blog, but in the meantime snark will suffice.
TechCrunch has been slamming Twitter and Rails . Let’s have a look at what they’re really saying.
(With apologies to Mark Pilgrim and John Gruber )
[Twitter’s] small team contains a handful of engineers, with only a person or two committed to infrastructure and architecture. He goes on to point out that at Digg the team for network and systems alone is bigger than the total engineering team at Twitter
I have never read the Mythical Man Month. More people are better.
I’ll also ignore the fact that Scribd has more web traffic than Twitter with only 11 total employees, information which I can get from my own affiliate site
The problems at Twitter are often attributed to their use of RubyOnRails, a web development framework.
This sentence gives me an easy way of saying I never blamed RoR for Twitter But of course it’s Rails’ fault, it’s “often attributed”! I will also ignore Evan Williams’s statement ‘Lots of our code is not in RoR, already, though.’
Twitter is almost certainly the largest site running on Rails
There are various ways of interpreting “almost certainly the largest”. One of which is ‘6th largest’.
Utilizing a framework that has never conquered large-scale territory [...]
I don’t consider the 474th or 775th ranked Alexa sites ‘large-scale territory’.
As an out-of-the box framework, Rails certainly doesn’t lend itself to large-scale application development.
I shall now pull terms with no meaning like “out-of-the-box framework” out of my ass. Oh and in case you still haven’t gotten my point, Twitter doesn’t scale because Rails doesn’t scale. Marvel at my skilful use of the non causa pro causa and cum hoc ergo propter hoc logical fallacy combo!
[...] But the old adage of “Good, Fast, Cheap – pick two” certainly applies;
Will trotting out a cliche let me get away with making subjective assertions? Hell yeah!
Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks
I shall now compare this “out-of-the-box framework” to two languages, without any substantiation. And look at how I wriggled out of that one… I have the “would do itself no harm by conceding” qualifier! I never said it isn’t a competitive platform. And if you read the comments, I’ll show you the intensive tasks I’m talking about that are total CPU-killers, such as waiting on database requests.
What we see at Twitter today is a very useful and popular service, but one with very complex underlying technical challenges to overcome. Twitter will require not only a new architecture approach and a big injection of the best minds they can find [...]
Damn it, I still haven’t hit my required word count. Here, enjoy this poo-poo platter of my best verbiage.
And later in the comments
It wouldn’t surprise me if the DB is falling over on those requests. Also judging by message ID’s and what I have seen so far, their data is in no way segmented – so you have one BFT (big fucking table)
Sorry, did I say their problem was Rails? It’s actually their database architecture! No wait, all of the above! Oh and my superpower is divining a site’s database architecture based the IDs it uses. Why, just the other day I saw an ID on Google Docs and it hit me: “Those guys use BigTable”.
I’ll make random unsubstantiated guesses about their technology instead of investigating with other sites their usage of Twitter’s API and what they’ve found out, or benchmarking Starling .
You want proper investigative journalism or insightful technical analysis? Sorry you’re in the wrong place. This is the “hearsay and hyperbole” section. You’re looking for somewhere else .
Twitter is def the largest RoR app – unless your awesome alexa-based ranking somehow manages to now count API hits (~80% of twitter). I dont know what I will do if one more person sends me a link showing Twitter on Alexa and says ‘see, not that big!’... Twitter isnt just the pretty web pages you see.
I never said Rails doesnt scale, nor that it is to blame, you have mis-interpreted the whole post. I actually said that most people blame Rails, not realizing just how hard it is to do what Twitter are doing. I also pointed out that there is a specific function here that is very different to other sites that ‘just’ serve pages or images etc.
I actually did speak to somebody very close to the company, who was my source – the story originated with them.
I happed to disagree with Nik that Twitter is the kind of site that’s hard to scale, and have commented on it several places. Twitter partitions reasonably easily the moment you accept a need to make copies of data. That said, I have had scaling nightmares myself – while getting the architecture right can be “easy”, that doesn’t mean it’s easy to do it while you at the same time have a lot of conflicting priorities and lots of things taking up your time. The Twitter guys needs to keep the system running and add features at the same time as they sort out the scaling issues, and they have a ton of traffic and users to keep happy while they re-engineer the underlying system. That massively complicates what would’ve been an easy job if they’d know what they went into when they started.
That’s probably the biggest problem. When I did Edgeio, we had a ton of scaling issues too, but the biggest challenge we had was not sorting out scaling, but sorting out scaling while adding features, keeping everything running, dealing with personality clashes, changing direction (several times) as we found out what worked and what didn’t.
Scaling itself isn’t hard, most of the time. Scaling when you don’t have enough time to spend on it, or don’t get approval for the right amount of hardware is.
That said, I have to take issue with the quip about message id’s:
You can tell a lot based on message id’s. Specifically, if they are monotonically increasing, and whether they are increasing by one at a time or have gaps or if messages id’s appear “jumbled” when sorted by time, but only by certain increments. You can make very good guesses about whether a data-set is partitioned or not at least by message id’s. If message id order is also posting order and the message id’s are monotonically increasing by one, then you can usually assume the database is not partitioned, for example.
@Scott: I’m aware of that line being a crucial point of that meme, but didn’t find a place I could use it justifiably!
@Henry: Yes, the obsession with Twitter is amusing. Actually with everyone and their grandmother blogging about how their solution can save Twitter, one might be tempted to argue that Twitter’s operations team numbers in the thousands… =P
@Nik:
How sure are you that the API is served purely by RoR? Quick example: I’m sure their SMS gateway has more direct access to the site than having to go through the web API.
Mea culpa for misinterpreting ”...certainly doesn’t lend itself to large-scale application development” and “Twitter is at a cross-roads as an application and Rails has served its purpose very well to date, but…” as alluding to Rails not scaling or being to blame. Silly me.
Finally if your source is that close to Twitter, wouldn’t he know more about the exact nature of their technical problems rather than a vague “they weren’t prepared” or “the problem is complex”?
@Vidar: Thanks for that catch—allows me to explain myself. That id point wa more alluding to the information on Twitter’s database is publicly available (one MySQL master, one read-only slave, per HighScalability Jan ‘08), so there’s no necessity to make a big show of guessing it. Kind of hard to add that qualifier while “translating” though!