This is a very simple benchmark comparing the response times of a few different webservers for an extremely simple response: just reply with a snippet of static json. It came up in discussion of a real-life service: in the actual server, a long-running thread/process periodically updates the state from a database, but requests will be served with the data directly from memory. It is imperative, though, that the latencies be extremely low for this service.
This comparison was partly inspired by this blog post.
The code for the various servers may be found here. To conduct each test, I ran the server on a Linux desktop machine and then ran ab (apachebench) against the server from another machine connected via gigabit ethernet on the local network.
Server specs:
- Ubuntu 12.04
- Intel Core i5-2500K CPU (3.30GHz, 4 cores)
- 8GB RAM
For each test, I made 20,000 requests with 1, 10, 100, and 1000 concurrent connections. I did 3 warm-up runs before collecting data. Below are the mean, median, 90th percentile, 99th percentile, and max latencies.
I looked at both using Sinatra and writing Rack applications directly. For each of these options, I tested with Racer, Thin, and Unicorn. (This comparison isn't quite fair because I ran Unicorn with 8 workers, whereas Racer and Thin only have 1 worker.) Also in Ruby-land, I tested Goliath.
I was also interested in trying some JRuby servers, so I ran the plain Rack app under Trinidad and mizuno as well.
Additionally, I made a Scala app (using Scalatra) and a Go app (using the net/http standard library
package).
Software versions:
- Ruby 1.9.3-p194
- JRuby 1.7.0-rc1
- Scala 2.9.2
- OpenJDK 1.7.0
- Go 1.0.3
The c value is the number of concurrent requests. 90% and 99% are the 90th and 99th percentiles. All
values are milliseconds.
| Server | mean | median | 90% | 99% | max |
|---|---|---|---|---|---|
| Sinatra + Racer | 0 | 0 | 0 | 1 | 11 |
| Sinatra + Thin | 1 | 0 | 1 | 1 | 9 |
| Sinatra + Unicorn | 1 | 1 | 1 | 7 | 12 |
| Rack + Racer | 0 | 0 | 0 | 0 | 7 |
| Rack + Thin | 0 | 0 | 0 | 0 | 12 |
| Rack + Unicorn | 1 | 1 | 1 | 5 | 7 |
| Goliath | 0 | 0 | 0 | 1 | 9 |
| JRuby + Mizuno | 1 | 1 | 1 | 1 | 12 |
| JRuby + Trinidad | 0 | 0 | 0 | 1 | 9 |
| Scalatra | 1 | 1 | 1 | 1 | 3 |
| Go | 0 | 0 | 0 | 1 | 4 |
| Server | mean | median | 90% | 99% | max |
|---|---|---|---|---|---|
| Sinatra + Racer | 2 | 2 | 3 | 8 | 10 |
| Sinatra + Thin | 3 | 2 | 3 | 9 | 14 |
| Sinatra + Unicorn | 1 | 1 | 2 | 17 | 49 |
| Rack + Racer | 1 | 1 | 1 | 2 | 5 |
| Rack + Thin | 1 | 1 | 1 | 4 | 8 |
| Rack + Unicorn | 1 | 1 | 2 | 14 | 47 |
| Goliath | 2 | 2 | 2 | 8 | 12 |
| JRuby + Mizuno | 1 | 1 | 2 | 8 | 25 |
| JRuby + Trinidad | 1 | 0 | 1 | 4 | 13 |
| Scalatra | 3 | 2 | 4 | 5 | 9 |
| Go | 1 | 1 | 1 | 1 | 3 |
| Server | mean | median | 90% | 99% | max |
|---|---|---|---|---|---|
| Sinatra + Racer | 18 | 3 | 7 | 12 | 3618 |
| Sinatra + Thin | 25 | 25 | 29 | 30 | 33 |
| Sinatra + Unicorn | 17 | 15 | 22 | 34 | 47 |
| Rack + Racer | 5 | 1 | 2 | 4 | 1623 |
| Rack + Thin | 10 | 10 | 13 | 17 | 18 |
| Rack + Unicorn | 14 | 13 | 16 | 27 | 42 |
| Goliath | 23 | 24 | 25 | 25 | 33 |
| JRuby + Mizuno | 10 | 5 | 10 | 27 | 1214 |
| JRuby + Trinidad | 6 | 6 | 7 | 14 | 17 |
| Scalatra | 22 | 20 | 25 | 30 | 1016 |
| Go | 4 | 4 | 5 | 7 | 9 |
<tr><td>Sinatra + Thin</td><td colspan='5'>server failed</td></tr>
<tr><td>Sinatra + Unicorn</td><td colspan='5'>server failed</td></tr>
<tr><td>Rack + Racer</td><td>45</td><td>1</td><td>3</td><td>1460</td><td>6459</td></tr>
<tr><td>Rack + Thin</td><td>90</td><td>13</td><td>19</td><td>2468</td><td>6462</td></tr>
<tr><td>Rack + Unicorn</td><td colspan='5'>server failed</td></tr>
<tr><td>Goliath</td><td colspan='5'>server failed</td></tr>
<tr><td>JRuby + Mizuno</td><td>76</td><td>6</td><td>29</td><td>1229</td><td>2462</td></tr>
<tr><td>JRuby + Trinidad</td><td colspan='5'>server failed</td></tr>
<tr><td>Scalatra</td><td>231</td><td>238</td><td>273</td><td>1150</td><td>1414</td></tr>
<tr><td>Go</td><td>20</td><td>17</td><td>21</td><td>34</td><td>642</td></tr>
| Server | mean | median | 90% | 99% | max |
|---|---|---|---|---|---|
| Sinatra + Racer | 160 | 4 | 1001 | 2438 | 6068 |
- At this time, racer seems much more like a proof of concept than a serious production-ready webserver.
- Rack is great, because it's super easy to drop in various webservers to run your app.
- JRuby is really easy to use, and plays well with rbenv and bundler.
- Deployment for JRuby apps may get complex, what with xml files and tomcat configuration and who knows what. Projects like Warbler that turn your whole project into a war file may help a lot though.
- JRuby startup time is really annoying.
- Scala, as usual, is a massive pain to set up and get running. The "minimal" example project for Scalatra required three different tools to set up and configure, and the sbt configuration makes the whole thing a real mess that's very newcomer-unfriendly. There are 13 files in this project, compared with around 2-4 per other implementation.
- I wanted to try Lift as well but the setup was too daunting. sbt is awful.
- Go is really great for this kind of thing. The server is dead-simple, configuration is non-existent, and the app is built to a single binary ready to be deployed to a server.
- Avoid Sinatra if latency on the order of a dozen milliseconds matters more than ease of development.
- Racer has really low latency, but also has a nasty habit of serving a small percentage of requests really slowly as load increases (see c=100, where the rack/racer combo served 99% of requests in 4ms but the max latency was 1623ms).
- Assuming we rule out racer (due to lack of documentation/polish/project momentum), thin seems like the best choice for low-latency performance if you want to stick with MRI.
- Both JRuby servers performed well; Trinidad, in particular, seems like a good bet for low latency. It did stop responding to requests once the concurrent requests got up to 1000, though.
- If you're optimizing for latency, Goliath is a strictly worse choice than, say, Rack + Thin.
- Scalatra is a poor choice for latency-critical applications. This could be due to all the framework code (the same as Sinatra), or because my app was not tuned properly.
- I'd be interested to see how a better Scala app (or a Java app) would perform in the context of a properly-tuned, high-performance Java webserver.
- Go completely dominates this comparison. It would be a great choice for a small standalone webserver like this where latency matters. Note how the implementation doesn't even use any third-party libraries, yet fits into a 30-line file and builds with no extra configuration files whatsoever.