Thursday, July 28, 2011

Microbenchmarking of luajit-based server (again: this time against lighttpd)

The other day inbetween the meetings I've ported my toy event loop experiment to use ljsyscall library.

Here are the results of running of ab -n 100000 -c 1000. First let's make a baseline:

lighttpd



# ab -n 100000 -c 1000 http://localhost:80/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software: lighttpd/1.4.19
Server Hostname: localhost
Server Port: 80

Document Path: /
Document Length: 10 bytes

Concurrency Level: 1000
Time taken for tests: 7.854757 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 24331590 bytes
HTML transferred: 1001300 bytes
Requests per second: 12731.14 [#/sec] (mean)
Time per request: 78.548 [ms] (mean)
Time per request: 0.079 [ms] (mean, across all concurrent requests)
Transfer rate: 3025.05 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 39 281.0 15 3023
Processing: 5 19 16.5 21 740
Waiting: 3 14 15.9 14 734
Total: 9 58 287.6 38 3746

Percentage of the requests served within a certain time (ms)
50% 38
66% 40
75% 42
80% 43
90% 45
95% 47
98% 50
99% 94
100% 3746 (longest request)
#

luajit


Now let's try it on the primitive event loop.

# ab -n 100000 -c 1000 http://localhost:12345/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software:
Server Hostname: localhost
Server Port: 12345

Document Path: /
Document Length: 13 bytes

Concurrency Level: 1000
Time taken for tests: 8.232656 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 5503355 bytes
HTML transferred: 1300793 bytes
Requests per second: 12146.75 [#/sec] (mean)
Time per request: 82.327 [ms] (mean)
Time per request: 0.082 [ms] (mean, across all concurrent requests)
Transfer rate: 652.77 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 44 336.4 6 3016
Processing: 4 10 13.9 10 766
Waiting: 3 8 13.9 7 762
Total: 10 55 341.0 16 3767

Percentage of the requests served within a certain time (ms)
50% 16
66% 17
75% 17
80% 18
90% 20
95% 22
98% 28
99% 3018
100% 3767 (longest request)


Upon multiple runs the numbers vary slightly, of course, but they stay within the same ballpark. I think this shows luajit is a very viable platform for server development.

----

Update:

curious cat as I am, I've added a HTTP parser that is made by a yet-unpublished-and-inefficient-and-incomplete patch for ragel that generates Lua state machines (and a ragel code from an earlier post about http parser.


ab -n 100000 -c 100 http://localhost:12345/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:
Server Hostname: localhost
Server Port: 12345

Document Path: /
Document Length: 19 bytes

Concurrency Level: 100
Time taken for tests: 21.980 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 6100000 bytes
HTML transferred: 1900000 bytes
Requests per second: 4549.62 [#/sec] (mean)
Time per request: 21.980 [ms] (mean)
Time per request: 0.220 [ms] (mean, across all concurrent requests)
Transfer rate: 271.02 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 3
Processing: 1 22 3.1 21 30
Waiting: 1 22 3.1 21 30
Total: 3 22 3.1 21 30

Percentage of the requests served within a certain time (ms)
50% 21
66% 25
75% 25
80% 25
90% 25
95% 26
98% 27
99% 27
100% 30 (longest request)


This is not stellar, but I did not see yet where the bottlenecks are and if I can speed it up. Either way the 4.5K is still not too bad.

Update2:

with concurrency of 1000, I notice connection resets... After fixing up the mixing SIGPIPE handler of SIG_IGN, they still happen, apparently. Further debugging pending...

2 comments:

agentzh said...

Are you testing hello world servers? The "lighttpd" case in your benchmark is running mod_magnet? or just a bare lighttpd emitting a 10-byte disk file?

BTW, your "luajit" case emits a 13-byte response, which is larger than your "lighttpd" case (10-bytes), so it's not so fair.

It may be interesting for you to try out nginx + ngx_lua + luajit2.0 too ;) As one of the main developers of the ngx_lua module, I think the number should be nice there :)

Andrew Yourtchenko said...

I tweaked the luajit code to use a library for poll, so was curious how good/bad it performs and if there is any obvious bottleneck there.

lighttpd was just spitting out the static text.. though it might be entertaining indeed to compile mod_magnet with luajit support, could be fun!


The "real" request handling performance by my modded helloworld server (see the github for the latest version) was about 4K requests/sec on the same hardware. But I must say that the state machine code generation backend that I made for ragel is quite inefficient :-)

(this is another reason for this eventloop server existence - to ensure I can get ragel to have efficient Lua state machine backend).

As for nginx+ngx_lua_+luajit2.0 - indeed, I'll definitely try it out one of these days! :)