Excited to have this new community forum to hang out in!
I’m writing a web service in go that will be handling many small-ish, but concurrent requests over HTTP. As expected, as I turn up the number of concurrent requests on my webserver, the average latency for the requests increase (as does the 99% / long tail). Are there any configuration options I should be looking at to help keep latency down?
Here’s a link to a simple example that is just almost completely stdlib code.
how many concurrent requests are you able to make before things are able to get out of control? You can use a worker/dispatcher model to regulate many requests at once
Thanks for the replies Conrad & Jason. I think these numbers will answer both of your questions. As I mentioned, nothing seems out of the ordinary since you would expect that more I/O => more latency. The same happens with node, for example. Just wondering if there are any options that might provide a win.
The service i’m writing will be “in-line” for requests in a webapp, so every millisecond is valuable
Also, these numbers are for 1.5.1
ab -n 10000 -c 10 -k 127.0.0.1:4567/
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.3 0 10
Waiting: 0 0 0.3 0 10
Total: 0 0 0.3 0 10
Percentage of the requests served within a certain time (ms)
50% 0
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 10 (longest request)
100 simultaneous
ab -n 10000 -c 100 -k 127.0.0.1:4567/
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 5
Processing: 0 4 2.2 4 19
Waiting: 0 4 2.2 4 19
Total: 0 4 2.2 4 19
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 5
80% 6
90% 7
95% 8
98% 9
99% 10
100% 19 (longest request)
1000 simultaneous
ab -n 10000 -c 1000 -k 127.0.0.1:4567/
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 3.0 0 40
Processing: 0 45 11.5 43 126
Waiting: 0 45 11.5 43 126
Total: 0 45 11.4 43 126
Percentage of the requests served within a certain time (ms)
50% 43
66% 48
75% 51
80% 53
90% 59
95% 64
98% 77
99% 88
100% 126 (longest request)
Looking at those numbers, you’ll be better off redesigning the webapp to require less network round-trips that block the user experience. Network latency is going to dwarf the amount of work your server is doing.
Also note that you may be limited by the throughout of your sockets since you’re benchmarking localhost. I recommend using wrk (https://github.com/wg/wrk) over ApacheBench as well since it behaves much more sanely.
The alternative would be connecting the webapp to this service over UDP, but I’d rather have the reliability. Fortunately, i’m not currently expected numbers in the 1000s of concurrent requests range so the ~10ms is OK for the time being.
Thankfully Go doesn’t require you to dig through 500 different configuration options to tweak performance to something “reasonable”
Given that, all of the normal tricks apply: limit IO, cache what you can, and where possible avoid expensive network setups. If your Go server is “middleware” between the client and a backend service you may be able to set longer keep-alives between Go and the backend, and/or have them communicate over a Unix socket instead of localhost if they’re on the same machine.
Huh? UDP can’t get rid of the speed of light. I’m talking about eliminating HTTP requests — can you combine multiple results into a single request? do we actually need this data? can we do these requests in the background? — not about reducing the # of packets.
UDP doesn’t get rid of the speed of light, but it would allow the client to “send and forget” such that it doesn’t wait for a response to come back so the latency cost is just the time to send network packets onto the wire.
I’ve been playing around with combining multiple requests into a single request but that actually is ending in more timeouts so maybe the latency due to concurrency i see in benchmarks isn’t what I’m hitting in practice.
Unfortunately there’s no way to do these requests in the background (thanks PHP…) I think all of your suggestions are totally reasonable and implementations I could have gone for but at the end of the day software development is the art of making compromises and they weren’t tradeoffs I wanted to make.
