I have deployed my Go server on AWS EC2. I’m currently using a t2.micro instance with Ubuntu Server 18.04 LTS (HVM), SSD Volume Type and 8GB of hard drive.
Go Code:
The code I’m using is the following:
package main
import (
"fmt"
"log"
"net/http"
"strings"
)
var (
chars_10 string
)
func main() {
chars_10 = strings.Repeat("a", 10)
http.HandleFunc("/experiment-1/10-characters", handler_10_chars)
log.Fatal(http.ListenAndServe(":8888", nil))
}
func handler_10_chars(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, chars_10)
}
Benchmark Code:
The benchmark tool I’m using is K6, and the code for the benchmark is the following:
import http from "k6/http";
import { check } from "k6";
export let options = {
vus: 2048,
duration: "60s"
};
let url = ""; // Here I define my AWS EC2 instance endpoint.
export default function() {
let res = http.get(url);
check(res, {
"status was 200": (r) => r.status == 200,
});
};
This code will be executed on my notebook, hitting to AWS EC2 endpoint.
The experiment:
First of all, I just compiled Go code into a single binary with go build server.go command on terminal. Then, I run ./server on the terminal to start my server.
When I first stress tested my server, an error was thrown in the k6 benchmark:
WARN[0005] Request Failed error="Get http://aws-url:8888/: read tcp [::1]:60816->[::1]:8888: read: connection reset by peer
Where aws-url is the autogenerated url that aws provides and 8888 is the port I have enabled (and mapped to the port my server is listening for incoming requests).
After searching on the internet, I found that this was caused because SOMAXCONN variable had the value of 128. So what I did was to change this number to 65535. The way of doing this was editing /etc/sysctl.conf and add the following line at the end of the file:
net.core.somaxconn=65535
Then, I run again K6 benchmark. What I noticed was that the Connection reset by peer error disappeared. This time there was a new error on the server side:
Accept error: too many open files.
Again, I searched how to solve this new error and found that it was because of the amount of file descriptors on the system. So, what I’ve done was to edit again the /etc/sysctl.conf and add the following line at the end of the file:
fs.file-max = 1000000
Also, I run the following command on the terminal:
ulimit -n 99999
Once again, I run K6 benchmark. This time there was no error. But I noticed something wrong: in every benchmark, CPU % usage and Memory % usage was always under 20% (I couldn’t make my server use 100% of CPU) and TTFB was increasing to a max of 2 seconds average. After hours of trying different configurations of Ubuntu, I found this post: Slow Server? This is the Flow Chart You're Looking For | Scout APM Blog. Again, I run the benchmark and noticed that my CPU performed as step 3:
Step 3: IO wait is low and idle time is high
So I conclude that Go is not using all of the CPU capacity for handling requests. My question is: how can I make Go use 100% of CPU capacity, so I can handle the maximum amount of requests per second? What other bottlenecks do you think that could be limiting the server to reach 100% CPU usage?