Try to Find my Http API server bottleneck

Getting Help
1

Hi:
I deployed my API Server at Alibaba cloud in China. I am using my PC with 2 Hyper-V VM from Taiwan.

tested with following:
ab -n 3000 -c 3000 http://ip:7000/api/0.2/weather (2 VMs, total 6000)

My server is 1 cpu (10% usage), 4GRam (10% usage)
bandwidth Max 3.5 Mbps, 3K pps, TCP Connections 6K

But when I use -c 4000, I got an error.
"apr_socket_recv: Connection timed out (110)"
Since all usages are low, while I can’t get higher request?

I use the following package:

go get github.com/robfig/cron
go get github.com/go-gorp/gorp
go get github.com/julienschmidt/httprouter
go get github.com/go-sql-driver/mysql
go get github.com/mssola/user_agent
go get github.com/astaxie/beego/session

The weather function doesn’t use database. when function is called, simple return a data struct in memory.

server side config.
$vi /etc/sysctl.conf

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2000
net.ipv4.tcp_max_syn_backlog=2048
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1

// additional command
ulimit -n 102400
$ifconfig eth0 txqueuelen 5000
$echo “/sbin/ifconfig eth0 txqueuelen 5000” >> /etc/rc.local

I use $watch -n 1 "netstat -anp|grep 7000|wc -l"
There are 6000 connections on my server.

Thank you in advance.
Steve

2

what do you expect? you’re running out of tcp connections

3

Did you tweak http server?

https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/

4

Thanks vitr.

The max tcp connection for one ip should be around 30K, I used 2 clients (one is in China, one is in Taiwan), to fire ab test. I got the same result. Could you tell me the linux command that can show I run out of tcp connections?
thanks.

5

maybe try to inspect the server logs, e.g. https://serverfault.com/questions/477749/apr-socket-recv-connection-timed-out-110

6

Thanks acim.

I am using go 1.10.1
Do you suggest I use prometheus to monitor my simple api server? I have alibab cloud monitor tool and some linux command. Are there some specific parameters that I have to watch?

7

Thanks vitr.

I don’t have an error message about “nf_conntrack”, but I have this one “request_sock_TCP: Possible SYN flooding on port 7000. Sending cookies”.
My /etc/sysctl.conf is following:
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time=120

see details in https://help.aliyun.com/knowledge_detail/39428.html

net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2

see details in https://help.aliyun.com/knowledge_detail/41334.html

net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1

for Http Api server

net.core.somaxconn = 2048
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 5000

net.core.wmem_max = 12582912
net.core.rmem_max = 12582912

net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912

net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 16384

8

Prometheus is cool, you could find many information once you start tracking metrics. We use it to monitor our http clients so this is from another side. Even if you do not track anything special, Prometheus will show some preset metrics and then you may add more custom stuff. Can if be that some firewall limits your connections or something else? I mean at Alibaba infrastructure, they must have such things.

9

Hi acim:
It’s possible that the Alibaba cloud has some limits. but I just can’t find the document address the problem.

10

I’d turn off the SYN cookies first
net.ipv4.tcp_syncookies = 0
read more https://help.marklogic.com/Knowledgebase/Article/View/182/0/possible-syn-flooding-messages-in-system-logs

11

Thank you, vitr.

I tried it. I found no difference. But I use 5 client to fire ab test, My server can handle up to 13K request. That’s the best I can get.

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.