Program breaks after creating around 7600 WebSockets

[BigBoulard]

Hi guys,

I’m working on a stress test script that creates WebSockets.
I don’t know why, but the creation of the web sockets breaks after around 7,600 calls to this function.

If I run the script in a docker container, I can accumulate the number of web sockets created up to 90 000, so it seems to be a host-related issue.

Regarding the resources, it’s working pretty well on each docker instance, so I’m thinking about a configuration of the compiler but I don’t know much about the capabilities yet (I’m currently digging into it)

I know that the maximum number of web socket creations is limited by the number of ports available (client-side) but I’m far from this limit it should be around 64,500.

Any idea?

func getWSConn(u *userdom.User, roomID string) (*websocket.Conn, error) {
	// Create a custom WebSocket dialer with custom TLS settings
	dialer := &websocket.Dialer{
		TLSClientConfig: &tls.Config{
			InsecureSkipVerify: true, // disable certificate verification for testing
		},
	}

	url := url.URL{
		Scheme: "wss",
		Host:   "localhost:8090",
		Path:   "/ws",
		RawQuery: url.Values{
			"uid":                 []string{u.UID},
		}.Encode(),
	}
	conn, _, err := dialer.Dial(url.String(), nil)
	if err != nil {
		return nil, fmt.Errorf("can't connect to %s %w", url.String(), err)
	}
	return conn, err
}

I’ve ran pprof to get insight into some numbers maybe to correlate with some compiler values. This are the numbers for 7,500 connections (just before it breaks):

[telo_tade]

@BigBoulard

accumulate the number of web sockets created up to 90 000, so it seems to be a host-related issue.

Regarding the resources, it’s working pretty well on each docker instance, so I’m thinking about a configuration of the compiler but I don’t know mu

Do you have the number of file descriptors configured on your machine ?

[BigBoulard]

Hi @telo_tade

Very good question now I’m thinking about it … you nailed it!

On my local computer (macOS)

# file descriptor limit
ulimit -n                                                          
10496

# limit of processes 
ulimit -u
2666

# soft / hard limit
launchctl limit maxfiles
maxfiles    256   unlimited

I changed the soft and hard values in the stress test container through the Compose file

stress-test:
   image: my-go-stress-test-build
   ulimits:
      nproc: 65535
      nofile:
        soft: 30000
        hard: 40000

The image was built from Scratch like below (https://hub.docker.com/_/scratch/), so I didn’t have a shell to check the numbers:

FROM golang:1.21-alpine3.17 AS builder
...
FROM scratch AS final
...
ENTRYPOINT ["/app/bin/stress-test"]

I just changed the Dockerfile so the image is based on the Golang/Alpine image:
FROM golang:1.21-alpine3.17 AS final

$ ulimit -n
30000

Now it is working, I just have this tls handshake error, but it’s happening randomly after 18 000 to 22 000 connections despite this TLSConfig that is set to ignore some security verifications … but it’s another story.

tls: handshake message of length 2243106 bytes exceeds maximum of 65536 bytes

TLSConfig: &tls.Config{
			Certificates:       []tls.Certificate{conf.HTTPServer.Certificate},
			InsecureSkipVerify: true, // WARN remove in prod
		},

Thank you for you help @telo_tade