HTTP requests to a headless k8s service not getting load balanced with go1.22.7

Getting Help
1

Hi,

I have a headless k8s service for which the local DNS resolver gives all the IPs of the pods sitting behind the service. The requests made to the service are not getting load balanced with go1.22.7 and all the requests go to a single pod. The HTTP client is invoked using the following snippet:

	for {
		client := &http.Client{
			Timeout: 95 * time.Second,
		}

		req, err := http.NewRequest("GET", "http://<service-name>:<port>", nil)
		if err != nil {
			fmt.Println("Error creating request:", err)
			return
		}

		resp, err := client.Do(req)
		if err != nil {
			fmt.Println("Error making request:", err)
			return
		}
		defer resp.Body.Close()
		time.Sleep(5 * time.Second)
	}

Requests made from the above with go1.23.1 are getting load balanced appropriately(not sure if it’s round robin though). I compared the transport logic for both the versions but couldn’t find anything concrete which could cause this problem. Can you experts please help me out?

2

If you are aware of the presence of a cache for DNS resolution, you can either pass in a new DNS resolver or write a simple DNS process for resolving domain names.

3

IIUC, DNS cache (if at all used) should be present in the library only. I am trying to understand the reason for the difference between the two versions. Just to clarify - I have the run the exact same code (shared above) on the same setup and experienced difference in resolution w.r.t different go versions.

4

Well, it’s common to have problems with go version differences (I’ve had it a few times). I don’t think there’s any other way to do it than if you look at the differences in the source code.
I think it’s imperative to avoid such a serious discrepancy (or to update your own Go development environment)

5

The difference is because of this PR where we were invoking getaddrinfo system call with 1.22.7 because of the cgo mode being invoked.

6

Since headless services return the individual pod IPs, the client (in this case, your Go application) is responsible for implementing load balancing. If the Go application is only hitting one of the pods consistently, it may appear that load balancing is not occurring.