Kicking off Goroutines as fast as possible?

[jrswab (Jaron R. Swab)]

I have been working on a terminal app and need to load data from a 1,250 URLs currently I am doing the following:

var wg = sync.WaitGroup
var lock = sync.RWMutex

for _, text := range urls {
    time.Sleep(time.Millisecond * 1)
    wg.Add(1)
    go parseData(text, wg, lock)
}
wg.Wait()

func parseData(url string, wg sync.WaitGroup, lock sync.RWMutex) {
    defer wg.Done()
    defer lock.Unlock()

    htmlBody := readHTML(url)

    lock.Lock()
    addToMap(htmlBody)
}

Is there a better way to spin up these Goroutines so they could execute as fast as the connection allows? Without the sleep function it creates the Goroutines much too fast and gives a “too many connections” error.

However, if I have the lock around the readHTML function and let them spin up all “at once” it takes far too long to get all the data.

Something I noticed with the time.Sleep() function in this usage is that occasionally the HTML still errors out. It’s rare, maybe 1 in 100, but still undesirable.

Any and all thoughts are welcome, thanks!

[NobbZ (Norbert Melzer)]

Find out what your connection limit is and use a pool of workers.

My assumption is that you hit a limit of file descriptors in your operating system or the other side denies connections because you are two fast.

---

Edit

Instead of a worker pool you could also use other means of rate limiting, eg a token bucket implementation.

[acim (Boban Acimovic)]

Instead of the mutex, you may want to use sync.Map and you should use some rate limiter anyways, as @NobbZ said, in order not to saturate your resources because the list of url‘s may grow.

[jrswab (Jaron R. Swab)]

Thanks @NobbZ and @acim I’ll read up on these solutions and give them a try.

#AlwaysLearning