Help Speeding Up Program With Concurrency

(Joe Dan) #1

Hello,

I am working on a program that could potentially make a lot of network calls out which I think may really slow it down or potentially do worse things. The code that makes me the most worried is the snippet below:

    for hostbuff.Scan() { //Buffer From File With Hostnames
        assetips, err := net.LookupIP(hostbuff.Text()) //Lookup IP And Put in Array
        // go assetips, err := net.LookupIP(hostbuff.Text()) //Would this work?
        if err != nil {
            fmt.Println("We had IP lookup error")
        }

    for _, ip := range assetips {
        //do stuff

With net.LookupIP I am resolving a lot of hostnames to get IP addresses to put them in an array. With those IP addresses I am performing different types of work. I am most concerned with the net.LookupIP function. Is there a recommended way or ideas to make this concurrent such as using Goroutines or other methods. The first thing that comes up in my mind is to just go assetips, err := net.LookupIP(hostbuff.Text()) but I am worried that could cause problems.

Thanks in advance,

Joe

(George Calianu) #2

Mkay, are a few things to mention around because this is an interesting subject. Basically doing some network calls concurrent worth only if you start all those calls in goroutines each of them having a infinite loop that make the call and periodically collect data somewhere. Otherwise, if the network calls are too short launching in gouroutines can have the same effect as serial execution because can happen to end first call before other goroutines start.
Supposing you are in the first case is better to know that the operating system permit a finite number of file descriptors (eg.65535 on Linux) and you must increase it if you want to make much more network calls over this limit. Is good to know that handling file descriptors, bandwidth and TCP zombie connections are pretty hard things.
Another point is where you collect the data, the simple and cheap method is using maps but you must take care to the locks.
Knowing those things I think you can but in balance if worth or not to make things concurrent in your particular case.