Problem
I want to use a GO routine every time I do the function call but as I understood the program main() function is exiting before the for loop finish is there anyway to use go routines to call concurrently the function that get_page() without using the sync lib?
What I have
package main
import (
"bufio"
"fmt"
"net/http"
"os"
)
func main() {
var site string
if len(os.Args) > 1 {
site = os.Args[1]
} else {
fmt.Println("Need the host. Example > http://www.google.com/")
fmt.Println("wfuzz http://www.google.com/ wordlist.txt")
site = "http://www.google.com"
os.Exit(1)
}
f, err := os.Open(os.Args[2])
if err != nil {
fmt.Println("error opening file!", err)
}
scanner := bufio.NewScanner(f)
for scanner.Scan() {
a := site + scanner.Text()
//would love to do the following function in a concurrent way
get_page(a)
}
if err := scanner.Err(); err != nil {
fmt.Println("error", err)
}
var input string
fmt.Scanln(&input)
fmt.Println("done")
}
func get_page(site string) string {
a := "\n"
res, err := http.Get(site)
if err != nil {
fmt.Println("error:", err)
}
if res.StatusCode != 404 {
fmt.Println("Page Found!! +::", site, " ::+", res.StatusCode)
}
return a
}
I know you asked for a solution without the sync lib, but this is literally exactly what sync.WaitGroup is for. The example in the documentation shows how easy it is to spawn N goroutines in a loop and then wait on them all to complete using that construct. Sure, you could build something nearly identical with channels, but why would you?
With all due respect to your preferences, please don’t fight the language. If you can clearly articulate why sync.WaitGroup is a bad fit for you, then I’d be happy to help you find a more suitable approach, but “I don’t wanna” doesn’t give me enough information to give you the right alternative. More importantly, it isn’t a good enough reason to avoid the idiomatic way of doing things.
Realistically it would be preferable to use sync.WaitGroup or something similar for this, since that is the reason it exists in the standard library, but there are of course other ways you can accomplish the same thing without the sync package.
Here is one simple example, only for learning purposes, but sync.WaitGroup should really be used:
package main
import "fmt"
func main() {
urls := []string{"http://urlone.com", "http://urltwo.com", "http://urlthree.com"}
received := make(chan struct{})
// Call getPage for each url concurrently and pass it the received channel.
for _, url := range urls {
go getPage(url, received)
}
done := make(chan struct{})
go func() {
// Receive from the received channel for each url in the urls slice
for range urls {
<-received
}
// Close the done channel now that it's ok to exit.
close(done)
}()
// Wait to receive from done before main exits.
<-done
}
func getPage(url string, received chan struct{}) {
fmt.Printf("Getting page: %s\n", url)
// Send a value on received when getPage is done.
received <- struct{}{}
}