I try to pull the data of a forum with the logic on this page. In this way, I can pull the data in the pages without any problems. But when I create a multi-threading thread for the For loop, the program will exit some time after pulling the data and cannot pull all the data.
Expample : If there are 300 data in total, it will dig 120 data and end the program.
I waiting your solution suggestions. Thank You For Advance
Colly sample: http://go-colly.org/docs/examples/reddit/
My Codes Steps :
1- Select urls to be scanned from the database
2- Execute “c.Visit (URL)” in for loop
3 - Scrap with “c.OnHtml”
My Codes Sample:
counter := 0
resultsLen := len(results)
var wg sync.WaitGroup
wg.Add(resultsLen)
for _, result := range results {
go func(result primitive.M) {
defer wg.Done()
counter++
mainForumUrl := result["url"].(string)
fmt.Println("##### Looking Main Forum Url #####",counter,":",mainForumUrl)
c.Visit(mainForumUrl)
c.Wait()
}(result)
}
wg.Wait()