Parallel transactions for Gocolly 'do not work properly

I try to pull the data of a forum with the logic on this page. In this way, I can pull the data in the pages without any problems. But when I create a multi-threading thread for the For loop, the program will exit some time after pulling the data and cannot pull all the data.
Expample : If there are 300 data in total, it will dig 120 data and end the program.

I waiting your solution suggestions. Thank You For Advance

Colly sample: http://go-colly.org/docs/examples/reddit/

My Codes Steps :

1- Select urls to be scanned from the database
2- Execute “c.Visit (URL)” in for loop
3 - Scrap with “c.OnHtml”

My Codes Sample:

	counter := 0
	resultsLen := len(results)
	var wg sync.WaitGroup
	wg.Add(resultsLen)
	for _, result := range results {
		go func(result primitive.M) {
			defer wg.Done()
			counter++
			
			mainForumUrl := result["url"].(string)
			fmt.Println("##### Looking Main Forum Url #####",counter,":",mainForumUrl)
			c.Visit(mainForumUrl)
			c.Wait()
		}(result)

	}
	wg.Wait()

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.