Cancel http request

[alex925 (alex)]

How cancel all goroutines processing http requests?

package main

import (
   "bytes"
   "errors"
   "fmt"
   "github.com/levigross/grequests"
   "golang.org/x/net/html/charset"
   "io/ioutil"
)

var workerComplete = make(chan []string)


func fetch(url string) (string, error) {
   response, err := grequests.Get(url, nil)
   if err != nil {
      return "", errors.New("Error request: " + err.Error())
   }

   defer response.ClearInternalBuffer()

   raw_page, err := charset.NewReader(bytes.NewReader(response.Bytes()), response.Header.Get("Content-Type"))
   if err != nil {
      return "", errors.New("Encoding error: " + err.Error())
   }

   unicodePage, err := ioutil.ReadAll(raw_page)
   if err != nil {
      return "", errors.New("Response read error : " + err.Error())
   }

   return string(unicodePage), nil
}

func downloadWebPage(url_ string, pageDownloaded chan []string) {

   html, err := fetch(url_)
   if err != nil {
      err = errors.New("Fetch error " + url_ + " : " + err.Error())
      pageDownloaded <- []string{"", err.Error()}
   }
   pageDownloaded <- []string{html, ""}
}

func main() {
   urls := []string{"http://golang-book.ru/chapter-10-concurrency.html",
      "http://golang-book.ru/chapter-10-concurrency.html", "https://gobyexample.com/json"}

   urlsCount := 0

   for _, url_ := range urls {
         urlsCount += 1
         go downloadWebPage(url_, workerComplete)
   }


   for i := 0; i < urlsCount; {
      result := <-workerComplete
      html, err := result[0], result[1]

      if err != "" {
         fmt.Println(err)
      }

      fmt.Println(html[:10])
      i += 1
   }

[christophberger]

Depending on what you need to achieve, here are two possible options:

1. http.Client has a property named Timeout that can be set to a time.Duration value to have the request time out after that duration.
   (I see you use a custom HTTP client library, but I guess this one also provides options for setting a timeout.)

2. To cancel a goroutine based on an external event, create a cancellable context:

ctx, cancel := context.WithCancel(context.Background())

Pass ctx to each goroutine that you want to cancel, and call the returned function cancel() when you want to cancel all goroutines. Every goroutine needs to listen on the “done” channel and return if they receive something:

select {
	case <-ctx.Done():
		return
	// more cases

(Full example here.)