Custom API Client : How to read net/http.Response that has been closed?

Hi,

Hi I am learning go and encountering a problem when trying to read a net/http.Response.Body a second time. I have created the sample program below to provide an explanation for what I am trying to do.

The sample program makes a request from https://jsonplaceholder.typicode.com/posts and formats the data in the response into a simple struct graph. The response.Body is closed once this operation has been successfully performed. Furthermore, the original net/http.response struct is stored within the struct graph to allow client developers access to the original request and headers etc.

The problem that I am encountering is that within my unit tests I am reading the response.Body a second time to minify the JSON response for comparison with an expected JSON response. However, I am finding that this raises an error, presumably because I previously issued a net/http.Response.Body.Close() after unmarshalling the HTTP response body into a struct graph.

Is it possible to read a net/http.Response.Body after it is has been closed? If not, then I guess I could store the original response bytes separately within the return struct graph. However, this seems to be redundant since the bytes have already been unmarshalled. In which case I should document to users of the library that the response body is closed?

Sample program

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

// constants for HTTP request header
const (
	acceptHeaderKey   string = "Accept"
	acceptHeaderValue string = "application/json"
)

// struct to simulate an object graph created from API response
// for brevity for demo including raw JSON string here rather than
// doing the extra unmarshaling from json into struct
type Payload struct {
	Data string
}

// struct to encaspulate data response
type Data struct {
	Payload Payload
	Resp    *http.Response
}

// main method
func main() {
	fmt.Println("CREATING CLIENT!")

	client := createClient()

	fmt.Println("PERFORMING REQUEST!")
	data, dErr := doGETRequest(*client)
	if dErr != nil {
		fmt.Printf("Error encountered performing HTTP request : %v\n", dErr)
	} else {
		fmt.Println("The data retrieved is", data.Payload.Data)
	}

	fmt.Println("READING ORIGINAL RESPONSE BODY")
	bytes, rdErr := ioutil.ReadAll(data.Resp.Body)
	if rdErr != nil {
		fmt.Println("Error encountered reading original response body :", rdErr)
	} else {
		respBytes, minErr := minify(bytes)
		if minErr != nil {
			fmt.Println("Error encountered while minifying response : ", minErr)
			fmt.Printf("Length of bytes read from body using 'ioutil.ReadAll' is %d\n\n", len(bytes))
		} else {
			fmt.Println("Minified response is : ", string(respBytes))
		}
	}
}

// create a http client instance
func createClient() *http.Client {
	client := &http.Client{
		Transport: &http.Transport{
			MaxIdleConnsPerHost: 30,
		},
		Timeout: time.Duration(10) * time.Second,
	}

	return client
}

// make a request to jsontest.com
func doGETRequest(client http.Client) (*Data, error) {

	// create the url for the request
	u, err := url.Parse("https://jsonplaceholder.typicode.com/posts")
	if err != nil {
		fmt.Println("ERROR CREATING URL")
		return nil, err
	}

	// prepare the HTTP request and add the headers
	req, rErr := http.NewRequest("GET", u.String(), nil)
	if err != nil {
		return nil, rErr
	}

	addRequestHeaders(req)

	// perform the HTTP request
	resp, dErr := client.Do(req)
	if dErr != nil {
		fmt.Println("ERROR PERFORMING REQUEST")
		return nil, dErr
	}

	// ensure response body is closed when exit function
	defer resp.Body.Close()

	bytes, rdErr := ioutil.ReadAll(resp.Body)
	if rdErr != nil {
		fmt.Println("ERROR READING RESPONSE BODY")
		return nil, rdErr
	}

	return &Data{
		Payload: Payload{Data: string(bytes)},
		Resp:    resp,
	}, nil
}

// add headers for json accept
func addRequestHeaders(req *http.Request) {
	req.Header.Add(acceptHeaderKey, acceptHeaderValue)
}

// minify json bytes
func minify(jsonB []byte) ([]byte, error) {

	var buff *bytes.Buffer = new(bytes.Buffer)
	errCompact := json.Compact(buff, jsonB)
	if errCompact != nil {
		newErr := fmt.Errorf("failure encountered compacting json := %v", errCompact)
		return []byte{}, newErr
	}

	b, err := ioutil.ReadAll(buff)
	if err != nil {
		readErr := fmt.Errorf("read buffer error encountered := %v", err)
		return []byte{}, readErr
	}

	return b, nil
}

You can’t read a response body twice. A response body is a stream of packets coming over the network. As you call Read, the data is copied into the []byte buffer passed into Read and then the underlying network stack frees its copy of the data. If you want to re-read the data, you have to store it in some temporary location by yourself by first such as by copying it to a *bytes.Buffer or *os.File.

1 Like

Ok, understood, thanks @skillian. Appreciated :slight_smile: