HTTP aggregator

[drafdev (Batle go)]

Hi there,
I’m doing simple aggregation http with golang follow the instructur from book. it’s working but returning invalid json. anyone can help?

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"os"
	"time"
)

var timeoutMilliseconds = 5000

type barrierResp struct {
	Err  error
	Resp string
}

func barrier(endpoints ...string) {

	requestNumber := len(endpoints)

	in := make(chan barrierResp, requestNumber)
	defer close(in)

	responses := make([]barrierResp, requestNumber)

	for _, endpoint := range endpoints {
		go makeRequest(in, endpoint)
	}

	var hasError bool
	for i := 0; i < requestNumber; i++ {
		// Block the execution waiting for data from the in channel
		resp := <-in
		if resp.Err != nil {
			fmt.Println("ERROR: ", resp.Err)
			hasError = true
		}
		responses[i] = resp
	}

	if !hasError {
		for _, resp := range responses {
			fmt.Println(resp.Resp)
		}
	}

}

// makeRequest performs HTTP GET request to an url and accepts a channel to output barrierResp values
func makeRequest(out chan<- barrierResp, url string) {
	res := barrierResp{}

	// Create the HTTP client and set the timeout
	client := http.Client{
		Timeout: time.Duration(time.Duration(timeoutMilliseconds) * time.Millisecond),
	}

	// Perform the HTTP GET request
	resp, err := client.Get(url)
	if err != nil {
		res.Err = err
		out <- res
		return
	}

	// Parse the response to a []byte
	byt, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		res.Err = err
		out <- res
		return
	}

	// Send it throught the channel
	res.Resp = string(byt)
	out <- res
}

// captureBarrierOutput will capture the outputs in stdout
func captureBarrierOutput(endpoints ...string) string {
	// Create a pipe that allows us to connect
	// an io.Writer interface to an io.Reader interface
	reader, writer, _ := os.Pipe()

	os.Stdout = writer

	out := make(chan string)

	// Copies reader input to a bytes buffer before sending
	// the contents of the buffer through a channel
	go func() {
		var buf bytes.Buffer
		io.Copy(&buf, reader)
		out <- buf.String()
	}()

	barrier(endpoints...)

	// Close the writer to signal the Goroutine that
	// no more input is going to come to it
	writer.Close()
	temp := <-out
	return temp
}

func main() {

	endpoints := []string{"http://httpbin.org/headers", "http://httpbin.org/user-agent"}

	http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
		result := captureBarrierOutput(endpoints...)
		rs, er := json.Marshal(result)
		if er != nil {
			http.Error(w, er.Error(), http.StatusInternalServerError)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		w.Write(rs)
	})

	log.Fatal(http.ListenAndServe(":8030", nil))

}

[Giulio_Iotti]

If I followed your code correctly, it reads two full responses (from the two endpoints) and writes them back (appending one after the other) in the final response. Is that correct?

If I’m correct, then the most correct solution is to json.Unmarshal each of the endpoints responses, then create a slice []interface{} and append all unmarshalled responses there. Finally, json.Marshal the response (make the empty interface a valid JSON string) and write that as final response to your HTTP request.

Please do not ever write “_” where an error is returned. Errors are returned very often, and should always be handled!

[drafdev (Batle go)]

Thanks for your advice. I don’t know how to try your solution on my code. can you please provide some example?

[Giulio_Iotti]

No, sorry. If there is anything I wrote that is unclear, I can explain further. But writing code is your part of the job

[drafdev (Batle go)]

hahah thank, I mean where to put slice of the interface is that when get data from endpoint or when all request done and then convert it as slice of iterface?

[nathankerr]

An alternative to json.Unmarshal for this use case is to convert the original []byte (read from resp.Body) to json.RawMessage.

This avoids much of the unmarshal then marshal cost by leaving the data in JSON.

[Giulio_Iotti]

Thanks for the suggestion Nathan, makes sense. I didn’t suggest it as there are some subtleties like commas present or not that need to be taken care of. It can be the next step for “drafdev” to implement this optimization.

[Giulio_Iotti]

You need to fetch from all endpoints, return either a byte slice or directly the unmarshalled interface{}, and then make an array with all the (unmarshalled) responses. Then you encode to JSON and write to the HTTP response.