Http client io reader changed file Magic Number

Getting Help Code Review
1

I encountered a very strange situation.

this is the code

func LoadHttpImage(url string) (image.Image, error) {
	client := http.Client{
		Timeout: 5 * time.Second,
	}
	resp, err := client.Get(url)
	if err != nil {
		return nil, err
	}
	defer func() {
		err = resp.Body.Close()
	}()
	if resp.StatusCode != http.StatusOK {
		err = fmt.Errorf("status code %d", resp.StatusCode)
	}
	if err != nil {
		return nil, err
	}
	pic, _, err := image.Decode(resp.Body)
	if err != nil {
		fmt.Println(err)
	}
	return pic, nil
}

and the test case is

LoadHttpImage("https://fb.fenbike.cn/api/tarzan/images/17d6a6a52635d89.png?width=700")

the std io print error :“image: unknown format”

After debugging the code, I found that the problem was probably in this location(package: image/format.go:72)：

// sniff determines the format of r's data.
func sniff(r reader) format {
	formats, _ := atomicFormats.Load().([]format)
	for _, f := range formats {
		b, err := r.Peek(len(f.magic))
		if err == nil && match(f.magic, b) {
			return f
		}
	}
	return format{}
}

It appears to iterate through all file formats using magic numbers.

However, when I use Postman to view the binary stream of this file, it is indeed a PNG file.

image
image720×430 40.7 KB

What’s even stranger is that only this one file is like this. Could you please help me check if this is a bug in Go?

2

Hey,

This took me a few minutes before I realised why this one was actually pretty confusing. Anyway, if you’re interested, I can let you know how I got to working it out, but pretty much, I noticed that the response encoding wasn’t always the same and neither was the response body size.

The issue is that it’s mostly returning a Brotli encoded response and once in a while it will actually return gzip I believe, so it was working here and there for me, but when it returns as Brotli encoded, it needs to be decoded before image/png will work on it.

Here’s a working example for you.

package main

import (
	"fmt"
	"image"
	"io"
	"log"
	"net/http"

	// Need this to decode into png for it's init side effects.
	_ "image/png"

	"github.com/andybalholm/brotli"
)

type BrotliTransport struct {
	Base http.RoundTripper
}

type brotliReadCloser struct {
	io.Reader
	Closer io.Closer
}

func (brc brotliReadCloser) Close() error {
	return brc.Closer.Close()
}

func (t *BrotliTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	req.Header.Set("Accept-Encoding", "br, gzip, deflate")

	base := t.Base
	if base == nil {
		base = http.DefaultTransport
	}

	resp, err := base.RoundTrip(req)
	if err != nil {
		return nil, err
	}

	if resp.Header.Get("Content-Encoding") == "br" {
		brReader := brotli.NewReader(resp.Body)
		resp.Body = brotliReadCloser{
			Reader: brReader,
			Closer: resp.Body,
		}
		resp.Header.Del("Content-Encoding")
	}

	return resp, nil
}

func LoadHTTPImage(url string) (image.Image, error) {
	client := &http.Client{
		Transport: &BrotliTransport{
			Base: http.DefaultTransport,
		},
	}

	resp, err := client.Get(url)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	img, _, err := image.Decode(resp.Body)
	if err != nil {
		return nil, err
	}
	return img, nil
}

func main() {
	i, err := LoadHTTPImage("https://fb.fenbike.cn/api/tarzan/images/17d6a6a52635d89.png?width=700")
	if err != nil {
		log.Fatalln(err)
	}

	fmt.Println(i.Bounds())
}
2 Likes
3

think you very mach!

I didn’t notice the Brotli encoding issue.

4

No problem :slight_smile:

It was pretty subtle, and I easily could have missed it when checking the response headers.

Have a lovely day!