jpeg.Decode fails on many images

Technical Discussion
#1

I have successfully implemented downloading images on my web app. They show up fine in the browser; however, when I attempt to decode them for resizing, I get errors. There is a post on GitHub about this issue:
https://github.com/golang/go/issues/45902
It is not resolved there. For me, it seems to occur more frequently with larger images. These are images which work fine in most applications; for example: Affinity photo and Mac Photos. I edited one of the files in Affinity and reduced the size. After I got it down to a width of 800, I was able to decode and resize it. Originally, it was 4000px wide and exported from Mac Photos. It fails to decode with error: unexpected EOF. The photo was taken with an iPhone 12. I see this error on most photos. After working with quite a few photos, I can consistently decode them only after I get the width down to around 800px. In every case of decode failing, the image shows fine in the browser.
I tried downloading this image:
https://hiconsumption.com/fastest-cars-in-the-world/
which is 1000px wide and get the same error on decode.
I would think this issue would be widely reported unless I am doing something wrong in my code.

var decodedImage image.Image
var err error
if fileType == "jpg" {
	buf := bytes.NewBufferString(imageData)
	reader := base64.NewDecoder(base64.StdEncoding, buf)

	config, err := jpeg.DecodeConfig(reader)

	if err != nil {
		log.Println("error jpeg.DecodeConfig: ", err)
		return err
	} else {
		log.Println("Width:", config.Width, "Height:", config.Height)
	}
	buf = bytes.NewBufferString(imageData)
	reader = base64.NewDecoder(base64.StdEncoding, buf)
	decodedImage, err = jpeg.Decode(reader)
	if err != nil {
		log.Println("jpeg.Decode failed error: ", err)
		return err
	}
}
#2

nevermind, I read the problem wrong. I thought the browser wouldn’t read your processed image.

#3

OK, no worries. Yeah the error occurs in the above code, on
jpeg.DecodeConfig() and jpeg.Decode()

#4

I don’t have a good suggestion other than isolation. Can you download the large image to a .jpg file and write Go code that directly reads and decodes the file? If so, can you write Go code to base-64 encode the downloaded file and compare the size of the result to what you receive in your service. You may have something somewhere that’s truncating the request data.

#5

Ah, good suggestion. In the browser, the base64-encoded image displays fine. However, if I just save the that string to the database, retrieve it and send it back to the browser, it is cut off. I added code to do a checksum on the data sent from the browser and it fails with unexpected EOF, too. It comes to the server as a post request and I use

	request.ParseForm()
	p.DataBodyMap = request.Form

to populate the url.Values, which is a
map[string][]string
Somewhere between the browser posting and the request.ParseForm() the base64 string is getting truncated.

#6

“If the request Body’s size has not already been limited by MaxBytesReader, the size is capped at 10MB.”

#7

Thanks, good to know what the limit is but none of my images are that large. The one I’m currently playing with(and displays fine in the browser) I under 4MB. When it arrives at the server it is 524288 bytes. That explains why, when it is sent back to the browser, only about the top 10% displays. I’m not using MaxBytesReader so the limit should be 10MB. I wonder if the browser limits the amount of data to send. I am using Safari on a Mac and also on an iPad.

#8

Now, when you say “save the string to the database”, what are you talking about? What kind of database and what is your data type on the column you’re trying to save the base64-encoded image in? Are you sure the problem is in the request and not in your database?

I have a production app that deals with image uploads/resizing and we routinely deal with much larger images. It’s also extremely unlikely that the browser would limit upload size. I have run in to a lot of funky browser stuff in my years as a web developer but that would be new to me.

Can you show your HTML form and the code on your server that consumes the upload? Are you using multipart/form-data?

#9

I was not using a multipart form, but this morning I have been looking into it and have successfully gotten the data for the file type element using the multipart form. I’m now working on modifying my processImage() function. I will update this thread as I get further along with it.

#10

OK, I got it working fine now. Here is the code:

case http.MethodPost:
		contentType := request.Header.Get("Content-Type")
		if strings.HasPrefix(contentType, "multipart/form-data") {
			p.DataBodyMap = make(url.Values, 0)
			request.ParseMultipartForm(1000)
			for key, value := range request.MultipartForm.Value {
				p.DataBodyMap[key] = value
			}
			photoFile, header, err := request.FormFile("photoBytesString")
			if err == nil {
				log.Println(header, err)
				photoContentType := header.Header["Content-Type"][0]
				log.Println(photoContentType)

				photoBytes, _ := ioutil.ReadAll(photoFile)
				log.Println("len(photoBytes): ", len(photoBytes))
				encodedImage64 := base64.StdEncoding.EncodeToString(photoBytes)
				encodedImage64WithPrefix := "data:" + photoContentType + ";base64," + encodedImage64
				p.DataBodyMap["photoBytesString"] = []string{encodedImage64WithPrefix}
			}

This works fine and does not truncate any of the data.

#11

Glad you found some success! FWIW, ioutil is deprecated.

#12

Oh, ok, what do you recommend instead?

#13

Replacements are described here ioutil package - io/ioutil - pkg.go.dev

#14

I didn’t know that, either! Looks like simple fixes, though: ioutil.ReadAll becomes io.ReadAll, ioutil.ReadFile → os.ReadFile, etc. I was worried for a sec!

#15

Yeah - there’s no plan to make breaking changes to my knowledge. If you look at ioutil.ReadAll it just calls the now-preferred io version:

// ReadAll reads from r until an error or EOF and returns the data it read.
// A successful call returns err == nil, not err == EOF. Because ReadAll is
// defined to read from src until EOF, it does not treat an EOF from Read
// as an error to be reported.
//
// As of Go 1.16, this function simply calls io.ReadAll.
func ReadAll(r io.Reader) ([]byte, error) {
	return io.ReadAll(r)
}

Per the docs (emphasis mine):

As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details.

I have slowly refactored ioutil out of my projects when I find it but I haven’t made it a priority since there’s no pressing need (or even a non-pressing need! Just a slight preference).

#16

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.