Why can't the memory generated by parsing gifs be released?

Getting Help
1

I tried to work with image/gif, which caused too much memory usage for my project and it wouldn’t be freed, and finally I found out that it was a problem with this library?

You can take a look at the simple code below, he occupies too much physical memory, and most of it is sharing memory, and even manual gc can’t release it, I use a tool for prometheus to show the actual physical memory, which is very abnormal.

go version go1.20.11 linux/amd64
22.04.3-Ubuntu

package main

import (
	"bytes"
	"fmt"
	"github.com/prometheus/procfs"
	"image/gif"
	"ntest/res"
	"os"
	"runtime"
	"time"
)

func init() {
	gif.DecodeAll(bytes.NewReader(res.GIF))
}

func main() {
	proc, _ := procfs.NewProc(os.Getpid())
	for {
		time.Sleep(1 * time.Second)
		runtime.GC()
		stat, _ := proc.Stat()
		fmt.Println(float64(stat.ResidentMemory()) / 1024 / 1024)
	}
}
2

I can’t recreate it because I don’t have ntest/res, but here are a few questions:

  • What is the size of res.GIF?
  • What happens if you move the decoding to main()?
  • How much memory does the process utilize vs. how much is available in the system?

Also, perhaps you might find “A Guide to the Go Garbage Collector” useful in your quest.

Cheers

3

This res. The gif is only 20MB, it’s just irrelevant, according to my tests, this problem doesn’t occur on my macos, the object memory is freed, and on my ubuntu, the object memory is also freed, but the shared memory is not freed (in macos this part is also freed), which makes me wonder, obviously, the example code is very simple, the object is also gc, but the actual physical memory, on ubuntu and macos does not behave the same.

4

This will happen whether it is main or init, at first I modified the source code to read the picture frame in streaming, and then I found that as I continued to read the picture frame, the shared memory would gradually become larger, and the gif library was just read once in a for loop, and the result was the same.

5

Interesting that you mention MacOS vs. Linux difference. It may actually be “working as designed”. I found this StackOverflow answer that may be relevant to your case.

I thought of this because of how Apple advertises that 8GB is enough. :laughing: They might just be throwing some of the POSIX standard out the window for efficiency’s sake.

6

Interesting, but I still don’t know why this shared memory gets bigger and bigger as I parse more image frames, and when I start the second process and repeat it, the shared memory also starts from scratch and gradually grows larger, which looks like the process has monopolized the shared memory and becomes a proprietary memory that cannot be freed.