High virtual memory allocation by golang

lelewqd · September 25, 2017, 3:53pm

I have a large golang project (based on docker source code) that runs on Raspberry Pi and the virtual memory that the go process request is huge:

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  728     1 root     S     932m 401%   8% /usr/bin/mygoap

There is still free physical memory which I can verify using free -m, why go vmalloc so much memory?

nathankerr · September 25, 2017, 4:08pm

https://golang.org/doc/faq#Why_does_my_Go_process_use_so_much_virtual_memory

lelewqd · September 25, 2017, 4:32pm

Where does this allocation happens in the golang project? https://github.com/golang/go

nathankerr · September 25, 2017, 4:39pm

I think it is here (I haven’t hacked on the runtime yet):

github.com

golang/go/blob/master/src/runtime/malloc.go#L255


	var p, pSize uintptr
	var reserved bool


	// The spans array holds one *mspan per _PageSize of arena.
	var spansSize uintptr = (_MaxMem + 1) / _PageSize * sys.PtrSize
	spansSize = round(spansSize, _PageSize)
	// The bitmap holds 2 bits per word of arena.
	var bitmapSize uintptr = (_MaxMem + 1) / (sys.PtrSize * 8 / 2)
	bitmapSize = round(bitmapSize, _PageSize)


	// Set up the allocation arena, a contiguous area of memory where
	// allocated data will be found.
	if sys.PtrSize == 8 {
		// On a 64-bit machine, allocate from a single contiguous reservation.
		// 512 GB (MaxMem) should be big enough for now.
		//
		// The code will work with the reservation at any address, but ask
		// SysReserve to use 0x0000XXc000000000 if possible (XX=00...7f).
		// Allocating a 512 GB region takes away 39 bits, and the amd64
		// doesn't let us choose the top 17 bits, so that leaves the 9 bits
		// in the middle of 0x00c0 for us to choose. Choosing 0x00c0 means