Hi guys,
I’m running a go program which loads the data from mysql into a map for long living cache purpose. Data will be queried from db in a loop and update the struct querying few records each time. The amount of data loaded into the struct will be around 2gb when written to disk in json format.
After the loading is complete and gc is triggered, I see the following memory stats using runtime.readMemStats()
"Alloc": 11107201856,
"TotalAlloc": 438869045528,
"Sys": 54348311616,
"Lookups": 0,
"Mallocs": 6589668524,
"Frees": 6448030009,
"HeapAlloc": 11107201856,
"HeapSys": 51584040960,
"HeapIdle": 38203949056,
"HeapInuse": 13380091904,
"HeapReleased": 37270716416,
"HeapObjects": 141638515,
"StackInuse": 1703936,
"StackSys": 1703936,
"MSpanInuse": 224454240,
"MSpanSys": 658674288,
"MCacheInuse": 34800,
"MCacheSys": 46800,
"BuckHashSys": 2067482,
"GCSys": 2037865776,
"OtherSys": 63912374,
"NextGC": 22214904160,
"LastGC": 1688029114272696800,
"PauseTotalNs": 20106946,
and the memory consumed by pod
$k top po test-pod --containers=true
POD NAME CPU(cores) MEMORY(bytes)
test-pod app-container 1m 17367Mi
The struct definition is like below.
type AppCache struct {
FieldA map[string]map[string][]map[string]string
}
and data in struct looks like this
{
"fieldA": {
"xyz": [
{"123": "abc"}, {"456": "def"}
],
...
}
}
I would like to know where the remaining memory is used apart from the heapAllocation. Is there a way optimize this to reduce memory footprint?