I am a developer for an application. Said application runs on a server. It appears we have a memory leak, on the order of a 5% (0.8 GB) increase over 3 days. net/http/pprof is set up (with a firewall so it’s not externally accessible) and I’m capturing profiles with go tool pprof once an hour via cronjob.
However nothing obvious is appearing in the profiles. Diffing two profiles (with -diff_base) doesn’t help - usage fluctuates over time so the contents of profile A and profile B may vary significantly because the application was doing different work at those two times.
I’m hoping there’s some way of crunching the data across many profiles and analyzing trends. For example, if function Foo shows up in some profiles with a large memory footprint and does not appear at all in others, I want to ignore it since it’s likely a transient load. However if function Bar shows up in every profile with an increasing footprint each time, that’s probably my leak.
Of course I could dump the data into CSVs and use Python or R or something to look for patterns, but I can’t be the first person who’s run into an issue like this so I’m hoping there are existing tools.