Goroutine Memory Leak Causing Gradual Performance Degradation on My Website Backend Built With Go

Hello Go Community,

I am currently facing a persistent performance issue with my website backend that is built using Go, and I am hoping to get some advice from experienced developers who may have encountered a similar situation. The core problem is that the application performance gradually becomes slower over time until the service needs to be restarted. The website initially works perfectly after deployment, with fast response times and normal resource usage, but after running continuously for several hours or days, memory consumption steadily increases and the number of active goroutines continues growing. Eventually, the application starts responding slowly to user requests, even though the traffic level remains relatively stable. Restarting the Go service temporarily resolves the issue, but the problem returns again after extended uptime.

The issue appears to be related to how background processes and concurrent tasks are handled within my application. The website uses Go routines for several asynchronous operations, such as processing user requests, handling API communication, updating database records, and running scheduled background jobs. These tasks are created successfully and complete normally in most cases, but monitoring shows that the total number of goroutines does not return to its original level after certain operations finish. Over time, the application accumulates more active goroutines than expected, which appears to contribute to increased memory usage and reduced performance. I have reviewed the code paths where goroutines are created, but I have not yet identified which specific process is failing to clean up properly.

I have already used Go’s built-in profiling tools, including pprof, to investigate the memory usage and goroutine activity. The profiles indicate that there are many long-running goroutines remaining active, but the stack traces are not immediately clear enough for me to determine the exact source of the leak. I have checked common causes such as missing channel closures, blocked channel operations, and forgotten context cancellation, but I have not found an obvious issue. The application uses multiple services and packages, so isolating the exact component responsible has been challenging. The problem also does not happen immediately during testing because the application needs to run under normal user activity for a longer period before the resource usage becomes noticeable.

Another confusing part of this issue is that the application does not crash or produce obvious error messages when the problem occurs. The Go runtime continues operating normally, and there are no panic logs indicating a fatal problem. Instead, the degradation happens gradually, with slower API responses, increased garbage collection activity, and higher memory consumption. Database queries and external service requests continue functioning, which makes it difficult to determine whether the issue is caused by my application logic, a dependency, or an incorrect concurrency pattern. I have added additional logging around goroutine creation and completion, but the logs have not yet revealed a clear reason why certain routines remain active indefinitely.

I have also reviewed the architecture of the website backend to ensure that resources are released correctly after each request. Database connections are managed through the standard Go database package, HTTP clients are reused properly, and context cancellation is implemented in many areas where operations may take longer than expected. However, because the application handles multiple concurrent users and performs background processing, I suspect there may still be a hidden lifecycle issue where certain goroutines continue waiting for events that never occur. I am particularly interested in understanding whether there are recommended patterns for managing long-running goroutines, worker pools, and background services in production Go applications to prevent this type of gradual resource exhaustion.

I would appreciate guidance from the Go community on how to properly identify and resolve this goroutine memory leak issue. Specifically, I would like to know the best methods for analysing goroutine profiles, finding blocked or abandoned goroutines, and structuring concurrent code so that all background tasks terminate correctly when they are no longer needed. Any recommendations regarding context management, channel handling, worker lifecycle design, or debugging techniques would be extremely valuable. My goal is to ensure that my Go-powered website backend can run reliably for long periods without memory growth, increasing goroutine counts, or requiring manual restarts. Sorry for long post!

One way to mitigate the immediate issue is to limit the lifetime of any go-routine your service starts anywhere. Set a Maximum Lifetime on your DB connection pool. Always create context for routines with hard time-outs. Make sure everything has a fixed maximum time to live.

I had a similar problem once and found the root with manual book-keeping. Every place where I started a routine I counted up an atomic counter and added a defer with decreasing the counter. I added a simple API which reads the current value of all the counters. After some time, I could see which counters hovered around zero (good) and which counters steadily increased - there was a bug.

Also make sure to defer close all your rows in the database. Always use the context of your requests for issued operations and always defer close the response.Body. These are the most common causes for leaked routines.

1 Like

Nice tips from Falco, especially the lifetime limits that’s saved me before too.
One thing that helped me a lot when I was chasing something similar: don’t just look at the goroutine count, pull the full stack traces and diff them over time.

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines_t1.txt

…wait a few hours

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines_t2.txt

diff goroutines_t1.txt goroutines_t2.txt

debug=2 gives you the actual stack for every single goroutine, not just numbers. If you suddenly see 400 goroutines all stuck on the exact same line, you basically found it right there.

Also honestly, before doing manual counters everywhere (which is a good technique btw) I’d just throw go tool pprof http://localhost:6060/debug/pprof/goroutine at it and run top or traces. pprof groups everything by stack automatically, so the noisy one usually just floats to the top.

A few specific things :

time.After() inside a select in a loop — that timer doesn’t get GC’d until it fires, so if the loop runs often, it just quietly piles up. Switch to time.NewTimer() + defer timer.Stop().
A goroutine sitting there waiting to read from a channel, but the thing that was supposed to write to it already returned early (error path, etc.) and never sent anything. Super common and super sneaky.
Spawning a goroutine per request for background stuff instead of using a fixed worker pool. Each one “finishes eventually” on its own, sure, but under load they just keep stacking up faster than they drain.

What ended up working for me: every background job gets a hard context.WithTimeout, no exceptions, and anything async goes through a fixed-size pool instead of spawning freely. Doesn’t necessarily fix the root cause overnight, but at least it turns a silent leak into a visible queue backup, which is 100x easier to debug than “memory just keeps climbing and idk why.”

Good luck, this kind of bug is always annoying to track down.

1 Like

Are you using SQL transactions at all? I’ve had goroutines pile up when I was failing to rollback transactions in certain error states.