Goroutine Memory Leak Causing Gradual Performance Degradation on My Website Backend Built With Go

Getting Help
1

Hello Go Community,

I am currently facing a persistent performance issue with my website backend that is built using Go, and I am hoping to get some advice from experienced developers who may have encountered a similar situation. The core problem is that the application performance gradually becomes slower over time until the service needs to be restarted. The website initially works perfectly after deployment, with fast response times and normal resource usage, but after running continuously for several hours or days, memory consumption steadily increases and the number of active goroutines continues growing. Eventually, the application starts responding slowly to user requests, even though the traffic level remains relatively stable. Restarting the Go service temporarily resolves the issue, but the problem returns again after extended uptime.

The issue appears to be related to how background processes and concurrent tasks are handled within my application. The website uses Go routines for several asynchronous operations, such as processing user requests, handling API communication, updating database records, and running scheduled background jobs. These tasks are created successfully and complete normally in most cases, but monitoring shows that the total number of goroutines does not return to its original level after certain operations finish. Over time, the application accumulates more active goroutines than expected, which appears to contribute to increased memory usage and reduced performance. I have reviewed the code paths where goroutines are created, but I have not yet identified which specific process is failing to clean up properly.

I have already used Go’s built-in profiling tools, including pprof, to investigate the memory usage and goroutine activity. The profiles indicate that there are many long-running goroutines remaining active, but the stack traces are not immediately clear enough for me to determine the exact source of the leak. I have checked common causes such as missing channel closures, blocked channel operations, and forgotten context cancellation, but I have not found an obvious issue. The application uses multiple services and packages, so isolating the exact component responsible has been challenging. The problem also does not happen immediately during testing because the application needs to run under normal user activity for a longer period before the resource usage becomes noticeable.

Another confusing part of this issue is that the application does not crash or produce obvious error messages when the problem occurs. The Go runtime continues operating normally, and there are no panic logs indicating a fatal problem. Instead, the degradation happens gradually, with slower API responses, increased garbage collection activity, and higher memory consumption. Database queries and external service requests continue functioning, which makes it difficult to determine whether the issue is caused by my application logic, a dependency, or an incorrect concurrency pattern. I have added additional logging around goroutine creation and completion, but the logs have not yet revealed a clear reason why certain routines remain active indefinitely.

I have also reviewed the architecture of the website backend to ensure that resources are released correctly after each request. Database connections are managed through the standard Go database package, HTTP clients are reused properly, and context cancellation is implemented in many areas where operations may take longer than expected. However, because the application handles multiple concurrent users and performs background processing, I suspect there may still be a hidden lifecycle issue where certain goroutines continue waiting for events that never occur. I am particularly interested in understanding whether there are recommended patterns for managing long-running goroutines, worker pools, and background services in production Go applications to prevent this type of gradual resource exhaustion.

I would appreciate guidance from the Go community on how to properly identify and resolve this goroutine memory leak issue. Specifically, I would like to know the best methods for analysing goroutine profiles, finding blocked or abandoned goroutines, and structuring concurrent code so that all background tasks terminate correctly when they are no longer needed. Any recommendations regarding context management, channel handling, worker lifecycle design, or debugging techniques would be extremely valuable. My goal is to ensure that my Go-powered website backend can run reliably for long periods without memory growth, increasing goroutine counts, or requiring manual restarts. Sorry for long post!

2

One way to mitigate the immediate issue is to limit the lifetime of any go-routine your service starts anywhere. Set a Maximum Lifetime on your DB connection pool. Always create context for routines with hard time-outs. Make sure everything has a fixed maximum time to live.

I had a similar problem once and found the root with manual book-keeping. Every place where I started a routine I counted up an atomic counter and added a defer with decreasing the counter. I added a simple API which reads the current value of all the counters. After some time, I could see which counters hovered around zero (good) and which counters steadily increased - there was a bug.

Also make sure to defer close all your rows in the database. Always use the context of your requests for issued operations and always defer close the response.Body. These are the most common causes for leaked routines.

3

Nice tips from Falco, especially the lifetime limits that’s saved me before too.
One thing that helped me a lot when I was chasing something similar: don’t just look at the goroutine count, pull the full stack traces and diff them over time.

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines_t1.txt

…wait a few hours

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines_t2.txt

diff goroutines_t1.txt goroutines_t2.txt

debug=2 gives you the actual stack for every single goroutine, not just numbers. If you suddenly see 400 goroutines all stuck on the exact same line, you basically found it right there.

Also honestly, before doing manual counters everywhere (which is a good technique btw) I’d just throw go tool pprof http://localhost:6060/debug/pprof/goroutine at it and run top or traces. pprof groups everything by stack automatically, so the noisy one usually just floats to the top.

A few specific things :

time.After() inside a select in a loop — that timer doesn’t get GC’d until it fires, so if the loop runs often, it just quietly piles up. Switch to time.NewTimer() + defer timer.Stop().
A goroutine sitting there waiting to read from a channel, but the thing that was supposed to write to it already returned early (error path, etc.) and never sent anything. Super common and super sneaky.
Spawning a goroutine per request for background stuff instead of using a fixed worker pool. Each one “finishes eventually” on its own, sure, but under load they just keep stacking up faster than they drain.

What ended up working for me: every background job gets a hard context.WithTimeout, no exceptions, and anything async goes through a fixed-size pool instead of spawning freely. Doesn’t necessarily fix the root cause overnight, but at least it turns a silent leak into a visible queue backup, which is 100x easier to debug than “memory just keeps climbing and idk why.”

Good luck, this kind of bug is always annoying to track down.

4

Are you using SQL transactions at all? I’ve had goroutines pile up when I was failing to rollback transactions in certain error states.

5

Best practice: Always use a deferred cancelled context with each and every transaction:

func doDBStuff(ctx context.Context, ...) {
  txCtx, cancel := context.WithCancel(ctx)
  defer cancel() // This will kill the Transaction after function exit
  tx, err := database.BeginTx(txCtx)
  ...
}
6

Yeah - on this specific project, I inherited it from a somewhat junior dev team. It was literally their first Go project (and they actually did a really good job for it being their first Go project!). And it used Gorm throughout. It was a little messy and over the years I have tracked down most of the cruft.

The official/usual way of handling it is to just defer a rollback:

Defer the transaction’s rollback. If the transaction succeeds, it will be committed before the function exits, making the deferred rollback call a no-op. If the transaction fails it won’t be committed, meaning that the rollback will be called as the function exits.

But I agree this is something where a context with a reasonable timeout makes sense!

7

Some great tips in this thread, just prompted me to look through some old code to see what I was doing wrong, and the Go docs to work out what I should be doing. Apparently time.After() is no longer the problem that it used to be.

Before Go 1.23, this documentation warned that the underlying Timer would not be recovered by the garbage collector until the timer fired, and that if efficiency was a concern, code should use NewTimer instead and call Timer.Stop if the timer is no longer needed. As of Go 1.23, the garbage collector can recover unreferenced, unstopped timers. There is no reason to prefer NewTimer when After will do.

8

You’re right, thanks for the clarification! I checked the official Go docs and can confirm: as of Go 1.23, the GC can recover unreferenced timers even if they haven’t fired or been stopped. So the classic memory leak I described no longer applies on newer versions.

There’s still a real reason to prefer time.NewTimer() + timer.Reset() over time.After() in loops, but it’s about performance, not leaks:

Memory allocations: every time.After() call allocates a new object on the heap that the GC then has to collect.
If performance and allocs/op are critical, creating a single time.NewTimer() outside the loop and reusing it with timer.Reset(d) remains more efficient, since it avoids putting extra pressure on the GC.

9

@joeroot did you find the issue? let us know how this ordeal panned out..

10

The docs seem a little ambiguous - the memory leak problem refers to the function that takes a time.Duration:
func After(d Duration) <-chan Time
there is also a function/method on type time.Time:
func (t Time) After(u Time) bool
I have been using the latter, and I don’t think this has the same problem.
if time.Now().After(stopTime) …

11

That makes sense, especially the idea of tracking goroutine counts by the location where they are started. I had been focusing mostly on pprof and stack traces, but keeping separate counters should make it much easier to identify which part of the application is steadily accumulating goroutines over time. I’ll also review the database code to make sure rows are always closed, request contexts have appropriate timeouts, and every response.Body is properly closed. Setting sensible maximum lifetimes for background operations and checking the connection pool limits should also help prevent resources from remaining active indefinitely. Thanks for sharing the practical approach and the common leak points to investigate.

12

Since the goroutine count keeps increasing, I would also compare multiple goroutine profiles taken at different times rather than looking at just one snapshot. If the same stack traces keep appearing in larger numbers, that can make the leaking code path much easier to spot.

It may also be worth enabling block and mutex profiling temporarily, because a goroutine that appears to be “leaking” is often simply stuck waiting on a channel, lock, timer, or network operation that never completes.