I want to create a goroutine that translates any data from an incoming channel to an outgoing channel. I need to tolerate creating many of these goroutines, so I want the whole thing to be garbage-collectable without having to close the incoming channel (I don’t have access to any lifetime information or control over the channel I get). Essentially, I want to fully decouple the GC from the incoming channel reference in the repeater function, and detect when the incoming channel is GC’d, close the outgoing channel and exit.
Weak references would be a perfect fit for this task, but they only support pointers and I need this to work on bare channels.
Judging from the Go documentation alone, a weak channel reference shouldn’t be anything unusual, as it categorizes channels under the same category as pointers.
Here is a working example of what I’d like to achieve, using pointers (which I can’t use for my actual problem, because I have no control of where the channel comes from) [playground]:
package main
import (
"fmt"
"runtime"
"time"
"weak"
)
func repeater(out *chan int, in *chan int) {
go func() {
t := time.NewTicker(100 * time.Millisecond)
defer t.Stop()
defer func() { fmt.Println("repeater stopped") }()
weakIn := weak.Make(in) // we want to be able to close out when in is GC'd
for {
in := weakIn.Value()
if in == nil {
fmt.Println("in was GC'd")
close(*out)
return
}
select {
case v, ok := <-*in:
if !ok {
close(*out)
return
}
*out <- v
case <-t.C:
// spin to detect GC of in
}
}
}()
}
func main() {
_out, _in := make(chan int), make(chan int)
out, in := &_out, &_in
repeater(out, in)
go func() {
*in <- 1
*in <- 2
}()
fmt.Println(<-*out, <-*out) // should be 1 2
done := make(chan struct{})
runtime.AddCleanup(out, func(struct{}) {
fmt.Println("out cleaned up")
done <- struct{}{}
}, struct{}{})
runtime.AddCleanup(in, func(struct{}) {
fmt.Println("in cleaned up")
done <- struct{}{}
}, struct{}{})
runtime.GC()
<-done
// We need 2 GC cycles here for some reason.. whatever.
runtime.GC()
time.Sleep(200 * time.Millisecond)
runtime.GC()
time.Sleep(200 * time.Millisecond)
<-done
}
IDK if this is actually doable in Go, but I’d appreciate any feedback or discussion regarding this, TIA
Why don’t you want just to read from a channel in for loop, as soon as in is closed loop is exited and the next step is to close out as well. After that both of them will be GC-ed. Very questionable use of runtime package and channels in general.
package main
import (
"fmt"
)
func repeater(out chan int, in chan int) {
go func() {
defer func() {
close(out)
fmt.Println("repeater stopped")
}()
for v := range in {
out <- v
}
}()
}
func main() {
out, in := make(chan int), make(chan int)
repeater(out, in)
go func() {
in <- 1
in <- 2
close(in)
}()
for v := range out {
fmt.Println(v)
}
}
Your solution is NOT GC’d - it’s what I currently have, but it’s leaking memory.
You can verify by setting GODEBUG=gctrace=1 and spawning 100k of these so they actually show up in the memory stats. You’ll see they’re not getting GC’d.
How I understand it, Go’s GC doesn’t clean up goroutines, which is why my solution makes it exit manually.
EDIT: The point is I can’t ensure the channels are closed - Go should guarantee unused channels are GC’d, but the presence of a repeater seems to break that. (I don’t have control over the channels I receive and what they do)
With my current understanding, this usage of runtime and channels is necessary given any other attempt leaked memory. But please do point out anything in particular that I could so better
Can you be more specific what is not getting GC-ed? Even with gctrace I see cleanups and after benchmarking I can see everything getting collected as expected. GC cleans channels as any other object as soon as it is closed and goroutine as soon as it exits.
Maybe you can explain what you are trying to do in details? You do not control channels or what?
TLDR: No, I don’t control the channels and cannot ensure they’re closed properly (sorry, I should have been more clear about that).
To be more specific: I am building a binding generator for a scripting language that uses its own representation of data types. Channels in the scripting language have an underlying type of basically chan *AnyTypeInTheScriptingLang, which I have to convert between with the Go type e.g. chan int. The way I do this is to spawn a goroutine that reads data from one chan, translates it and sends it into the other chan. This is also why I don’t control when the channel closes - I don’t control what the user’s script does, and I don’t control what the channel sender from the Go library does.
Well that looks like exactly not your responsibility then. And the solution is specifically state in docs, that it is user’s task to close in channel on one end, so the out channel get close as well. There are many examples in go’s std as well. We always need to call Close on so many objects, connections, channels on our own and it’s a mistake if we don’t. IMHO it’s an error if channel is not closed but somehow got collected by GC. I can’t actually imagine this situation. Another solution is a custom type for description of the script language channels. If this scripting language can somehow signalize that they close the channel then you need to learn how to track it from go and wrap your go implementation as a struct with explicit isOpen() bool.
I agree that it’s bad practice to leave a channel open, but that is an option you have in Go (see go - Is it OK to leave a channel open? - Stack Overflow, which links to a google groups discussion where Ian Lance Taylor himself says closing is just a signal).
I guess simply documenting this memory-leaking behavior is what I’ll have to do for now. Doing GC from the side of the scripting language is likely possible - perhaps even the using runtime.Weak like in the example (or a custom GC further into the future); however this still doesn’t account for the Go library possibly not closing its channel.
This unfortunately leads me to believe doing this isn’t entirely possible in the current state of Go.
Anyways, thanks a bunch for your time in discussing this
