What happens to "wasted" space in slices?

Hi, all,

Anyone know what happens to the wastedSpace slice here? Does the garbage collector “know” that the whole slice isn’t being used?

I imagine it’s wasted. You could run a program allocating a slice of bytes with a length 1 and one that allocates a slice of bytes with a length of 10 million and see how memory is allocated on the heap when run. Then do a test where you fill all 10 million bytes and a test where you fill the first and last in the slice. See if memory allocation on the heap differs.

Now I say heap. Go might allocate it on the stack :upside_down_face: But same methodology counts.

It will probably be handled by the only way it can be handled…

References into the slice exist, so the slice can’t be garbage collected, as only this way you can guarantee that the references won’t become stale accidentaly and avoid memory fragmentation.

In C you wouldn’t free the underlying array as well, as you were unable to guarantee to be able to tell the allocator that there are still places which shouldn’t be freed…

1 Like

@NobbZ I wouldn’t call it the only way it could be handled; I can think of another.
However based on the mention here, it seems the entire slice is indeed preserved. This tells me that the GC keeps track of the “root” allocation and preserves the entire allocation as long as 1 or more references point into it, instead of recognizing that it could “simplify” the allocation from the []int slice to just two *ints and free the space between.

Is that documented anywhere? Can I depend on this space being preserved as long as a pointer into the slice exists?

I’m not aware of any allocator, that could free only parts of an allocation block.

I understand that tracking pointers within allocations and putting handling in place to free around an allocation could drastically degrade the performance of the garbage collector and/or allocator. I was just interested in knowing if this was a solved problem in computer science or not (I’m not a computer scientist). I understand garbage collectors to a limited extent, but I couldn’t think of how to handle the situation I described in my initial post. I was wondering if someone else in this community knew of an (efficient) way to handle this!

This has nothing to do with garbage collectors, but memory allocators. The garbage collector is limited by how the allocator can allocate and free memory blocks.

I don’t understand what you mean. I know that garbage collection and memory allocation are different things, but they both cooperate to manage memory:

  • The garbage collector needs to know what memory it is “allowed” to free. Go’s GC doesn’t seem to collect allocations made from C.malloc, so it needs to somehow “know” what allocations came from the Go runtime’s allocator (or allocators?).

  • Go’s GC is concurrent and while a collection is running, other goroutines need to communicate to the garbage collector when pointers are being updated, so the allocator in turn needs to know about the garbage collector.

Because of this somewhat tight coupling between two parts of Go’s memory management system, I didn’t think it was unreasonable to wonder if Go’s runtime could “know” that there are no references to data within the slice so it could possibly flag the [1:65535] region of the slice as available to the allocator.

As I was writing this, though, I thought of unsafe.Pointer which I think makes my whole point irrelevant. If there’s an unsafe.Pointer to anywhere inside of that slice, the GC has no idea how much data might be referenced, so it cannot partially free the slice.

Nothing happen, you are saying that the length of the slice is 65536, so you have 65536 of capacity, if you initialize an array with this size the memory not used inside of the array will be free or unused for the entire program life, if you don’t need such amount of memory you need to initialize the slice with small lentgh and left the “go” handle the increasing size of the slice,
Sorry my english.

The garbage collector can’t collect that unused space. Any pointer to an allocation will keep that whole allocation alive. At least, that’s true with the current collector; maybe someday we’ll make it better.

This is particularly relevant in cases like this:

t1 := new(T)
t2 := new(T)
s := []*T{t1, t2}
s = s[1:]

Even though t1 is no longer accessible, the garbage collector still considers it live, because s points to an allocation (a [2]*T), and that allocation points to t1. If t1 points to yet other things, it may keep an arbitrary amount of storage live. So when T has pointers in it, it may make sense to do this instead:

t1 := new(T)
t2 := new(T)
s := []*T{t1, t2}
s[0] = nil
s = s[1:]

To make sure the backing store of s no longer points to t1.

2 Likes