Filtering a slice by value vs reference

[kahunacohen (Aaron Cohen)]

I have an “event” struct that contains a slice of duration structs. I need to filter out the durations that are expired, or have already happened.

The use-case is in a very high-traffic micro service running gin. It’s not user-facing, but is consumed by a user-facing restful API service. I expect that these slices will have not much more than, say, 20 elements, and each request may have about 20 events structs.

I’m not sure, given the use case, if in the example where I pass by reference, and leverage the slice’s underlying array is worthwhile in terms of memory footprint. The example where I pass by value and allocate a new array for the filtered durations is simpler to read and probably less error prone. I realize that the structs in the slice are lightweight, given they are just ints.

The question is whether in my use-case the savings in memory is worthwhile, given that the the server is high traffic. I’m not sure given the expected size of the data and the fact that it’s in a server context that passing by reference and re-using the underlying array is worthwhile. There’s also the issue of consistent API. Let’s say that the events have more complicated data, and I expect to do multiple steps of post-processing, filtering on other properties etc and thus very well may add other functions that do similar things. I’m not so much fretting over every bit of memory, but it’s the first time I’ve used a language with pointers in production code. Thoughts?

Example:
playground

package main

import "fmt"

type duration struct {
	ValidFrom  int64
	ValidUntil int64
}
type event struct {
	Recurrences []duration
}

func filterExpired(evt event, now int64) event {
	var recurrences []duration
	for _, r := range evt.Recurrences {
		if r.ValidUntil >= now {
			recurrences = append(recurrences, r)
		}
	}
	evt.Recurrences = recurrences
	return evt
}
func filterExpired2(evt *event, now int64) {
	// Take a zero-length slice of the underlying array.
	temp := evt.Recurrences[:0]
	for _, r := range evt.Recurrences {
		if r.ValidUntil >= now {
			temp = append(temp, r)
		}
	}
	evt.Recurrences = temp
}

func main() {
	evt1 := event{Recurrences: []duration{{ValidFrom: 100, ValidUntil: 200}, {ValidFrom: 250, ValidUntil: 300}}}
	fmt.Println(filterExpired(evt1, 201))
	evt2 := event{Recurrences: []duration{{ValidFrom: 100, ValidUntil: 200}, {ValidFrom: 250, ValidUntil: 300}}}
	filterExpired2(&evt2, 201)
	fmt.Println(evt2)
}

[mje (Jeff Emanuel)]

I would go with your instinct of creating a new slice until GC pressure becomes a real problem. You even mention the likelihood of multiple filtering steps. That will definitely complicate reusing the slice if your multiple steps run concurrently.

@kahunacohen

first time I’ve used a language with pointers in production code

Not Go, but much earlier in my career I spent many long days debugging from core dumps errors caused by manipulating pointers. Now I do anything I can to produce code that is correct now and resilient to changes in the future.

[kahunacohen (Aaron Cohen)]

I am also conflating passing by reference and not allocating an extra array. Both of which are not necessary in this context as far as I can tell.

[GonzaSaya (Gonzalo Sayago)]

something like this?

func filterExpired3(evt *event, now int64) {
	newSize := 0
	for i := 0; i < len(evt.Recurrences); i++ {
		if evt.Recurrences[i].ValidUntil >= now {
			if newSize != i {
				evt.Recurrences[newSize] = evt.Recurrences[i]
			}
			newSize++
		}
	}
	evt.Recurrences = evt.Recurrences[:newSize]
}

[stand_chan]

I’m confused that it may same as “var recurrences duration”，and there is one thing easily misunderstood in Golang. where you pass value ( slice or map ) as param , it always pass by reference, whatever you use “evt event” or “evt *event”

@kahunacohen

temp := evt.Recurrences[:0]