I have a small hobby project that in the realm of event sourcing and stream processing. Basically, it is something similar to https://geteventstore.com. Please keep in mind that this is just a for fun project and I’m not trying to reinvent the wheel, but more deeply understand some of the existing wheels
I’m currently using boltdb for the storage layer and wanted a bit of insight on how best to implement the data structures.
My naive solution is to have a $all bucket, where all the events are stored in sequential order and then a bucket per aggregate/stream by name where it will store version number and the id of the record in the $all stream. The quick implementation can be found here: https://play.golang.org/p/l_QxlgIOhA
The two main read use cases that should perform well are a full scan of the $all stream and then full scans of each of the streams by name.
The current approach performs well enough, however, is there a better way to handle those two use cases well? I feel as though I may be missing an obvious way to utilize the range and prefix scans for the stream by name use case.
Just as a point of data, we use BoltDB in production as an event store and just store all the events for a domain context in the “events” bucket. (Using the bucket sequence as you do in your example, our strongly-typed event model has ‘type’ field to distinguish between objects.) We don’t have a standalone event store service except as a read-only cache. The canonical event store is part of the command handler. We save domain aggregate changes and events in the same boltdb transaction. We leave it as an exercise for the consumer to filter out events they are not interested in.
