[SOLVED] Boltdb data structure for an event store

I have a small hobby project that in the realm of event sourcing and stream processing. Basically, it is something similar to https://geteventstore.com. Please keep in mind that this is just a for fun project and I’m not trying to reinvent the wheel, but more deeply understand some of the existing wheels :slight_smile:

I’m currently using boltdb for the storage layer and wanted a bit of insight on how best to implement the data structures.

My naive solution is to have a $all bucket, where all the events are stored in sequential order and then a bucket per aggregate/stream by name where it will store version number and the id of the record in the $all stream. The quick implementation can be found here: https://play.golang.org/p/l_QxlgIOhA

The two main read use cases that should perform well are a full scan of the $all stream and then full scans of each of the streams by name.

The current approach performs well enough, however, is there a better way to handle those two use cases well? I feel as though I may be missing an obvious way to utilize the range and prefix scans for the stream by name use case.


The full scan of $all has to be in eventId order?

If yes, I don’t see other solution. If not, then you could use “bucket:eventId” keys, and save off obe indirection.

Makes sense to me at a basic level.

Just as a point of data, we use BoltDB in production as an event store and just store all the events for a domain context in the “events” bucket. (Using the bucket sequence as you do in your example, our strongly-typed event model has ‘type’ field to distinguish between objects.) We don’t have a standalone event store service except as a read-only cache. The canonical event store is part of the command handler. We save domain aggregate changes and events in the same boltdb transaction. We leave it as an exercise for the consumer to filter out events they are not interested in.

Thanks folks. Just wanted to bounce the idea off a few people before committing :slight_smile:

@Chad_Maine out of curiosity have you ran into any unforeseen problems with your system yet? Sound like we are going down a very similar path here.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.