What you describe sounds almost like a database file…
Edit: What I mean is that writing and reading quickly from and to a (well, more or less) binary file is what a datatbase also provides.
If you are rather searching for a way to implement a specific binary file format, then a mmap package might be what you are looking for. (mmap = memory mapped file) (A Web search for golang mmap returns a few such packages; I have not used mmap yet so I cannot comment on any of these.)