Thanks for the response @skillian!
But I don’t understand what you mean by that; what’s “arbitrary memory corruption?” If you have a hardware error in your memory or there’s some strong electromagnetic disturbance, that could potentially corrupt your memory regardless of the memory page safety.
In this case, I just meant that if some other buggy code (e.g. a C library wrapped as a Golang package) extremely infrequently and/or randomly writes a byte into memory then if that memory is read-only then the corruption would get caught immediately.
Could you not refactor them to something like this: … This gets you compile-time safety at the cost of “clunkier” syntax.
Thanks for the suggestion!
Yeah, the problem is that the structures in reality are much larger and can be hierarchical. The current usage pattern is:
- Many functions read and write to a larger struct until a certain point in time AKA construction phase.
- At a certain point in time then the larger structure becomes fixed / frozen / unchangeable.
- After being frozen then same functions should work as before to navigate the structures, and no more updating should happen.
It’s guaranteeing the last part “and no more updating should happen” which is the tricky bit. This is currently done in the code by taking a cryptographic hash of the serialized data structure. In theory the hash only has to be taken once and uniquely identifies the frozen structure. But in reality the hash gets calculated multiple times to ensure that the frozen data structure has not changed somehow, e.g. by buggy code or due to memory corruption or whatever.
The read-only memory map is interesting and could work as long as you never put any pointers into your structs that reference Go memory, which may or may not include strings.
Thanks, and that’s also what I was thinking too. I have a hacky PoC which copies structs, slices, and strings into an own memory map. In this case, there are pointers but they are pointers to memory I have created myself (and not Golang) in the memory map. So this means the data structure in the memory map DOES contain pointers but all the pointers point to things inside the memory map and all those things have been created by me.
This seems to work, BUT… I wanted to implement the same thing for Golang maps and now the problems start. If the arbitrary struct contains a pointer to a map then I’d like to recreate the map in the memory map too. However, the multiple data structures for the map, and biz logic to navigate them, are all hidden away in Golang’s own
runtime package. So not sure how to recreate / copy a Golang managed map inside an own memory map. Because the Golang managed map uses multiple structs which are hidden inside the
runtime package, I have no way to recreate them inside the memory map.
The only idea I have so far – and likely difficult to implement – is as follows:
- For a given version of Golang on a given platform, my package
- Grab the
runtime source files and copy them into my package
- Auto change the copied source files so that their package is
xyz instead of
- Assuming I can get it to compile then now I have a ‘copy’ of runtime built into my package.
- That also means I can see all the hidden map internal data structures from my package.
- In theory I can auto change the
map.go copy so that when it allocates its structures, they do not go on the heap or stack, but go in my memory map
As a side functionality, the code will also be able to dump not only structures but also all the internal Golang memory associated with those structures. For example, packages like
spew already dump arbitrary complex Golang data structures, but only in relatively high level format. Yes, there it dumps pointer values, but it doesn’t dump the memory and pointers associated with the internal Golang data structures such as struct internal padding, slices, strings, and finally the more complicated map internal structs.
So my plan is:
- Step 1: Use the copy
runtime technique to get access to the internal map structs.
- Step 2: Finish off my ‘dumper’ which dumps all memory associated with an arbitrary user struct.
- Step 3: Use the
runtime copy to recreate maps at an arbitrary memory location, and use the step 2 dumper to check that all memory associated with the cloned data structure is in the memory map of my choosing.
Lots of problems to solve here, but in theory after I recreate an arbitrary map at a data location of my choosing – e.g. in my memory map – in theory, if I will be able to access that map in memory map with the ‘real’
runtime map functions and it’ll just work. If writing it will fail either because the memory map is read-only, or because it tries to update non-Golang managed internal structures.
What do you think? And is there an alternative way?