I’m trying to understand the limits of atomic access to bytes in a []byte.
I modeled the following in the code below:
- I have a data buffer of []byte where the low order nibble of each byte is an owner id obviously ranging from 0 - 15
- I have 16 go routines given ids 0 - 15. Imagine each go routine owns the bytes that have a low order nibble == their id.
- The routines constantly and concurrently scan all the data looking for their bytes, and when they find one, they change the high order nibble of the byte in the buffer.
Importantly, 15 routines can access the same byte concurrently while one routine can write to the byte, but since one and only one routine will ever write to a given byte, there is no race condition. Just a question of atomic loading and setting, I believe.
It seems to work fine running for a long time on 16+ core AMD x64 hardware (both Ryzen and Threadripper). If this does actually work, it will save us a huge synchronization cost.
Is this actually safe, or is it hardware-dependent, or am I just getting lucky in this test?
Thanks,
Warren
package main
import (
"fmt"
"sync"
"sync/atomic"
"time"
)
func changeMyData(myID byte, data []byte, running *int64, wg *sync.WaitGroup) {
// each worker will only change bytes that match
// their id in the low order nibble
var passNum byte
for {
// All workers are concurrently reading all of data, and all workers
// are concurrently writing to data, but just to the bytes they own.
for i, b := range data {
// if id is in the low order nibble, then this byte is owned by this routine.
if b&0x0F == myID {
// put passNum in the high order nibble and myID back in the low
data[i] = passNum<<4 | myID
}
}
// we only check for a stop between complete passes through the data
// so all all bytes with this id have the same passNum in the high nibble.
if atomic.LoadInt64(running) == 0 {
wg.Done()
return
}
passNum++
}
}
func main() {
// Make the length shorter (min 32) to increse the velocity of changes at every byte.
// Make longer to have a better chance catching write errors with wrong passNum.
data := make([]byte, 64)
var id byte
for i := range data {
data[i] = byte(i & 0x0F) // just set the low order nibble to incrementing 0-15
}
// turn on the running flag
running := new(int64)
*running = 1
var wg sync.WaitGroup
for id = 0; id < 0x0F; id++ {
wg.Add(1)
go changeMyData(id, data, running, &wg)
}
// Make this longer to increase the chance of a conflict
time.Sleep(10 * time.Second)
atomic.StoreInt64(running, 0)
// Wait for the workers to finish
wg.Wait()
// Now make sure the data is what we expect...
// make sure the low order nibble is still correct for every byte.
for bNum, b := range data {
// expected low order nibble
id = b & 0x0F
if byte(bNum&0x0F) != id {
fmt.Printf("Error: owning ID of bNum %v=%X changed to %X\n", bNum, bNum, id)
}
}
// make sure the high order nibble is the same for every byte owned by a given id.
// This just means all bytes for an id are the same: same high and same low nibble.
for id = 0; id < 0x0F; id++ {
expectedByte := data[int(id)]
// loop over each byte owned by this id.
for bNum := int(id); bNum < len(data); bNum += 0x10 {
if data[bNum] != expectedByte {
fmt.Printf("Error: byte at bNum=%X got %X but expected %X\n", bNum, data[bNum], expectedByte)
}
}
}
fmt.Printf("ByteNum:")
for bNum, _ := range data {
fmt.Printf("%3X", bNum)
}
fmt.Printf("\n Value:")
for _, b := range data {
fmt.Printf("%3X", b)
}
fmt.Println()
}