I’m trying my first real concurrency project in GO.
I just want to import a few thousand rows of data into postgres using GORM.
I start some worker goroutines and then iterate over the file and pass each line to a channel that a worker goroutine processes using range.
Seemed like all was working well but I found that when I use more than one worker db.Create(&c) does not populate customer © with the correct ID and sometimes it even updates another customer with the phone numbers from the current line of data.
I don’t notice this happening with 1 goroutine so I have to believe it is related to having more than one worker and the data getting crossed up. But I don’t see how.
GORM is supposed to be Thread safe. I am also aware of the pitfalls of passing iterated values into goroutines.
So I have avoided that. I even moved gorm.open() into the worker instead of using a global.
Anybody got any ideas?
importFamilies() reads the file and sets everything up.
createFamily() is the worker
link to code:
https://play.golang.org/p/18yai_J5NV
UPDATE and ANSWER to my own question:
I ended up updating GORM to the latest commit. I was using a version from Aug 2016. All my problems went away.
Not only that but my performance went from about 4 minutes single worker or 1.5 minutes with 4 workers to under 18 seconds single worker or <5 seconds with 4 workers. 3.8 secs with 12 workers.
Either my installation had problems or they made some big improvements.
I am a bit relieved because I thought even 1.5 minutes was quite slow for a few thousand records.
Plus it wasn’t even working with goroutines. I was considering changing SQL libraries.