When I first saw the announcement for pargo, I though it could be useful.
When I saw there weren’t any examples, I though, I know just the thing!. I had all sorts of things going for me:
- it is embarrassing parallel
- processing is the bottleneck
- I have written and improved several parallel versions
- I have tests to make sure still works after using the package
- I have benchmarks that will tell me how well it performs
- Its a good excuse to update my article, maybe even using go1.9beta2.
- I saw this neat
parallel.Range function that looked like it would do just what I needed.
Then I tried to figure out how to use
threshold. Since I missed the comments in the introduction that gave an idea of how to use
threshold, I was left the descriptions of how threshold was calculated in the comments for
pargo.ComputeEffectiveThreshold. This didn’t help me understand how to use it, so I read the code for both functions. I ignored the the seemingly inefficient recursive divide-and-conquer implementation of
Finally, I wrote a program to try and figure things out. Its output wasn’t helpful for understanding how to use
threshold, though it did uncover a bug (now fixed).
I asked questions and got some good answers.
However, I still don’t know why you think
threshold is a better way to decide how many parts the range should be split into. Your advice was:
It’s difficult to come up with good recommendations here that work well for all cases, so you have to experiment.
The docs now say to start with a threshold of 4. From the ranges in my example above that would use 8 goroutines.
Now to experiment a bit. Maybe change
threshold to 3. Oh. It still uses 8 goroutines. Even though threshold changed, that change did not affect performance.
Ok, then. I’ll try again. May use 5 this time. Now there are twice as many goroutines. For my embarrassingly parallel example with a computational bottleneck, I expect this to drastically reduce performance.
So, the smallest possible change in either direction from the starting recommendation of 4 will either have no effect or double the number of sub-ranges the work is split into.
I have expended time and effort trying to figure how threshold is better than directly setting the number of sub-ranges/goroutines.
Asking several times hasn’t produced arguments or evidence for
threshold. Do you have any?