There are already many good Goroutine pool implementations, and I love their designs so much.
Here I post my own. I made this library several months ago for the parallel execution of the Merkle Tree library.
Although the overhead of Goroutine is significantly lower than a regular thread, process, pooling, and reusing goroutines property can still help improve the program performance (and it did help with the performance of the Merkle Tree library).
The methods of my goroutine pool are just like Python ThreadPoolExecutor:
Submit: Submit a task and return the result (if any).
AsyncSubmit: Submit a task and return a future of the result (if any). The future is the result channel.
Map: Submit a bundle of tasks and return the results in order (if any).
AsyncMap: Submit a bundle of tasks and return the futures of the results (if any). The futures are the result channels.
To use it, you need to define the following:
Handler function: handler func(A) R, and
Argument: arg A.
With types A and R being arbitrary generic types.
When creating a new pool, you can specify the number of workers numWorkers and the task queue size cap.
Please feel free to take a look and give some comments. Thanks!
Your library is pretty nice, and could be a valuable tool for many. But, it could also benefit from the following improvements:
add examples in the readme, examples on how to use it
add unit tests, for both pool and worker (each public method should have a unit test)
add an examples folder with a sample app using it
add package comments
add method comments
add more info in the readme (how to test, what are the limitations, benchmarks)
add benchmarks
add graceful shutdown
add a way to configure the pool with the “max amount of time a job is allowed to run, before it gets cancelled”; if your jobs take too long they will hog up the workers and all blocks
test its limits, add a diagram showing the CPU count on one axis, and the max jobs capable to run (for the same mock job) without blocking or crashing
add a smoke test (this is in a separate file)
add an interface with the methods expose to the user, and a method to create an object implementing it
add linters
add a CI pipeline config for it
add a Dockerfile to build an image of it, and upload that image to a registry
