Yet Another Goroutine Pool - Simple Implementation, Generics

Releases
1

GitHub Link: https://github.com/txaty/gool

There are already many good Goroutine pool implementations, and I love their designs so much.

Here I post my own. I made this library several months ago for the parallel execution of the Merkle Tree library.

Although the overhead of Goroutine is significantly lower than a regular thread, process, pooling, and reusing goroutines property can still help improve the program performance (and it did help with the performance of the Merkle Tree library).

The methods of my goroutine pool are just like Python ThreadPoolExecutor:

  • Submit: Submit a task and return the result (if any).
  • AsyncSubmit: Submit a task and return a future of the result (if any). The future is the result channel.
  • Map: Submit a bundle of tasks and return the results in order (if any).
  • AsyncMap: Submit a bundle of tasks and return the futures of the results (if any). The futures are the result channels.

To use it, you need to define the following:

  • Handler function: handler func(A) R, and
  • Argument: arg A.

With types A and R being arbitrary generic types.

When creating a new pool, you can specify the number of workers numWorkers and the task queue size cap.

Please feel free to take a look and give some comments. Thanks!

1 Like
2

Hi @tommytim0515 ,

Your library is pretty nice, and could be a valuable tool for many. But, it could also benefit from the following improvements:

  • add examples in the readme, examples on how to use it
  • add unit tests, for both pool and worker (each public method should have a unit test)
  • add an examples folder with a sample app using it
  • add package comments
  • add method comments
  • add more info in the readme (how to test, what are the limitations, benchmarks)
  • add benchmarks
  • add graceful shutdown
  • add a way to configure the pool with the “max amount of time a job is allowed to run, before it gets cancelled”; if your jobs take too long they will hog up the workers and all blocks
  • test its limits, add a diagram showing the CPU count on one axis, and the max jobs capable to run (for the same mock job) without blocking or crashing
  • add a smoke test (this is in a separate file)
  • add an interface with the methods expose to the user, and a method to create an object implementing it
  • add linters
  • add a CI pipeline config for it
  • add a Dockerfile to build an image of it, and upload that image to a registry
  • fix the bugs (ask chatGPT about this)
  • add better concurrency support
1 Like
3

Good work :+1:

4

Your detailed comment is highly appreciated!

5

Thanks!