Dynamic functions in worker pool

Hi
I have a worker pool setup with heap. Nothing fancy.
I put a simple Http rest Ali on top and can pass a request/job to the queue. The problem is that different request/jobs needs different functions to be called.

For example:
Api request GET GetInfo with data : entity=abc
The worker needs to run the function
GetInfo(entity string) *entity {}
And another request could be POST SetData with entity =cde and data="something something. The worker should the run the function:
SetData(entity string, data string) result {}

So I need the worker to be able to run any function and return a response through the rest API.

I were thinking of making a request struct att pass the data as interfaces with the function intended.
type request struct {
Fnc func(data interface{}) interface{}
}

Is this the way to Go or how should I go about solving this.

Thanks
Mathias.

Here are a few ways to manage this. From your description I don’t know what is best. You might need to combine these techniques or use some other ones (if, for example, your jobs need to be persistent.

closures

If you just need to queue things up, then define your jobs as something like:

type Job func()

job := func() {
    e := GetInfo("entity")
    response(e)
    ///...
}

interfaces

If you need additional behavior for the jobs (e.g., checking progress), then use an interface:

type Job interface{
    Run()
    Progress() float64
}

Then implement the interface for each job type you have. These can also be nicely queued.

semaphores

If your main goal is to control the number of jobs that are happening at the same time (i.e., order of execution does not matter), then using a channel as a semaphore along with launching goroutines to queue work.

semaphore := make(chan struct{}, 10) // replace 10 with number of concurrent tasks

Then your job functions need to work with the semaphore:

func GetInfo(entity string) *entity {
    <-semaphore
    defer func() {
        semaphore <-struct{}
    }

    // do the work
    // ...
}

Thanks for a quick response!
I’m going to try to implement your suggestions and se how far that takes me!
Else I think I might need to post a clearer description with examples.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.