Job/ worker problem

Hi,

Recently I have read one blog where they have used the go job/worker 2 tier channel concept to handle the 1 million request per minute.
They have 2 tier channel concept where jobqueue channel will have the job to be done and there is dispatcher which will have worker pool and job channel . dispatcher will check any jobchannel is free in worker pool and if so it send the job for worker to process otherwise it will wait for workers to be free.
Reference :https://medium.com/smsjunk/handling-1-million-requests-per-minute-with-golang-f70ac505fcaa
My question is that why do we need this two tier channel concept , I can have a jobqueue channel where jobs will be there., I will create 10 or 20 worker go routines, which will read from jobqueue channel to process it.
can you explain whether I’m missing anything here?
thank you so much for the help

3 Likes

The blog is related to modelling against Ruby’s sidekiq job/worker working model. The actual goal is not just about achieving concurrency; it’s about guarantees of job executions.

Back in time due to a bunch of limitation from app engines (e.g. Heroku, Google App Engine), session concurrency must comply to the main thread’s deadline (e.g. 30 / 60 seconds). If the main thread is killed, the worker’s job stops regardless of jobs completion. Hence, there is no guarantees of the worker’s job is done.

The idea is 1 tier records the job with persistent memory; the other as execution. Hence, every time when there is a new job scheduled from main thread, the worker can resumes its last execution.


Doing this leads a number of design questions like where to store those jobs persistently; when to do persistent job/worker; when not to.

1 Like

Hi Hollaway,

Thank you for the response.
so what I understood , we can also do like have the constant number of workers read from same channel to perform concurrent operations.
Nothing more is benefit in using the 2 tier channel concept as in blog than the above approach.
Please correct me if I understood wrong

2 Likes

No harm using either methods as long as you benchmark your concurrency to meet the deadline and they are achieving the requirements. I disagree with the 2nd part as it has its own benefits.


If we use your implementation, concurrency does work but in situation where some jobs are incomplete (e.g. abrupt termination, crashes), the scheduled job is lost. Also, the concurrency implementations is wrapped under the application.

job/worker approach is to ensure even in those jobs are managed even under those unexpected events: the unfinished jobs can be resumed. Hence, the “guarantee” and “managed”. You can imagine it as “concurrency abstract” or an “engine”. You can further upgrade the “engine” with other monitoring or job tracking tools without changing all your project codes.


Keep in mind that Go simplified concurrency by A LOT compared to Ruby and C environments, which is why sometimes it made us questioning the needs of 2-tiers. There is no strict rule saying it is either or but rather select the one that is simple and comprehensibly efficient.

E.g. for a production web application, job/worker approach will be sensible; for simple concurrent data processing like parsing a string to multiple words, it would be overkill.

1 Like

Even in job/worker approach , error can happen (panic) and jobs can be incomplete rite.
In 1 tier channel concept, I can put back in jobqueue channel to process again incase of panic.

so, how job/worker approach will be sensible in production instead normal

sorry for making a lot of questions.

1 Like

If the job is created by the main program and it has the initial data to re-initialize the run, then yes, it is possible to reschedule.

However, there are cases where the jobs are not initialize by the main program. Take “reset password’s email” for an example. These emails are triggered by users’ call and we usually send it to worker as emails take a long time to send to avoid blocking.

In those cases, on the program side, the program does not know precisely what emails are scheduled at a given time. Hence, when unforeseen circumstances occurs, it’s easier said than done to recover the scheduled jobs without some kind of job management.

Of course, one can propose to integrate the management into the main program instead of abstracting it into a module. That works as well in exchange for app-specific design. job/worker design exchanges more codes for concurrency management benefits.


One more benefit: job/worker approach allows you to build “worker” farm, where you can manage the workers scaling without modifying all your main programs.

3 Likes

Thank you so much for your explanation. I will try it out in different way and let you know if I get any doubts.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.