Hey guys,
Excited to share my project, gojob! It’s a simple Task Scheduler library for GoLang designed for task sharding, retry on failure, logging, and status reporting.
Whether you’re building a web app, a microservice, or a data processing pipeline, gojob makes task management a breeze.
I’m all ears for feedback on design, functionality, or usability. Professionals’ insights will help refine this project and my skills to meet industry standards.
Features:
- Concurrent Task Execution: Handles multiple tasks concurrently.
- Logging Support: Tracks task execution and errors seamlessly.
- Status Reporting: Real-time monitoring with customizable options.
- Ease of Use: Simple API and intuitive design for easy integration.
- Monitoring Integration: Works smoothly with Prometheus, Grafana, and more.
Check out this quick example (a concurrent HTTP crawler) to see how simple gojob can used:
Basically you just need to implement the Task interface and submit it to the scheduler. Here’s an example of a concurrent http crawler using gojob:
// Task is an interface that defines a task
type Task interface {
// Do starts the task, returns error if failed
// If an error is returned, the task will be retried until MaxRetries
// You can set MaxRetries by calling SetMaxRetries on the scheduler
Do() error
}
package main
import (
"fmt"
"time"
"github.com/WangYihang/gojob"
)
type MyTask struct {
Url string `json:"url"`
StatusCode int `json:"status_code"`
}
func New(url string) *MyTask {
return &MyTask{
Url: url,
}
}
func (t *MyTask) Do() error {
response, err := http.Get(t.Url)
if err != nil {
return err
}
t.StatusCode = response.StatusCode
defer response.Body.Close()
return nil
}
func main() {
var numTotalTasks int64 = 256
scheduler := gojob.New(
gojob.WithNumWorkers(8),
gojob.WithMaxRetries(4),
gojob.WithMaxRuntimePerTaskSeconds(16),
gojob.WithNumShards(4),
gojob.WithShard(0),
gojob.WithTotalTasks(numTotalTasks),
gojob.WithStatusFilePath("status.json"),
gojob.WithResultFilePath("result.json"),
gojob.WithMetadataFilePath("metadata.json"),
).
Start()
for i := range numTotalTasks {
scheduler.Submit(New(fmt.Sprintf("https://httpbin.org/task/%d", i)))
}
scheduler.Wait()
}
Find more info, examples (e.g. concurrent tcp port scanner), and the source code on GitHub. Your feedback and suggestions are welcome. Let’s make coding more enjoyable together!