How to control the concurrency in http handlers

[ep4sh (Pasha Radchenko)]

Hey folks, I definitely need a help - looks like I have collisions in my mind

I have HTTP server written in go (via std lib), and one handler is for metadata generation.

func handler(w http.ResponseWriter, r *http.Request) {
    path, ok := r.URL.Query()["path"]

    // ....
    // checks and 
    // other stuff

    err := BuildMetadata(path)

    // ....
    // checks and 
    // other stuff
}

Each client can trigger metadata generation, but I have to be sure, that the only 1 process of metadata generation is running at the same time for then particular path, and it could be multiple for different paths.

My BuildMetadata() signature looks like the following:

func BuildMetadata(path string) error {

	tasks := make(chan string, 1)
	done := make(chan struct{})
	errors := make(chan error, 1)
	tasks <- path


	go func(tasks chan string, done chan struct{}, errors chan error) {
		fp := <-tasks
		log.Printf("New task for %s metadata has been created\n", fp)

        // the less interesting happens here
		if err := rebuildMeta(fp); err != nil {
			errors <- err
		}
		done <- struct{}{}
	}(tasks, done, errors)

	select {
	case err := <-errors:
		return err
	case <-done:
		log.Printf("Metadata has been created\n")
		return nil
	}
}

I’m not sure that I’ve satisfied my own requirement (only a single metadata generation is possible for the particular path)

AFAIK, each handle is running in its own goroutine:
https://go.dev/src/net/http/server.go#L3071

I’m afraid of I’m doing wrong with concurrency - when BuildMetadata will be performed, a bunch of new channels will be created for each goroutine, independently of each other, isn’t it?

How do can I control such case and not reinvent the wheel (most important for me as for Go learner)

I appreciate any suggestion, link, blame - please help me to understand Go.

Thank you,
Pash

[mje (Jeff Emanuel)]

Build a map of paths to task chans. Write to the chan based on the path. I think you want unbuffered chans.

[ep4sh (Pasha Radchenko)]

Thank you, Jeff, for your reply, really appreciate it.

I hope I got the idea, smth like that:

type Task map[string]struct{}

func BuildMetadata(path string) error {

	tasks := make(chan Task)
	done := make(chan struct{})
	errors := make(chan error, 1)

	go func(tasks chan Task, done chan struct{}, errors chan error) {
		for fp := range <-tasks {
			log.Printf("New task for %s metadata has been created\n", fp)

			if err := f.rebuildMetadata(p); err != nil {
				errors <- err
			}
			done <- struct{}{}
		}
	}(tasks, done, errors)

	t := Task{path: struct{}{}}
	tasks <- t

	select {
	case err := <-errors:
		return err
	case <-done:
		log.Printf("Metadata has been created\n")
		return nil
	}
}

}

Is it enough good code to be accepted? (not sure) =)

[mje (Jeff Emanuel)]

By creating a new chan and goroutine for each request, you are not enforcing any mutual exclusion.

I would do something like this, albeit untested:

type Task struct{
	resultChan chan Result
	errorChan chan error
	taskParameters interface{}
}

var chansPerPath map[string]chan Task = make(map[string]chan Task)

func addPath(path string) {
	ch, ok := chansPerPath[path]
	if !ok {
		pathChan = make(chan Task)
		resultChan = make(chan Task)
		chansPerPath[ch] = ch
		go func(taskChan chanTask) {
			for t:= range taskChan {
				result,err := performTask(t.taskParameters)
				if err!=nil {
					t.errorChan <- err
				} else {
					t.resultChan <- result
				}
			}
		}
	}
}

// Presumes this runs concurrently for each request.
func handlePath(path string) {
	addPath(path)

	task := Task{
		resultChan: make(chan Result)
		errorChan: make(chan error)
		taskParameters: getTaskParameters(path)
	}

	// No more than one task per path can be handled.
	chansPerPath[path] <- task
	select {
		case result := <-task.resultChan:
			// Do something with result
		case err := <-task.errChan:
			// Do something with result
	}

}