Why http response packet blocked by exec.Command?

What version of Go are you using (go version)?

go version go1.5.1 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH=“amd64"
GOBIN=”“
GOEXE=”"
GOHOSTARCH=“amd64"
GOHOSTOS=“linux"
GOOS=“linux"
GOPATH=”/data/songtianyi/golang/own:/data/songtianyi/golang/go:/data/songtianyi/golang/own/src/dtf-manager/“
GORACE=”“
GOROOT=”/data/songtianyi/golang/go"
GOTOOLDIR=”/data/songtianyi/golang/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=”"
CC=“gcc"
GOGCCFLAGS=”-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED=“1”

What did you do?

package main

import (
	"fmt"
	"net/http"
	"os/exec"
)

func Hello(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("Hello!!"))

	out, err := exec.Command("sleep", "10").CombinedOutput()
	fmt.Println(string(out))
	if err != nil {
		fmt.Println(err)
		return
	}
}

func main() {
	http.HandleFunc("/hello", Hello)
	addr := "0.0.0.0:8083"
	if err := http.ListenAndServe(addr, nil); err != nil {
		fmt.Println(err)
		return
	}
}

steps to replay the wired “bug”

  1. build and run the code above
  2. execute the following command
time curl http://localhost:8083/hello

What did you expect to see?

server response “Hello!!” and curl exit immediately

What did you see instead?

server response after exec exit!! wired!

Can’t be reproduced.
It works as it should be.
The test phrase displayed after delay

go version go1.8 linux/amd64

GOARCH=“amd64"
GOBIN=”“
GOEXE=”"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GCCGO=“gccgo"
CC=“gcc"
GOGCCFLAGS=”-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build056269949=/tmp/go-build -gno-record-gcc-switches"
CXX=“g++“
CGO_ENABLED=“1"
PKG_CONFIG=“pkg-config"
CGO_CFLAGS=”-g -O2"
CGO_CPPFLAGS=”“
CGO_CXXFLAGS=”-g -O2"
CGO_FFLAGS=”-g -O2"
CGO_LDFLAGS=”-g -O2”

I was able to reproduce (go version go1.8 darwin/amd64). There are two things going on.

  1. http.ResponseWriter waits for enough data to be written before flushing to the client. This can be triggered manually with http.Flusher.
  2. The first write will trigger ResponseWriter.WriteHeader, which uses the first 512 bytes to figure out the Content-Type.

I have been able to get an immediate response with:

func Hello(w http.ResponseWriter, r *http.Request) {
	text := [512]byte{}
	for i := range text {
		text[i] = 'b'
	}
	text[511] = 'c'
	w.Write(text[:])

	w.(http.Flusher).Flush()

	out, err := exec.Command("sleep", "10").CombinedOutput()
	fmt.Println(string(out))
	if err != nil {
		fmt.Println(err)
		return
	}
}

text is designed to be noticeable along with its ending.

The trick seems to be:

  1. Write at least 512 bytes of data (511 does not give immediate display in the browser)
  2. Flush the ResponseWriter after enough data has been written.

I tried circumventing the content type checker by setting the Content-Type header and calling w.WriteHeader() before writing “Hello!!”, but still encountered the delay. This was done both with and without flushing w.

1 Like

Would the data be flushed to the client before the goroutine exit when the response size is less than 512 byte?
What make me confused is why go runtime didn’t do this before exec?

@dfc
Hi,
Would you mind explaining this?

the terminal command would still hang up for 10s even we flush http response manually.

Hey @songtianyi,

This is occurring because you are running a command and using combined output which waits for the command to be completed before continuing, so it’s a blocking call.

I haven’t really tested this and it might not be the best thing to do, but maybe you want something similar to this example which pipes any output from the running command and also doesn’t hold up the terminal.

func Hello(w http.ResponseWriter, r *http.Request) {
	// Create a new command.
	c := exec.Command("ls", "-la")

	stdoutPipe, err := c.StdoutPipe()
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Start the command.
	if err := c.Start(); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Copy all output from the running command to stdout.
	go func() {
		io.Copy(os.Stdout, stdoutPipe)

		if err := c.Wait(); err != nil {
			panic(err)
		}
	}()

	// Do stuff while the command is running.
	fmt.Println("doing stuff")
}

In that example I’m only using an ls command so you can see how it prints the output, but you can try it with a sleep command and see that it shouldn’t be blocking.

Please show some code.

FWIW, running an external program is unrelated.

func Hello(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello!!"))
        time.Sleep(10 * time.Second)
}

Will probably exhibit the same behaviour.

Conclusion:
It’s not about go scheduler, it’s just http server behavior.
Flush will be called after Handler function returned or called by user manually.

1 Like

After reading your responses and the original post again, I think I misunderstood what you wanted.

You expect running:

time curl http://localhost:8083/hello

to result in:

Hello!!curl http://localhost:8083/hello  0.01s user 0.01s system 39% cpu 0.056 total

but you get:

Hello!!curl http://localhost:8083/hello  0.01s user 0.01s system 0% cpu 10.066 total

The difference being the total time.

This can be accomplished by running the command in a goroutine:

func Hello(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("Hello!!"))

	go func() {
		out, err := exec.Command("sleep", "10").CombinedOutput()
		fmt.Println(string(out))
		if err != nil {
			fmt.Println(err)
			return
		}
	}()
}

The command is started when the http request is received. Using a goroutine like this runs the command in the background so Hello does not block until the command completes.

I think this is the behavior you desired. Please tell me if I am wrong.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.