Error reading from external command fatal error all goroutines are asleep - deadlock

Getting Help
1

I want to write a mime/multipart message in Python to standard output and read that message in Golang using the mime/multipart package. This is just a learning exercise.

I tried simulating this example

I asked the same question question on StackOverflow : link

output.py

```
#!/usr/bin/env python2.7
import sys
s = "--foo\r\nFoo: one\r\n\r\nA section\r\n" +"--foo\r\nFoo: two\r\n\r\nAnd another\r\n" +"--foo--\r\n"
print s 


main.go

package main

    import (
        "fmt"
        "io"
        "io/ioutil"
        "log"
        "mime/multipart"
        "os"
        "os/exec"
        "sync"

        "github.com/pkg/errors"
    )

    func readCommand(cmdStdout io.ReadCloser, wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
        defer wg.Done()
        defer close(errc)
        defer close(resc)

        mr := multipart.NewReader(cmdStdout, "foo")

        for {
            part, err := mr.NextPart()
            if err != nil {
                if err == io.EOF {
                    fmt.Println("EOF")
                } else {
                    errc <- errors.Wrap(err, "failed to get next part")
                }

                return
            }

            slurp, err := ioutil.ReadAll(part)
            if err != nil {
                errc <- errors.Wrap(err, "failed to read part")
                return
            }

            resc <- slurp
        }
    }

    func main() {
        cmd := exec.Command("python", "output.py")
        cmd.Stderr = os.Stderr
        pr, err := cmd.StdoutPipe()
        if err != nil {
            log.Fatal(err)
        }

        var wg sync.WaitGroup
        wg.Add(1)

        resc := make(chan []byte)
        errc := make(chan error)
        go readCommand(pr, &wg, resc, errc)

        if err := cmd.Start(); err != nil {
            log.Fatal(err)
        }

        for {
            select {
            case err, ok := <-errc:
                if !ok {
                    errc = nil
                    break
                }

                if err != nil {
                    log.Fatal(errors.Wrap(err, "error from goroutine"))
                }

            case res, ok := <-resc:
                if !ok {
                    resc = nil
                    break
                }

                fmt.Printf("Part from goroutine: %q\n", res)
            }

            if errc == nil && resc == nil {
                break
            }
        }

        cmd.Wait()
        wg.Wait()
    }


It's giving me this error : error from goroutine: failed to get next part: multipart: NextPart: EOF exit status 1
Please help me understand this concept and make this sample program work.
2

You could try to put the logic for running the command and getting the output streams inside the go routine. It belongs there because you can’t really have two or more go routines reading from the same command output (not without strange effects at least)

3

I added exec.Command() inside the readCommand() function.

func readCommand(wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
    defer wg.Done()
    defer close(errc)
    defer close(resc)

    cmd := exec.Command("python", "output.py")
    cmd.Stderr = os.Stderr
    pr, err := cmd.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }
    if err := cmd.Start(); err != nil {
        log.Fatal(err)
    }
mr := multipart.NewReader(pr, "foo")

/ * other stuff */
cmd.Wait()
}

But it is till not working.

4

Hm. Does it make any difference if you don’t closes the channels? Maybe they get closed before the main go routine is able to read from them.

5

I am getting EOF error which is passed via channel , therefore channel are working fine. There is some other issue . I copy pasted the mime message from golang example , so i don’t think it is wrong.

can you try and do this in your PC.

6

I try tomorrow. Time for bed now.

7

I am on windows 10 , 64 bit , go version go1.11.1 windows/amd64 . I thought that the guy on stackoverflow must be using linux , so i live booted linux mint and tried my code there . To my surprise , it worked there,but i have to make it work on windows.
Golang is suppose to be cross platform ,so why this behavior ?

8

To recover Stdout and Stderr, using o ,e := cmd.CombinedOutput() seems sufficient as you wait for the command to end. Piping seems more complicated then needed.

9

can’t use CombinedOutput() because it will return me bytearray , which i can’t use to create io.Reader , hence can’t use it to get the message part by part using Nextpart.

10

Hi. Tested now and the problem is python. By default seems python to add an extra \r for each \n printed to stdout. Just to compensate for line-ending being \r\n instead of \n on windows. So the output from output.py is

“–foo\r\r\nFoo: one\r\r\n\r\r\nA section\r\r\n–foo\r\r\nFoo: two\r\r\n\r\r\nAnd another\r\r\n–foo–\r\r\n”

You must find a way to make python output the string in raw format or just use \n in the python program

#!/usr/bin/env python2.7
import sys
s = "--foo\nFoo: one\n\nA section\n" +"--foo\nFoo: two\n\nAnd another\n" +"--foo--\n"
print s
11

wow , thank you so much .It worked. You can’t imagine how happy I am. You are great !!
Thanks again

1 Like
12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.