exec.Cmd Stdin pipe vs file

Getting Help
1

Hello Gophers,

I noticed some unexpected behavior when feeding a buffer to an exec.Command.

There is a tool to create image thumbnails, called “vipsthumbnail”, that I am using in a pipeline architecture like this:

vipsthumbnail stdin -o thumbnail.webp < original.jpeg

It creates the thumbnail as expected.

The go equivalent also works the same way:

original, err := os.Open("original.jpeg")
if err != nil {
	log.Fatal(err)
}

cmd := exec.Command("vipsthumbnail", "stdin", "-o", "thumbnail.webp")
cmd.Stdin = original
if err := cmd.Run(); err != nil {
	log.Fatal(err)
}

However, using a buffer does not work:

original, err := os.ReadFile("original.jpeg")
if err != nil {
	log.Fatal(err)
}

cmd := exec.Command("vipsthumbnail", "stdin", "-o", "thumbnail.webp")
cmd.Stdin = bytes.NewReader(original)
if err := cmd.Run(); err != nil {
	log.Fatal(err)
}

Using a buffer, the vipsthumbnail tool fails with some generic “unable to read from stdin” message.

From the Go docs we get:

If Stdin is an *os.File, the process’s standard input is connected directly to that file.

Otherwise, during the execution of the command a separate goroutine reads from Stdin and delivers that data to the command over a pipe.

Is this an implementation error in the vipsthumbnail software not beeing able to handle pipes properly? How does it make a difference for that software at all? Is there anything I can do in my Go code?

Thanks for all the help!

1 Like
2

Did you try with the golang package?

3

I tried to use vipsthumbnail as an example for better understanding the problem.

This issue is about the differences of passing a buffer vs using a file handle for the exec.Command. In my understanding, using a buffer should not affect the behavior of the command call.

1 Like
4

My initial thought was that this has something to do with vipsthumbnail itself. I don’t have vipsthumbnail, so I tried to apply your thinking/coding to grep, and it looks like cmd.Exec does what you would expect.

I get the following:

With file:         "Hello, 世界\n"
With bytes reader: "Hello, 世界\n"

when I run this, which I tried to keep with the intent of your code/question:

func main() {
	f, _ := os.Create("input.txt")
	f.WriteString("Hello, 世界")
	f.Close()

	withFile()
	withStdin()
}

func run(r io.Reader) []byte {
	cmd := exec.Command("grep", "-i", "hello")
	cmd.Stdin = r
	out, err := cmd.Output()
	if err != nil {
		panic(err)
	}
	return out
}

func withFile() {
	f, _ := os.Open("input.txt")
	fmt.Printf("With file:         %q\n", run(f))
}

func withStdin() {
	b, _ := os.ReadFile("input.txt")
	fmt.Printf("With bytes reader: %q\n", run(bytes.NewReader(b)))
}

Have you tried any other command-line program?

5

Thanks for testing this! I was expecting it to work as you said.

With some more testing, I can confirm that it is a problem with vipsthumbnail, and it also fails on windows only.

This works on both Windows and Linux:

vipsthumbnail stdin -o thumbnail.webp < "image.jpg"

This fails on windows (on Windows use type instead of cat) cmd:

cat "image.jpg" | vipsthumbnail stdin -o thumbnail.webp

The same fails in Windows PowerShell:

Get-Content -Path "image.jpg" -AsByteStream | vipsthumbnail stdin -o thumbnail.webp

Thank you for testing @Zachary_Young.

As this is clearly a vipsthumbnail issue, I will close this topic now.

1 Like