Code explanation

Getting Help
1

I was going over the source code of azcopy and I cannot figure out how the “os.Stdin” part of the below code is working. Is anyone kind enough to explain?

    func (cca *cookedCopyCmdArgs) processRedirectionUpload(blobResource common.ResourceString, blockSize int64) error {
	ctx := context.WithValue(context.TODO(), ste.ServiceAPIVersionOverride, ste.DefaultServiceApiVersion)

	// if no block size is set, then use default value
	if blockSize == 0 {
		blockSize = pipingDefaultBlockSize
	}

	// step 0: initialize pipeline
	p, err := createBlobPipeline(ctx, common.CredentialInfo{CredentialType: common.ECredentialType.Anonymous()})
	if err != nil {
		return err
	}

	// step 1: parse destination url
	u, err := blobResource.FullURL()
	if err != nil {
		return fmt.Errorf("fatal: cannot parse destination blob URL due to error: %s", err.Error())
	}

	// step 2: leverage high-level call in Blob SDK to upload stdin in parallel
	blockBlobUrl := azblob.NewBlockBlobURL(*u, p)
	_, err = azblob.UploadStreamToBlockBlob(ctx, os.Stdin, blockBlobUrl, azblob.UploadStreamToBlockBlobOptions{
		BufferSize: int(blockSize),
		MaxBuffers: pipingUploadParallelism,
	})

	return err
}

The source code is at: https://github.com/Azure/azure-storage-azcopy/blob/master/cmd/copy.go

2

azblob.UploadStreamToBlockBlob takes an io.Reader interface and reads it until the end, all the while uploading it to a blob.

os.Stdin implements the io.Reader interface so can be used as input to this function.

There are lots of things which implement this interface, eg bytes.Buffer and it is one of the main ways of doing io in Go.

I hope that was what you wanted to know, if not let me know!

3

@ncw
Nick, thank you for your answer. Seems that my question was too ambiguous. I have used azblob.UploadStreamToBlockBlob function myself, what I do not understand is how are the data bytes from the file sent to the Stdin pipe to be picked up by the function.

Example
azcopy copy "C:\somefile.txt" "https://account.blob.core.windows.net/mycontainer

4 
	// step 2: leverage high-level call in Blob SDK to upload stdin in parallel
	blockBlobUrl := azblob.NewBlockBlobURL(*u, p)
	_, err = azblob.UploadStreamToBlockBlob(ctx, os.Stdin, blockBlobUrl, azblob.UploadStreamToBlockBlobOptions{
		BufferSize: int(blockSize),
		MaxBuffers: pipingUploadParallelism,
	})

In ths step, note how the os.Stdin is passed to azblob.UploadStreamToBlockBlob

os.Stdin implements an io.Reader which reads the bytes from standard input.

5

processRedirectionUpload is a wrapper for UploadStreamToBlockBlob. Why do you expect your example to execute processRedirectionUpload? You offer no proof that it does. For your example, is cca.isRedirection() true?

6

Good point :slight_smile: It was the only implementation of azblob.Upload that I could find, but they could use bloburl methods (I am remotely now and cannot check).
But the question remains, how could someone get byte data from a file into the stdin? Or does it need to be piped in the command line?

7 
$ cat stdin.go
package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    s := bufio.NewScanner(os.Stdin)
    for s.Scan() {
	    b := s.Bytes()
	    fmt.Println(string(b))
    }
    if err := s.Err(); err != nil {
	    fmt.Println(err)
    }
}

$ cat stdin.file
This is data from a file.

.

$ go build stdin.go

$ ./stdin < stdin.file
This is data from a file.

$ cat stdin.file | ./stdin
This is data from a file.
1 Like
8

Excellent, thank you!

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.