Using Go routines to download multiple files using FTP

Getting Help
1

I am trying to download multiple files concurrently using the github.com/jlaffaye/ftp library.

Non concurrently (works just fine)

func main() {
	c, err := ftp.Dial(remoteHost, ftp.DialWithTimeout(5*time.Second))
	if err != nil {
		fmt.Println(err)
	}
	defer c.Quit()
	
	err = c.Login(loginID, loginPasw)
	if err != nil {
		fmt.Println(err)
	}
	
	entries, err := c.List(remoteDir)
	if err != nil {
		fmt.Println(err)
	}
	
	for _, entry := range entries {
	
		res, err := c.Retr(remoteDir + entry.Name)
		if err != nil {
			fmt.Println(err)
		}
	
		outFile, err := os.Create(localDir + entry.Name)
		if err != nil {
			fmt.Println(err)
		}
		defer outFile.Close()
	
		_, err = io.Copy(outFile, res)
		if err != nil {
			fmt.Println(err)
		}
		res.Close()
	}
}

Attempt to run the downloads concurrently

func main() {
	c, err := ftp.Dial(remoteHost, ftp.DialWithTimeout(5*time.Second))
	if err != nil {
		fmt.Println(err)
	}
	defer c.Quit()

	err = c.Login(loginID, loginPasw)
	if err != nil {
		fmt.Println(err)
	}

	entries, err := c.List(remoteDir)
	if err != nil {
		fmt.Println(err)
	}

	var wg sync.WaitGroup

	for _, entry := range entries {

		wg.Add(1)
		go func(entry *ftp.Entry) {
			defer wg.Done()

			res, err := c.Retr(remoteDir + entry.Name)
			if err != nil {
				fmt.Println(entry.Name)
				fmt.Println(err)
			}
			defer res.Close()

			outFile, err := os.Create(localDir + entry.Name)
			if err != nil {
				fmt.Println(err)
			}
			defer outFile.Close()

			_, err = io.Copy(outFile, res)
			if err != nil {
				fmt.Println(err)
			}

		}(entry)

	}
	wg.Wait()
}

Segfault

panic: runtime error: invalid memory address or nil pointer dereference
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5bca98]

goroutine 22 [running]:
github.com/jlaffaye/ftp.(*Response).Close(0xf?)
	/home/jeanluc/golang/src/github.com/jlaffaye/ftp/ftp.go:1134 +0x18
panic({0x5dcdc0, 0x77a340})
	/usr/lib/go-1.19/src/runtime/panic.go:884 +0x212
github.com/jlaffaye/ftp.(*Response).Read(0x10?, {0xc00016c000?, 0xc0000169c0?, 0xc000054c00?})
	/home/jeanluc/golang/src/github.com/jlaffaye/ftp/ftp.go:1128 +0x19
io.copyBuffer({0x668100, 0xc0000169c0}, {0x6679e0, 0x0}, {0x0, 0x0, 0x0})
	/usr/lib/go-1.19/src/io/io.go:427 +0x1b2
io.Copy(...)
	/usr/lib/go-1.19/src/io/io.go:386
os.genericReadFrom(0xc000063e38?, {0x6679e0, 0x0})
	/usr/lib/go-1.19/src/os/file.go:162 +0x67
os.(*File).ReadFrom(0xc000014060, {0x6679e0, 0x0})
	/usr/lib/go-1.19/src/os/file.go:156 +0x1b0
io.copyBuffer({0x667c00, 0xc000014060}, {0x6679e0, 0x0}, {0x0, 0x0, 0x0})
	/usr/lib/go-1.19/src/io/io.go:413 +0x14b
io.Copy(...)
	/usr/lib/go-1.19/src/io/io.go:386
main.main.func1(0xc0000a0320)
	/home/jeanluc/golang/src/jeanluc/myftp2/myftp2.go:60 +0x28a
created by main.main
	/home/jeanluc/golang/src/jeanluc/myftp2/myftp2.go:44 +0x1ff
exit status 2

Line 44 : go func(entry *ftp.Entry) {
Line 60: _, err = io.Copy(outFile, res)

When I keep only the file creation part, it works and empty files are created in the localDir

	for _, entry := range entries {

		wg.Add(1)
		go func(entry *ftp.Entry) {
			defer wg.Done()

			outFile, err := os.Create(localDir + entry.Name)
			if err != nil {
				fmt.Println(err)
			}
			defer outFile.Close()

		}(entry)

	}
	wg.Wait()
}

I have used the go func() literal functions in the past and never had such a problem. Since I can’t fully grasp the concept of Go channels, I prefer using Go routines as I’ve attempted to do above.

Is the FTP connection non-shareable?

Any ideas?

2

Looks like it is not safe to use the same connection.
Dial() returns a ServerConn

ServerConn represents the connection to a remote FTP server. A single connection only supports one in-flight data connection. It is not safe to be called concurrently.

1 Like
3

You can also check standard sftp package here. It supports concurrent reads and writes in chunks.

4

Unfortunately, the protocol I have to use is FTPS, the TLS version of SFTP.

5

Indeed. The ftp.Dial function has to be in the go routines. One ServerConn per routine.

func main() {

	c, err := ftp.Dial(remoteHost, ftp.DialWithTimeout(5*time.Second))
	if err != nil {
		fmt.Println(err)
	}
	defer c.Quit()

	err = c.Login(loginID, loginPasw)
	if err != nil {
		fmt.Println(err)
	}

	entries, err := c.List(remoteDir)
	if err != nil {
		fmt.Println(err)
	}

	var wg sync.WaitGroup
	for _, entry := range entries {

		wg.Add(1)
		go func(entry *ftp.Entry) {
			defer wg.Done()

			c, err := ftp.Dial(remoteHost, ftp.DialWithTimeout(5*time.Second))
			if err != nil {
				fmt.Println(err)
			}
			defer c.Quit()

			err = c.Login(loginID, loginPasw)
			if err != nil {
				fmt.Println(err)
			}

			res, err := c.Retr(remoteDir + entry.Name)
			if err != nil {
				fmt.Println(entry.Name)
				fmt.Println(err)
			}
			defer res.Close()

			outFile, err := os.Create(localDir + entry.Name)
			if err != nil {
				fmt.Println(err)
			}
			defer outFile.Close()

			_, err = io.Copy(outFile, res)
			if err != nil {
				fmt.Println(err)
			}
		}(entry)

	}
	wg.Wait()
}
``
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.