Faster file renaming?

Hello.

<- Newbie in go…

I often need to rename files. Not just a few, but up to 100k at a time. This time the current extension must be changed into a new one…

This is the current code:

package main

import (
	"flag"
	"log"
	"os"
	"path/filepath"
	"strings"
)

var flagPath = flag.String("path", "", "path to traverse in search of jpg files")
var flagExt = flag.String("ext", "", "new file extension (dot must be included)")

func visit(path string, f os.FileInfo, err error) (e error) {
	dir := filepath.Dir(path)
	base := filepath.Base(path)
	ext := filepath.Ext(path)
	if ext != "" {
		newname := filepath.Join(dir, strings.Replace(base, ext, *flagExt, 1))
		os.Rename(path, newname)
	}
	return
}

func init() {
	flag.Parse()
}

func main() {
	if *flagPath == "" || *flagExt == "" || !strings.Contains(*flagExt, ".") {
		flag.Usage()
		return
	}
	log.Println("begin")
	filepath.Walk(*flagPath, visit)
	log.Println("end")
}

All files were in a ram disk in Windows (10, x64) in a single directory. Number of files = ~65k
It took 17 seconds to do the renaming

Are there any ways to make this process (even) faster?

Regards

Please include error handling before going any further. The unhandled errors are:

  • err argument to visit
  • error returned by os.Rename
  • error returned by filepath.Walk

https://pocketgophers.com/error-checking-while-prototyping/ describes my method for handling errors without having to think too much about them (until they are encountered). https://github.com/kisielk/errcheck is a tool to help find unhandled errors.

Now, onto you question.

Start by profiling your program. Profiling will help you know what part of your program is using the most time, memory, etc. You need to know what is happening to know what to improve or even if there are useful improvements you can do. https://blog.golang.org/profiling-go-programs gives an example of how to do so and the thinking required.

I have not profiled your code, but have made some guesses based on the principle that the fastest code is the code that does not exist.

strings.Replace searches the string to find where the extension is. The extension is always at the end, and since the extension is already known (it is stored in ext), newname := path[:len(path)-len(ext)] + *flagExt will give the same result without needing filepath.Dir, filepath.Base, filepath.Join, or strings.Replace.

Also, the code you posted does not do what it indicates it should do. The description for flagPath says it should be searching for “jpg files”, but if ext != "" actually checks for files with extensions.

1 Like

As mentioned, measure. But I would be quite surprised if the majority of the time was not spent in the os.Rename call - that is, waiting for the filesystem layer.

1 Like

Thanks guys!

@nathankerr
That’s quite a lot to process for me. Go is my first “high-level” language :wink:
Thanks for providing so much detailed information, it will take a while
to get into all of this.

Changing the newname := …
already trimmed down the processing time by about 2 seconds

“path to traverse in search of jpg files” -> An oversight…

@calmh
That’s probably the issue but I get to this once I know more about profiling…

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.