The loop with one go-exiftool instance hangs on big amount of files

Getting Help
1

hi everybody,

I’m looping big ~10k files using go-exiftool. I’m using one instance of the go-exiftool to get info for all required files. This code is called 10k times in the loop, where the file is always different.

fileInfos := et.ExtractMetadata(file)

After the ~7k loops the program hangs. I debugged go-exiftool and found that it hangs in https://github.com/barasher/go-exiftool/blob/master/exiftool.go#L121 on the line:
fmt.Fprintln(io.WriteCloser, "-execute")
if i understood correct io.WriteCloser has instance of exec.Command(binary, initArgs...).StdinPipe()

i tried to run the same file in 10k loops. Works fine. It looks like it runs out of memory, can it be? I see no problem in the system memory graph. Or stdin is overflowed. Have no idea how to check that.

another case: if i ignore the last broken file reading, the it stops on the next one.

and more over it works perfekt in debug mode.

so, the questions are:

  1. Does exec.Command has a limit of execution?
  2. If 1) - not, what can be the reason else?
  3. Does it depends on the file sizes? I tried another folder and it worked with 35k files and then hanged. How to check that?
2

Do you read the files or is the othe tool reading them after you tell it a name?

Is that program closing the files?

If you are on Linux you can use lsof to check when the program hangs.

Grep the list for your own binary, as well as the external tool you are using.

3

if i understand correct, this code does writing file to the writer?
fmt.Fprintln(io.WriteCloser, filePath)

and io.WriteCloser is closed only then all 10k files are looped.

4

Seems the problem comes from exiftool indeed. It hangs for reason. I’ll do workaround for that. thx for the tips!

5

That just writes a filename to the stub of the exiftool, which then opens that file, but it seems to not close it, have you checked lsof output as I suggested?

6

i tried to use alternative to lsof for macos. I could not understand what i have to see there. It stopping working pretty fast with using it.

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.