Faster way to redirect console logs to a file

Getting Help
1

I am working on an application which uses a C library using cgo binding. The library by default dumps logs on console.

To redirect the logs from console, we have used the following code

file, err := os.OpenFile(logFile, os.O_WRONLY|os.O_CREATE|os.O_SYNC, 0644)
if err != nil {
return errors.New(“error creating log file, using stderr”)
}
syscall.Dup2(int(file.Fd()), 1)
syscall.Dup2(int(file.Fd()), 2)

This works well but slows down the whole program considerably. If the logs are huge, the program execution is delayed.
If I run this code from RAM disk(RAM disk I/O is much faster than disk I/O), things are good. but if logs are huge, having all the logs on RAM will not make much sense given that RAM disk has limited space.
For the code that we have written in go, we have used logrus for logging which works well.

Can someone please suggest a faster way to redirect the logs from console to a file. I can’t change anything in the C library.

1 Like
2

One update here. Tried Dup3 variant. That is even more slower.

3

You say that the C library “by default” dumps logs to the console. I understand that you cannot modify the C library, but is there any way to configure it to dump to a file directly?

I’m curious if the slowdown is because of Go or if it’s perhaps because dumping to disk is just slow. If you don’t do anything in your code to handle the redirection from stdout or stderr to disk and then redirect from your console, does it still slow down or is it fine?

4

No the library doesn’t support writing to a file.

Further about the redirection, if i disable above two lines, the speed is increased.

I am yet to check the behavior when “the redirection from stdout or stderr to disk and then redirect from your console”.

5

Stop asking for writes to be really, really, really slow. Don’t O_SYNC.

$ man open

O_SYNC

Write operations on the file will complete according to the  re‐
quirements  of  synchronized  I/O  file integrity completion (by
contrast with the synchronized  I/O  data  integrity  completion
provided by O_DSYNC.)

By  the  time write(2) (or similar) returns, the output data and
associated file metadata have been transferred to the underlying
hardware  (i.e.,  as though each write(2) was followed by a call
to fsync(2)).
1 Like
6

Thanks @petrus. That solves my problem.

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.