I’m writing a high performance application which will get hit hard via the network and need to implement logging (to a text file) as well.
Is it a good idea to have a buffer with a separate goroutine slowly draining the buffer to a file, or is log.Println (to a file) already optimised enough to avoid this being necessary?
Looking at the implementation of log.Println, I see it just delegates to fmt.Sprintln and ultimately calls (*log.Logger).Output which just calls Write on the logger’s configured output here. There doesn’t seem to be any sort of optimization. If the underlying io.Writer implementation doesn’t buffer its writes, this could be a bottleneck if you’re doing a lot of logging. Note also that *log.Logger has a sync.Mutex to make sure writes are atomic, so only one log message can be written at a time.
You could try using (*log.Logger).SetOutput to set the logging output to a *bufio.Writer which should let most logging operations complete more quickly, however you have to make sure to call (*bufio.Writer).Flush so that all the log messages actually get flushed to the underlying writer. Maybe it has to go in a defer statement in case of a panic.
How do you intend to do this? You could possibly use (*log.Logger).SetOutput to set the output to, e.g. a *bytes.Buffer, but then how would you communicate to another goroutine to flush the buffer? Maybe you could use a wrapper around (*bytes.Buffer).Write that checks the buffer capacity and uses, e.g., a channel to signal to the writer goroutine to flush the buffer, but then you also would have to wait for the flush to complete. I don’t think that gets you much over the *bufio.Writer alternative other than allowing the last goroutine that filled the buffer to proceed while the writer goroutine actually flushes the buffer. However, the next call from any goroutine to any of the logging functions will block until that writer goroutine completes the flush.
If you have a lot of requests with a lot of log events per request, perhaps you could use separate *log.Logger instances with separate (perhaps buffered) outputs. It would result in separate files which may or may not be a deal-breaker, but the performance of writing out to the files wouldn’t be bottlenecked by one *log.Logger’s mutex or by the underlying writer.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.