`bufio.Scanner: token too long` for reading large 10GB file

While reading large 10GB text file where each of the lines are long nested JSON.
I am reading the content using the following

scanner := bufio.NewScanner(file)

bufio.Scanner: token too long

Need help to resolve this or, any alternative snippet.

Can you show us the line that produces this error?

You could try bufio.NewReader(file) and then ReadString("/n")


Use https://golang.org/pkg/bufio/#Scanner.Buffer to set a large enough buffer for reading.

1 Like

Thanks but it required more changes in the code this way then just changing the buffer limit. :sunny: :shushing_face:

Thanks. I did the same. :sunny: :shushing_face:

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.