`bufio.Scanner: token too long` for reading large 10GB file

Getting Help
1

While reading large 10GB text file where each of the lines are long nested JSON.
I am reading the content using the following

scanner := bufio.NewScanner(file)
scanner.Scan()

bufio.Scanner: token too long

Need help to resolve this or, any alternative snippet.

2

Can you show us the line that produces this error?

3

You could try bufio.NewReader(file) and then ReadString("/n")

4

Hi

Use https://golang.org/pkg/bufio/#Scanner.Buffer to set a large enough buffer for reading.

1 Like
5

Thanks but it required more changes in the code this way then just changing the buffer limit. :sunny: :shushing_face:

6

Thanks. I did the same. :sunny: :shushing_face:

1 Like
7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.