While reading large 10GB text file where each of the lines are long nested JSON.
I am reading the content using the following
scanner := bufio.NewScanner(file)
bufio.Scanner: token too long
Need help to resolve this or, any alternative snippet.
Can you show us the line that produces this error?
You could try bufio.NewReader(file) and then ReadString("/n")
Use https://golang.org/pkg/bufio/#Scanner.Buffer to set a large enough buffer for reading.
Thanks but it required more changes in the code this way then just changing the buffer limit.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.