Database/sql - query rows without sql cursor?

Hello Go community,

I am not sure if I understand it correctly. Reading the documentation about the database/sql module from Go suggests that DB.Query returns Rows and Rows is using the sql cursor under the hood. My program recently fired a sql query that should have returned several GB of rows but my memory consumption was very low, which matches the documentation.

For a lot of rows (~1e8) I find my program waiting for the next row a lot, having only 40-50 % of CPU usage while iterating over the rows. Is it possible to get batches of rows or the whole dataset at once without using the row by row cursor? I could use this to reduce the network and database overhead while iterating over the results.

Thanks a lot!

Carl

Yes and No.

You can aggregate the the query and send one row containing all rows. But I doubt that this makes sense to move the problem to the server:

https://play.golang.org/p/kmu1l75u2N4
http://94.237.92.101:5053/

1 Like

Thank you for the suggestions. You’re right, JSON encoding for this large amount of data wouldn’t make much sense for my application.

Using two goroutines each working on half of the data might be the best way.