Dealing with large .go files & slices

[Rob_S (Rob)]

I have a project where I need to get a large number of data into a slice.

The data would be IMDB type graph db. Movies and actors are nodes. Movie<->Actor would be an edge.

The data would be read-only. No writing required.

I realize the best way to do this would just be to use a database and then read them into the go app at launch. However, I’d really like to do all of this in a Google App Engine type scenario and not require a server or something to house the DB.

My questions are:

• How do you all deal with large files? I find VS Code locks up if I have HUGE golang files.
• Are there any Go native databases that you’d recommend? I found Dagger. But looks like I’d still need to read the data into it. It is 100% memory resident.
• Really any advice from your experience, reading large amounts of data into a slice would be valued.

I realize this is a strange question that makes me sound like a noob programmer. I’m just trying to think outside the box here to keep a project as inexpensive to host, with very responsive searches, as possible.

Thank you!

[christophberger]

Hi @Rob_S,

There is no recommended size of Go source files as far as I am aware, but when you say they are huge’, then maybe you’d want to break them in to smaller pieces, and if it is only for making VSCode happy

VSCode is indeed not exactly known for high performance and CPU efficiency – after all, it is built on Electron.

If you are familiar with using vi, you could also switch to (Neo-)Vim. I daresay both of them are happy with huge files. I once edited large log files with Vim, and it works when highlighting is off.

(Speaking of highlighting, try switching that off in VSCode and see if this helps for editing huge files. I don’t say this is a solution but I’d bi curious if highlighting is the culprit)

Go native databases: CockroachDB comes to mind, a distributed SQL database with Postgres wire protocol. And there is a pure Go port (or rather, a transpile) of SQLite - modernc/sqlite. And I guess there is a good number of non-SQL DB’s available also, from KV stores like bolt or bbolt to more sophisticated ones (Dgraph etc).

Apologies for not including links but I write this on my mobile. Maybe awesome-go.com has some more useful DB links.

[Hultan (Per Hultqvist)]

And while we are talking editors/IDE:s, if you are serious about programming Go, my recommendation is to take a look at Goland (GoLand: A Clever IDE to Go by JetBrains). It is not free, but it is awesome, and it is just a guess, but it probably would handle large files much better than VSCode.

I do not know how large files we are talking about, but I just tested a 2MB (~5700 lines) file and it opened instantly.

Can unfortunately not help you with the DB questions…

[zekro (Ringo Hoffmann)]

When dealing with such large static data, you can put it inside a JSON file and then serialize it using the encoding/json package. Though, this would require the program to serialize the database every time again on startup, but depending on the number of entries, this should not take that long.

Here is an example how to serialize a JSON file containing some data objects.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"
	"time"
)

const dataFile = "data.json"

type Movie struct {
	Title       string    `json:"title"`
	Actors      []string  `json:"actors"`
	ReleaseDate time.Time `json:"release_date"`
}

func main() {
	f, err := os.Open(dataFile)
	if err != nil {
		log.Fatal(err)
	}

	var movies []*Movie
	err = json.NewDecoder(f).Decode(&movies)
	if err != nil {
		log.Fatal(err)
	}

	for _, m := range movies {
		fmt.Printf("%+v\n", m)
	}
}

This also has the advantage that you can drop in another data source later if you want to read your JSON from a REST API or something like that, for example.

I hope this was somewhat helpful to you.

[Rob_S (Rob)]

Thanks for the reply.

I started out with Json. But that was just a different large file that was blowing up vs code. That’s why I figured just use a struct because that’s how it will end up anyway. Get rid of that overhead.