My program uses lots of RAM

Hello there.

I’m trying to build a small http API which received a file and perform some operations on it. I can successfully receive the base64 of the file, decode it and it to disk, but for some reason, my program does not free up memory afterwards.

Here is the complete code of the server:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"encoding/base64"
	"os"
	"io/ioutil"
	"path"
	"os/exec"
	"fmt"
)

const progName =  "dummy"

type Request struct {
	FileBase64 string `json:"base64,omitempty"`
	FileName string `json:"filename,omitempty"`
	Auth    string `json:"auth,omitempty"`
}

type Reply struct {
	Success bool
	Status  string
}


func main() {
	router := mux.NewRouter()
	router.HandleFunc("/put", processFile).Methods("POST")
	log.Fatal(http.ListenAndServe("localhost:8000", router))
}

func processFile(w http.ResponseWriter, r *http.Request) {
	var cmd Request
	_ = json.NewDecoder(r.Body).Decode(&cmd)
	log.Printf("request received")

	// decode incoming data
	dec, err := base64.StdEncoding.DecodeString(cmd.FileBase64)
	if err != nil {
		log.Println(r.RemoteAddr, "-", "cannot decode base64:", err)
		reply := Reply{false, "cannot decode base64"}
		json.NewEncoder(w).Encode(reply)
		return
	}
	log.Printf("request decoded")

	// create temp directory
	// we need it to dump the input file and run AppImage on it.
	tempDir, err := ioutil.TempDir("", progName+"-")
	if err != nil {
		log.Println(r.RemoteAddr, "-", "cannot create temp directory:", err)
		reply := Reply{false, "cannot create temp directory"}
		json.NewEncoder(w).Encode(reply)
		return
	}
	defer os.RemoveAll(tempDir)
	log.Printf("created temp dir")

	// create output file
	zipFile, err := os.Create(path.Join(tempDir, cmd.FileName))
	if err != nil {
		log.Println(r.RemoteAddr, "-", "cannot create output file:", err)
		reply := Reply{false, "cannot create output file"}
		json.NewEncoder(w).Encode(reply)
		return
	}
	defer zipFile.Close()
	log.Printf("output file created")

	// write content
	written, err := zipFile.Write(dec)
	if err != nil {
		log.Println(r.RemoteAddr, "-", "cannot write output file:", err)
		reply := Reply{false, "cannot write output file"}
		json.NewEncoder(w).Encode(reply)
		return
	}
	log.Printf("content written")

	// flush buffer
	err = zipFile.Sync()
	if err != nil {
		log.Println(r.RemoteAddr, "-", "error while flushing buffer:", err)
		reply := Reply{false, "error while flushing buffer"}
		json.NewEncoder(w).Encode(reply)
		return
	}
	log.Printf("buffer flushed")

	reply := Reply{true, fmt.Sprintf("written %d bytes", written)}
	json.NewEncoder(w).Encode(reply)
}

To send a file to the api, I use this command:

(echo -n '{"filename": "test.zip", "base64": "'; base64 500M.bin; echo '"}') | curl -H "Content-Type: application/json" -d @-  http://127.0.0.1:8000/put

When testing with a 500MB file, the server component uses lots of RAM but more importantly, it does not releases it when the function processFile exits.

Any idea on what I’m doing wrong?

Thanks a lot
Matteo

you are missing a call to r.Body.Close() in processFile.

actually, make it defer r.Body.Close() :slight_smile:

also: you’re not checking your errors. you shouldn’t do that. please check all your errors… (your future-self will thank you)

Thanks, I’ve added the instruction but my still does not release the memory. Do you have an idea of what could be happening?

How do you measure RAM usage?

Have you forced a GC run?

And even if the GC collected memory used and cleaned it, it will only given back to the operating system after a certain time (if at all).

I’ve never had problems because of this, but I’ve heard often, that the Go runtime is not very keen on handing back memory to the operating system…

2 Likes

Its in Nature of Go.
==============

The Go memory allocator reserves a large region of virtual memory as an arena for allocations. This virtual memory is local to the specific Go process; the reservation does not deprive other processes of memory.

To find the amount of actual memory allocated to a Go process, use the Unix top command and consult the RES (Linux) or RSIZE (Mac OS X) columns.

https://golang.org/doc/faq#Why_does_my_Go_process_use_so_much_virtual_memory

2 Likes

actually, the documentation states that you do not need to close the r.Body. I still consider it good style, but you will get away with it if you forget it

https://golang.org/pkg/net/http/#Request

The Server will close the request body. …

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.