Troubleshooting Go Server Problems

[joemarti (Joe Daniel)]

Hello,

I think my question at least at first is more generic and I lack developer experience as I am a sysadmin/network guy by trade. Anyhow, I have built a golang server with about 200 lines of code. The program has a web server but also another go routine that accepts JSON data on another port and does things with the data to display. The code seems to work fine as it does exactly what I want it to do as far as I can tell. The problem is the server goes down after a couple days running. Or the most recent change I made it stays up for maybe half a day before being killed. What are ways to troubleshoot, build defensive code, etc. I am not used to building applications that stay running as standalone for long periods of time.

Thanks again,

Joe

[adriel]

I think, better to post your important part of your code especially for receiving and accepting data.

[calmh (Jakob Borg)]

In production, have some service manager run your program. Systemd is adequate and there are other choices out there that are easier to work with (runit is my favorite). Make sure that it logs both stdout and stderr from your program. This way it will catch and log any panics, fatal errors or out of memory crashes that may happen, log them so you can figure out what happened, and restart your process. Add logging (to stdout) at relevant points in your program. Inspect the logs now and then and act on any irregularities.

At scale, add metrics instead perhaps using the Prometheus client library and run a Prometheus or similar server to collect and view them. This lets you trend memory usage, requests, errors etc and get an understanding of how your server works over time.

[joemarti (Joe Daniel)]

Thanks calmh,
I think you saved in a lot of time in researching. I will try this.

[joemarti (Joe Daniel)]

Adriel,

The below is relevant code. Thanks for your help.

func handleConnection(conn net.Conn) {
    getinfo()
    var mutex = &sync.Mutex{}
    mutex.Lock()
    dec := gob.NewDecoder(conn)
    p := &MinerResults{}
    dec.Decode(p) //working
    num := (len(MinerResultArray) + 1)
    if p.UID == "one" {
		num = 0
    } else if p.UID == "two" {
		num = 1
    } else if p.UID == "three" {
		num = 2
    } else if p.UID == "four" {
		num = 3
    }
    if num < 4 {
		MinerResultArray[num].CpuAvgHashRate = p.CpuAvgHashRate
		MinerResultArray[num].GpuAvgHashRate = p.GpuAvgHashRate
		MinerResultArray[num].TotalEntries = p.TotalEntries
		MinerResultArray[num].Err = p.Err
		MinerResultArray[num].TimeStamp = p.TimeStamp
		MinerResultArray[num].UID = p.UID
		MinerResultArray[num].CpuCurrTemp = p.CpuCurrTemp
		MinerResultArray[num].GpuPowerCurr = p.GpuPowerCurr
		MinerResultArray[num].GpuTempCurr = p.GpuTempCurr
    }

    mutex.Unlock()
    conn.Close()
}

func starthttp() {
    http.HandleFunc("/", printCliWalt)
    if err := http.ListenAndServe("IP:PORT", nil); err != nil {
        panic(err)
    }

}

func main() {
	var servermode string
	if len(os.Args) != 3 {
		fmt.Println("Please supply 2 arguments")
		os.Exit(3)
    } else {
        Adress = os.Args[1]
	    servermode = os.Args[2]
        fmt.Println("Args do line up!!!")
        fmt.Println(servermode)
    }
    for _, i := range MinerResultArray {
		i.CpuAvgHashRate, i.GpuAvgHashRate, i.TotalEntries = 0, 0, 0
		i.Err, i.TimeStamp, i.UID = "", "", ""
    }
    fmt.Println("start");
	if servermode == "client-enabled" {
		go starthttp()
		ln, err := net.Listen("tcp", "IP1:PORT")
		if err != nil {
			fmt.Println(err)
		}
		for {
			conn, err := ln.Accept()
			if err != nil {
				continue
			}
			go handleConnection(conn)
		}
	} else if servermode == "stand-alone" {
		fmt.Println("stand-alone enabled")
		http.HandleFunc("/", printJstWalt)
		if err := http.ListenAndServe("IP2:PORT", nil); err != nil {
			panic(err)
		}
	} else {
		fmt.Println("You did not enter a correct server mode")
	}
}

[stangeorge]

Where are you running this code on? If it is heroku and its idle for 30 minutes, they will let the instance sleep: https://devcenter.heroku.com/articles/free-dyno-hours#dyno-sleeping

Based on your variables such as MinerResultArray, GpuAvgHashRate etc, I am guessing this may be crypto mining related. If thats the case some services such as Google Compute Engine do not allow it in their Free Tier.

If neither of those apply to you, then do some profiling using go tool pprof every few hours. It may point to a memory leak, timing bug or some process gone awry.

[Rio_M]

At least sync.Mutex is used wrongly. Try this: https://play.golang.org/p/LoVw-DJT6IN

Correct: https://play.golang.org/p/N4I2nnRXV5o

[joemarti (Joe Daniel)]

First of all I am running the machine on a stand-alone server running a Kali Linux 64 bit OS.

I am not sure what all your thoughts are on this but I stripped about half that code out to create a mini-server and I still have problems with it the program existing. The last thing I did was when I start the program I did this:

    nohup ./miniserver

When using nohub it at least ran over night without exiting. I am not sure if this is an ideal method but I am going to let it run to see how long it stays up.

Thanks again