My Http API Server is terminated. I can't find the reason

Getting Help
1

Hi:
My http api server run about 3 ~ 10 hours, then it terminated. There were no error messages. Could somebody help me to find the problem.

The program run on a ECS instance @ AliYun Cloud in China.

The program use the following libraries.

"robfig/cron"
"go-gorp/gorp"
"julienschmidt/httprouter"
_ "go-sql-driver/mysql"

I use

resp, err := http.Get("http://api.wunderground.com/api/xxx)

to get weather data every 10 mins.
then store in a struct.

The client can access at the following http get request.
“http://myproj-name.xyz:7000/api/0.2/weather”

response body like the following.

{
  "Observation_time": "2018-04-03T10:41:57+08:00",
  "Create_time": "2018-04-03T10:50:00.000139373+08:00",
  "City": "Changzhou",
  "Country_name": "China",
  "Temp_c": 26,
  "Relative_humidity": "62",
  "Icon": "clear",
  "Weather": "Clear",
  "Wind_dir": "SSE",
  "Wind_degrees": 0,
  "Wind_kph": 11,
  "Is_weather_site_alive": true
}

I store MySQL also on RDS @ AliYun Cloud in China

type Weather struct {                                               
	Observation_time 		time.Time
	Create_time 			time.Time
	City					string	`db:",size:64"`
	Country_name			string	`db:",size:64"`
	
	Temp_c					int32						// for some problem that gorp reflect float32 to double schema
	Relative_humidity		string	`db:",size:32"`
	
	Icon					string	`db:",size:32"`		// correct name for weather icon
	Weather					string	`db:",size:64"`		// "Clear"
	Wind_dir				string	`db:",size:32"`		// "South", "SSE"
	Wind_degrees			int32
	Wind_kph				int32
	Is_weather_site_alive	bool 
}

It pass the stress test
$ab -n 10000 -c 1000 my-http-site-above

The cpu usage is about 5%, there are still 3GB avail memery on 4GB RAM.

Could someone help me to find the problem?
Thank you in advance.

2

If your program stop without any message surely is because you don’t properly handle the errors and you have a panic somewhere. So, first of all you must resolve this otherwise is hard to say what’s happen, can be thousants of reasons. Also, some code will help.

3

Thank you for your reply.

The following is a part of my service script.

#!/bin/bash
### BEGIN INIT INFO
# Provides:          Steve
# Required-Start:    
# Required-Stop:     
# Default-Start:   2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     false
# Short-Description: 
# Description:       Start/stop a Service
### END INIT INFO

# Variables
PROG_NAME=HttpApiServer

#Start the server
start() {
        #Verify if the service is running
        pgrep -f HttpApiServer > /dev/null

        if [ 0 = $? ]
        then
                echo "The Service is alreay running."
        else
                echo "Starting" $PROG_NAME "..."
                # run the server under account home dir
                #cd ~/go/bin
                #./HttpApiServer > output.txt 2>&1 & disown
                nohup /root/go/bin/HttpApiServer & disown
                sleep 5

                # verify if the server is running...
                if [ 0 = $? ]
                then
                        echo "Service was successfully started."
                else
                        echo "Failed to start service."
                fi
        fi
        echo
}

…
Nohup.txt didn’t store error info. How do I find the error message from golang? if there is a panic causes the program crash.
Thank you.

4

I had similar problem on linux hosting. Finally it was problem with memory - OS simply kill my process. I found record about it in system logs.

5

I tried $cat /var/log/syslog, but it only shows events today. My program crashed yesterday. how do I view the log yesterday or even several days earlier?

6

I’m talking about error handling in your Go program. You must handle this errors in your code and log output of those errors.

7

Hi @Steve.Tsai,

If you can show more of your code it may be useful in debugging the problem.

For example, if you aren’t calling checking your error or calling resp.Body.Close() after checking your error, your issue may lay there.

For example:

resp, err := http.Get("http://api.wunderground.com/api/xxx")
if err != nil {
	log.Fatal(err)
}
defer resp.Body.Close()

Edit: By the way, the reason I’m being specific about closing the response’s body is because you may have a resource leak with the connection’s file descriptor remaining open.

8 
func DecodeWeatherData(data map[string]interface{}) bool {
	// 2nd level of json struct is also decoded as a map struct
	loc := data["location"].(map[string]interface{})

	if loc == nil {
		return false
	}

	g_weather.Country_name = loc["country_name"].(string)
	g_weather.City = loc["city"].(string)

	// real weather data is here,
	observ := data["current_observation"].(map[string]interface{})

	if observ == nil {
		return false
	}

	var err error
	// the weather underground api use the following pre-defined format
	// RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700
	g_weather.Observation_time, err = time.Parse(time.RFC1123Z, observ["observation_time_rfc822"].(string))

	if err != nil {
		fmt.Println("!!! Error. " + err.Error())
		fmt.Println(" Decode weather observation_time = " + observ["observation_time_rfc822"].(string))
		return false
	}

	g_weather.Relative_humidity = observ["relative_humidity"].(string)

	// !!! Crash here
	// !!! when request weather api over 10 times/sec.
	// !!! panic: interface conversion: interface {} is nil, not float64
	g_weather.Temp_c = int32(observ["temp_c"].(float64))
	g_weather.Icon = observ["icon"].(string)
	g_weather.Weather = observ["weather"].(string)
	g_weather.Wind_degrees = int32(observ["wind_degrees"].(float64))
	g_weather.Wind_dir = observ["wind_dir"].(string)
	g_weather.Wind_kph = int32(observ["wind_kph"].(float64))

	fmt.Println(g_weather)
	return true
}

func DecodeRequestBody(r io.Reader) map[string]interface{} {
	body, err := ioutil.ReadAll(r)
	if err != nil {
		// keep last status, no change
		return nil
	}
	// copy slice string
	bodyBytes := make([]byte, len(body), (cap(body)+1)*2) // == nil, len() = 0, cap() = 0
	copy(bodyBytes, body)

	// parsing json string to get needed data for simple form
	var data map[string]interface{}

	// decoding json byte stream, and a check for associated errors.
	if err := json.Unmarshal(bodyBytes, &data); err != nil {
		fmt.Println("!!! Error. Decode weather data.")
		return nil
	}
	return data
}

func RequestOutWeather(t time.Time) {

	g_weather.Is_weather_site_alive = false

	// an article said that to avoid hang forever by default Client,
	// specify timeout to stop and output error.
	var netClient = &http.Client{
		Timeout: time.Second * 30,
	}

	resp, err := netClient.Get("http://api.wunderground.com/api/mykeyxxxxx/geolookup/conditions/q/autoip.json?geo_ip=49.80.123.123.json")
	if err != nil {
		fmt.Printf("!!! Get Weather info Error.")
		g_weatherBody = nil
		return
	}
	defer resp.Body.Close()

	g_weatherBody := DecodeRequestBody(resp.Body)

	// if successfully decode all
	g_weather.Is_weather_site_alive = DecodeWeatherData(g_weatherBody)
}

// Is this a correct way when an error happen ?
// or should I log it and continue to run?
func CheckErr(err error, msg string) {
    if err != nil {
        log.Fatalln(msg, err)
    }
}

Could anyone suggest a consistent and better way to log and handle the error?
ps. Could anyone help me to quote program text correctly ?

9

log.Fatalln will cause your program to die. It depends, if process is able to continue, don’t use Fatalln, just Warn or something. And where is your server code? Do you log the return from http.ListenAndServe?

1 Like
10

create a *log.Logger
and do logger.Println(err), you may set your logger to be std err, or a file, or whatever io.Writer

1 Like
11

Thank you acim. I found that I didn’t catch all possible errors. One of the reasons that my program dies is that I didn’t handle the return error from httpClient correctly.

My code is the following:

import (
	"os"
	"fmt"
	"log"
	"net"
	"time"
	"flag"			// command line flags
	"net/http"
	"database/sql"
	"encoding/json"

	"github.com/go-gorp/gorp"
	_ "github.com/go-sql-driver/mysql"
	"github.com/julienschmidt/httprouter"
)

func main() {
	// 1. Parse command line flags
	ParseCmdFlags()
	
	// 2. read config file, set by provision engineer
	ReadConfig () 

	// 3. connect to db		// old: ConnectDb ()
	g_dbMap = InitDb()
	defer g_dbMap.Db.Close()

	// 4. Declare db & clear db data if necessary
	DeclareDb (g_dbMap, *g_cmdFlags.pClearDb)
	
	RepeatRequestWeather ();
	
	// Set Router & Handlers
	router := httprouter.New()

	HttpSetHandlers (router)
	
	http.ListenAndServe(":" + g_config.ServerPort, router)
}

I didn’t handle the error. But the code from golang official document is the following:

log.Fatal(http.ListenAndServe(":12345", nil))

log.Fatal always exist the program. Should it handle different kind of errors? not necessary exist?

12

It is OK to use it here because this means your server won’t start anyways, but don’t use it in places you can recover and continue. There you can use log.Println, log.Print and log.Printf. Many people use the following logger instead of stdlib one:

This one has methods like Warn, Info, etc.

13

Thanks acim. Nice to know the existence of level logger. Let you know if I have any further problems.

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.