Best Way to unMarshal a date values JSON

fstn · December 29, 2021, 2:00am

Hi the community, I would like to unmarshal this kind of data:

 "ranks": {
      "2021-12-22T00:00:00Z": "Over page 5",
      "2021-12-23T00:00:00Z": "Over page 5",
      "2021-12-24T00:00:00Z": "Over page 5",
      "2021-12-25T00:00:00Z": "112",
      "2021-12-26T00:00:00Z": "152",
      "2021-12-27T00:00:00Z": "138",
      "2021-12-28T00:00:00Z": "105",
      "2021-12-29T00:00:00Z": "Pending data refresh",
      "2021-12-30T00:00:00Z": "No data"
    },

I declared my struct like this:

Ranks          map[string]string `json:"ranks"`

SO I’m going to unmarshal it as string and after look on this map to transform the iso8601 key as date to get a struct like this:

Ranks          map[Time.time]string `json:"ranks"`

Any better ideas?

skillian · December 29, 2021, 3:57am

It depends on what you want to do with the data after you unmarshal it.

eudore · December 29, 2021, 6:06am

// Ranks          map[Time]string `json:"ranks"`

type Time time.Time

func (d *Time ) UnmarshalJSON(b []byte) error {
	str := string(b)
	if str != "" && str[0] == '"' && str[len(str)-1] == '"' {
		str = str[1 : len(str)-1]
	}

	// parse string
	t, err := time.Parse(format,str)
	if err == nil {
		*d = Time(t)
		return nil
	}
	return fmt.Errorf("invalid duration type %T, value: '%s'", b, b)
}

skillian · December 29, 2021, 11:40am

My point is that when you ask, “Any better ideas?” The answer could be, “no, that seems like an idiomatic way to do what you want,” or it could be, “you might want to consider doing x.” The only way we can give you a good answer is by knowing what you’re going to do with this data after you unmarshal it. If the timestamps are sparse and you need to check for exact time (e.g. you will check for the value of 2021-12-26T00:00:00Z, but not 2021-12-25T19:00:00+500 which is the same instant in time but a different zone). If you’re instead going to iterate through the times and/or do something based on the closest time, there’s a different solution. If the times are always UTC and increment by exactly one day, you could instead store the first date in a data structure and index into a slice of the values by thw number of days since the first day, etc… We can only give you (potentially) better ideas if we know what you’re doing.

fstn · December 29, 2021, 12:31pm

Thank you, You’re right.

After that, what I need to do is to display this data on a React graph, graph is taking this kind of input:

[
  {
    "Date": "2010-01",
    "scales": 1998
  },
  {
    "Date": "2010-02",
    "scales": 1850
  },
  {
    "Date": "2010-03",
    "scales": 1720
  },
  {
    "Date": "2010-04",
    "scales": 1818
  },
  {
    "Date": "2010-05",
    "scales": 1920
  },
  {
    "Date": "2010-06",
    "scales": 1802
  },

So my point was to return it as Ranks map[Time.time]string json:“ranks”`` and transform to an object with date and scales at properties on the JS side.

I would need to store it into mongodb too.

fstn · December 29, 2021, 12:33pm

Thank you so much can you specify your unmarshallJSON func into your fields tags? Or this is going to be applied automatically on all Time of my App?

fstn · December 29, 2021, 12:49pm

It would be something like that:


type Rank struct {
	Date time.Time `json:"date"`
	Rank int       `json:"rank"`
}

func (b *Rank) UnmarshalJSON(data []byte) error {

	var v []byte
	if err := json.Unmarshal(data, &v); err != nil {
		return err
	}
	str := string(v[0])
	if str != "" && str[0] == '"' && str[len(str)-1] == '"' {
		str = str[1 : len(str)-1]
	}

	t, err := time.Parse("2006-01-02T15:04:05-0700", str)
	if err == nil {
		return nil
	}
	b.Date = t
	rankAsStr := string(v[1])
	switch rankAsStr {
	case NO_DATA:
		b.Rank = 0
	case PENDING:
		b.Rank = -1
	case RANK_REPORT_OUT_OF_RANGE:
		b.Rank = 150
	default:
		b.Rank, err = strconv.Atoi(string(v[1]))
		if err == nil {
			return nil
		}
	}
	return nil
}

fstn · December 29, 2021, 12:58pm

This seems to not be called It might be because

 {
    "projectName": null,
    "asin": "B07Z842WBF",
    "marketplace": "France",
    "keyword": "macbook",
    "searchVolume": 135000,
    "ranks": {
      "2021-12-22T00:00:00Z": "9",
      "2021-12-23T00:00:00Z": "10",
      "2021-12-24T00:00:00Z": "8",
      "2021-12-25T00:00:00Z": "7",
      "2021-12-26T00:00:00Z": "7",
      "2021-12-27T00:00:00Z": "6",
      "2021-12-28T00:00:00Z": "7",
      "2021-12-29T00:00:00Z": "Pending data refresh",
      "2021-12-30T00:00:00Z": "No data"
    },
    "sponsoredRanks": {
      "2021-12-22T00:00:00Z": "Over page 5",
      "2021-12-23T00:00:00Z": "Over page 5",
      "2021-12-24T00:00:00Z": "Over page 5",
      "2021-12-25T00:00:00Z": "Over page 5",
      "2021-12-26T00:00:00Z": "Over page 5",
      "2021-12-27T00:00:00Z": "Over page 5",
      "2021-12-28T00:00:00Z": "Over page 5",
      "2021-12-29T00:00:00Z": "Pending data refresh",
      "2021-12-30T00:00:00Z": "No data"
    }
  },

is un object and I tried to decode it as an array:


type RankReport struct {
	ProjectName    interface{} `json:"projectName"`
	Asin           string      `json:"asin"`
	Marketplace    string      `json:"marketplace"`
	Keyword        string      `json:"keyword"`
	SearchVolume   int         `json:"searchVolume"`
	Ranks          []Rank      `json:"ranks"`
	SponsoredRanks []Rank      `json:"sponsoredRanks"`
}

type Rank struct {
	Date time.Time `json:"date"`
	Rank int       `json:"rank"`
}

fstn · December 29, 2021, 1:13pm

I also tried:


type RankReport struct {
	ProjectName    interface{} `json:"projectName"`
	Asin           string      `json:"asin"`
	Marketplace    string      `json:"marketplace"`
	Keyword        string      `json:"keyword"`
	SearchVolume   int         `json:"searchVolume"`
	Ranks          Ranks       `json:"ranks"`
	SponsoredRanks Ranks       `json:"sponsoredRanks"`
}

type Ranks = []Rank

type Rank struct {
	Date time.Time `json:"date"`
	Rank int       `json:"rank"`
}

func (ranks *Ranks) UnmarshalJSON(data []byte) error {

	var mapOfValues map[string]interface{}
	if err := json.Unmarshal(data, &mapOfValues); err != nil {
		return err
	}

	for k, v := range mapOfValues {
		if k != "" && k[0] == '"' && k[len(k)-1] == '"' {
			k = k[1 : len(k)-1]
		}

		t, err := time.Parse("2006-01-02T15:04:05-0700", k)
		if err == nil {
			return errors.InternalServerError.NewWithSecret("Unable to read DataHawk date", err)
		}
		rankAsStr, ok := v.(string)
		if !ok {
			return errors.InternalServerError.NewWithDetails("Unable to read DataHawk value", "")
		}
		rankValue := 0
		switch rankAsStr {
		case NO_DATA:
			rankValue = 0
		case PENDING:
			rankValue = -1
		case RANK_REPORT_OUT_OF_RANGE:
			rankValue = 150
		default:
			rankValue, err = strconv.Atoi(rankAsStr)
			if err == nil {
				return nil
			}
		}
		rank := Rank{
			Date: t,
			Rank: rankValue,
		}
		*ranks = append(*ranks, rank)
	}
	return nil
}

But it seems to be impossible on array

fstn · December 29, 2021, 1:44pm

I finally did that :


type RankReport struct {
	ASIN           string  `json:"asin"`
	Marketplace    string  `json:"marketplace"`
	Keyword        string  `json:"keyword"`
	SearchVolume   float64 `json:"searchVolume"`
	Ranks          []Rank  `json:"ranks"`
	SponsoredRanks []Rank  `json:"sponsoredRanks"`
}

type Rank struct {
	Date time.Time `json:"date"`
	Rank int       `json:"rank"`
}

func (rankReport *RankReport) UnmarshalJSON(data []byte) error {

	var mapOfValues map[string]interface{}
	if err := json.Unmarshal(data, &mapOfValues); err != nil {
		return err
	}

	rankReport.ASIN = mapOfValues["asin"].(string)
	rankReport.Marketplace = mapOfValues["marketplace"].(string)
	rankReport.Keyword = mapOfValues["keyword"].(string)
	rankReport.SearchVolume = mapOfValues["searchVolume"].(float64)
	rankReport.Ranks = []Rank{}
	ranks, ok := mapOfValues["ranks"].(map[string]interface{})
	if !ok {
		return errors.InternalServerError.NewWithDetails("Unable to read DataHawk ranks", "mapOfValues[\"ranks\"].")
	}
	for k, v := range ranks {
		rank, err := rankReport.extractRank(k, v)
		if err != nil {
			return errors.InternalServerError.NewWithSecret("Unable to read DataHawk ranks", err)
		}
		if rank != nil {
			rankReport.Ranks = append(rankReport.Ranks, *rank)
		}
	}
	sponsoredRanks, ok := mapOfValues["sponsoredRanks"].(map[string]interface{})
	if !ok {
		return errors.InternalServerError.NewWithDetails("Unable to read DataHawk sponsoredRanks", "mapOfValues[\"sponsoredRanks\"]")
	}
	for k, v := range sponsoredRanks {
		rank, err := rankReport.extractRank(k, v)
		if err != nil {
			return errors.InternalServerError.NewWithSecret("Unable to read DataHawk sponsoredRanks", err)
		}
		if rank != nil {
			rankReport.SponsoredRanks = append(rankReport.SponsoredRanks, *rank)
		}
	}
	return nil
}

func (rankReport *RankReport) extractRank(k string, v interface{}) (rank *Rank, err error) {
	if k != "" && k[0] == '"' && k[len(k)-1] == '"' {
		k = k[1 : len(k)-1]
	}

	t, err := time.Parse("2006-01-02T15:04:05Z", k)
	if err != nil {
		return nil, err
	}
	rankAsStr, ok := v.(string)
	if !ok {
		return nil, err
	}
	rankValue := 0
	switch rankAsStr {
	case NO_DATA:
		rankValue = 0
	case PENDING:
		rankValue = -1
	case RANK_REPORT_OUT_OF_RANGE:
		rankValue = 150
	default:
		rankValue, err = strconv.Atoi(rankAsStr)
		if err != nil {
			return nil, err
		}
	}

	return &Rank{
		Date: t,
		Rank: rankValue,
	}, nil
}

What I don’t like, is that it’s going to ignore all json tags…

mje · December 29, 2021, 4:30pm

You want a []Rank, but ranks in your JSON is an object. Try

type RanksObject struct {
  Ranks []Rank
}
type RankReport struct {
  ...
  Ranks *RanksObject `json:"ranks"`  
  ....
}

and define UnmarshalJSON for *RanksObject.

system · March 29, 2022, 4:31pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.