Implementing custom marshalers/unmarshalers on imported types

Getting Help
1

In my project, I have two go packages, world and database and two go programs client and server:

gopath
  - src
    - world (imports encoding/json)
    - database (imports world, imports dynamodb)
    - client (imports world, encoding/json)
    - server (imports world, database, encoding/json)

The world package contains many such enumerators that are common to client and server:

const (
    HammerTool ToolType = iota
    TrowelTool
    FrameTool
    PistolTool
    CannonTool
)

And since encoding/json is used to communicate between client and server, world contains the associated JSON marshalers and unmarshalers for each enumerator.

func (toolType ToolType) MarshalText() ([]byte, error) {
    switch toolType {
    case HammerTool:
        return []byte("hammer"), nil
    case TrowelTool:
        return []byte("trowel"), nil
    case FrameTool:
        return []byte("frame"), nil
    case PistolTool:
        return []byte("pistol"), nil
    case CannonTool:
        return []byte("cannon"), nil
    default:
        return nil, errors.New("Unknown tool " + fmt.Sprint(toolType))
    }
}

func (toolType *ToolType) UnmarshalText(text []byte) error {
    switch string(text) {
    case "hammer":
        *toolType = HammerTool
    case "trowel":
        *toolType = TrowelTool
    case "frame":
        *toolType = FrameTool
    case "pistol":
        *toolType = PistolTool
    case "cannon":
        *toolType = CannonTool
    default:
        return errors.New("Unknown tool " + string(text))
    }
    return nil
}

The problem arises when the enumerators are saved to DynamoDB in the database package. DynamoDB has its own marshalers and unmarshalers:

// Note: These methods were never tested because they were not compiled.
// They may contain errors but should convey the general idea
func (toolType ToolType) MarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
   txt, err := toolType.MarshalText()
   if err != nil {
      return err
   }
   av.S = aws.String(txt)
   return nil
}

func (toolType *ToolType) UnmarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
    return toolType.UnmarshalText([]byte(*av.S))
}

These methods are thin wrappers of the JSON marshalers/unmarshalers. However, they cannot be compiled into the database package as Go does not allow adding additional receivers on an imported type. Additionally, they cannot be added into the world package as world is imported by client and client cannot afford to be megabytes larger due to an import of the DynamoDB package exposing the dynamodb.AttributeValue type. After all, client gets compiled to js/wasm and downloaded to web browsers over the network…and has no use for DynamoDB.

My question is: How can I implement custom marshalers/unmarshalers on types that I import from other packages?

Notes:

  • I don’t want to keep marshaling enumerators into DynamoDB as integers because then I couldn’t add additional enumerator values without risking corruption
  • It would be nice if everything related to DynamoDB was limited to the database package, including all DynamoDB specific marshalers.
  • The DynamoDB SDK provides a string struct tag but, for example, it would encode TrowelTool as {S:"1"} not {S:"trowel"}, as opposed the normal encoding of {N:"1"}
2

How so, as long as you only add new entries at the bottom of the list it should continue to increase and they will stay unique. iota resets only at the next const keyword.

You can also give them a String function, this is a “best practice”

func (t ToolType) String() string {
    return [...]string{"Hammer", "Trowel", "Frame", "Pistol", "Cannon"}[t]
}

Then put the array where UnMarshall could use it to map the String representation back to the index.

1 Like
3

You could define another ToolType in your database package:

package database

import (
    "world"
)

type ToolType world.ToolType

func (toolType ToolType) MarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
   txt, err := world.ToolType(toolType).MarshalText()
   if err != nil {
      return err
   }
   av.S = aws.String(txt)
   return nil
}

func (toolType *ToolType) UnmarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
    return ((*world.ToolType)(toolType)).UnmarshalText([]byte(*av.S))
}

And then server handles the conversion between world.ToolType and database.ToolType.

EDIT: Call to UnmarshalText was wrong.

1 Like
4

Thanks! I realize that I might be able to get away with adding entries at the end in certain cases but, in the case I used as the example, the order of the entries is what determines their order in the user interface. If I add another item, I need full flexibility of where to put it so that my user interface has the best possible order.

5

Thanks! I did consider making a mirror of my struct hierarchy in the database package, with special types for each enumerator like you demonstarated, and doing one unsafe cast at the top level. I’m pretty sure this would work but it introduces new issues such as the two packages must be kept in sync (or a code generator would have to be implemented). It might be the only way though.

I am curious though: If I was to make a mirror of every struct in the hierarchy and each type in need of custom marshaling, would that introduce extreme redundancy into the compiled binary or is that some thing the compiler could optimize?

6

You realize you that there is nothing stopping you from ordering anything in a gui to be in a different order as anything in the Go code.

1 Like
7

You’re correct. The GUI is implemented in Javascript and it could define it’s own order for each type. I was hoping to avoid that though. I could use _ to reserve a few numbers in between each entry but, then again, I was hoping for a more elegant solution (Edit: and that would have the side effect of making it harder to validate the type since I could no longer do valid = 0 < x && x <= CannonTool.

Also, storing the enumerators as strings in DynamoDB makes the database human-readable.

8

You don’t need to use an unsafe cast this way. You can just convert the value back to it’s world package counterpart whenever necessary like how I demonstrated above. The only logic that has to be kept in sync from the database package is anything related to marshaling or unmarshaling to/from the *dynamodb.AttributeValue which I think is the basic definition of your problem. I can’t quite think of any other simplification other than moving the problem (i.e. moving (Un)MarshalDynamoDBAttributeValue into world) or generalizing it, e.g. if all types you’re marshaling implement encoding.TextMarshaler and encoding.TextUnmarshaler, you could to this:

type TextMarshaler interface {
    encoding.TextMarshaler
    encoding.TextUnmarshaler
}

type DBMarshaler struct {
    TextMarshaler
}

func (m DBMarshaler) MarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
    txt, err := m.MarshalText()
    if err != nil {
        return err
    }
    av.S = aws.String(txt)
    return nil
}

func (m DBMarshaler) UnmarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
    return m.TextUnmarshaler.UnmarshalText([]byte(*av.S))
}

And then before marshaling your world.ToolType, do this:

tt := world.HammerTool
av, err := dynamodbattribute.Marshal(DBMarshaler{tt})
// ...
err := dynamodbattribute.Unmarshal(av, DBMarshaler{tt})
1 Like
9

Tying the GUI order to the back end ensures the worst of tight coupling and low cohesion at the same time. You want loose coupling and high cohesion. Google it if you are unfamiliar with the terms/concepts.

10

Not that this is very important but there is a third metric: Redundancy. I didn’t want to repeat myself (in terms of the order of the entries) unless absolutely necessary.

11

This won’t work in the general case of defining a custom marshaler on imported types but I found a third-party DynamoDB package that supports the TextMarshaler interface. I think it is the most elegant solution in my specific case to switch to this package.

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.