How to learn and work with Go's types?

Getting Help
1

I made a couple smaller Go apps and studied Go books. But I still struggle tremendously with Go’s type and the different conversion paths between them. How can I learn those and code confidently, without having to copy/paste online code?

Here’s an example to illustrate my struggles. In my app I’m working with NewDecoder(), which takes an io.Reader.

I make a HTTP request with http.Get(), which seems to return an Response if I understand the docs right. The Body field of that type returns io.ReadCloser.

So to make both work, I need to go from io.ReadCloser → io.Reader. But where do I find that function?

I found here that no conversion path is needed for io.ReadCloser → io.Reader. But my NewDecoder() errors with “unexpected EOF” when I give it Response.Body, so I probably need a reader in between them that can properly handle EOF without erroring. :thinking:

Earlier I made NewDecoder work with os.Open(), which returns a File according to the docs. But that doc page doesn’t define the struct’s field, and I couldn’t find online what type File.Body is.

Edit: I “fixed” the problem in this specific case by putting a new(bytes.Buffer) and ReadFrom() function between the HTTP request and the NewDecoder().

This is of course not a real solution since I only found it through trail & error, copying online code, a lot of time (1+ hour), and a decent bit of frustration.

So how do I learn Go’s types and the conversion paths, so I have a much more pleasant and productive Go experience?

2 Likes
2

There is no way you can memorize every Go types in the planet :joy:. For basic understanding and for now, all you need to remember is that “a type must match a type” like a square block can only be fitted in a square mould, not circle or triangle.

The reason behind this is the defensive control of computing memory which itself can be a standalone topic for computer science on its own.

The trick, is how to hunt for the right “mould” and “block” when needed.

Now, to find the correct “mould” and “block”, you do it by reading the package documentations before use. Depending on packages, the documentations are solely dictated by the package maintainers. Here are some examples:

  1. Standard package is documented here: Standard library - Go Packages
  2. Community contributed packages will follows the maintainer’s documentations. Example:
    GitHub - goreleaser/nfpm: nFPM is Not FPM - a simple deb, rpm, apk and arch linux packager written in Go → https://nfpm.goreleaser.com/.
  3. If all else fails, should the source code is available for read (e.g. open source), your last resort is to read the code.

Where to find it at the first place? Improve your Google / DuckDuckGo searches.

Asking in forum (depending on which one) can sometimes attract very toxic and demoralizing replies that sometimes you might question what happened to our society.

Say you have not idea with io package’s ReadCloser(...) and Reader(...) function, the first thing to do is to find the io package (io package - io - Go Packages) by searches. You will find that they are both called interface type which points you to understand how interface works in Go. Hence, proceed to go hunt down the necessary documentations and digest them via understanding.

You’re asking an experience-based question. Hence, there is no “one” solution to rule them all. What I can tell you is that you haven’t found you way of learning the programming paradigm. Have patience and explore your way.

Like for me, I prefer reading through text, codes, and research paper (mainly because I can speed read and my past experience) alongside hands-on experimentation. Hence, attempting https://gobyexample.com/ and reading through all the boring specifications in Documentation - The Go Programming Language is highly effective for me.

Some people can’t take it so they prefer mentoring by reading guided book, video tutorial, or attending boot-camp. Some are natural godlike talents who can absorb anything without first referring anything (I only physically met 2 in my life so far) .

Because programming was not meant to “copy and paste” at the first place.

In short, it is an art of:

  1. analyze, breakdown, and understand a problem
  2. being resourceful, especially with your searching skills.
  3. derive a systematic, repeatable approach to solve problem.

To be good in art, it takes time so don’t give up. Continue with your exploration and take break when needed. :upside_down_face:

4 Likes
3

I’m going to add to Holloway’s answer with some info that I hope will be useful:

There are two “classifications”* of types in Go: concrete and interface types.

Concrete types are:

  • All of Go’s built in types like bool, int, string, etc.
  • All structs
  • All arrays
  • All pointers
  • All slices
  • All maps
  • All functions

That is, concrete types are everything. Everything in Go that has a type is of a concrete type. If you want to convert from one concrete type to another, you have to use a conversion.

Interface types are descriptions of required methods that must exist on a concrete type in order to implement the interface. For example:

io.Reader is an interface type and therefore consists of just a method set:

type Reader interface {
        Read(b []byte) (n int, err error)
}

That is to say that any concrete type that has a Read method with the same signature (one []byte slice parameter and returns (int, error)), then that concrete type implements io.Reader. io.ReadCloser is also an interface type:

type ReadCloser interface {
        Read(b []byte) (n int, err error)
        Close() error
}

Because the io.ReadCloser interface defines the same Read method as io.Reader, then if your type currently implements io.Reader and you add a Close method that returns an error now your type also implements io.ReadCloser.

Putting it together a bit more:

The Body field of the http.Response type is of type io.ReadCloser. We know now that io.ReadCloser is just a method set. You never actually have a value of type io.ReadCloser, the value has to have a real, concrete type, and that type must have Read and Close methods just like the io.ReadCloser interface requires.

If you have a value (of some concrete type, which we don’t actually care about) that implements io.ReadCloser and you want to use it as an io.Reader, no conversion is necessary because you’re going from an interface type that contains a Read method to another interface type that contains the same Read method.

Consider the following example:

package main

import (
	"fmt"
	"io/ioutil"
)

type myType struct{}

func (myType) Read(a []byte) (b int, err error) {
	return 0, fmt.Errorf("TODO!")
}

type myReader interface {
	Read(b []byte) (n int, err error)
}

func main() {
	var x myReader = myType{}
	bs, err := ioutil.ReadAll(x)
	fmt.Println("bs:", bs, "err:", err)
}

I made my own myReader type for no reason here other than to demonstrate that interface types just define method sets and if the compiler knows at compile time that my x variable is of some type that implements the myReader method set, then the compiler knows that ioutil.ReadAll requires a value that implements the io.Reader method set which is the same set as myReader, so it just works.

An *os.File doesn’t have a Body field because an *os.File itself already has the Read and Close methods on it.

* Classification is a word I chose here but isn’t an official term for this divide, so I suspect you won’t see it called this way anywhere else.

4 Likes
4

Thanks so much for the positive, encouraging and in-depth replies! This is amazing. :slightly_smiling_face: I was a little worried coming back to this topic but not even my parents are this nice to me (this is a joke, don’t worry).

(I’ll reply to the statements I have questions about rather than make this reply into a long ‘thanks’ reply.)

Thanks, this makes a lot of sense. More reading and taking your time.

Thanks, I agree I still have plenty to learn here. Can you recommended resources to learning the above points? Trail-and-error is also possible of course, but that will take decent time and frustration. :slightly_smiling_face:

Thank you. This makes sense to me and is similar to how my understanding of C#'s interfaces is: ‘an interface is a promise/contract to implement something’.

Thanks, so if I understand properly the implementation of a type is done implicitly? Because you say that any type that has a Read() method with the same signature, is considered to be an io.Reader, without it explicitly saying so?

This is an awesome statement that clarifies a lot.

Thanks so much for this clarification. All this time I was looking for information on how to do conversions and which conversion path to take. All I had to do is look for similarly named methods?

In my app I fixed NewDecoder() by giving it a value from new(bytes.buffer). That was an io.Reader interface for the first and an Buffer interface for the second. Now I see why they are compatible!

Buffer has this method according to the docs, which is what io.Reader expects.

func (b *Buffer) Read(p []byte) (n int, err error)

But I could also have used Reader.Read() from the strings package, because that one has this method:

func (r *Reader) Read(b []byte) (n int, err error)

If that understanding of me is correct, then does it matter which option I go with? Is one better than the other? Or is this redundancy simply there for convenience?

So the type doesn’t really matter, as ioutil.ReadAll() simply wants a Read() method with a certain signature to be happy? You could have made any type and all is good as long as Read() is there.

That would mean that me looking for type conversions is actually a bit silly, since Go doesn’t care about that. This is simplified, but: Go simply wants to match methods together.

5

Yes, exactly.

If you have a string of JSON and you want to decode it, you could do dec, err := json.NewDecoder(strings.NewReader(jsonString))

If you want to decode your JSON directly from the request, you might be able to just say dec, err := json.NewDecoder(req.Body).

You cannot do dec, err := json.NewDecoder(new(bytes.Buffer)) because the buffer is empty, so when you try to decode, you’ll get an io.EOF error. If you want to use a buffer, you should first load it with whatever data you’re buffering, e.g. for an (*http.Request).Body:

// req defined somewhere else
req, err := http.Default.Do(req)
if err != nil {
    return err
}
defer req.Body.Close()
b := new(bytes.Buffer)
if _, err = io.Copy(b, req.Body); err != nil {
    return err
}
dec, err := json.NewDecoder(b)

There are pros and cons to using the buffer. By reading the whole response body into a buffer and decoding afterward, if any network errors occur, you’ll get them while buffering instead of while decoding, so you could make a new request to retry. If you want to avoid loading the whole JSON into a buffer in memory, you could use the request body directly in the call to json.NewDecoder, but understand that when you call dec.Decode(&myVar) later, if the *json.Decoder encounters an error while reading from the network, it will report that error back to you in the return from dec.Decode and you’ll have to handle it.

This is true when you’re talking about interfaces, but like I showed in my other post, concrete types must be the exact same type.

A common idiom in Go is to have your functions accept interfaces as parameters and return concrete types. That way, your functions essentially self-document the methods they expect on their parameters. Then by returning a concrete type, you get full access to all the methods available on that return type.

For completeness’ sake I also want to mention that there are exceptions:

  • There is a very small overhead when converting concrete values to interfaces and then calling functions through those interfaces. It’s very small, but it is reasonable to use concrete types as parameters when you know that the functions will only ever be called with instances of one specific type to a function. I only ever do this for some functions deep down in my internal code bases, almost never in publicly exposed functions.

  • If your function is some sort of factory function that returns different types depending on its parameters, then you can’t return a concrete type, so it of course makes sense to return an interface.

1 Like
6

At starting point until you found your way, it is frustrating but necessary. Try look beyond the codes, like concept, programming structure, and etc.

Resources wise, since you’re working on Go, you can start off by:

  1. Standard library - Go Packages, Documentation - The Go Programming Language, and https://gobyexample.com/

  2. Try solving simple problem to gain experience alongside. E.g. first 10 questions from https://projecteuler.net/
    2.1. While solving, whenever you bump into problem, instead of hunting for answer, try to understand the problem first. In the case you got the answer, try to understand how others fix the problem before ‘pasting’’ it. Something like how these hackers break things down: https://youtu.be/0uejy9aCNbI

  3. Bookwise, there are many (GitHub - dariubs/GoBooks: List of Golang books).
    4.1. The one worked for me (very advanced for tech transfer but outdated) is: Introduction · Build web application with Golang
    4.2. Common practices in coding: Effective Go - The Go Programming Language

2 Likes
7

Thanks again for the in-depth replies. I’m really grateful for the time you both put into helping a random stranger. :slightly_smiling_face:

Thanks for your clear explanation and the example. I could follow along your explanation and now also realise I made a thought mistake earlier, when I asked about the function redundancy for Read().

Because even though both types have the same method with the same signature, the actual implementation/behaviour of Read() is of course different. Since an interface doesn’t specify what should be inside Read(), but only that there needs to be a Read() method with a certain signature, both methods can still do something slightly different!

Like in your example, Buffer’s Read() is different from Reader’s Read() in terms of how they work with the computer memory and when they can error.

Thanks again. :slightly_smiling_face:

Ah I see. That is quite smart, because it also makes the function more flexible in terms of what inputs it accepts, which prevents the need for code duplication.

So the best approach is to have functions accept type inputs. In my app I have for instance this function:

// getFileHash() opens a file and calculates that file's SHA256 hash
func getFileHash(path string) string { ... }

Would it be more idiomatic Go use if I do:

// getFileHash() opens a file and calculates that file's SHA256 hash
func getFileHash(path Reader) string { ... }

(As an discussion example, I could not make this work nor find the type what the string concrete type is.)

Thanks for the list! :slightly_smiling_face:

1 Like
8

Yes, it would be more idiomatic, take interface as a parameter, return a concrete type.

To get good with data types fast, it can help if you make a cheat sheet. Write down in a file each interesting/useful struct you encounter, and a few words about what it does (like one liner examples).

1 Like
9

Thanks for the reply and the advice Les. :slightly_smiling_face:

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.