Go REST gRPC API?

I am trying to do a REST API build by Go and gRPC. This seems not to be the simplest way to build REST API, but hopefully the fastest…

This is only basically covered almost everywhere. I have not found any clear roadmap or simple how-to. So far I think it may(?) contains of four parts:

  1. The web-page (text based AJAX-json?)
  2. Bridge proxy (Envoy or Nginx) - text to binary translation?
  3. go Something (go-web or grpc-web?) with endpoints?
  4. Postgresql database

For an example: The installation of Envoy without Docker demands hacker skills. Relaying on Javascript, Python, Bazel and so on…

Any tip, clue or roadmap how to do this is highly appreciated.

Is it even possible?

TIA

You could look at https://github.com/brocaar/lora-app-server and check its structure. The frontend is a React app that talks to a REST API through a swagger client, which in turn is generated from gRPC using https://github.com/grpc-ecosystem/grpc-gateway.

2 Likes

There are lots of pieces in the puzzle that I do not understand. What piece does what and in which order.

I have played with traditional REST API that basically consists of 2 pieces:

The Browser

1. GET users

The Go server

2. Connect to Postgresql
3. Listen and serve (Start server)
4. Route to an endpoint
5. Fetch SQL data
6. Parse the result to JSON and return to the browser

How do I translate this into gRPC? Which pieces do I use? In which order?

Thank you in advance for any clue or link that makes this a bit simpler.

I’ll try to make a quick summary of everything involved but I won’t bother with where things should be and some relations: you should really look up and read about the different pieces, and then try to follow the repo’s structure to see how everything adds up.

First, you’ll want to create some protos describing your services and their messages, e.g.:

syntax = "proto3";

package api;

import "google/api/annotations.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";

// UserService is the service managing the user access.
service UserService {

	// Get data for a particular user.
	rpc Get(GetUserRequest) returns (GetUserResponse) {
		option(google.api.http) = {
			get: "/api/users/{id}"
		};
	}

	// Create a new user.
	rpc Create(CreateUserRequest) returns (CreateUserResponse) {
		option(google.api.http) = {
			post: "/api/users"
			body: "*"
		};
	}
}


message User {
	// User ID.
	// Will be set automatically on create.
	int64 id = 1;

	// Username of the user.
	string username = 2;

	// The session timeout, in minutes.
	int32 session_ttl = 3 [json_name = "sessionTTL"];

	// Set to true to make the user a global administrator.
	bool is_admin = 4;

	// Set to false to disable the user.
	bool is_active = 5;

	// E-mail of the user.
	string email = 6;

	// Optional note to store with the user.
	string note = 7;
}

message CreateUserRequest {
	// User object to create.
	User user = 1;

	// Password of the user.
	string password = 2;

	// Add the user to the following organizations.
	repeated UserOrganization organizations = 3;
}

message CreateUserResponse {
	// User ID.
	int64 id = 1;
}

message GetUserRequest {
	// User ID.
	int64 id = 1;
}

message GetUserResponse {
	// User object.
	User user = 1;

	// Created at timestamp.
	google.protobuf.Timestamp created_at = 2;

	// Last update timestamp.
	google.protobuf.Timestamp updated_at = 3;
}

After that you’ll want to generate your gRPC code from it and optionally the REST gateway and swagger definitions, which could be achieved with a script like this after you’ve installed protoc (protobuffer compiler) and the needed plugins (https://github.com/grpc-ecosystem/grpc-gateway):

#!/usr/bin/env bash

GRPC_GW_PATH=`go list -f '{{ .Dir }}' github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway`
GRPC_GW_PATH="${GRPC_GW_PATH}/../third_party/googleapis"

# generate the gRPC code
protoc -I../vendor -I${GRPC_GW_PATH} -I. --go_out=plugins=grpc:. \
    user.proto \
    other_model.proto

# generate the JSON interface code
protoc -I../vendor -I${GRPC_GW_PATH} -I. --grpc-gateway_out=logtostderr=true:. \
    user.proto \
    other_model.proto

# generate the swagger definitions
protoc -I../vendor -I${GRPC_GW_PATH} -I. --swagger_out=json_names_for_fields=true:./swagger \
    device.proto \

# merge the swagger code into one file
go run swagger/main.go swagger > ../static/swagger/api.swagger.json

If we run the script it’ll generate our gRPC code, the transalation to a JSON REST API and a nice swagger definition to use at the frontend. Great! Now we need to implement real logic for the endpoints:

// UserAPI exports the User related functions.
type UserAPI struct {
	validator auth.Validator
}

// NewUserAPI creates a new UserAPI.
func NewUserAPI(validator auth.Validator) *UserAPI {
	return &UserAPI{
		validator: validator,
	}
}

// Create creates the given user.
func (a *UserAPI) Create(ctx context.Context, req *pb.CreateUserRequest) (*pb.CreateUserResponse, error) {
	if req.User == nil {
		return nil, grpc.Errorf(codes.InvalidArgument, "user must not be nil")
	}

	if err := a.validator.Validate(ctx,
		auth.ValidateUsersAccess(auth.Create)); err != nil {
		return nil, grpc.Errorf(codes.Unauthenticated, "authentication failed: %s", err)
	}

	// validate if the client has admin rights for the given organizations
	// to which the user must be linked
	for _, org := range req.Organizations {
		if err := a.validator.Validate(ctx,
			auth.ValidateIsOrganizationAdmin(org.OrganizationId)); err != nil {
			return nil, grpc.Errorf(codes.Unauthenticated, "authentication failed: %s", err)
		}
	}

	user := storage.User{
		Username:   req.User.Username,
		SessionTTL: req.User.SessionTtl,
		IsAdmin:    req.User.IsAdmin,
		IsActive:   req.User.IsActive,
		Email:      req.User.Email,
		Note:       req.User.Note,
	}

	isAdmin, err := a.validator.GetIsAdmin(ctx)
	if err != nil {
		return nil, helpers.ErrToRPCError(err)
	}

	if !isAdmin {
		// non-admin users are not able to modify the fields below
		user.IsAdmin = false
		user.IsActive = true
		user.SessionTTL = 0
	}

	var userID int64

	err = storage.Transaction(func(tx sqlx.Ext) error {
		userID, err = storage.CreateUser(tx, &user, req.Password)
		if err != nil {
			return err
		}

		for _, org := range req.Organizations {
			if err := storage.CreateOrganizationUser(tx, org.OrganizationId, userID, org.IsAdmin); err != nil {
				return err
			}
		}

		return nil
	})
	if err != nil {
		return nil, helpers.ErrToRPCError(err)
	}

	return &pb.CreateUserResponse{Id: userID}, nil
}

// Get returns the user matching the given ID.
func (a *UserAPI) Get(ctx context.Context, req *pb.GetUserRequest) (*pb.GetUserResponse, error) {
	if err := a.validator.Validate(ctx,
		auth.ValidateUserAccess(req.Id, auth.Read)); err != nil {
		return nil, grpc.Errorf(codes.Unauthenticated, "authentication failed: %s", err)
	}

	user, err := storage.GetUser(storage.DB(), req.Id)
	if err != nil {
		return nil, helpers.ErrToRPCError(err)
	}

	resp := pb.GetUserResponse{
		User: &pb.User{
			Id:         user.ID,
			Username:   user.Username,
			SessionTtl: user.SessionTTL,
			IsAdmin:    user.IsAdmin,
			IsActive:   user.IsActive,
			Email:      user.Email,
			Note:       user.Note,
		},
	}

	resp.CreatedAt, err = ptypes.TimestampProto(user.CreatedAt)
	if err != nil {
		return nil, helpers.ErrToRPCError(err)
	}
	resp.UpdatedAt, err = ptypes.TimestampProto(user.UpdatedAt)
	if err != nil {
		return nil, helpers.ErrToRPCError(err)
	}

	return &resp, nil
}

Now we need to serve the gRPC service and the HTTP gateway API:

func setupAPI(conf config.Config) error {
	validator := auth.NewJWTValidator(storage.DB(), "HS256", jwtSecret)
	rpID, err := uuid.FromString(conf.ApplicationServer.ID)
	if err != nil {
		return errors.Wrap(err, "application-server id to uuid error")
	}

	grpcOpts := helpers.GetgRPCLoggingServerOptions()
	sizeOpts := []grpc.ServerOption{
		grpc.MaxRecvMsgSize(math.MaxInt32),
		grpc.MaxSendMsgSize(math.MaxInt32),
		grpc.MaxMsgSize(math.MaxInt32),
	}
	grpcOpts = append(grpcOpts, sizeOpts...)
	grpcServer := grpc.NewServer(grpcOpts...)
	api.RegisterUserServiceServer(grpcServer, NewUserAPI(validator))
	api.RegisterOtherModelServiceServer(grpcServer, NewOtherModelAPI(validator))
// setup the client http interface variable
	// we need to start the gRPC service first, as it is used by the
	// grpc-gateway
	var clientHTTPHandler http.Handler

	// switch between gRPC and "plain" http handler
	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
			grpcServer.ServeHTTP(w, r)
		} else {
			if clientHTTPHandler == nil {
				w.WriteHeader(http.StatusNotImplemented)
				return
			}

			if corsAllowOrigin != "" {
				w.Header().Set("Access-Control-Allow-Origin", corsAllowOrigin)
				w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
				w.Header().Set("Access-Control-Allow-Headers", "Accept, Content-Type, Content-Length, Accept-Encoding, Grpc-Metadata-Authorization")

				if r.Method == "OPTIONS" {
					return
				}
			}

			clientHTTPHandler.ServeHTTP(w, r)
		}
	})

	// start the API server
	go func() {
		log.WithFields(log.Fields{
			"bind":     bind,
			"tls-cert": tlsCert,
			"tls-key":  tlsKey,
		}).Info("api/external: starting api server")

		if tlsCert == "" || tlsKey == "" {
			log.Fatal(http.ListenAndServe(bind, h2c.NewHandler(handler, &http2.Server{})))
		} else {
			log.Fatal(http.ListenAndServeTLS(
				bind,
				tlsCert,
				tlsKey,
				h2c.NewHandler(handler, &http2.Server{}),
			))
		}
	}()

	// give the http server some time to start
	time.Sleep(time.Millisecond * 100)

	// setup the HTTP handler
	clientHTTPHandler, err = setupHTTPAPI(conf)
	if err != nil {
		return err
	}

	return nil
}

//This serves and routes the HTTP API and static files (e.g., the React frontend files).
func setupHTTPAPI(conf config.Config) (http.Handler, error) {
	r := mux.NewRouter()

	// setup json api handler
	jsonHandler, err := getJSONGateway(context.Background())
	if err != nil {
		return nil, err
	}

	log.WithField("path", "/api").Info("api/external: registering rest api handler and documentation endpoint")
	r.HandleFunc("/api", func(w http.ResponseWriter, r *http.Request) {
		data, err := static.Asset("swagger/index.html")
		if err != nil {
			log.WithError(err).Error("get swagger template error")
			w.WriteHeader(http.StatusInternalServerError)
			return
		}
		w.Write(data)
	}).Methods("get")
	r.PathPrefix("/api").Handler(jsonHandler)

	// setup static file server
	r.PathPrefix("/").Handler(http.FileServer(&assetfs.AssetFS{
		Asset:     static.Asset,
		AssetDir:  static.AssetDir,
		AssetInfo: static.AssetInfo,
		Prefix:    "",
	}))

	return wsproxy.WebsocketProxy(r), nil
}

//This registers the gRPC handlers JSON gateways to pass them over to the HTTP API.
func getJSONGateway(ctx context.Context) (http.Handler, error) {
	// dial options for the grpc-gateway
	var grpcDialOpts = []grpc.DialOption{
		grpc.WithDefaultCallOptions([]grpc.CallOption{
			grpc.MaxCallRecvMsgSize(math.MaxInt32),
			grpc.MaxCallSendMsgSize(math.MaxInt32),
		}...),
	}

	if tlsCert == "" || tlsKey == "" {
		grpcDialOpts = append(grpcDialOpts, grpc.WithInsecure())
	} else {
		b, err := ioutil.ReadFile(tlsCert)
		if err != nil {
			return nil, errors.Wrap(err, "read external api tls cert error")
		}
		cp := x509.NewCertPool()
		if !cp.AppendCertsFromPEM(b) {
			return nil, errors.Wrap(err, "failed to append certificate")
		}
		grpcDialOpts = append(grpcDialOpts, grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{
			// given the grpc-gateway is always connecting to localhost, does
			// InsecureSkipVerify=true cause any security issues?
			InsecureSkipVerify: true,
			RootCAs:            cp,
		})))
	}

	bindParts := strings.SplitN(bind, ":", 2)
	if len(bindParts) != 2 {
		log.Fatal("get port from bind failed")
	}
	apiEndpoint := fmt.Sprintf("localhost:%s", bindParts[1])

	mux := runtime.NewServeMux(runtime.WithMarshalerOption(
		runtime.MIMEWildcard,
		&runtime.JSONPb{
			EnumsAsInts:  false,
			EmitDefaults: true,
		},
	))

	if err := pb.RegisterUserServiceHandlerFromEndpoint(ctx, mux, apiEndpoint, grpcDialOpts); err != nil {
		return nil, errors.Wrap(err, "register user handler error")
	}
	if err := pb.RegisterotherModelServiceHandlerFromEndpoint(ctx, mux, apiEndpoint, grpcDialOpts); err != nil {
		return nil, errors.Wrap(err, "register other model handler error")
	}
 
	return mux, nil
}

Finally, in your React frontend you may have stores for each service, e.g.:


import { EventEmitter } from "events";

import Swagger from "swagger-client";

import sessionStore from "./SessionStore";
import {checkStatus, errorHandler } from "./helpers";
import dispatcher from "../dispatcher";


class UserStore extends EventEmitter {
  constructor() {
    super();
    this.swagger = new Swagger("/swagger/user.swagger.json", sessionStore.getClientOpts());
  }

  create(user, password, organizations, callbackFunc) {
    this.swagger.then(client => {
      client.apis.UserService.Create({
        body: {
          organizations: organizations,
          password: password,
          user: user,
        },
      })
      .then(checkStatus)
      .then(resp => {
        this.notify("created");
        callbackFunc(resp.obj);
      })
      .catch(errorHandler);
    });
  }

  get(id, callbackFunc) {
    this.swagger.then(client => {
      client.apis.UserService.Get({
        id: id,
      })
      .then(checkStatus)
      .then(resp => {
        callbackFunc(resp.obj);
      })
      .catch(errorHandler);
    });
  }
}

And then those are used in your frontend views:

import React, { Component } from "react";
import UserForm from "./UserForm";
import UserStore from "../../stores/UserStore";

class CreateUser extends Component {
  constructor() {
    super();
    this.onSubmit = this.onSubmit.bind(this);
  }

  onSubmit(user) {
    UserStore.create(user, user.password, [], resp => {
      this.props.history.push("/users");
    });
  }

  render() {
    return(
              <UserForm
                submitLabel="Create user"
                onSubmit={this.onSubmit}
              />
    );
  }
}

export default withRouter(CreateUser);

Now, this is in no way a guide on how to do this, just a general idea of what pieces need to be coded: proto definitions, endpoints logic, serving code and frontend. Also, there’s no need to use React, swagger definitions to specify the API are unnecessary sugar, etc. I’m sure I’ve seen a bunch of guides or tutorials posted at reddit and other sites which will probably be more helpful, just look up some of those regarding the actual technologies that you want to use.

Good luck!

Thank you! But where do I put my SQL queries?

If you look at the code in the Users API, you’ll see this at the Get method:

user, err := storage.GetUser(storage.DB(), req.Id)

That’s just calling the storage package GetUser function, which in this case is located at lora-app-server/internal/storage/user.go, and looks like this:

package storage

//Some imports and vars.

//The user struct
// User represents a user to external code.
type User struct {
	ID           int64     `db:"id"`
	Username     string    `db:"username"`
	IsAdmin      bool      `db:"is_admin"`
	IsActive     bool      `db:"is_active"`
	SessionTTL   int32     `db:"session_ttl"`
	CreatedAt    time.Time `db:"created_at"`
	UpdatedAt    time.Time `db:"updated_at"`
	PasswordHash string    `db:"password_hash"`
	Email        string    `db:"email"`
	Note         string    `db:"note"`
}

const externalUserFields = "id, username, is_admin, is_active, session_ttl, created_at, updated_at, email, note"
const internalUserFields = "*"

/// GetUser returns the User for the given id.
func GetUser(db sqlx.Queryer, id int64) (User, error) {
	var user User
	err := sqlx.Get(db, &user, "select "+externalUserFields+" from \"user\" where id = $1", id)
	if err != nil {
		if err == sql.ErrNoRows {
			return user, ErrDoesNotExist
		}
		return user, errors.Wrap(err, "select error")
	}

	return user, nil
}

Again, this is just one way of doing it. In this particular project there’s an internal api package and another storage one, which gets called by the api implementation to create and consume the data (i.e., data validations, execute necessary queries, etc.). There are lots of ways to structure a project, it’s even been discussed in the forum a few times (search for project layout or similar).

1 Like

Searching for “project layout grpc” in this forum gave 1 result. This question :slight_smile:

But I do not see the forest for all trees. So I do an attempt to get a birds wiew of this. Here is my attempt:

As I interpret all this needs 3 servers? Web server, gateway server and finally, the gRPC server. Correct?

The Browser (web server)

1. GET users

The Gateway server (translate from JSON to binary and vice versa)?

6. Parse the result to JSON and return to the browser

The gRPC server

2. Connect to Postgresql
3. Listen and serve (Start server)
4. Route to an endpoint
5. Store SQL and fetch SQL data

And I suppose that the same structs are use both by gateway and gRPC server? (Generated mysteriously by proto something).

Did I get it right? Please correct me if I am wrong!

I meant Go projects in general, not gRPC. You see, it being a gRPC or and HTTP API shouldn’t change how you abstract and access your data layer, should it? In this case, if you took either one away, the internal/storage/... structure for data models would remain the same.

Actually, no, there’s just 2 servers: the gRPC and the HTTP ones. The JSON api is just the handler that you pass to the HTTP one, but it is not a server itself.

Again, the gateway is just a translation between gRPC and a JSON HTTP API: whenever you get and HTTP request in the HTTP server, it’ll internally make a call to the gRPC server through the gateway, get its answer and then send an HTTP response. So yeah, data structures are “shared”, as the logic is really only implemented once.

If you find it mysterious, then I believe you should really read about it and protocol buffers before trying to use them blindly.

So the gateway is not a server? What is it then?

This is grpc-gateway’s first paragraph:

The grpc-gateway is a plugin of the Google protocol buffers compiler protoc. It reads protobuf service definitions and generates a reverse-proxy server which translates a RESTful JSON API into gRPC. This server is generated according to the google.api.http annotations in your service definitions.

In short, grpc-gateway will create an HTTP server that will call gRPC under the hood when it gets any request (thus the gateway name). So 2 servers, not 3.

I am using golang as a web server as well. So if my webpage is on a “web-server” and “grpc-gateway will create an HTTP server”.

I count to 3 servers:

  1. The original go web server http://94.237.25.207:8080/
  2. The gateway (“grpc-gateway will create an HTTP server”)
  3. The gRPC server

Or will the gateway server replace my original server?

Well yes, but what does your “original go web server http://94.237.25.207:8080/” have to do with the gRPC + HTTP gateway ones? For what I know, you could have 10, 20 or 30 more Go web servers running that are completely irrelevant to what I’m telling you about.

Thank you for your patience. But to me this is about getting the whole picture. You made it much more clearer.

Sorry if that came out wrong, what I meant is that any other web facing applications you are running, be them in Go, Ruby, Python, etc., are totally independent of this particular web facing application you are trying to build that happens to serve gRPC and HTTP services in it, or as we put it, implements 2 servers: one gRPC, one HTTP.

Getting back to the core issue, I think a good summary is:

  1. Check project layout strategies in Go, and web projects to be more specific, in order to arrange your internal packages (such as storage).
  2. Ask yourself why you want to use gRPC in the first place. What’s wrong with a typical web app? What are your needs or requirements to consider it? Do you just want to lear about gRPC? Why do you need the HTTP API translation then?
  3. Now read and learn about protocol buffers and gRPC so that defining and understanding proto services becomes natural.
  4. Check the grpc-gateway repo and their documentation so you get comfortable with what it does and how it does it.
  5. Write some html templates or use some frontend library/framework that calls your API.
  6. Rest.
1 Like

To describe the chain of pizza ordering, you cannot ignore the customer. It is about understanding the big picture. Not a technical issue.

The “customer” part (web server) I have a good clue how to organize templates etc. On the “REST” side it is unclear how to store and retrieve tons of SQL queries…

That is a good question. My final application will rely heavily on traffic from/to the Postgresql server. And speed is the main issue. I do not have the experience yet to chose between common REST service or gRPC. Does the parsing of JSON differ in speed between these two? Any advice is appreciated.

I think that the protocol buffer generates Go structs semi automatically and it will be easier to update changes in structs with gRPC. Correct me if I am wrong.

I will…

I have played with Angular and Go as frontend. Go is at least twice as fast. So until I know better, I will stick with Go.

I see a bunch of misconceptions in your answer: Web server is a customer (of what)? The REST side is separate from the web server? Go as frontend (using gopherjs or wasm I’ll guess)? JSON parsing speed depending on unrelated protocols/technologies? I’m not making much sense of this.

It is a metaphor. Customer > Waiter > Chef. Very common language to explain to a complete newbie (me). It is a metaphor for: ask the question (order) > translate into binary > Fetch from the database. Or Web > gateway > gRPC server.

I cannot interpret your question…

No. Plain Vanilla Javascript is the goal.

Well, you are talking to a newbie. I maybe is not that good to ask qualified questions…

I see, just didn’t notice that both answers were related. In any case, I still think you are confused. Other web applications aren’t part of any big picture because they have no relation with the one we are discussing: they are not the “customer”, they don’t make any requests. Requests come from a web client and are served by the HTTP server in the application, which then makes the gRPC call and so on. There’s no middleman nor other component involved. That’s not to say there couldn’t be (you could be behind a reverse proxy such as nginx, for example), but there’s no need for one.

There’s no problem with being a “newbie”, we all started as one, I’m just kind of lost with some of the questions. For instance, how is it that you are running Go in frontend and claim it to be twice as fast as Angular? What do you mean by frontend in this case?

I agree :slight_smile:

So the gateway is integrated to the webserver? AJAX calls its own server? The same server that serves the html pages?

A newbie does not know how to ask the right question. And being professional you sometimes takes things for granted.

This is my test front end server that needs to ask REST questions. Basically serving the same stuff as Angular. Twice as fast.

ar tpl *template.Template

func init() {
	tpl = template.Must(template.ParseGlob("public/templates/*.html"))
}

func main() {
	http.HandleFunc("/", index)
	http.Handle("/img/", http.StripPrefix("/img/", http.FileServer(http.Dir("./public/img"))))
	http.Handle("/css/", http.StripPrefix("/css/", http.FileServer(http.Dir("./public/css"))))
	http.Handle("/icn/", http.StripPrefix("/icn/", http.FileServer(http.Dir("./public/icn"))))

	http.ListenAndServe(":8080", nil)
}

func index(w http.ResponseWriter, r *http.Request) {....

https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2F94.237.25.207%3A8080%2F

Ok, now I get it, I really thought you were talking about a separate application. If you check my post where I showed a bunch of example files, there was this one part that served static files for the React frontend:

// setup static file server
	r.PathPrefix("/").Handler(http.FileServer(&assetfs.AssetFS{
		Asset:     static.Asset,
		AssetDir:  static.AssetDir,
		AssetInfo: static.AssetInfo,
		Prefix:    "",
	}))

Well, there doesn’t need to be a React or any other fancy frontend really, Go html templates are perfectly fine. You just have to serve the templates instead and make ajax calls to the API endpoints from them using JS. So the HTTP server will act in 2 ways: it’ll provide server side rendered html and also expose the REST API for the ajax calls. So yeah, the calls just go to the same application.