Writing a microservice in Golang which communicates over gRPC

This is a guest post by Bitbucket user Milan Savaliya, Senior software engineer at GSLab.

This is the first installment of my 3 part long series on:

gRPC powered micro-service in golang
Protecting golang micro-service with keycloak
Writing java client to talk with our golang micro-service.

Let's start!. 🙂

What is gRPC: gRPC stands for Google Remote Procedure Call. It's a remote communication protocol created by Google which lets different services communicate with each other easily and efficiently. It offers synchronous and asynchronous communication to/from services. To learn more about gRPC, visit https://gRPC.io/

gRPC is best suitable for internal communications. It makes client calls much cleaner and we don't need to worry about serialization, type safety and all those things as gRPC does this for us.

gPRC uses protobuf, a type-safe binary transfer format that is designed for efficient network communications. To learn more about protobuf, visit this link.

Performance benchmark results indicate that gRPC is a far better choice than http/http2 if performance and native call like experience is what the developer wants. You can find out more from the discussion here.

Building a Microservice in Golang

Let's use an example of the Bitbucket repository creation flow: user signs up for a Bitbucket account > selects the plan > creates repository.

We want to create a service designed to handle various operations on the repository. When the user creates a repository, our service will be called to serve that operation. If user is modifying repository settings, our service will be called to serve that operation. Basically anything related to repository management will be served by our service.

We've chosen Golang (also referred to as Go) as our programming language for this service, gRPC as the communication protocol for other services to talk with our service and keycloak to protect our service using the proven OpenId identity layer on OAuth 2.0 protocol.

Create the Message

To do this, first we need to create a simple entity representation in gRPC called message. A message in gRPC terminology is something which can be used as a message(message is defined using protobuf syntax) to one service from another service. You can imagine a pigeon from one service carrying a message called "Winter has come" to another service and that service is consuming that message to carry out the contextual operation.

Now, in the above example, the service which sent the pigeon is the gRPC client, "Winter has come" is our message and service consuming the message is the gRPC server listening to that message. Nice thing about a message is that it can be transferred back and forth.

message Repository {
	int64 id  = 1;
	string name = 2;
	int64 userId = 3;
	bool isPrivate = 4;
}

Define the gRPC Service

Now that we have created a Repository named message to be used in communication, the next step is to define the gRPC service.

service RepositoryService {
	//For now we'll try to implement "insert" operation.
	rpc add (Repository) returns (AddRepositoryResponse);
}
 
message AddRepositoryResponse {
	Repository addedRepository = 1;
	Error error = 2;
}
message Error {
	string code = 1;
	string message = 2;
}

Here, we are telling the gRPC compiler that the piece of code starting with the "service" keyword should be treated as a gRPC service. The method preceded with the "rpc" keyword indicates that it's a Remote Procedure Call and the compiler should generate appropriate stubs for client and server runtime.

We've also defined 2 more messages to tell our pigeon to return a success response or an error response after performing the operation.

Create folder structure for the Golang Service

I am assuming you have Go runtime set up. If you don't, please follow the steps in their official documentation at https://golang.org/doc/install#install

We'll also be using dep as our dependency management tool for our project. Dep is a mature solution for managing external dependencies in a golang project. We use dep because Go module support is not yet officially released.

If you're a windows user, put the path to the dep installation in your environment's PATH variable. This makes it easier as you'll be able to use it without specifying the full path to the executable file.

Once Go runtime is installed, follow the below steps.

Create directory called "bitbucket-repository-management-service" at $GOPATH/src

Create these subpackages into that directory. The package layout I'm proposing is according to go standards for package layout.

build
   → proto-gen.bat
   → proto-gen.sh
cmd
   → gRPC
           → server
               → main.go
           → client
               → main.go
Internal
   → gRPC
   → proto-files
           → domain
                   → repository.proto
           → service
                   → repository-service.proto
pkg

Navigate to the root directory of project and execute below command
1. If windows, "dep.exe init"
2. If linux, "dep init"
The above command will create a folder called "vendor" along with "Gopkg.lock" and "Gopkg.toml" file. These two files are important to manage different dependencies of our project.
Our next step would be to place our proto files into the "internal" folder as these files are strictly bound to our application. Later, we'll create a separate repository for this if we want to use the same files for different services in different programing languages. But for simplicity, we'll put those in the same directory for now.
Create folder called "proto-files" in the "internal" package as shown in the below image.
Inside the "proto-files" folder, create two sub folders
1. domain
2. service

So final project's package layout will look like this.

Note: I am using VS Code for demonstration. Any other code editor or IDE is fine.

Next we'll paste the below code in the file called "repository.proto". This code defines a skeleton message, written in protobuf syntax, which will be exchanged between grpc client and server.

syntax = "proto3";

package domain;

option go_package = "bitbucket-repository-management-service/internal/gRPC/domain";

message Repository {
	int64 id  = 1;
	string name = 2;
	int64 userId = 3;
	bool isPrivate = 4;
}

After that, we'll paste below code in the file called "repository-service.proto". This code defines the grpc service definition. It defines the operation our grpc server will support along with possible input and return types.

syntax = "proto3";

package service;

option go_package = "bitbucket-repository-management-service/internal/gRPC/service";

import "bitbucket-repository-management-service/internal/proto-files/domain/repository.proto";

//RepositoryService Definition
service RepositoryService {
	rpc add (domain.Repository) returns (AddRepositoryResponse);
}
 
message AddRepositoryResponse {
	domain.Repository addedRepository = 1;
	Error error = 2;
}
message Error {
	string code = 1;
	string message = 2;
}

**Install gRPC compiler — The Great "*PROTOC"***

Without a gRPC compiler installed in our system, we won't be able to generate stubs.

To install the protoc compiler,

navigate to this link,
Select latest release tag, make sure that you select a stable release.
Download the appropriate binary for your operating system.
Once downloaded, extract it to the location which is being scanned by the path variable of your operating system.

For windows,
"Environment Variables" → edit Path variable → add your folder location where you've extracted the downloaded binary.

For Linux family,
Append to "/etc/profile" file to add extracted folder to PATH like this:

export PATH=$PATH:<<folder-location-to-protoc-binaries>>

The above steps will install the protoc compiler which we'll use in a bit. There is one more thing we need to make available into our system → "Go bindings for protocol-buffers".

Install Go bindings & Generate Stubs

Without Go bindings, our stubs are useless. Go bindings provides helper structs, interfaces, and functions which we can use to register our gRPC service, marshal and unmarshal binary messages etc.

To do that, we first need to add very simple Go code into our server.go file because by default dep (our dependency management tool) doesn't download any libraries if there is no go code in the project.

To satisfy our beloved dep's requirement, we'll put some very basic go code in our cmd → gRPC → server → main.go file

package main

import "fmt"

func main() {
    fmt.Println("gRPC In Action!")
}

Now we are all good to install go bindings for proto buffers. We'll execute the below command to install it.

Linux

dep ensure --add google.golang.org/gRPC github.com/golang/protobuf/protoc-gen-go

Windows

dep.exe ensure -add google.golang.org/gRPC github.com/golang/protobuf/protoc-gen-go

The above command will download go bindings into the "vendor" folder.

Now it's time to generate stubs.
If you are on windows, execute this command.

protoc.exe -I $env:GOPATH\src --go_out=$env:GOPATH\src $env:GOPATH\src\bitbucket-repository-management-service\internal\proto-files\domain\repository.proto

protoc.exe -I $env:GOPATH\src --go_out=plugins=gRPC:$env:GOPATH\src $env:GOPATH\src\bitbucket-repository-management-service\internal\proto-files\service\repository-service.proto

If you are on linux, execute this command.

protoc -I $GOPATH/src --go_out=$GOPATH/src $GOPATH/src/bitbucket-repository-management-service/internal/proto-files/domain/repository.proto

protoc -I $GOPATH/src --go_out=plugins=gRPC:$GOPATH/src $GOPATH/src/bitbucket-repository-management-service/internal/proto-files/service/repository-service.proto

The above command will generate stubs in the below marked sub-directories.

Implement gRPC Service Stub

Next, to write our own implementation,

We will create a package called "impl" in the "internal → gRPC" directory.
We will create a struct called RepositoryServiceGrpcImpl,
Make sure that our struct implements all the gRPC stub methods.

So as we know that our gRPC service has a method called add. We wrote its definition in our proto file earlier in the process.

rpc add (domain.Repository) returns (AddRepositoryResponse);

To Implement it's service contract we'll start by declaring a struct which will be responsible for RepositoryService's implementation.

package impl

import (
	"bitbucket-repository-management-service/internal/gRPC/domain"
	"bitbucket-repository-management-service/internal/gRPC/service"
	"context"
	"log"
)

//RepositoryServiceGrpcImpl is a implementation of RepositoryService Grpc Service.
type RepositoryServiceGrpcImpl struct {
}

//NewRepositoryServiceGrpcImpl returns the pointer to the implementation.
func NewRepositoryServiceGrpcImpl() *RepositoryServiceGrpcImpl {
	return &RepositoryServiceGrpcImpl{}
}

//Add function implementation of gRPC Service.
func (serviceImpl *RepositoryServiceGrpcImpl) Add(ctx context.Context, in *domain.Repository) (*service.AddRepositoryResponse, error) {
	log.Println("Received request for adding repository with id " + strconv.FormatInt(in.Id, 10))

	//Logic to persist to database or storage.
	log.Println("Repository persisted to the storage")

	return &service.AddRepositoryResponse{
		AddedRepository: in,
		Error:           nil,
	}, nil
}

Now it's time to write server configurations, port configurations and a minimal test-client which we can execute to verify the overall flow. 🙂

Let's start with gRPC server first.

Configure gRPC Server

We'll create an instance of a `RepositoryServiceGrpcImpl`
`repositoryServiceImpl := impl.NewRepositoryServiceGrpcImpl()`

We'll create net.Listener

func getNetListener(port uint) net.Listener {
	lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
		panic(fmt.Sprintf("failed to listen: %v", err))
	}

	return lis
}

We'll create gRPC server.

gRPCServer := gRPC.NewServer()

We'll register our service implementation to gRPC server.

service.RegisterRepositoryServiceServer(gRPCServer, repositoryServiceImpl)

We'll bind net.Listener and gRPC server to let it communicate from specified port.

// start the server
	if err := gRPCServer.Serve(netListener); err != nil {
		log.Fatalf("failed to serve: %s", err)
	}

If we wire up all the things, we'll have something like this:

package main

import (
    "bitbucket-repository-management-service/internal/gRPC/impl"
    "bitbucket-repository-management-service/internal/gRPC/service"
    "fmt"
    "log"
    "net"

    "google.golang.org/gRPC"
)

func main() {
    netListener := getNetListener(7000)
    gRPCServer := gRPC.NewServer()

    repositoryServiceImpl := impl.NewRepositoryServiceGrpcImpl()
    service.RegisterRepositoryServiceServer(gRPCServer, repositoryServiceImpl)

    // start the server
    if err := gRPCServer.Serve(netListener); err != nil {
        log.Fatalf("failed to serve: %s", err)
    }

}

func getNetListener(port uint) net.Listener {
    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
        panic(fmt.Sprintf("failed to listen: %v", err))
    }

    return lis
}

Configure gRPC Client

To configure the client:

We'll create connection to gRPC server.

serverAddress := "localhost:7000"
conn, e := gRPC.Dial(serverAddress, gRPC.WithInsecure())

We'll pass that connection to gRPC client.

client := service.NewRepositoryServiceClient(conn)

We'll call our gRPC method. 🙂

client.Add(context.Background(), &repositoryModel);

If we wire up things here too, it'll be like:

package main

import (
	"bitbucket-repository-management-service/internal/gRPC/domain"
	"bitbucket-repository-management-service/internal/gRPC/service"
	"context"
	"fmt"

	"google.golang.org/gRPC"
)

func main() {
	serverAddress := "localhost:7000"

	conn, e := gRPC.Dial(serverAddress, gRPC.WithInsecure())

	if e != nil {
		panic(e)
	}
	defer conn.Close()

	client := service.NewRepositoryServiceClient(conn)

	for i := range [10]int{} {
		repositoryModel := domain.Repository{
			Id:        int64(i),
			IsPrivate: true,
			Name:      string("Grpc-Demo"),
			UserId:    1245,
		}

		if responseMessage, e := client.Add(context.Background(), &repositoryModel); e != nil {
			panic(fmt.Sprintf("Was not able to insert Record %v", e))
		} else {
			fmt.Println("Record Inserted..")
			fmt.Println(responseMessage)
			fmt.Println("=============================")
		}
	}
}

Test the flow

To run the gRPC server, execute below command from your project's root directory.

go run .\cmd\gRPC\server\main.go

To run client, ( use new terminal window for this please. 🙂 )

go run .\cmd\gRPC\client\main.go

You should see something like this on client's standard output stream.

And on the server side,

For now we are not persisting any of the requests to the proper storage engine.

Summary

We've created a minimal flow with best practices in mind for gRPC request to response. On one end, our gRPC server is listening and serving requests and on the other end the client is sending requests to the server. We are using our custom messages to pass to/from the gRPC server/client.

Our implementation above is synchronous. We have not yet addressed asynchronous and streaming of responses from the server. I'll try to do that in my next post where I will talk about various features of gRPC protocols.

Till that time, Namaste. 🙏

You'll find all the code hosted at this bitbucket repository.

Author bio: Milan Savaliya: Milan is passionate problem solver. He currently works as a Senior software engineer at GSLab. He is also a Yoga practitioner and teacher for Isha Yoga Foundation. Connect with Milan on Twitter @MilanSavaliyaM9

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Building a Microservice in Golang

Create the Message

Define the gRPC Service

Create folder structure for the Golang Service

Install gRPC compiler — The Great "PROTOC"

Install Go bindings & Generate Stubs

Implement gRPC Service Stub

Configure gRPC Server

Summary

How to integrate security checks into your deployment workflow

Ensure your API contracts are secure by adding 42Crunch to your pipeline

Introducing Workspaces to Bitbucket Cloud

**Install gRPC compiler — The Great "*PROTOC"***