GCP – Bigtable with Web Application

GCP

Target

Develop simple web application using Bigtable and golang

Post data and Get data using postman

Final source codes are here https://github.com/DJ110/worldt (my git hub repo)

For simple sample application (not web application) please check GCP – BigTable Hello world golang

Steps

  1. Develop an application
  2. Create an instance from console
  3. Create a table and column family from console
  4. Build and Run application
  5. POST data from postman
  6. GET data (rowkey, and all versions) from postman

Develop an application

This sample application saves World City temperature every 1 hour, and save into BigTable, support Insert and get latest (last) log data get all data in the target day. We expect to create following 3 end points

  • /update (POST) : Post parameters(day, city) to insert temperature into Bigtable
  • /get (GET) : Get latest(last log) temperature using city and the target day
  • /getall (GET) : Get all temerature using city and the target day (return array)

We did not keep hour info, Bigtable has version when the data is inserted. We expect every 1 hour, insert data, so have 24 data in one data, one data one version, one hour

Prepare the project (import library)

Let’s prepare project and libraries

mkdir worldt
cd worldt
go mod init worldt
go get cloud.google.com/go/bigtable 
go get github.com/labstack/echo/v4 

I use echo for web application development for golang

Next is development, just shows final codes

package main

import (
	"bytes"
	"context"
	"encoding/binary"
	"flag"
	"fmt"
	"log"
	"net/http"
	"strconv"

	"cloud.google.com/go/bigtable"
	"github.com/labstack/echo/v4"
)

type TempItem struct {
	City        string
	Day         string
	Hour        string
	Temperature int16
}

var project *string
var instance *string

const tableName = "citytemp"
const columnFamily = "temperature"
const columnQualifier = "temperature"

func main() {
	project = flag.String("project", "", "The Google Cloud Platform project ID. Required.")
	instance = flag.String("instance", "", "The Google Cloud Bigtable instance ID. Required.")
	flag.Parse()
	e := echo.New()
	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello World!")
	})
	e.POST("/update", postTemp)
	e.GET("/get", getLatestTempDay)
	e.GET("/getall", getTempDayAll)
	e.Logger.Fatal(e.Start(":1323"))
}

func getBigTableClient() (*bigtable.Client, context.Context) {
	ctx := context.Background()

	client, err := bigtable.NewClient(ctx, *project, *instance)
	if err != nil {
		log.Fatalf("Could not create admin client: %v", err)
	}
	return client, ctx
}

func getTempDayAll(c echo.Context) error {
	city := c.QueryParam("city")
	day := c.QueryParam("day")
	// Get Temp data from BigTable
	client, ctx := getBigTableClient()
	defer client.Close()
	tbl := client.Open(tableName)
	rowKey := city + "#" + day // tokyo#2024-07-12
	var s []uint64
	// Read the row
	err := tbl.ReadRows(ctx, bigtable.PrefixRange(rowKey), func(row bigtable.Row) bool {
		// Iterate over the column families
		version := 0
		for _, ris := range row[columnFamily] {
			if ris.Column == fmt.Sprintf("%s:%s", columnFamily, columnQualifier) {
				// Print the version timestamp and the cell value
				version++
				data := binary.BigEndian.Uint64(ris.Value)
				s = append(s, data)
				fmt.Printf("Version %d, Timestamp: %v, Value: %d\n", version, ris.Timestamp, data)
			}
		}
		return true
	}, bigtable.RowFilter(bigtable.ColumnFilter(columnQualifier)))
	if err != nil {
		log.Fatalf("Could not read row: %v", err)
		return c.String(http.StatusBadRequest, "Bad request city and day")
	}
	return c.JSON(http.StatusOK, s)
}

func getLatestTempDay(c echo.Context) error {
	city := c.QueryParam("city")
	day := c.QueryParam("day")

	// Get Temp data from BigTable
	client, ctx := getBigTableClient()
	defer client.Close()
	tbl := client.Open(tableName)
	rowKey := city + "#" + day // tokyo#2024-07-12
	row, err := tbl.ReadRow(ctx, rowKey, bigtable.RowFilter(bigtable.ColumnFilter(columnQualifier)))
	if err != nil {
		log.Fatalf("Could not read row with key %s: %v", rowKey, err)
		return c.String(http.StatusBadRequest, "Bad request city and day")
	}
	data := binary.BigEndian.Uint64(row[columnFamily][0].Value)
	log.Printf("\t%s = %s\n", rowKey, strconv.FormatUint(data, 10))

	return c.JSON(http.StatusOK, strconv.FormatUint(data, 10))
}

func postTemp(c echo.Context) error {
	temp := new(TempItem)
	if err := c.Bind(temp); err != nil {
		return c.String(http.StatusBadRequest, "bad request")
	}
	// Save data into bigtable
	client, ctx := getBigTableClient()
	defer client.Close()

	// Create a mutation
	rowKey := temp.City + "#" + temp.Day // tokyo#2024-07-12

	mut := bigtable.NewMutation()
	buf := new(bytes.Buffer)
	binary.Write(buf, binary.BigEndian, uint64(temp.Temperature))
	mut.Set(columnFamily, columnQualifier, bigtable.Now(), buf.Bytes())

	// Apply the mutation
	tbl := client.Open(tableName)
	err := tbl.Apply(ctx, rowKey, mut)
	if err != nil {
		log.Fatalf("Could not apply mutation: %v", err)
		return c.String(http.StatusBadRequest, "failed to write into bigtable")
	}
	println(rowKey)
	println(uint64(temp.Temperature))
	return c.JSON(http.StatusOK, temp)
}

Create an instance from console

This step is almost same as GCP – BigTable Hello world golang Please check how to create a cluster / instance. In this time, explained next step

Bigtable is not super cheap, minimum instance, SSD, around 500SGD per month, once finished your usage, recommend to remove if you won’t run as an actual application.

Create a table

Different from last sample, we need Table (I did not add table add codes into my sample)

From CCP console, “テーブルを作成” Create a table, and set table name, That’s it.

Next is column family. Need columnfamily to insert temperature value. I set Garbage collection (expire data) in 2 days.

We can change Garbage collection rule later. Above example is to switch version base policy. I set 24 versions (1 day temp data)

It’s ready to test

Build and Run application

Build and run are also same step as last blog

go build
gcloud auth application-default login
./worldt --project (GCP Project ID) --instance (Big Table Instance name)

Please use your GCP project ID and Bigtable instance ID (Don’t forget to access gcloud auth if you want to run on the local machine)

Now running application under localhost port 1323

POST data from postman

Let’s insert some data

Insert tokyo#2024-07-21 as rowkey, temperature 33 as value.

Let’s check from GCP console, Bigtable Studio, but cannot see number properly. (I tested using string, in this case, can see number as string, properly.

Try to get all data, can see 33 as array. Maybe, Bigtable Studio has something wrong or does not support to show int (because inserted value as binary)

Insert one more data

Can get 2 data

Insert different city

Can see 2 rows

Get from GET API properly

コメント

タイトルとURLをコピーしました