Target
Develop simple web application using Bigtable and golang
Post data and Get data using postman
Final source codes are here https://github.com/DJ110/worldt (my git hub repo)
For simple sample application (not web application) please check GCP – BigTable Hello world golang
Steps
- Develop an application
- Create an instance from console
- Create a table and column family from console
- Build and Run application
- POST data from postman
- GET data (rowkey, and all versions) from postman
Develop an application
This sample application saves World City temperature every 1 hour, and save into BigTable, support Insert and get latest (last) log data get all data in the target day. We expect to create following 3 end points
- /update (POST) : Post parameters(day, city) to insert temperature into Bigtable
- /get (GET) : Get latest(last log) temperature using city and the target day
- /getall (GET) : Get all temerature using city and the target day (return array)
We did not keep hour info, Bigtable has version when the data is inserted. We expect every 1 hour, insert data, so have 24 data in one data, one data one version, one hour
Prepare the project (import library)
Let’s prepare project and libraries
mkdir worldt
cd worldt
go mod init worldt
go get cloud.google.com/go/bigtable
go get github.com/labstack/echo/v4
I use echo for web application development for golang
Next is development, just shows final codes
package main
import (
"bytes"
"context"
"encoding/binary"
"flag"
"fmt"
"log"
"net/http"
"strconv"
"cloud.google.com/go/bigtable"
"github.com/labstack/echo/v4"
)
type TempItem struct {
City string
Day string
Hour string
Temperature int16
}
var project *string
var instance *string
const tableName = "citytemp"
const columnFamily = "temperature"
const columnQualifier = "temperature"
func main() {
project = flag.String("project", "", "The Google Cloud Platform project ID. Required.")
instance = flag.String("instance", "", "The Google Cloud Bigtable instance ID. Required.")
flag.Parse()
e := echo.New()
e.GET("/", func(c echo.Context) error {
return c.String(http.StatusOK, "Hello World!")
})
e.POST("/update", postTemp)
e.GET("/get", getLatestTempDay)
e.GET("/getall", getTempDayAll)
e.Logger.Fatal(e.Start(":1323"))
}
func getBigTableClient() (*bigtable.Client, context.Context) {
ctx := context.Background()
client, err := bigtable.NewClient(ctx, *project, *instance)
if err != nil {
log.Fatalf("Could not create admin client: %v", err)
}
return client, ctx
}
func getTempDayAll(c echo.Context) error {
city := c.QueryParam("city")
day := c.QueryParam("day")
// Get Temp data from BigTable
client, ctx := getBigTableClient()
defer client.Close()
tbl := client.Open(tableName)
rowKey := city + "#" + day // tokyo#2024-07-12
var s []uint64
// Read the row
err := tbl.ReadRows(ctx, bigtable.PrefixRange(rowKey), func(row bigtable.Row) bool {
// Iterate over the column families
version := 0
for _, ris := range row[columnFamily] {
if ris.Column == fmt.Sprintf("%s:%s", columnFamily, columnQualifier) {
// Print the version timestamp and the cell value
version++
data := binary.BigEndian.Uint64(ris.Value)
s = append(s, data)
fmt.Printf("Version %d, Timestamp: %v, Value: %d\n", version, ris.Timestamp, data)
}
}
return true
}, bigtable.RowFilter(bigtable.ColumnFilter(columnQualifier)))
if err != nil {
log.Fatalf("Could not read row: %v", err)
return c.String(http.StatusBadRequest, "Bad request city and day")
}
return c.JSON(http.StatusOK, s)
}
func getLatestTempDay(c echo.Context) error {
city := c.QueryParam("city")
day := c.QueryParam("day")
// Get Temp data from BigTable
client, ctx := getBigTableClient()
defer client.Close()
tbl := client.Open(tableName)
rowKey := city + "#" + day // tokyo#2024-07-12
row, err := tbl.ReadRow(ctx, rowKey, bigtable.RowFilter(bigtable.ColumnFilter(columnQualifier)))
if err != nil {
log.Fatalf("Could not read row with key %s: %v", rowKey, err)
return c.String(http.StatusBadRequest, "Bad request city and day")
}
data := binary.BigEndian.Uint64(row[columnFamily][0].Value)
log.Printf("\t%s = %s\n", rowKey, strconv.FormatUint(data, 10))
return c.JSON(http.StatusOK, strconv.FormatUint(data, 10))
}
func postTemp(c echo.Context) error {
temp := new(TempItem)
if err := c.Bind(temp); err != nil {
return c.String(http.StatusBadRequest, "bad request")
}
// Save data into bigtable
client, ctx := getBigTableClient()
defer client.Close()
// Create a mutation
rowKey := temp.City + "#" + temp.Day // tokyo#2024-07-12
mut := bigtable.NewMutation()
buf := new(bytes.Buffer)
binary.Write(buf, binary.BigEndian, uint64(temp.Temperature))
mut.Set(columnFamily, columnQualifier, bigtable.Now(), buf.Bytes())
// Apply the mutation
tbl := client.Open(tableName)
err := tbl.Apply(ctx, rowKey, mut)
if err != nil {
log.Fatalf("Could not apply mutation: %v", err)
return c.String(http.StatusBadRequest, "failed to write into bigtable")
}
println(rowKey)
println(uint64(temp.Temperature))
return c.JSON(http.StatusOK, temp)
}
Create an instance from console
This step is almost same as GCP – BigTable Hello world golang Please check how to create a cluster / instance. In this time, explained next step
Bigtable is not super cheap, minimum instance, SSD, around 500SGD per month, once finished your usage, recommend to remove if you won’t run as an actual application.
Create a table
Different from last sample, we need Table (I did not add table add codes into my sample)
From CCP console, “テーブルを作成” Create a table, and set table name, That’s it.
Next is column family. Need columnfamily to insert temperature value. I set Garbage collection (expire data) in 2 days.
We can change Garbage collection rule later. Above example is to switch version base policy. I set 24 versions (1 day temp data)
It’s ready to test
Build and Run application
Build and run are also same step as last blog
go build
gcloud auth application-default login
./worldt --project (GCP Project ID) --instance (Big Table Instance name)
Please use your GCP project ID and Bigtable instance ID (Don’t forget to access gcloud auth if you want to run on the local machine)
Now running application under localhost port 1323
POST data from postman
Let’s insert some data
Insert tokyo#2024-07-21 as rowkey, temperature 33 as value.
Let’s check from GCP console, Bigtable Studio, but cannot see number properly. (I tested using string, in this case, can see number as string, properly.
Try to get all data, can see 33 as array. Maybe, Bigtable Studio has something wrong or does not support to show int (because inserted value as binary)
Insert one more data
Can get 2 data
Insert different city
Can see 2 rows
Get from GET API properly
コメント