Publish Minio events via NATS

Introduction

Minio server supports Amazon S3 compatible bucket event notification for following targets AMQP, Elasticsearch , Redis, nats.io, PostgreSQL and Apache Kafka. Part 4 of this blog series covers sending bucket notifications using NATS.

Steps to Configure NATS

I used Ubuntu Linux 16.04, gnatsd and nats package. It requires go package installed, for installation steps follow https://docs.minio.io/docs/how-to-install-golang

$ go get github.com/nats-io/gnatsd
$ go get github.com/nats-io/nats


Start NATS server.

$ gnatsd --user yourusername --pass yoursecret
[1510] 2016/10/12 06:14:48.706544 [INF] Starting nats-server version 0.9.4
[1510] 2016/10/12 06:14:48.707441 [INF] Listening for client connections on 0.0.0.0:4222
[1510] 2016/10/12 06:14:48.708332 [INF] Server is ready


Steps to configure Minio

Latest Minio server binary can be downloaded from https://minio.io/downloads/

$ wget https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
In this case “myphotos” is used as my data directory for Minio server.
$ ./minio server myphotos
Endpoint: http://10.1.10.150:9000 http://127.0.0.1:9000
AccessKey: 7I6R5G576YI641GS9J9F
SecretKey: SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c
Region: us-east-1
...
...


The default location of Minio server configuration file is~/.minio/config.json. Update the NATS configuration block in ‘config.json’ as follows

"nats": {
"1": {
"enable": true,
"address": "0.0.0.0:4222",
"subject": "bucketevents",
"username": "yourusername",
"password": "yoursecret",
"token": "",
"secure": false,
"pingInterval": 0
}
},


Restart the Minio server to reflect config changes made above. “bucketevents” is the subject used by NATS in this example.

Enable bucket notification using Minio client

Step 1: Download and install Minio client

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc


Step 2: Add Minio server host alias information

Configure Minio client with access and secret keys pointing to the Minio server.

$ ./mc config host add myminio http://localhost:9000 7I6R5G576YI641GS9J9F SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c


Step 3: Setup bucket notification

In this example we will enable bucket events only when JPEG images are uploaded or deleted from ‘images’ bucket on ‘myminio’ server. Here ARN value is arn:minio:sqs:us-east-1:1:nats. To understand more about ARN please follow AWS ARN documentation.

$ ./mc mb myminio/images
$ ./mc events add myminio/images arn:minio:sqs:us-east-1:1:nats --suffix .jpg
$ ./mc events list myminio/images
arn:minio:sqs:us-east-1:1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”


Step 4: Testing on NATS

Using this program below we can log the bucket notification added to NATS.

package main

// Import Go and NATS packages
import (
"runtime"
"log"
"github.com/nats-io/nats"
)

func main() {

// Create server connection
natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222")
log.Println("Connected")

// Subscribe to subject
log.Printf("Subscribing to subject 'bucketevents'\n")
natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) {

// Handle the message
log.Printf("Received message '%s\n", string(msg.Data) + "'")
})

// Keep the connection alive
runtime.Goexit()
}
$ go run nats.go
2016/10/12 06:39:18 Connected
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'


Open another terminal and upload a JPEG image into “images” bucket.

$ ./mc cp myphoto.jpg myminio/images


Below terminal running nats.go program prints event notification on console.

$ go run nats.go
2016/10/12 06:51:26 Connected
2016/10/12 06:51:26 Subscribing to subject 'bucketevents'
2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"}



For further questions and comments join our Slack chat at: https://slack.minio.io