jscharber/1. microk8s-openfaas-kafka-triggers.md

OpenFaas with kakfa triggers on microk8s

Clone the repo from GitHub

git clone [email protected]:openfaas-incubator/kafka-connector.git
Cloning into 'kafka-connector'...
Warning: Permanently added the RSA host key for IP address '192.30.255.113' to the list of known hosts.
remote: Enumerating objects: 32, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 1778 (delta 7), reused 19 (delta 5), pack-reused 1746
Receiving objects: 100% (1778/1778), 13.73 MiB | 3.58 MiB/s, done.
Resolving deltas: 100% (431/431), done.

Replace the build.sh and push.sh

Th The build.sh and push.sh files need t

Build and push the connector

The provided code does not read the TAG evnrionment correct and will break. In addition, it does not support local repositories.

cd kafka-connector/
export TAG=0.3.3
export REPO=localhost:32000
make build push

./build.sh 0.3.3
Sending build context to Docker daemon  21.07MB
Step 1/13 : FROM golang:1.10 as build
1.10: Pulling from library/golang
741437d97401: Pull complete 
34d8874714d7: Pull complete 
0a108aa26679: Pull complete 
7f0334c36886: Pull complete 
d35724ed4672: Pull complete 
c0eaf021aeaf: Pull complete 
d3d9c96611f1: Pull complete 
Digest: sha256:6d5e79878a3e4f1b30b7aa4d24fb6ee6184e905a9b172fc72593935633be4c46
Status: Downloaded newer image for golang:1.10
 ---> 6fd1f7edb6ab
Step 2/13 : RUN mkdir -p /go/src/github.com/openfaas-incubator/kafka-connector
 ---> Running in 9e05944eaeb1
Removing intermediate container 9e05944eaeb1
 ---> 0dc11a5d0a40
Step 3/13 : WORKDIR /go/src/github.com/openfaas-incubator/kafka-connector
 ---> Running in 05e10aa01991
Removing intermediate container 05e10aa01991
 ---> 35852a12b787
Step 4/13 : COPY vendor     vendor
 ---> 15194390521d
Step 5/13 : COPY main.go    .
 ---> c5d1b25a9c4d
Step 6/13 : RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*"))"
 ---> Running in 76a1827b114f
Removing intermediate container 76a1827b114f
 ---> 4f9796ebf17b
Step 7/13 : RUN go test -v ./...
 ---> Running in f0167a9def61
?   	github.com/openfaas-incubator/kafka-connector	[no test files]
Removing intermediate container f0167a9def61
 ---> 2bd69e19a389
Step 8/13 : RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags "-s -w" -installsuffix cgo -o /usr/bin/producer
 ---> Running in d375f13e735c
Removing intermediate container d375f13e735c
 ---> e33c2e3e40d8
Step 9/13 : FROM alpine:3.9 as ship
 ---> 055936d39205
Step 10/13 : RUN apk add --no-cache ca-certificates
 ---> Running in 90b08876fa34
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/1) Installing ca-certificates (20190108-r0)
Executing busybox-1.29.3-r10.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 6 MiB in 15 packages
Removing intermediate container 90b08876fa34
 ---> ef0eb5cd23bf
Step 11/13 : COPY --from=build /usr/bin/producer /usr/bin/producer
 ---> 0088bc640745
Step 12/13 : WORKDIR /root/
 ---> Running in ea51db715959
 emoving intermediate container ea51db715959
 ---> b60db60b3c9e
Step 13/13 : CMD ["/usr/bin/producer"]
 ---> Running in fa4709f47658
Removing intermediate container fa4709f47658
 ---> 770c79b53563
Successfully built 770c79b53563
Successfully tagged openfaas/kafka-connector:0.3.3
./push.sh 0.3.3
The push refers to repository [docker.io/openfaas/kafka-connector]
The push refers to repository [localhost:32000/openfaas/kafka-connector]
32386e17be3b: Pushed 
401b9e685849: Pushed 
f1b5933fe4b5: Mounted from hello-world 
0.3.3: digest: sha256:7231b8106a8ec3466b2f167502c5594f882ef953ed8cfeb51159d6b4ba3ccff5 size: 949

Get the broker container name

$ BROKER=$(kubectl get pods -n openfaas -l component=kafka-broker -o name|cut -d'/' -f2)
$ echo $BROKER
kafka-broker-8779986d-7npbg
$

Deploy figlet to listen for "faas-request" topic

faas-cli store deploy figlet --annotation topic="faas-request"

Make sure all the pods are running

You can use kubens to only look in the openfaas namespace.

kubens
container-registry
default
kube-node-lease
kube-public
kube-system
**openfaas**
openfaas-fn
jscharber@ubuntu:~$ kubectl get po
NAME                                 READY   STATUS    RESTARTS   AGE
alertmanager-6d98dfd8f-zvmmk         1/1     Running   2          46h
basic-auth-plugin-7f858fdcc9-rhzv8   1/1     Running   2          46h
faas-idler-6b68895d78-r5ddd          1/1     Running   35         46h
gateway-6874f578f8-np6jk             2/2     Running   28         46h
kafka-broker-8779986d-7npbg          1/1     Running   155        17h
kafka-connector-587968d57-v496q      1/1     Running   7          17h
nats-79ffb4bb68-jcbpj                1/1     Running   2          46h
prometheus-6667dc86f4-tfnl5          1/1     Running   2          46h
queue-worker-f94f87444-6bkqt         1/1     Running   32         46h
zookeeper-74f8767679-7kfzn           1/1     Running   1          4h

Use the broker to publish messages

At the carrot, type a line of text to send as a message. Hit CTL-C when you are done.

kubectl exec -n openfaas -t -i $BROKER -- /opt/kafka_2.12-0.11.0.1/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic faas-request
>test 1
>test 2
>hello kafka
>

Use the UI or CLI to verify the function fired

UI, or use faas-cli list

faas-cli list 
Function                      	Invocations    	Replicas
face-detect-opencv            	0              	1    
figlet                        	3              	1    
haveibeenpwned                	0              	1    
hello-world                   	0              	1    
qrcode-go                     	0              	1    
test1                         	0              	1

	#!/bin/sh
	set -e

	export dockerfile="Dockerfile"
	export arch=$(uname -m)

	if [ "$arch" = "armv7l" ]; then
	dockerfile="Dockerfile.armhf"
	TAG="latest-armhf-dev"
	fi

	if [ "$1" ]; then
	TAG=$1
	if [ "$arch" = "armv7l" ]; then
	TAG="$1-armhf"
	fi
	fi

	if [ -z "$NAMESPACE" ]; then
	NAMESPACE="openfaas"
	fi

	if [ -z "$TAG" ]; then
	TAG="latest"
	fi

	if [ -z "$REPO" ]; then
	docker build -t $NAMESPACE/kafka-connector:$TAG . -f $dockerfile --no-cache
	else
	docker build -t $REPO/$NAMESPACE/kafka-connector:$TAG . -f $dockerfile --no-cache
	fi

	#(cd yaml && docker service rm kafka_connector ; docker stack deploy kafka -c connector-swarm.yml)
	~

	#!/bin/sh
	set -e

	if [ -z "$NAMESPACE" ]; then
	NAMESPACE="openfaas"
	fi

	if [ -z "$REPO" ]; then
	docker push $NAMESPACE/kafka-connector:$TAG
	else
	docker push $REPO/$NAMESPACE/kafka-connector:$TAG