Steps/actions needed to be done to deploy loki logging backend on staging aks cluster.

README.md

Deploying Loki Stack on AKS staging cluster

How to deploy the stack

This process assume you already have:

• Kubectl and helm tools installed
• Assuming that all files are in the same directory you can execute:

make loki-install

This command will do:

• initialize the environment variables at variables.sh
• Execute the setup-loki-fluentbit.sh to do the following actions:
  • Set or enable the Caffeine - Flood Risk Tool Subscription.
  • Connect to the rhd-caff-stag-aks-xxxx K8s cluster.
  • Add the Grafana helm chart repo and update local helm index.
  • Create the monitoring namespace.
  • Install Loki stack via Helm chart.

Modyfying Fluentbit configmap collector agent.

As long the loki stack is deployed, the configmap/loki-fluent-bit-loki resource should be modified to allow the fluentbit collect logs at the /var/log/containers/*.log directory over all aks nodes (it is being deployed as a Daemonset) A kubernetes filter was added to check K8s metadata like labels, namespaces, containers etc. in order to get the calculationId label as an index label to filter logs based on it.

Then, the fluentbit-configmap.yaml file should be executed:

kubectl apply fluentbit-configmap.yaml

After this, will be necessary to restart the fluentbit service which is running as a daemonset, so is neccesary to apply:

kubectl rollout restart daemonset loki-fluent-bit-loki -n observability
daemonset.apps/loki-fluent-bit-loki restarted

---

NOTE:

Perhaps before to execute make loki-install you can give execution permissions to variables.sh and setup-loki-fluentbit.sh files:

chmod +x ./variables.sh
chmod +x ./setup-loki-fluentbit.sh

fluentbit-configmap.yaml

apiVersion: v1
data:
  fluent-bit.conf: |-
    [SERVICE]
        HTTP_Server    On
        HTTP_Listen    0.0.0.0
        HTTP_PORT      2020
        Flush          1
        Daemon         Off
        Log_Level      warn
        Parsers_File   parsers.conf
    [INPUT]
        Name           tail
        Tag            kube.*
        Path           /var/log/containers/*.log
        Parser         docker
        DB             /run/fluent-bit/flb_kube.db
        Mem_Buf_Limit  1000MB
    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc:443
        Merge_Log On
        K8S-Logging.Exclude Off
        K8S-Logging.Parser Off
    [Output]
        Name grafana-loki
        Match *
        Url http://loki:3100/loki/api/v1/push
        TenantID ""
        BatchWait 1
        BatchSize 1048576
        Labels {job="fluent-bit"}
        RemoveKeys kubernetes,stream
        AutoKubernetesLabels false
        LabelMapPath /fluent-bit/etc/labelmap.json
        LineFormat json
        LogLevel warn
  labelmap.json: |-
    {
      "kubernetes": {
        "container_name": "container",
        "host": "node",
        "labels": {
          "app": "app",
          "release": "release",
          "calculationId": "calculationId"
        },
       "namespace_name": "namespace",
       "pod_name": "instance"
      },
      "stream": "stream"
    }
  parsers.conf: |-
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L 
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: loki
    meta.helm.sh/release-namespace: observability
  creationTimestamp: "2022-01-17T21:57:25Z"
  labels:
    app: fluent-bit-loki
    app.kubernetes.io/managed-by: Helm
    chart: fluent-bit-2.3.0
    heritage: Helm
    release: loki
  name: loki-fluent-bit-loki
  namespace: observability 

grafana-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: lets-encrypt-prod # letsencrypt-staging
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: grafana-ingress
  namespace: monitoring
spec:
  tls:
    - hosts:
      - grafana-fr-staging.rhdhv.io
      secretName: tls-grafana
  rules:
    - host: grafana-fr-staging.rhdhv.io
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: loki-grafana
              port:
                number: 80

Makefile

loki-install:
          sh ./variables.sh
          sh ./setup-loki-fluentbit.sh

setup-loki-fluentbit.sh

#! /bin/bash


# Exit immediately if a command exits with a non-zero status.
# Print commands and arguments as they are being executed.
set -xe

# Setting up Caffeine - Flood Risk Tool Subscription
az account set --subscription $ARM_SUBSCRIPTION_ID 

# Connecting to aks cluster
az aks get-credentials --resource-group $RESOURCE_GROUP_NAME  --name $AKS_CLUSTER_NAME


# Adding Grafana helm chart repo
helm repo add grafana https://grafana.github.io/helm-charts
# Updating local index repos
helm repo update

# Creating monitoring namespace 
kubectl create ns monitoring

# Install Loki stack via Helm chart
# The loki stack involves a customized fluentbit agent collector made by grafana labs
# A persistent volume is created for loki, which one is backed in a storage account in an azurefile of 5GB
helm upgrade --install loki grafana/loki-stack \
 --set fluent-bit.enabled=true,promtail.enabled=false,grafana.enabled=true,loki.persistence.enabled=true,loki.persistence.storageClassName=azurefile,loki.persistence.size=5Gi --namespace=observability

variables.sh

#! /bin/bash


# export variables 

# Caffeine - Flood Risk Tool Subscription
export ARM_SUBSCRIPTION_ID="15288bef-c3c1-48ef-90bb-1fb178819cf3"
export RESOURCE_GROUP_NAME="rhd-caff-stag-main-r8hacl0nvxhyt"
export AKS_CLUSTER_NAME="rhd-caff-stag-aks-7ha9z30d6qvxy"