Pyroscope.md

Unlocking Performance Insights: A Guide to Grafana Profiles Drilldown with Rails on EKS

To enable Grafana Profiles Drilldown for your Rails application running in EKS, you'll need to introduce a continuous profiling backend to your existing stack. The key takeaway is that Grafana's Profiles Drilldown feature is powered by Grafana Pyroscope, not Loki. While your current setup with the OpenTelemetry Collector and Loki is excellent for logs, it needs to be extended to handle profiling data.

This guide will walk you through the necessary additions and modifications to your Terraform configuration to integrate Pyroscope and instrument your Rails application for profiling.

The Interconnected Pieces: How It All Works

Here's a high-level overview of how the components will work together to provide you with profiling data in Grafana:

opentelemetry-instrumentation-rails & pyroscope-otel: These Ruby gems will be added to your Rails application. The opentelemetry-instrumentation-rails gem provides the core OpenTelemetry integration, while pyroscope-otel specifically enables the collection and export of profiling data. Your application will send this profiling data to the OpenTelemetry Collector.
OpenTelemetry Collector (otel-collector): Your existing otel-collector will be configured to receive this new stream of profiling data from your Rails application via the OpenTelemetry Protocol (OTLP). It will then export this data to the Pyroscope server.
Grafana Pyroscope: This will be a new component in your architecture. Pyroscope is an open-source continuous profiling platform that will store and aggregate the profiling data from your Rails application.
Grafana: Your Grafana instance will be configured with a new data source pointing to the Pyroscope server. This will enable the "Profiles Drilldown" view, allowing you to visualize and analyze the collected profiling data, such as CPU utilization and memory allocation, in flame graphs.

Now, let's dive into the implementation steps.

Step 1: Deploying Grafana Pyroscope

First, you need to add Pyroscope to your EKS cluster. We'll use the official Grafana Helm chart for Pyroscope.

Here's a Terraform resource to deploy Pyroscope. You can add this to your existing Terraform files.

resource "helm_release" "pyroscope" {
  for_each = local.enable_kube_prometheus_stack ? { "create" = true } : {}

  name             = "pyroscope"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "pyroscope"
  namespace        = "pyroscope"
  version          = "0.5.0" # Use a recent version
  create_namespace = true
  wait             = true

  values = [
    <<-EOF
    service:
      type: ClusterIP
    
    # If you want to expose Pyroscope via an Ingress, you can configure it here.
    # ingress:
    #   enabled: true
    #   ingressClassName: alb
    #   annotations:
    #     external-dns.alpha.kubernetes.io/hostname: pyroscope.datocms.com
    #     alb.ingress.kubernetes.io/scheme: internal-facing
    #     # ... other ingress annotations
    #   hosts:
    #     - host: pyroscope.datocms.com
    #       paths:
    #         - path: /
    #           pathType: ImplementationSpecific
    EOF
  ]
}

This configuration will deploy Pyroscope in its own namespace. For simplicity, it's exposed as a ClusterIP service, meaning it will only be accessible from within the cluster.

Step 2: Instrumenting Your Rails Application for Profiling

Next, you'll need to add and configure the necessary gems in your Rails application's Gemfile.

# Gemfile
gem 'opentelemetry-sdk'
gem 'opentelemetry-instrumentation-all'
gem 'pyroscope-otel'

After running bundle install, create or update your OpenTelemetry initializer. This is typically located at config/initializers/opentelemetry.rb.

# config/initializers/opentelemetry.rb

require 'opentelemetry/sdk'
require 'opentelemetry/instrumentation/all'
require 'pyroscope/otel'

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'your-rails-app' # Replace with your application's name
  c.use_all() # Enables all available instrumentations

  # Configure the Pyroscope exporter to send data to the OpenTelemetry Collector
  # The endpoint should match the OTLP receiver in your collector config.
  pyroscope_exporter = Pyroscope::Otel::Exporter.new(endpoint: "http://otel-collector.loki.svc.cluster.local:4317") 

  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(pyroscope_exporter)
  )
end

Key Points:

Replace 'your-rails-app' with a descriptive name for your service.
The endpoint for the Pyroscope::Otel::Exporter should be the address of your OpenTelemetry Collector's OTLP gRPC receiver. Based on your provided Terraform, the service name is otel-collector in the loki namespace.

Step 3: Updating the OpenTelemetry Collector Configuration

Now, you need to modify your otel-collector's configuration to handle profiling data and send it to the newly deployed Pyroscope instance.

In your helm_release.otel_collector resource, you'll need to add a pipeline for profiles and an exporter for Pyroscope.

resource "helm_release" "otel_collector" {
  count            = local.enable_loki ? 1 : 0
  name             = "otel-collector"
  repository       = "https://open-telemetry.github.io/opentelemetry-helm-charts"
  chart            = "opentelemetry-collector"
  namespace        = "loki"
  version          = "0.126.0"
  create_namespace = false
  wait             = true

  depends_on = [helm_release.loki[0], helm_release.pyroscope["create"]]

  values = [
    <<-EOF
    image:
      repository: "otel/opentelemetry-collector-contrib" # Use the contrib image for more exporters

    mode: daemonset

    presets:
      logsCollection:
        enabled: true
      kubernetesAttributes:
        enabled: true

    config:
      receivers:
        otlp:
          protocols:
            grpc:
            http:

      processors:
        transform:
          log_statements:
            - context: log
              statements:
                # 1) If body looks like JSON, parse it...
                - set(attributes["parsed_json"], ParseJSON(body))
                  where IsMatch(body, "^\\s*\\{.*\\}\\s*$")
                # 2) ...and merge into attributes, inserting only non‑conflicting keys
                - merge_maps(attributes, attributes["parsed_json"], "insert")
                  where attributes["parsed_json"] != nil
                # 4) Remove our temporary parsed_json map
                - delete_key(attributes, "parsed_json")

      exporters:
        otlphttp:
          endpoint: http://loki-gateway.loki.svc.cluster.local/otlp
          tls:
            insecure: true
        
        # Add the Pyroscope exporter
        pyroscope:
          endpoint: http://pyroscope.pyroscope.svc.cluster.local:4040
          tls:
            insecure: true

      service:
        pipelines:
          logs:
            receivers: [otlp]
            processors: [transform]
            exporters: [otlphttp]
          
          # Add the profiles pipeline
          profiles:
            receivers: [otlp]
            processors: []
            exporters: [pyroscope]
    EOF
  ]
}

Important Changes:

Image: We've switched to the otel/opentelemetry-collector-contrib image. This version includes a wider range of exporters, including the one for Pyroscope.
Pyroscope Exporter: A new exporter named pyroscope is added. The endpoint points to the service for the Pyroscope instance you deployed in Step 1.
Profiles Pipeline: A new pipeline for profiles is defined. It receives data from otlp, has no processors in this example (but you could add some), and exports to pyroscope.
depends_on: We've added a dependency on the helm_release.pyroscope to ensure it's created before the collector.

Step 4: Configuring the Pyroscope Data Source in Grafana

The final step is to tell Grafana where to find your profiling data. You'll add Pyroscope as a new data source in your Grafana configuration.

In your helm_release.kube_prometheus_stack resource, you can add the Pyroscope data source under additionalDataSources.

resource "helm_release" "kube_prometheus_stack" {
  # ... other configuration ...

  values = [
    <<-EOF
    # ... other values ...

      additionalDataSources:
        - name: "AWS CloudWatch"
          type: cloudwatch
          jsonData:
            authType: arn
            defaultRegion: "eu-west-1"
            roleArn: "${aws_iam_role.grafana["create"].arn}"
        - name: "Loki"
          type: loki
          url: "http://loki-gateway.loki.svc.cluster.local"
          access: "proxy"
        
        # Add the Pyroscope data source
        - name: "Pyroscope"
          type: "pyroscope"
          url: "http://pyroscope.pyroscope.svc.cluster.local:4040"
          access: "proxy"

    # ... rest of your values ...
  EOF
  ]
}

This configuration adds a new data source named "Pyroscope" to your Grafana instance, pointing to the internal service URL of your Pyroscope deployment.

After applying these Terraform changes and deploying the updated Rails application, you should be able to navigate to the "Explore" section in Grafana, select the "Pyroscope" data source, and start exploring your application's profiles. You will also find the "Profiles Drilldown" in your Grafana navigation, providing a dedicated interface for analyzing this data.