xoelop

How to attribute ClickHouse query cost by endpoint, user/team and code path

This is the setup I would build if I wanted to answer these questions:

1. How many ClickHouse cores do we need?
2. What rate limit makes sense per endpoint, customer, or use case?
3. What is the max QPS ceiling we can safely give?
4. Which queries, users/teams, endpoints, and backend methods are using the most CPU?

Load tests against your own API are still useful, but this gives you the base database cost of each query shape in production.

xoelop

{
  "BTTPresetCreatorNotes" : "",
  "BTTPresetInfoURL" : "",
  "BTTPresetName" : "Xoel",
  "BTTPresetColor" : "205.206320, 241.230000, 106.141200, 255.000000",
  "BTTGeneralSettings" : {
    "BTTShowBTTWhenControlStripHidden" : true,
    "BSTTopMissionControlTreshold" : 34,
    "BTTDrawingAreaWidth" : 915,
    "BTTNotchBarNotchWidgetModeShowMiniMenubar" : true,

xoelop

repos:

  # Local hooks
  - repo: local
    hooks:
      - id: env-var-replacer
        name: Replace sensitive values with env variables
        description: Replace actual values from .env with their variable names
        entry: scripts/env_var_replacer.sh .env blur
        language: script

xoelop

# import pyperclip
import argparse
import os
import subprocess

import openai
from dotenv import load_dotenv

load_dotenv()

xoelop

job_title,url,company,location,remote,salary_string,min_annual_salary_usd,max_annual_salary_usd,salary_currency,countries,country,cities,continents,technology_names,date_posted,seniority,country_codes,created_at,final_url,normalized_title,manager_roles,matching_phrases,matching_words,company_url,company_linkedin,industry,company_size,probability_actual_domain_found,company_url_source,logo
Pleno Engenheiro de Software Java | Kotlin,https://indeed.com/viewjob?jk=3326b0400a6a9485,Impulso,"Remoto, Brazil",false,R$ 13.600 - R$ 14.400 por mês,32979.02,34918.97,BRL,,Brazil,,,"Cloud, Java, AWS, Amazon Web Services, Docker, Spring, Kubernetes, BEM, PostgreSQL, Spring Boot, Kotlin, Postgres",2023-09-11,Mid-Level,BR,2023-09-12T06:32:18.600117+00:00,https://impulso.team/en/profissionais/oportunidade/1938,engenheiro de software,,"É super importante ter conhecimento no uso de feature flags / feature toggles, DDD, Continuous integration e Continuous delivery, qualidade de código e testes automatizados",feature flags,https:

xoelop

"113375236","https://indeed.com/viewjob?jk=9b8f0538f79216b3","Asistente- Gestión De Documentos","Banco General, S.A.","PROPOSITO DEL CARGO Gestionar, clasificar, digitalizar y ordenar un conjunto de documentos que son necesarios para conservar una base de datos de los clientes; cumpliendo las normas y procedimientos del Banco, el Código de Ética y Valores, las normas y procedimientos de uniforme y de imagen del Banco; y así mantener un control de la producción de información, como de su manejo y edición. FUNCIONES GENERALES Archivo de carpeta o documentos de portafolio de inversión Preparar documentos de cuentas de inversión local o internacional de persona natural o jurídica para el proceso de digitalización Preparar documentos de cuentas bancarias de persona natural o jurídica para el proceso de digitalización Digitalizar cuentas bancarias y cuentas de inversión local o internacional de persona natural o jurídica Rearmado de carpeta de cuentas de inversión local o internacional de persona natural o jurídica

xoelop

name_text,newsletter_tags_list_option_category_keywords,profile_image_image,starting_price_number,subscribers_number,Slug,click_rate_number,newsletter_freq_option_newsletter_frequency,open_rate_number,status_option_newsletter_status,url_text,newsletter_other_category_text,newsletter_crosspromotion_option_cross_promotions,long_summary_text,owner1_list_user,verified_date,Modified Date,short_summary_text,facebook_url_text,twitter_url_text,newletter_category_option_category,flytier_url_text
Codementor's Newsletter,"['tech', 'productivity']",//s3.amazonaws.com/appforest_uf/f1658204152516x482178286177161700/NewLogo_circle_white%20%281%29.png,3000,471771,codementors-newsletter,0.12,week,0.21,approved,https://www.codementor.io/,Education,i_m_open_to_cross_newsletter_sponsorships0,"🚩 Codementor is a newsletter that helps developers learn and grow in programming knowledge through curated events and community articles. 

⌛️ Every week, Codementor sends a newsletter with upcoming virtual events and featured articles by o

xoelop

{
    "uuid": "60799e3a-81e6-11ed-8e5b-1377772fbf58",
    "page": {
        "id": 115259211,
        "data": {
            "appState": "[\"~#iR\",[\"^ \",\"n\",\"appTemplate\",\"v\",[\"^ \",\"isFetching\",false,\"plugins\",[\"~#iOM\",[\"textInputToken\",[\"^0\",[\"^ \",\"n\",\"pluginTemplate\",\"v\",[\"^ \",\"id\",\"textInputToken\",\"type\",\"widget\",\"subtype\",\"TextInputWidget2\",\"namespace\",null,\"resourceName\",null,\"resourceDisplayName\",null,\"template\",[\"^3\",[\"spellCheck\",false,\"readOnly\",false,\"iconAfter\",\"\",\"showCharacterCount\",false,\"autoComplete\",false,\"maxLength\",null,\"hidden\",false,\"customValidation\",\"\",\"patternType\",\"\",\"hideValidationMessage\",false,\"textBefore\",\"\",\"validationMessage\",\"\",\"textAfter\",\"\",\"showInEditor\",false,\"showClear\",false,\"pattern\",\"\",\"tooltipText\",\"\",\"labelAlign\",\"left\",\"formDataKey\",\"{{ self.id }}\",\"value\",\"\",\"labelCaption\",\"\",\"labelWidth\",\"33\",\"autoFill\",\"\",\"placeholder\",\"Enter value\",\"la

xoelop

import json

import requests
from dotenv import load_dotenv

load_dotenv(override=True)
import os

for i in range(3, 52):

xoelop

# if .env, source .env
if test -f .env; then
    source .env && 
    echo `date`: sourcing .env
fi

# mkdir if not exists
mkdir -p data/script_invoices

# download invoices created later or on this date (yyyy-mm-dd)