OCI Usage Analysis with MySQL

Manual download of OCI usage data

Using the OCI web console, go to the Menu > Account Management > Usage Report.

Example:

reports/usage-csv/0001000000072041.csv.gz
reports/usage-csv/0001000000071666.csv.gz
reports/usage-csv/0001000000071275.csv.gz

Reference: https://docs.cloud.oracle.com/iaas/Content/Billing/Concepts/usagereportsoverview.htm

Import to MySQL

Extract the files

$ gunzip reports_usage-csv_*.gz
$ ls *.csv
reports_usage-csv_0001000000071275.csv  
reports_usage-csv_0001000000072041.csv
reports_usage-csv_0001000000071666.csv
$ pwd
/Users/alastori/my-data

Check the encoding and terminators in CSV file

$ file reports_usage-csv_0001000000071275.csv
reports_usage-csv_0001000000071275.csv: ASCII text, with very long lines, with CRLF line terminators

Run MySQL using Docker

To bindmount the directory containing the CSV files, use the option -v:

docker pull mysql/mysql-server:8.0
export MY_PWD='Welcome1!'
docker run --name=mysqlserver -v ~/my-data:/mnt/mysql-files -e MYSQL_ROOT_PASSWORD=$MY_PWD -d mysql/mysql-server:8.0 --secure-file-priv=/mnt/mysql-files
docker ps

Connect to MySQL using MySQL Shell

docker exec -it mysqlserver /bin/bash
ls /mnt/mysql-files
mysqlsh --sql --socket -uroot -p$MYSQL_ROOT_PASSWORD

Create the schema in MySQL to import the data from the CSV files

DROP DATABASE IF EXISTS oci;
CREATE DATABASE oci;
USE oci;
CREATE TABLE oci_usage (
    li_referenceNo CHAR(90) NOT NULL,
    li_tenantId CHAR(100) NOT NULL,
    li_intervalUsageStart DATETIME NOT NULL,
    li_intervalUsageEnd DATETIME NOT NULL,
    p_service CHAR(50) NOT NULL,
    p_resource CHAR(50) NOT NULL,
    p_compartmentId CHAR(100),
    p_compartmentName VARCHAR(255),
    p_region VARCHAR(30),
    p_availabilityDomain VARCHAR(35),
    p_resourceId CHAR(100) NOT NULL,
    u_consumedQuantity BIGINT NOT NULL,
    u_billedQuantity BIGINT NOT NULL,
    u_consumedQuantityUnits VARCHAR(15),
    u_consumedQuantityMeasure VARCHAR(30),
    li_isCorrection BOOLEAN,
    li_backreferenceNo VARCHAR(90),
    meta JSON,
    PRIMARY KEY (li_referenceNo)
);

Note: maybe not all the csv files have the same number of columns, because the trailing tags/ may change along the time. Also lineItem/backreferenceNo is not present in old files. See the docs for details.

Load CSV data into MySQL

Example to remove trailing columns from CSV file

Only the first 16 columns will be imported. Run these commands in the terminal in the directory where the csv file is located:

# store the csv filename into the variable f
f=reports_usage-csv_0001000000026006.csv

# backup file, don't overwrite 
cp -n $f $f.bak        

# new csv file, now with 16 columns, no more, no less
awk 'BEGIN {FS=","; OFS=","; ORS="\r\n"} {print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16}' ${f}.bak > $f

Note: the column 17th lineItem/backreferenceNo is not present in old csv files.

Example to load the data from one single CSV file

In MySQL, run:

USE oci; TRUNCATE TABLE oci_usage;
LOAD DATA INFILE '/mnt/mysql-files/reports_usage-csv_0001000000026006.csv' 
INTO TABLE oci_usage 
CHARACTER SET ascii 
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'  
IGNORE 1 LINES
   (li_referenceNo,
    li_tenantId,
    @li_intervalUsageStart,
    @li_intervalUsageEnd,
    p_service,
    p_resource,
    p_compartmentId,
    p_compartmentName,
    p_region,
    p_availabilityDomain,
    p_resourceId,
    u_consumedQuantity,
    u_billedQuantity,
    u_consumedQuantityUnits,
    u_consumedQuantityMeasure,
    @li_isCorrection)
SET li_intervalUsageStart = STR_TO_DATE(@li_intervalUsageStart, '%Y-%m-%dT%H:%iZ'),
    li_intervalUsageEnd = STR_TO_DATE(@li_intervalUsageEnd, '%Y-%m-%dT%H:%iZ'),
    li_isCorrection = IF(@li_isCorrection = 'FALSE', 0, 1),
    meta = JSON_OBJECT('file', 'reports_usage-csv_0001000000026006.csv');

Note: li_backreferenceNo is not populated in this example because lineItem/backreferenceNo is not present in old csv files.

Example of shell script to import all CSV files in the directory

Create a file oci_usage_import.sh with the following content:

#!/bin/bash
#Bash script to import CSV files with OCI usage data into MySQL

set -e # stop script execution on any error 
BASENAME=/mnt/mysql-files
SQLFILE=./import-csv.sql
echo "USE oci; TRUNCATE TABLE oci_usage;" > $SQLFILE
for CSVFILE in *.csv
do
  echo "Processing file ${CSVFILE}..."
  cp -n $CSVFILE ${CSVFILE}.bak || :    # backup files, don't overwrite and don't emit error
  awk 'BEGIN {FS=","; OFS=","; ORS="\r\n"} {print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16}' ${CSVFILE}.bak > $CSVFILE  # new csv files, now with exact 16 columns
  echo "Generating SQL for file ${CSVFILE}..."
  cat <<EOF >>$SQLFILE
-- 
SELECT 'Loading data from ${BASENAME}/${CSVFILE}...';
LOAD DATA INFILE '${BASENAME}/${CSVFILE}' 
INTO TABLE oci_usage 
CHARACTER SET ascii 
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'  
IGNORE 1 LINES
   (li_referenceNo,
    li_tenantId,
    @li_intervalUsageStart,
    @li_intervalUsageEnd,
    p_service,
    p_resource,
    p_compartmentId,
    p_compartmentName,
    p_region,
    p_availabilityDomain,
    p_resourceId,
    u_consumedQuantity,
    u_billedQuantity,
    u_consumedQuantityUnits,
    u_consumedQuantityMeasure,
    @li_isCorrection)
SET li_intervalUsageStart = STR_TO_DATE(@li_intervalUsageStart, '%Y-%m-%dT%H:%iZ'),
    li_intervalUsageEnd = STR_TO_DATE(@li_intervalUsageEnd, '%Y-%m-%dT%H:%iZ'),
    li_isCorrection = IF(@li_isCorrection = 'FALSE', 0, 1),
    meta = JSON_OBJECT('file', '${CSVFILE}');
SELECT '...${CSVFILE} loaded!';
EOF
done
echo "File $SQLFILE generated."

Execute the oci_usage_import.sh shellscript to generate the import-csv.sql file.

chmod +x oci_usage_import.sh
./oci_usage_import.sh

Then execute the generated import-csv.sql file using MySQL Shell:

mysqlsh --sql --socket -uroot -p$MYSQL_ROOT_PASSWORD < /mnt/mysql-files/import-csv.sql

Sample queries

Using MySQL Shell:

mysqlsh --sql --socket -uroot -p$MYSQL_ROOT_PASSWORD

Files imported and intervals

USE oci;
SELECT 
  MIN(li_intervalUsageStart) AS start,
  MAX(li_intervalUsageEnd) AS end,
  li_tenantId AS tenancy,
  meta-> "$.file" AS file,
  COUNT(li_referenceNo) AS items
FROM oci_usage
GROUP BY file, tenancy
ORDER BY start;

Top 50 more expensive Block Storage resources in the last 30 days

SELECT p_resourceId, p_service, p_resource, p_compartmentName, SUM(u_billedQuantity) AS quantity 
FROM oci_usage 
GROUP BY p_resourceId, p_resource, p_service, p_compartmentName 
HAVING p_service='BLOCK_STORAGE' AND MAX(li_intervalUsageEnd) >= (NOW() - INTERVAL 30 DAY) 
ORDER BY quantity DESC LIMIT 50;

Price

We are going to create a support table with OCI price lists.

What are the Services and Resources already consumed?

SELECT DISTINCT p_service, p_resource, u_consumedQuantityUnits AS units, u_consumedQuantityMeasure AS measure FROM oci_usage ORDER BY p_service;
+-----------------------------+------------------------------------+-------+------------------+
| p_service                   | p_resource                         | units | measure          |
+-----------------------------+------------------------------------+-------+------------------+
| BLOCK_STORAGE               | PIC_BLOCK_STORAGE_STANDARD         | GB_MS | STORAGE_SIZE     |
| BLOCK_STORAGE               | PIC_BLOCK_STORAGE_STANDARD_FREE    | GB_MS | STORAGE_SIZE     |
| COMPUTE                     | PIC_COMPUTE_STANDARD_E2            | MS    | OCPUS            |
| COMPUTE                     | PIC_COMPUTE_VM_STANDARD            | MS    | OCPUS            |
| COMPUTE                     | PIC_COMPUTE_X7_VM_DENSEIO          | MS    | OCPUS            |
| COMPUTE                     | PIC_COMPUTE_X7_VM_STANDARD         | MS    | OCPUS            |
| FILESTORAGESERVICE          | PIC_FILE_STORAGE                   | GB_MS | STORAGE_SIZE     |
| NETWORK                     | PIC_COMPUTE_OUTBOUND_DATA_TRANSFER | BYTES | DATA_TRANSFERRED |
| OBJECTSTORE                 | PIC_COMPUTE_OUTBOUND_DATA_TRANSFER | BYTES | DATA_TRANSFERRED |
| OBJECTSTORE                 | PIC_OBJECT_STORAGE_REQUEST_TIERED  | COUNT | REQUESTS         |
| OBJECTSTORE                 | PIC_OBJECT_STORAGE_TIERED          | GB_MS | STORAGE_SIZE     |
| ORACLE_NOTIFICATION_SERVICE | PIC_NOTIFICATIONS_EMAIL_DELIVERY   | COUNT | EMAILS_SENT      |
| PUBLIC_DNS                  | PIC_DYN_DNS_QUERIES                | COUNT | DNS_QUERIES      |
| TELEMETRY                   | PIC_METRIC_CONSUMPTION             | COUNT | DATAPOINTS       |
+-----------------------------+------------------------------------+-------+------------------+
14 rows in set (0.0160 sec)

Create a price list table for the the consumed Services and Resources

Base table

USE oci;
CREATE TABLE oci_price (
    p_resource CHAR(50) NOT NULL,
    p_service CHAR(50) NOT NULL,
    units VARCHAR(15),
    measure VARCHAR(30),
    p_name VARCHAR(80),
    price DECIMAL(5,4),
    price_units VARCHAR(30),
    PRIMARY KEY (p_resource, p_service)
);

Populate the price table based on consumed resources

INSERT IGNORE INTO oci_price (
    p_resource, 
    p_service, 
    units, 
    measure 
) SELECT DISTINCT 
    p_resource, 
    p_service, 
    u_consumedQuantityUnits, 
    u_consumedQuantityMeasure 
FROM oci_usage 
ORDER BY p_service;

Update the prices

References:

-- VM.Standard.E2 = PIC_COMPUTE_STANDARD_E2
UPDATE oci_price SET price = 0.03, price_units = 'OCPU_HOUR', p_name = 'VM.Standard.E2' 
WHERE p_resource='PIC_COMPUTE_STANDARD_E2' AND p_service='COMPUTE'; 

-- (Deprecated) VM.Standard1 = PIC_COMPUTE_VM_STANDARD
UPDATE oci_price SET price = 0.0638, price_units = 'OCPU_HOUR', p_name = 'VM.Standard1' 
WHERE p_resource='PIC_COMPUTE_VM_STANDARD' AND p_service='COMPUTE'; 

-- VM.Standard2 = PIC_COMPUTE_X7_VM_STANDARD
UPDATE oci_price SET price = 0.0638, price_units = 'OCPU_HOUR', p_name = 'VM.Standard2' 
WHERE p_resource='PIC_COMPUTE_X7_VM_STANDARD' AND p_service='COMPUTE'; 

-- VM.DenseIO2 = PIC_COMPUTE_X7_VM_DENSEIO
UPDATE oci_price SET price = 0.1275, price_units = 'OCPU_HOUR', p_name = 'VM.DenseIO2' 
WHERE p_resource='PIC_COMPUTE_X7_VM_DENSEIO' AND p_service='COMPUTE'; 

-- Block Volumes = PIC_BLOCK_STORAGE_STANDARD
UPDATE oci_price SET price = 0.0425, price_units = 'GB_MONTH', p_name = 'Block Volumes' 
WHERE p_resource='PIC_BLOCK_STORAGE_STANDARD' AND p_service='BLOCK_STORAGE'; 

-- Block Volumes = PIC_BLOCK_STORAGE_STANDARD_FREE
UPDATE oci_price SET price = 0.0, price_units = 'GB_MONTH', p_name = 'Block Volumes (Free)' 
WHERE p_resource='PIC_BLOCK_STORAGE_STANDARD_FREE' AND p_service='BLOCK_STORAGE'; 

-- File Storage = PIC_FILE_STORAGE
UPDATE oci_price SET price = 0.3, price_units = 'GB_MONTH', p_name = 'File Storage' 
WHERE p_resource='PIC_FILE_STORAGE' AND p_service='FILESTORAGESERVICE'; 

-- Object Storage - Storage = PIC_OBJECT_STORAGE_TIERED
UPDATE oci_price SET price = 0.0255, price_units = 'GB_MONTH', p_name = 'Object Storage - Storage' 
WHERE p_resource='PIC_OBJECT_STORAGE_TIERED' AND p_service='OBJECTSTORE'; 

-- Object Storage - Requests = PIC_OBJECT_STORAGE_REQUEST_TIERED
UPDATE oci_price SET price = 0.0034, price_units = '10000_REQUESTS_MONTH', p_name = 'Object Storage - Requests' 
WHERE p_resource='PIC_OBJECT_STORAGE_REQUEST_TIERED' AND p_service='OBJECTSTORE'; 

-- [TBC] Object Storage - Data Transfer - Over 10 TB / Month = OBJECTSTORE/PIC_COMPUTE_OUTBOUND_DATA_TRANSFER
UPDATE oci_price SET price = 0.0085, price_units = 'GB_MONTH', p_name = 'Object Storage -  Outbound Data Transfer' 
WHERE p_resource='PIC_COMPUTE_OUTBOUND_DATA_TRANSFER' AND p_service='OBJECTSTORE'; 

-- Network - Outbound Data Transfer - Over 10 TB / Month = NETWORK/PIC_COMPUTE_OUTBOUND_DATA_TRANSFER
UPDATE oci_price SET price = 0.0085, price_units = 'GB_MONTH', p_name = 'Outbound Data Transfer - Over 10 TB / Month' 
WHERE p_resource='PIC_COMPUTE_OUTBOUND_DATA_TRANSFER' AND p_service='NETWORK'; 

-- Email Delivery Services - PIC_NOTIFICATIONS_EMAIL_DELIVERY 
UPDATE oci_price SET price = 0.0085, price_units = '1000_EMAILS_SENT', p_name = 'Email Delivery Services' 
WHERE p_resource='PIC_NOTIFICATIONS_EMAIL_DELIVERY' AND p_service='ORACLE_NOTIFICATION_SERVICE';

-- DNS Services (1000 Supported Zones, 25,000 Records per Zone) - PIC_DYN_DNS_QUERIES
UPDATE oci_price SET price = 0.85, price_units = '1000000_DNS_QUERIES_MONTH', p_name = 'DNS Services (1000 Supported Zones, 25,000 Records per Zone)' 
WHERE p_resource='PIC_DYN_DNS_QUERIES' AND p_service='PUBLIC_DNS';

-- [TODO] Telemetry - PIC_METRIC_CONSUMPTION
-- ?

Price list

SELECT LEFT(p_name, 35) AS product, price, price_units FROM oci_price WHERE price IS NOT NULL ORDER BY p_service;
+-------------------------------------+--------+---------------------------+
| product                             | price  | price_units               |
+-------------------------------------+--------+---------------------------+
| Block Volumes                       | 0.0425 | GB_MONTH                  |
| Block Volumes (Free)                | 0.0000 | GB_MONTH                  |
| VM.Standard.E2                      | 0.0300 | OCPU_HOUR                 |
| VM.Standard1                        | 0.0638 | OCPU_HOUR                 |
| VM.DenseIO2                         | 0.1275 | OCPU_HOUR                 |
| VM.Standard2                        | 0.0638 | OCPU_HOUR                 |
| File Storage                        | 0.3000 | GB_MONTH                  |
| Outbound Data Transfer - Over 10 TB | 0.0085 | GB_MONTH                  |
| Object Storage -  Outbound Data Tra | 0.0085 | GB_MONTH                  |
| Object Storage - Requests           | 0.0034 | 10000_REQUESTS_MONTH      |
| Object Storage - Storage            | 0.0255 | GB_MONTH                  |
| Email Delivery Services             | 0.0085 | 1000_EMAILS_SENT          |
| DNS Services (1000 Supported Zones, | 0.8500 | 1000000_DNS_QUERIES_MONTH |
+-------------------------------------+--------+---------------------------+

Normalized price

-- 1 hour is equal to 3,600,000 ms; 1 month of 744 hours is equal to 2,678,400,000 ms
SELECT LEFT(p_name, 35) AS product, price, price_units,
  CASE price_units 
    WHEN 'OCPU_HOUR' THEN (CAST(price AS DECIMAL(30,25)) / 3600000)
    WHEN 'GB_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 2678400000)
    WHEN '10000_REQUESTS_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 10000)
    WHEN '1000_EMAILS_SENT' THEN (CAST(price AS DECIMAL(30,25)) / 1000)
    WHEN '1000000_DNS_QUERIES_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 1000000)
    ELSE NULL
  END AS normalized_price, 
units, measure 
FROM oci_price 
WHERE price IS NOT NULL 
ORDER BY p_service;

Some queries

Total cost for the past 31 days

WITH  oci_price_normalized (price_normalized, p_resource, p_service) AS (
  SELECT 
    CASE price_units 
      WHEN 'OCPU_HOUR' THEN (CAST(price AS DECIMAL(30,25)) / 3600000)
      WHEN 'GB_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 2678400000)
      WHEN '10000_REQUESTS_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 10000)
      WHEN '1000_EMAILS_SENT' THEN (CAST(price AS DECIMAL(30,25)) / 1000)
      WHEN '1000000_DNS_QUERIES_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 1000000)
      ELSE NULL
    END, 
    p_resource, 
    p_service
  FROM oci_price 
  WHERE price IS NOT NULL
) 
SELECT 
  ROUND(SUM(usg.u_billedQuantity * pn.price_normalized),2) AS cost,
  MIN(usg.li_intervalUsageStart) AS start,
  MAX(usg.li_intervalUsageEnd) AS end, 
  COUNT(DISTINCT usg.meta->"$.file") AS files
FROM oci_usage AS usg 
LEFT JOIN oci_price_normalized AS pn 
  ON usg.p_resource=pn.p_resource AND usg.p_service=pn.p_service 
WHERE li_intervalUsageEnd >= (NOW() - INTERVAL 31 DAY) 
ORDER BY cost DESC;

Total cost per service in the last year

WITH  oci_price_normalized (price_normalized, p_resource, p_service) AS (
  SELECT 
    CASE price_units 
      WHEN 'OCPU_HOUR' THEN (CAST(price AS DECIMAL(30,25)) / 3600000)
      WHEN 'GB_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 2678400000)
      WHEN '10000_REQUESTS_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 10000)
      WHEN '1000_EMAILS_SENT' THEN (CAST(price AS DECIMAL(30,25)) / 1000)
      WHEN '1000000_DNS_QUERIES_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 1000000)
      ELSE NULL
    END, 
    p_resource, 
    p_service 
  FROM oci_price 
  WHERE price IS NOT NULL 
) 
SELECT 
  ROUND(SUM(usg.u_billedQuantity * pn.price_normalized),2) AS cost,
  pn.p_service AS service
FROM oci_usage AS usg
LEFT JOIN oci_price_normalized AS pn 
  ON usg.p_resource=pn.p_resource AND usg.p_service=pn.p_service 
GROUP BY service
HAVING MAX(li_intervalUsageEnd) >= (NOW() - INTERVAL 365 DAY) 
ORDER BY cost DESC;

Cost per Compartment in the last 7 days

WITH  oci_price_normalized (price_normalized, p_resource, p_service) AS (
  SELECT 
    CASE price_units 
      WHEN 'OCPU_HOUR' THEN (CAST(price AS DECIMAL(30,25)) / 3600000)
      WHEN 'GB_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 2678400000)
      WHEN '10000_REQUESTS_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 10000)
      WHEN '1000_EMAILS_SENT' THEN (CAST(price AS DECIMAL(30,25)) / 1000)
      WHEN '1000000_DNS_QUERIES_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 1000000)
      ELSE NULL
    END, 
    p_resource, 
    p_service
  FROM oci_price 
  WHERE price IS NOT NULL 
) 
SELECT 
  ROUND(SUM(usg.u_billedQuantity * pn.price_normalized),2) AS cost,
  usg.p_compartmentName AS compartment,
  GROUP_CONCAT(DISTINCT pn.p_service) AS services
FROM oci_usage AS usg
LEFT JOIN oci_price_normalized AS pn 
  ON usg.p_resource=pn.p_resource AND usg.p_service=pn.p_service 
GROUP BY compartment
HAVING MAX(li_intervalUsageEnd) >= (NOW() - INTERVAL 7 DAY) 
ORDER BY cost DESC;

Cost per Compute instance in the last 7 days

WITH  oci_price_normalized (p_name, price_normalized, p_resource, p_service) AS (
  SELECT 
    p_name,
    CASE price_units 
      WHEN 'OCPU_HOUR' THEN (CAST(price AS DECIMAL(30,25)) / 3600000)
      WHEN 'GB_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 2678400000)
      WHEN '10000_REQUESTS_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 10000)
      WHEN '1000_EMAILS_SENT' THEN (CAST(price AS DECIMAL(30,25)) / 1000)
      WHEN '1000000_DNS_QUERIES_MONTH' THEN (CAST(price AS DECIMAL(30,25)) / 1000000)
      ELSE NULL
    END, 
    p_resource, 
    p_service
  FROM oci_price 
  WHERE price IS NOT NULL 
) 
SELECT 
  ROUND(SUM(usg.u_billedQuantity * pn.price_normalized),2) AS cost,
  usg.p_compartmentName AS compartment,
  LEFT(pn.p_name, 35) AS product, 
  usg.p_resourceId AS ocid
FROM oci_usage AS usg
LEFT JOIN oci_price_normalized AS pn 
  ON usg.p_resource=pn.p_resource AND usg.p_service=pn.p_service 
GROUP BY ocid, compartment, product, usg.p_service
HAVING usg.p_service='COMPUTE' AND MAX(li_intervalUsageEnd) >= (NOW() - INTERVAL 7 DAY) 
ORDER BY cost DESC;

alastori/OCI Usage Analysis with MySQL.md