ISUCON TIME LINE

00-checklist.md

Start

Đọc hiểu vấn đề

• Đọc và xác định yêu cầu: Nắm rõ yêu cầu, cách tính điểm và lường trước các vấn đề của test case -> memo
  • Cách tính điểm
  • Các luật hạn chế
  • Các api và testcase

Cài đặt cơ bản

• Các thông số môi trường cần xác nhận trước khi làm
  • ruby, mysql, nginx, redis version
  • ruby, mysql, nginx, redis config file path
• Khởi đầu sẽ cài đặt db cho 1 server / web app cho 1 server / benchmark cho 1 server
  • Cài đặt github cho db server và webserver
  • Push code lên github
  • Cài đặt các batch script đã được chuẩn bị sẵn
• Thiết lập môi trường mặc định cho từng server
  • db server: my.cnf | mysql.service
  • webapp server: nginx.conf | redis conf | ruby service
  • bench server: target-url
• Cài đặt các tool bổ trợ
  • db-server: percona-toolkit
  • web-server:
    • https://github.com/wg/wrk benchmark http
    • netdata for monitoring
    • datadog for apm

P & Solution

Xác định bottleneck của hệ thống

• Chạy bench để tìm ra bottleneck
  • Quá tải cho db server:
    • db replica
    • ....
  • Quá tải cho web server:
    • loadbalancing
    • giảm số lượng worker
  • Đọc từ disk quá nhiều..vv

Thay đổi kiến trúc

• Kiến trúc hệ thống Webapp
• Thiết kế kiến trúc cho DB

Thực hiện thay đổi thiết lập module thành phần

• Thiết lập cho database: mysql config, version
• Thiết lập cho webapp: unicorn worker, unicorn config
  • Redis config / version
  • Rack Env config / Version
  • Os

Xác định bottleneck của code

Thay đổi thực thi

• Xác định | sửa các điểm đọc file
• Xác định | sửa các điểm N+1
• Xác định | sửa các xử lý thừa (code, query)
• Đánh lại index
  • Đánh cho dữ liệu nào, bảng nào, cột nào?
  • Thực hiện chỉnh sửa cho schema

Thay đổi kiến trúc DB

• Xác định các điểm có thể cached dữ liệu
  • Cache trên redis hay cache trên mysql

Other

• Đồng bộ database:

  • Dùng 1 database server
  • Tìm cách replica: writer, read
    • Setup for mysql + ruby
    • Làm thế nào để switch query giữa các db

• Đồng bộ code, nginx https://qiita.com/momotaro98/items/694000dfb736d0316441

  • Tạo 1 bash để đồng bộ code giữa 2 server

• Cuối game Xóa các thiết lập log, để tăng speed

01-config-mysql.md

mysql 設定

• Check Engine mysql select engine from information_schema.tables where table_schema = 'isuumo'
• /etc/mysql/my.cnf でMysqlの設定変数を追加する

[mysqld]
slow_query_log = 1 # enabled
slow_query_log_file=/var/lib/mysql/slow-query.log
long_query_time = 0.2 # 0.2 second 以上のクエリはログに入れる
log_queries_not_using_indexes=1 # If you enable this variable with the slow query log enabled, queries that are expected to retrieve all rows are logged.
max_allowed_packet=32M #  max_allowed_packet バイトより大きいパケットを受け取ると、ER_NET_PACKET_TOO_LARGE エラーが発行され、接続が失われます
innodb_buffer_pool_size = 1GB # ディスクイメージをメモリ上にバッファさせる値をきめる設定値
innodb_flush_log_at_trx_commit = 2 # 1に設定するとトランザクション単位でログを出力するが 2 を指定すると1秒間に1回ログファイルに出力するようになる
innodb_flush_method = O_DIRECT # データファイル、ログファイルの読み書き方式を指定する(実験する価値はある)

• slow-query.log ファイルを作成する

sudo touch /var/lib/mysql/slow-query.log
sudo chown mysql:mysql /var/lib/mysql/slow-query.log

• max_connections と open_files_limit設定

#/etc/systemd/system/mysql.service
[Service]
LimitNOFILE=65535

• restart daemon and mysql

sudo systemctl daemon-reload
sudo service mysql restart
SET GLOBAL max_connections = 10000;

• check mysql boot status

  • sudo journalctl -xe

• Check Mysql Error log:

  • /var/log/mysql/error.log

• 上記のログファイルを確認して、問題なければMysql Consoleで環境変数を確認する

sudo mysql -u isucon -pisucon
SET GLOBAL max_connections = 10000;
# check variables
SHOW VARIABLES LIKE '%slow_query%';
SHOW VARIABLES LIKE '%long_que%';
SHOW VARIABLES LIKE '%max_allowed_packet%';
SHOW VARIABLES LIKE '%innodb_buffer_pool_size%';
SHOW VARIABLES LIKE '%innodb_flush_log_at_trx_commit%';
SHOW VARIABLES LIKE '%innodb_flush_method%';
SHOW VARIABLES LIKE 'max_connections';
SHOW VARIABLES LIKE 'open_files_limit';

• 画面に表示された結果と同じであればOK

Slow ログ確認する

• Reading mysql slow log

mysqldumpslow -a /var/lib/mysql/slow-query.log

ロカルのSequel Proに繋がる手法

• Connect mysql on Sequel
  • https://aws.amazon.com/premiumsupport/knowledge-center/rds-mysql-ssh-workbench-connect-ec2/

Upgrade to Mysql 8.0

Decending Orderというような機能はMysql5.7以下にないため、Mysql8.0までアップグレードする必要がある

wget https://repo.mysql.com//mysql-apt-config_0.8.14-1_all.deb
sudo dpkg -i mysql-apt-config_0.8.14-1_all.deb
sudo apt-get update
sudo apt-get install mysql-server
sudo apt policy mysql-server
sudo systemctl start mysql.service

参考

• https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/
• https://phoenixnap.it/kb/improve-mysql-performance-tuning-optimization
• https://www.javacodemonk.com/upgrade-mysql-from-5-7-to-8-on-ubuntu-18-04-bd9b5e46
• https://mysqlserverteam.com/whats-new-in-mysql-8-0-generally-available/#:~:text=MySQL%208.0%20makes%20UTF8MB4%20the,8.0%20as%20compared%20to%205.7.&text=The%20default%20character%20set%20has,changed%20from%20latin1_swedish_ci%20to%20utf8mb4_800_ci_ai%20.

02-nginx.md

設定

• Webapp とSymbol Linkを作って、修正・管理しやすくなるため
• Nginxファイルリンク

touch /home/isucon/project/webapp/nginx/nginx.conf
sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx_copy.conf
sudo rm /etc/nginx/nginx.conf
cd /etc/nginx/
sudo ln -s /home/isucon/isuumo/webapp/nginx/nginx.conf /etc/nginx/nginx.conf

• Githubまでプッシューする
• /home/isucon/project/webapp/nginx/nginx.cnf を設定する。おすすめ設定は
  • Load Balancerとして扱うなら、upstreamのところを入れなきゃいけね
  • / と /api はsv2までリダイレクトする

user www-data;
worker_processes 1; # autoにしたら、コア数と同じ数まで増やすが、Isuconならただ1コアのCPUしか提供してくれないので、1にしても問題ない
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 16000; # worker_connectionsの4倍 程度


error_log /home/isucon/isuumo/logs/error.log error; # errorログパース設定

events {
    worker_connections 4096; 
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    # tcp
    tcp_nopush  on; # nginxはデフォルトでファイルを送り出す際にsendfileシステムコールを使いますが、sendfileの前に動くHTTPヘッダのパケット書き出しを遅延させ、ヘッダとファイルの中身を一度に送り出すように調整します
    tcp_nodelay on;

    # log
    log_format main "time:$time_local"
    "\thost:$remote_addr"
    "\tforwardedfor:$http_x_forwarded_for"
    "\treq:$request"
    "\tmethod:$request_method"
    "\turi:$request_uri"
    "\tstatus:$status"
    "\tsize:$body_bytes_sent"
    "\treferer:$http_referer"
    "\tua:$http_user_agent"
    "\treqtime:$request_time"
    "\truntime:$upstream_http_x_runtime"
    "\tapptime:$upstream_response_time"
    "\tcache:$upstream_http_x_cache"
    "\tvhost:$host";

    access_log  /home/isucon/isuumo/logs/access.log  main;

    # Proxy cache 設定。使いどころがあれば。1mでkey8,000個。1gまでcache。
    # proxy_cache_path /var/cache/nginx/cache levels=1:2 keys_zone=zone1:1m max_size=1g inactive=1h;
    # proxy_temp_path  /var/cache/nginx/tmp;
    
    # Use nginx like load balancer
    #upstream app {
    #  least_conn; # リクエストが少ないサーバーまで整理する
    #  server 127.0.0.1;
    #  server server_2_private_ip;
    #}

    # gzip
    client_max_body_size 10m;
    open_file_cache max=100 inactive=65s;
    client_body_buffer_size     8k;
    client_header_buffer_size   8k;
    large_client_header_buffers 4 64k;
    gzip              on;
    gzip_types        text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/atom_xml text/javascript application/javascript application/x-javascript application/json;
    gzip_disable      "msie6" "Mozilla/4";
    gzip_http_version 1.0;  # HTTP/1.0もgzip圧縮の対象とする
    gzip_vary         on;   # Accept-Encodingレスポンスヘッダを追加
    gzip_proxied      any;  # プロキシ経由でもgzip圧縮を有効にする
    gzip_buffers      4 8k; # gzip圧縮で使用するバッファサイズ（4 8k は 4 x 8k = 32k という意味）
    gzip_min_length   1100; # gzip圧縮する最小データサイズ
    sendfile        on;
    keepalive_timeout  125;
    keepalive_requests 500; # 1接続中に何回のリクエストまで受けるかの設定nginxのデフォルトは100しかないため
    server_tokens off;

    server {
        root /www/data;
        listen 80 default_server;
        listen [::]:80 default_server;


        if ( $http_user_agent ~ (Mediapartners-ISUCON|ISUCONCoffee|isubot|Isupider) ) {
          return 503;
        }


        if ( $http_user_agent ~ /ISUCONbot(-Mobile)?/ ) {
          return 503;
        }

        if ( $http_user_agent ~ /ISUCONFeedSeeker(Beta)?/ ) {
          return 503;
        }

        if ( $http_user_agent ~ /Isupider(-image)?\+/ ) {
          return 503;
        }

        # こちら、Isuconの問題により修正する必要
        # Loadbalancingが必要なら、こちらを修正
        location /api {
          proxy_pass http://localhost:1323;
        }

        location /initialize {
          proxy_pass http://localhost:1323;
        }

        location / {
          root /www/data;
        }
    }
}

参考資料

• http://nginx.org/en/docs/ngx_core_module.html
• /etc/nginx/nginx.conf に以下を設定。
  • worker_rlimit_nofile は fd 数上限。worker_connections の 3 ~ 4倍推奨
    • cf. http://www.1x1.jp/blog/2013/02/nginx_too_many_open_files_error.html
  • static file の配信( location ~ ^/(stylesheets|images)/ の部分)
  • unix domain socket ( server unix:/tmp/unicorn.sock の部分 )
  • tcp_nopush や tcp_nodelay が気になったら https://heartbeats.jp/hbblog/2012/02/nginx03.htmlや http://blog.nomadscafe.jp/2013/09/benchmark-g-wan-and-nginx.html?_=123 を見る
  • etag はキャッシュしたコンテンツを 200 で返すためのもの http://qiita.com/syokenz/items/e3c8ed4b4a6dfed51a83
  • index は / 終端を /index.html 終端として扱うための directive。app へ流せてるか確認しながら設定すべき。
  • gzip は ネットワークが圧迫してる時用。
    • http://qiita.com/cubicdaiya/items/2763ba2240476ab1d9dd#gzip_static%E3%83%A2%E3%82%B8%E3%83%A5%E3%83%BC%E3%83%AB%E3%81%A7%E5%9C%A7%E7%B8%AE%E6%B8%88%E3%81%BF%E3%81%AE%E3%82%B3%E3%83%B3%E3%83%86%E3%83%B3%E3%83%84%E3%82%92%E9%85%8D%E4%BF%A1%E3%81%99%E3%82%8B
• worker_processes はコア数で設定しても良い(multiproces へ)
• http://lxyuma.hatenablog.com/entry/2015/08/29/200443
• https://gist.github.com/koudaiii/735ef14b83ee31ac0967
• https://heartbeats.jp/hbblog/2012/06/nginx06.html
• https://serverfault.com/questions/763597/why-is-multi-accept-off-as-default-in-nginx
• https://qiita.com/iwai/items/1e29adbdd269380167d2
• http://tetsuyai.hatenablog.com/entry/20111220/1324466655
• https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user
• https://github.com/firehol/netdata/wiki/high-performance-netdata#2b-increase-open-files-limit-systemd
• https://www.nginx.com/blog/websocket-nginx/
• https://qiita.com/zaru/items/c41072e29b9550c2e6a8
• https://github.com/openresty/lua-resty-redis#set_keepalive
• https://www.nginx.com/blog/monitoring-nginx/
• https://github.com/wg/wrk

03-tools.md

• percona-toolkit

  • https://www.percona.com/software/database-tools/percona-toolkit

• netdata

  • https://learn.netdata.cloud/docs/dashboard/how-dashboard-works
  • https://learn.netdata.cloud/docs/get-started/#install-on-linux-with-one-line-installer-recommended

• Benchmark HTTP tool

  • https://github.com/wg/wrk

netdata by default listens on all IPs on port 19999,
so you can access it with:

  http://this.machine.ip:19999/

To stop netdata run:

  systemctl stop netdata

To start netdata run:

  systemctl start netdata

Uninstall script copied to: /usr/libexec/netdata/netdata-uninstaller.sh

 --- Installing (but not enabling) the netdata updater tool ---
Failed to disable unit: Unit file netdata-updater.timer does not exist.
Update script is located at /usr/libexec/netdata/netdata-updater.sh

 --- Check if we must enable/disable the netdata updater tool ---
Auto-updating has been enabled through cron, updater script linked to /etc/cron.daily/netdata-updater

If the update process fails and you have email notifications set up correctly for cron on this system, you should receive an email notification of the failure.
Successful updates will not send an email.

 --- Wrap up environment set up ---
Preparing .environment file
[/tmp/netdata-kickstart-EivSbPH6tn/netdata-v1.31.0-113-g512f9228e]# chmod 0644 /etc/netdata/.environment
 OK

Setting netdata.tarball.checksum to 'new_installation'

 --- We are done! ---

  ^
  |.-.   .-.   .-.   .-.   .-.   .  netdata                          .-.   .-
  |   '-'   '-'   '-'   '-'   '-'   is installed and running now!  -'   '-'
  +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->

  enjoy real-time performance and health monitoring...

 OK

10-references.md

• ISUCON10 予選でやったこと、失敗したこと : https://qiita.com/lostfind/items/40f77453f8c91d49014e

• https://gist.github.com/catatsuy/e627aaf118fbe001f2e7c665fda48146

• https://creators.oisix.co.jp/entry/2020/09/17/102711

• https://www.javacodemonk.com/upgrade-mysql-from-5-7-to-8-on-ubuntu-18-04-bd9b5e46