Skip to content

Instantly share code, notes, and snippets.

@thejhh

thejhh/ram-leaking.md

Created January 24, 2012 20:42
Download ZIP

Revisions

  1. thejhh revised this gist Jan 25, 2012. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -16,9 +16,9 @@ print_debug_line() {
    snapshot_create() {
    name="$1"
    print_debug_line "snapshot_create: before filesystem sync"
    btrfs filesystem sync "$ramdisk_dir"
    btrfs filesystem sync "$ramdisk_dir" >/dev/null || return 2
    print_debug_line "snapshot_create: before snapshot creation"
    btrfs subvolume snapshot "$snapshot_dir/$name" "$master_ram_git" >/dev/null || return 1
    btrfs subvolume snapshot "$ramdisk_dir" "$snapshot_dir/$name" >/dev/null || return 1
    #btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    print_debug_line "snapshot_create: after snapshot creation"
    return 0
  2. root revised this gist Jan 25, 2012. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -15,8 +15,8 @@ print_debug_line() {
    # Create working copy in a COW snapshot
    snapshot_create() {
    name="$1"
    print_debug_line "snapshot_create: before sync"
    sync
    print_debug_line "snapshot_create: before filesystem sync"
    btrfs filesystem sync "$ramdisk_dir"
    print_debug_line "snapshot_create: before snapshot creation"
    btrfs subvolume snapshot "$snapshot_dir/$name" "$master_ram_git" >/dev/null || return 1
    #btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
  3. root revised this gist Jan 25, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -18,7 +18,7 @@ snapshot_create() {
    print_debug_line "snapshot_create: before sync"
    sync
    print_debug_line "snapshot_create: before snapshot creation"
    btrfs subvolume snapshot "$master_ram_git" "$snapshot_dir/$name" >/dev/null || return 1
    btrfs subvolume snapshot "$snapshot_dir/$name" "$master_ram_git" >/dev/null || return 1
    #btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    print_debug_line "snapshot_create: after snapshot creation"
    return 0
  4. root revised this gist Jan 25, 2012. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -18,7 +18,8 @@ snapshot_create() {
    print_debug_line "snapshot_create: before sync"
    sync
    print_debug_line "snapshot_create: before snapshot creation"
    btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    btrfs subvolume snapshot "$master_ram_git" "$snapshot_dir/$name" >/dev/null || return 1
    #btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    print_debug_line "snapshot_create: after snapshot creation"
    return 0
    }
    @@ -30,7 +31,8 @@ snapshot_delete() {
    print_debug_line "snapshot_delete: Before changing pwd"
    cd "$snapshot_dir" || return 2
    print_debug_line "snapshot_delete: before deleting snapshot"
    btrfsctl -D "$name" .|grep -qxF 'operation complete' || status=1
    btrfs subvolume delete "$name" > /dev/null || status=1
    #btrfsctl -D "$name" .|grep -qxF 'operation complete' || status=1
    print_debug_line "snapshot_delete: before changing pwd back"
    cd "$orig_pwd"
    print_debug_line "snapshot_delete: after changing pwd back"
  5. thejhh revised this gist Jan 25, 2012. 2 changed files with 56 additions and 0 deletions.
    27 changes: 27 additions & 0 deletions ram-leaking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    Why does deleting a snapshot increase the used space on btrfs filesystem?

    # time ./transaction-test.sh
    DEBUG 452 work_begin: before snapshot creation
    DEBUG 452 snapshot_create: before sync
    DEBUG 408 snapshot_create: before snapshot creation
    DEBUG 412 snapshot_create: after snapshot creation
    DEBUG 412 work_begin: before changing pwd
    DEBUG 412 work_begin: after changing pwd
    DEBUG 412 do_test: before test
    DEBUG 412 do_test: after test
    DEBUG 412 work_commit: before git commit
    DEBUG 412 work_commit: before changing pwd to master ram git
    DEBUG 412 work_commit: before git pull
    DEBUG 412 work_commit: before changing pwd back
    DEBUG 412 work_commit: before changing pwd back
    DEBUG 412 snapshot_delete: Before changing pwd
    DEBUG 412 snapshot_delete: before deleting snapshot
    DEBUG 460 snapshot_delete: before changing pwd back
    DEBUG 460 snapshot_delete: after changing pwd back
    DEBUG 460 work_commit: after changing pwd back
    Commit successful.

    real 0m0.431s
    user 0m0.272s
    sys 0m0.228s

    29 changes: 29 additions & 0 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -5,50 +5,76 @@ snapshot_dir="$ramdisk_dir"/snapshot
    master_ram_git="$ramdisk_dir"/data
    orig_pwd="$(pwd)"

    # Output line for debuging
    print_debug_line() {
    msg="$1"
    ram="$(df -P /ramdisks/mnt/disk0|tail -n 1|awk '{print $3}')"
    echo "DEBUG\t$ram\t$msg" >&2
    }

    # Create working copy in a COW snapshot
    snapshot_create() {
    name="$1"
    print_debug_line "snapshot_create: before sync"
    sync
    print_debug_line "snapshot_create: before snapshot creation"
    btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    print_debug_line "snapshot_create: after snapshot creation"
    return 0
    }

    # Delete snapshot
    snapshot_delete() {
    name="$1"
    status=0
    print_debug_line "snapshot_delete: Before changing pwd"
    cd "$snapshot_dir" || return 2
    print_debug_line "snapshot_delete: before deleting snapshot"
    btrfsctl -D "$name" .|grep -qxF 'operation complete' || status=1
    print_debug_line "snapshot_delete: before changing pwd back"
    cd "$orig_pwd"
    print_debug_line "snapshot_delete: after changing pwd back"
    return "$status"
    }

    # Start working on snapshot
    work_begin() {
    name="$1"
    print_debug_line "work_begin: before snapshot creation"
    snapshot_create "disk0-$time"
    print_debug_line "work_begin: before changing pwd"
    cd "$snapshot_dir/$name/data"
    print_debug_line "work_begin: after changing pwd"
    return 0
    }

    # Commit changes to master ram git
    work_commit() {
    name="$1"
    status=0
    print_debug_line "work_commit: before git commit"
    git commit -q -a -m 'Changed data' || return 1
    print_debug_line "work_commit: before changing pwd to master ram git"
    cd "$master_ram_git" || return 3
    print_debug_line "work_commit: before git pull"
    git pull -q "$snapshot_dir/$name/data" master || status=2
    print_debug_line "work_commit: before changing pwd back"
    cd "$orig_pwd" || return 4
    print_debug_line "work_commit: before changing pwd back"
    snapshot_delete "$name" || return 5
    print_debug_line "work_commit: after changing pwd back"
    return "$status"
    }

    # Rollback, lcean up snapshot without saving data
    work_rollback() {
    name="$1"
    status=0
    print_debug_line "work_rollback: before changing pwd back"
    cd "$orig_pwd" || return 4
    print_debug_line "work_rollback: before snapshot deletion"
    snapshot_delete "$name" || return 5
    print_debug_line "work_rollback: after snapshot deletion"
    return "$status"
    }

    @@ -58,10 +84,13 @@ time="$(date +%Y%m%d-%H%M%S-%N)"

    # Test modifying data
    do_test() {
    print_debug_line "do_test: before test"
    echo 'New test line written '"$(date -u)" >> test.txt || return 1
    print_debug_line "do_test: after test"
    return 0
    }

    # Perform the test
    if work_begin "disk0-$time"; then
    if do_test; then
    if work_commit "disk0-$time"; then
  6. thejhh revised this gist Jan 24, 2012. 1 changed file with 40 additions and 0 deletions.
    40 changes: 40 additions & 0 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -70,3 +70,43 @@ For some strange reason this doesn't work:
    # btrfsctl -D disk0-20120124-2149 /ramdisks/mnt/disk0/snapshot
    ioctl:: No such file or directory

    Transaction tests
    -----------------

    # time ./transaction-test.sh
    Commit successful.

    real 0m0.220s
    user 0m0.112s
    sys 0m0.072s

    Test Platform
    -------------

    I am running these tests on a VIA ARTiGO A2000 Storage Server which has a VIA C7-D processor.

    # cat /proc/cpuinfo
    processor : 0
    vendor_id : CentaurHauls
    cpu family : 6
    model : 13
    model name : VIA C7-D Processor 1500MHz
    stepping : 0
    cpu MHz : 1499.885
    cache size : 128 KB
    fdiv_bug : no
    hlt_bug : no
    f00f_bug : no
    coma_bug : no
    fpu : yes
    fpu_exception : yes
    cpuid level : 1
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge cmov pat clflush acpi mmx fxsr sse sse2 tm nx up pni xtpr rng rng_en ace ace_en ace2 ace2_en phe phe_en pmm pmm_en
    bogomips : 2999.77
    clflush size : 64
    cache_alignment : 64
    address sizes : 36 bits physical, 32 bits virtual
    power management:

    This machine has 1 GB 800 MHz DDR-2 RAM.
  7. thejhh revised this gist Jan 24, 2012. 1 changed file with 39 additions and 9 deletions.
    48 changes: 39 additions & 9 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -8,6 +8,7 @@ orig_pwd="$(pwd)"
    # Create working copy in a COW snapshot
    snapshot_create() {
    name="$1"
    sync
    btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    return 0
    }
    @@ -25,29 +26,58 @@ snapshot_delete() {
    # Start working on snapshot
    work_begin() {
    name="$1"
    snapshot_create "disk0-$time"
    cd "$snapshot_dir/$name/data"
    return 0
    }

    # Commit changes to master ram git
    work_commit() {
    name="$1"
    git commit -q -a -m 'Changed data'
    cd "$master_ram_git"
    git pull -q "$snapshot_dir/$name/data" master
    cd "$orig_pwd"
    status=0
    git commit -q -a -m 'Changed data' || return 1
    cd "$master_ram_git" || return 3
    git pull -q "$snapshot_dir/$name/data" master || status=2
    cd "$orig_pwd" || return 4
    snapshot_delete "$name" || return 5
    return "$status"
    }

    # Rollback, lcean up snapshot without saving data
    work_rollback() {
    name="$1"
    status=0
    cd "$orig_pwd" || return 4
    snapshot_delete "$name" || return 5
    return "$status"
    }

    set -e

    time="$(date +%Y%m%d-%H%M%S-%N)"

    # Test modifying data
    snapshot_create "disk0-$time"
    work_begin "disk0-$time"
    do_test() {
    echo 'New test line written '"$(date -u)" >> test.txt || return 1
    return 0
    }

    echo 'New test line written '"$(date -u)" >> test.txt
    if work_begin "disk0-$time"; then
    if do_test; then
    if work_commit "disk0-$time"; then
    echo 'Commit successful.'
    else
    echo 'Error: Commit failed!' >&2
    work_rollback "disk0-$time"
    fi
    else
    echo 'Error: Test failed!' >&2
    work_rollback "disk0-$time"
    fi
    else
    echo 'Error: Failed to begin transaction!' >&2
    exit 1
    fi

    work_commit "disk0-$time"
    snapshot_delete "disk0-$time"

    # EOF
  8. thejhh revised this gist Jan 24, 2012. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,11 +1,11 @@
    #!/bin/sh -x
    #!/bin/sh

    ramdisk_dir=/ramdisks/mnt/disk0
    snapshot_dir="$ramdisk_dir"/snapshot
    master_ram_git="$ramdisk_dir"/data
    orig_pwd="$(pwd)"

    # Create snapshot working copy
    # Create working copy in a COW snapshot
    snapshot_create() {
    name="$1"
    btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    @@ -31,9 +31,9 @@ work_begin() {
    # Commit changes to master ram git
    work_commit() {
    name="$1"
    git commit -a -m 'Changed data'
    git commit -q -a -m 'Changed data'
    cd "$master_ram_git"
    git pull "$snapshot_dir/$name/data" master
    git pull -q "$snapshot_dir/$name/data" master
    cd "$orig_pwd"
    }

  9. jhh revised this gist Jan 24, 2012. 1 changed file with 53 additions and 0 deletions.
    53 changes: 53 additions & 0 deletions transaction-test.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,53 @@
    #!/bin/sh -x

    ramdisk_dir=/ramdisks/mnt/disk0
    snapshot_dir="$ramdisk_dir"/snapshot
    master_ram_git="$ramdisk_dir"/data
    orig_pwd="$(pwd)"

    # Create snapshot working copy
    snapshot_create() {
    name="$1"
    btrfsctl -s "$snapshot_dir/$name" "$master_ram_git"|grep -qxF 'operation complete' || return 1
    return 0
    }

    # Delete snapshot
    snapshot_delete() {
    name="$1"
    status=0
    cd "$snapshot_dir" || return 2
    btrfsctl -D "$name" .|grep -qxF 'operation complete' || status=1
    cd "$orig_pwd"
    return "$status"
    }

    # Start working on snapshot
    work_begin() {
    name="$1"
    cd "$snapshot_dir/$name/data"
    }

    # Commit changes to master ram git
    work_commit() {
    name="$1"
    git commit -a -m 'Changed data'
    cd "$master_ram_git"
    git pull "$snapshot_dir/$name/data" master
    cd "$orig_pwd"
    }

    set -e

    time="$(date +%Y%m%d-%H%M%S-%N)"

    # Test modifying data
    snapshot_create "disk0-$time"
    work_begin "disk0-$time"

    echo 'New test line written '"$(date -u)" >> test.txt

    work_commit "disk0-$time"
    snapshot_delete "disk0-$time"

    # EOF
  10. thejhh revised this gist Jan 24, 2012. 1 changed file with 19 additions and 7 deletions.
    26 changes: 19 additions & 7 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,12 +1,16 @@
    These are my notes while testing.

    Installing newer btrfs-tools
    ----------------------------

    Ubuntu LTS 10.04 doesn't support removing btrfs snapshots. It can be fixed by installing newer btrfs.

    cd /tmp
    wget "http://se.archive.ubuntu.com/ubuntu/pool/main/b/btrfs-tools/btrfs-tools_0.19+20100601-3ubuntu3_i386.deb"
    dpkg -i btrfs-tools_0.19+20100601-3ubuntu3_i386.deb

    Creating the master git repository (on a flash drive):
    ----------------------------

    mkdir -p /flash/disk0
    cd /flash/disk0
    @@ -15,32 +19,40 @@ Creating the master git repository (on a flash drive):
    git add test.txt
    git commit -a -m 'Added initial test data.'

    Initialising tmpfs:
    Initialising tmpfs
    ----------------------------

    mkdir /ramdisks
    mount -t tmpfs tmpfs /ramdisks
    mkdir /ramdisks/raw /ramdisks/mnt

    Creating images on ramdisk:
    Creating images on ramdisk
    ----------------------------

    dd if=/dev/zero of=/ramdisks/raw/disk0 bs=1M count=256

    Initializing btrfs image:
    Initializing btrfs image
    ----------------------------

    mkfs.btrfs /ramdisks/raw/disk0

    Mounting btrfs image:
    Mounting btrfs image
    ----------------------------

    mount -o loop /ramdisks/raw/disk0 /ramdisks/mnt/disk0

    Cloning flash to the ramdisk:
    Cloning flash to the ramdisk
    ----------------------------

    git clone /flash/disk0 /ramdisks/mnt/disk0/data

    Creating snapshot:
    Creating snapshot
    -----------------

    btrfsctl -s /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/data

    Removing snapshot:
    Removing snapshot
    -----------------

    cd /ramdisks/mnt/disk0/snapshot && btrfsctl -D disk0-20120124-2149 . && cd -

  11. thejhh revised this gist Jan 24, 2012. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -44,9 +44,17 @@ Removing snapshot:

    cd /ramdisks/mnt/disk0/snapshot && btrfsctl -D disk0-20120124-2149 . && cd -

    For some strange reason these do not work:
    For some strange reason this doesn't work:

    # btrfsctl -D /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/snapshot
    ioctl:: Invalid argument

    ...or any of these -- although these are probably wrong anyway:

    # btrfsctl -D disk0-20120124-2149 /ramdisks/mnt/disk0/data
    ioctl:: No such file or directory
    # btrfsctl -D /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/data
    ioctl:: Invalid argument
    # btrfsctl -D disk0-20120124-2149 /ramdisks/mnt/disk0/snapshot
    ioctl:: No such file or directory

  12. thejhh revised this gist Jan 24, 2012. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -43,3 +43,10 @@ Creating snapshot:
    Removing snapshot:

    cd /ramdisks/mnt/disk0/snapshot && btrfsctl -D disk0-20120124-2149 . && cd -

    For some strange reason these do not work:

    # btrfsctl -D disk0-20120124-2149 /ramdisks/mnt/disk0/data
    ioctl:: No such file or directory
    # btrfsctl -D /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/data
    ioctl:: Invalid argument
  13. thejhh revised this gist Jan 24, 2012. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -39,3 +39,7 @@ Cloning flash to the ramdisk:
    Creating snapshot:

    btrfsctl -s /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/data

    Removing snapshot:

    cd /ramdisks/mnt/disk0/snapshot && btrfsctl -D disk0-20120124-2149 . && cd -
  14. thejhh renamed this gist Jan 24, 2012. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → testing.md
    View file
    File renamed without changes.
  15. thejhh revised this gist Jan 24, 2012. 1 changed file with 25 additions and 0 deletions.
    25 changes: 25 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -6,10 +6,35 @@ Ubuntu LTS 10.04 doesn't support removing btrfs snapshots. It can be fixed by in
    wget "http://se.archive.ubuntu.com/ubuntu/pool/main/b/btrfs-tools/btrfs-tools_0.19+20100601-3ubuntu3_i386.deb"
    dpkg -i btrfs-tools_0.19+20100601-3ubuntu3_i386.deb

    Creating the master git repository (on a flash drive):

    mkdir -p /flash/disk0
    cd /flash/disk0
    git init
    echo 'This is a test file.' > test.txt
    git add test.txt
    git commit -a -m 'Added initial test data.'

    Initialising tmpfs:

    mkdir /ramdisks
    mount -t tmpfs tmpfs /ramdisks
    mkdir /ramdisks/raw /ramdisks/mnt

    Creating images on ramdisk:

    dd if=/dev/zero of=/ramdisks/raw/disk0 bs=1M count=256

    Initializing btrfs image:
    mkfs.btrfs /ramdisks/raw/disk0

    Mounting btrfs image:

    mount -o loop /ramdisks/raw/disk0 /ramdisks/mnt/disk0

    Cloning flash to the ramdisk:

    git clone /flash/disk0 /ramdisks/mnt/disk0/data

    Creating snapshot:

  16. thejhh created this gist Jan 24, 2012.
    16 changes: 16 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,16 @@
    These are my notes while testing.

    Ubuntu LTS 10.04 doesn't support removing btrfs snapshots. It can be fixed by installing newer btrfs.

    cd /tmp
    wget "http://se.archive.ubuntu.com/ubuntu/pool/main/b/btrfs-tools/btrfs-tools_0.19+20100601-3ubuntu3_i386.deb"
    dpkg -i btrfs-tools_0.19+20100601-3ubuntu3_i386.deb

    Mounting btrfs image:

    mount -o loop /ramdisks/raw/disk0 /ramdisks/mnt/disk0


    Creating snapshot:

    btrfsctl -s /ramdisks/mnt/disk0/snapshot/disk0-20120124-2149 /ramdisks/mnt/disk0/data