Skip to content

Instantly share code, notes, and snippets.

@kuntoaji
Last active August 29, 2015 14:04

Revisions

  1. kuntoaji revised this gist Aug 5, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion pilpres2014.rb
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,7 @@
    (1..149).each do |tps|

    # id kelurahan, bisa dimodifikasi
    (6959..100_000).each do |kel_id|
    (1..100_000).each do |kel_id|
    url = "http://pilpres2014.kpu.go.id/c1.php?cmd=download&tps=#{tps}&kel_id=#{kel_id}"

    # saved_url_list.txt add \n as new line
  2. kuntoaji created this gist Aug 5, 2014.
    51 changes: 51 additions & 0 deletions pilpres2014.rb
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,51 @@
    #!/usr/bin/env ruby
    require 'open-uri'

    user_agent = "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.16 Safari/537.36"
    saved_url_list = "saved_list.txt"
    accessed_urls = []

    accessed_urls = File.readlines(saved_url_list) if File.exists?(saved_url_list)

    # tps terbanyak 149
    # source: http://kpukotacimahi.com/joomla-overview/40-berita-depan/135-kelurahan-melong-memiliki-tps-terbanyak-dikota-cimahi
    # jumlah tps, bisa dimodifikasi
    (1..149).each do |tps|

    # id kelurahan, bisa dimodifikasi
    (6959..100_000).each do |kel_id|
    url = "http://pilpres2014.kpu.go.id/c1.php?cmd=download&tps=#{tps}&kel_id=#{kel_id}"

    # saved_url_list.txt add \n as new line
    unless accessed_urls.include?("#{url}\n")
    begin
    puts "Accessing #{url}"
    content = open(url, "User-Agent" => user_agent).read
    rescue
    puts "Retrying..."
    retry
    end

    file_name = "#{kel_id}_#{tps}.zip"
    unless File.exists?(file_name)
    if content.size > 7000
    File.open(file_name, 'w') {|f| f.write(content) }
    saved_file = File.open(saved_url_list, "a")
    saved_file.puts url
    saved_file.close
    puts "#{file_name} is successfully saved"

    sleep_in_seconds = Random.rand(10..15)
    puts "sleeping for #{sleep_in_seconds} seconds.."
    sleep sleep_in_seconds
    else
    puts "Empty"
    end
    else
    puts "#{file_name} is exist"
    end
    end
    end
    end

    puts "done"