Starscrape2 with Nokogiri

Tag: Code

Kay people, it’s Sunday and I’m a bit bored, so I wrote a ruby script to webscrape Starcraft 2 profile pages using the wonderful Nokogiri gem.
Well it’s done now and to be honest I have no idea what to do with it, so I’ll share it with you instead.

#starscrape.rb
require 'rubygems'
require 'nokogiri'
require 'open-uri'

def starcraft_profile(url)
  h = Nokogiri::HTML open(url)
  { 
      :name=>h.css('#profile-header h2').text,
      :achievement_points => h.css('#profile-header h3').text,
      :snapshot => {
          :favored_race => h.css('#season-snapshot .module-footer a').text,
          :highest_ranks => Hash[([1,2,3,4].map{|i| 
              ["#{i}on#{i}".to_sym,{
                  :league =>  h.xpath("//div[@id=\"best-team-#{i}\"]/div/strong/../@class").text.gsub('badge-',''),
                  :division=> h.xpath("//div[@id=\"best-team-#{i}\"]/div/strong/..").text.match(/Division: (.+)Rank: (\d+)Record: (\d+ - \d+)/)<a href="/posts/1">1</a>.strip,
                  :rank => h.xpath("//div[@id=\"best-team-#{i}\"]/div/strong/..").text.match(/Division: (.+)Rank: (\d+)Record: (\d+ - \d+)/)<a href="/posts/2">2</a>.strip,
                  :record => h.xpath("//div[@id=\"best-team-#{i}\"]/div/strong/..").text.match(/Division: (.+)Rank: (\d+)Record: (\d+ - \d+)/)<a href="/posts/3">3</a>.strip                
              }]
          })],
      },
      :career_stats => {
          :league_wins => h.css('#career-stats h2').text,
          :league_games => h.css('#career-stats ul li span')[0].text,
          :custom_games => h.css('#career-stats ul li span')<a href="/posts/1">1</a>.text,
          :coop_vs_ai => h.css('#career-stats ul li span')<a href="/posts/2">2</a>.text,
          :ffa => h.css('#career-stats ul li span')<a href="/posts/3">3</a>.text,
          :campaign_status=> h.css('#career-stats h4')<a href="/posts/2">2</a>.text
      },
      :achievements => Hash[h.css('#top-achievements .progress-tile').map{|p|
          [p.css('a')[0].text.gsub(/[^\w ]/,''),{
              :points=> p.css('span').text.strip,
              :progress => p.css('.progress-bar').attribute('style').value.match(/width: (\S+)%/)<a href="/posts/1">1</a>.to_f
          }]
      }],
      :showcase =>  h.css('.showcase-tile p a').map{|a| a.text}
  }
end

This code will return a pretty hash stored with all the pretty informaiton: (Yeah I know I’m not hotshot player, but still this is fun ;) ) 

starcraft_profile http://eu.battle.net/sc2/en/profile/836668/1/telamon/"=>
 {:achievement_points=>"2640", :showcase=>  ["Team Random 100",   "Wings of Liberty: Hard 25",   "Hot Shot",   "League Qualifier",   "Wings of Liberty"], :snapshot=>  {:favored_race=>"Random",   :highest_ranks=>    {:"1on1"=>      {:division=>"Roxara Foxtrot",       :rank=>"69",       :record=>"31 - 41",       :league=>"silver"},     :"2on2"=>      {:division=>"Immortal Mars",       :rank=>"7",       :record=>"26 - 21",       :league=>"silver"},     :"3on3"=>      {:division=>"Sarengo Foxtrot",       :rank=>"62",       :record=>"11 - 11",       :league=>"platinum"},     :"4on4"=>      {:division=>"Pridewater Nu",       :rank=>"15",       :record=>"22 - 28",       :league=>"silver"}}}, :career_stats=>  {:custom_games=>"50",   :coop_vs_ai=>"3",   :ffa=>"5",   :campaign_status=>"Normal Campaign Ace",   :league_wins=>"265",   :league_games=>"556"}, :name=>"numri", :achievements=>  {"Cooperative"=>{:points=>"0", :progress=>0.0},   "Liberty Campaign"=>{:points=>"1410", :progress=>88.679245283},   "Exploration"=>{:points=>"480", :progress=>100.0},   "Custom Game"=>{:points=>"10", :progress=>1.4705882353},   "Quick Match"=>{:points=>"220", :progress=>25.2873563218}}}