Categories
Business Entrepreneurship Main Programming Tech

Reminiscing about Provo411.com and Scraping the Course Catalog

One of my first web development projects and biz partnerships with Brian Stucki was Provo411.com. We were roommates at BYU and conceived of a website where students could share events — parties, concerts, football games, etc. We were already in our beds for the night when the idea came, but we couldn’t go to sleep before buying the domain. I think it was the first domain I ever bought. It was September 2002.

I developed a calendar in PHP and wrote a few scripts to scrape byucougars.com and retrieve the sports schedules. I also developed a WML app so Brian and I could add events to the calendar from our pre-iPhone mobile phones. I recall being at a party in south Provo, in a former dental office, and using my Nextel phone to add the party to Provo411. If you go back far enough, you can see events on the calendar. My brother Alan did the artwork.

I always wanted Provo411.com to have a course schedule alert system. Perhaps students would pay $3 to receive an email or SMS alert when hard-to-get classes had an opening. It shouldn’t have been hard technically, but the publicly available course catalog isn’t updated in real-time. I could have scraped the authenticated course catalog on Route Y, but BYU might have objected and it’d be a fragile business model.

My brother Michael recently came home from his mission and started school at CSN. The business classes he wanted were full, so I put the old “course schedule alert” idea to the test with some new tools — Ruby and Mac OS X’s speech. Here’s what I came up with:

#!/usr/bin/env ruby

# a list of course call numbers to check
call_numbers = %w{ 46405 46407 46409 46411 46415 46413 53252 53254 53256 53258 53260 53262 53268 53270 53272 53274 46423 46435 53276 46443 }

# auth_token obtained via Firefox+TamperData while my brother logged into CSN
auth_token = "123456789012345"

say "Checking"

call_numbers.uniq.sort.each do |call_number|
    c = `curl -si -d CONVTOKEN=#{auth_token} -d AUDITT=N -d CALLT=#{call_number} -d CONTINUE=Continue "https://bighorn.nevada.edu/sis_csn/XSMBWEBM/SIVRE04.STR"`
    print "Call number #{call_number}: "
    if (c =~ /<p class="p5">([^< &#93;+)<br\/>/m)
        if $1.strip.empty?
            puts "May have openings\n"
            3.times {say "Michael, class number #{call_number} may be open!"}
        else
            puts "#{$1.strip}\n"
        end
    else
        puts "could not find message"
        say "Help. I cannot access the C S N website."
        return
    end
    sleep 5
end

# Ouput an audible message via Mac OS X's speech function
def say(message)
    `say "#{message}"`
end

We set this to run every 15 minutes on the living room iMac, and we turned up the volume. Every 15 minutes we could hear “Checking” from the computer. A few hours later we heard the script announce that a class had opened up. Michael, I’m still waiting for my $3.