<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
		>
<channel>
	<title>Comments on: Reminiscing about Provo411.com and Scraping the Course Catalog</title>
	<atom:link href="http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/feed" rel="self" type="application/rss+xml" />
	<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog</link>
	<description></description>
	<lastBuildDate>Tue, 15 May 2012 02:38:28 +0000</lastBuildDate>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
	<item>
		<title>By: Brendon Beebe</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-174327</link>
		<dc:creator>Brendon Beebe</dc:creator>
		<pubDate>Mon, 14 Nov 2011 17:55:11 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-174327</guid>
		<description>Hey,  it seems a lot of people have had the same idea!  I am the admin at schedule snatcher and we are thinking of moving up to  BYU-I.  I was wondering if any of you guys would like to help out with the code you have already been using. :D</description>
		<content:encoded><![CDATA[<div class='microid-9900d903623bdfc96355d5310b4abcbb33507c29'>Hey,  it seems a lot of people have had the same idea!  I am the admin at schedule snatcher and we are thinking of moving up to  BYU-I.  I was wondering if any of you guys would like to help out with the code you have already been using. <img src='http://richardkmiller.com/wp-includes/images/smilies/icon_biggrin.gif' alt=':D' class='wp-smiley' /> </div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Richard K Miller</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-170911</link>
		<dc:creator>Richard K Miller</dc:creator>
		<pubDate>Thu, 07 Jul 2011 02:23:47 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-170911</guid>
		<description>Thanks, Mere! Someone just told me a couple weeks ago about a project from some BYU students that does this -- http://www.schedulesnatcher.com/ . Sweet validation. :)</description>
		<content:encoded><![CDATA[<div class='microid-781638ceff08110e27cb6063c5cd105a241b5e67'>Thanks, Mere! Someone just told me a couple weeks ago about a project from some BYU students that does this &#8212; <a href="http://www.schedulesnatcher.com/" rel="nofollow">http://www.schedulesnatcher.com/</a> . Sweet validation. <img src='http://richardkmiller.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </div>
]]></content:encoded>
	</item>
	<item>
		<title>By: mere</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-170907</link>
		<dc:creator>mere</dc:creator>
		<pubDate>Thu, 07 Jul 2011 00:52:07 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-170907</guid>
		<description>i dont understand script, but i love this story. i remember when you told me about this when you used it at BYU--i thought &quot;he should charge 3$ for this!&quot;</description>
		<content:encoded><![CDATA[<div class='microid-3928ff6db91127a1f801518b1d9d5c482f71b626'>i dont understand script, but i love this story. i remember when you told me about this when you used it at BYU&#8211;i thought &#8220;he should charge 3$ for this!&#8221;</div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Richard K Miller</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-170207</link>
		<dc:creator>Richard K Miller</dc:creator>
		<pubDate>Sat, 04 Jun 2011 23:24:14 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-170207</guid>
		<description>Hey Jesse, funny to run into you here! Hope you&#039;re doing great.</description>
		<content:encoded><![CDATA[<div class='microid-781638ceff08110e27cb6063c5cd105a241b5e67'>Hey Jesse, funny to run into you here! Hope you&#8217;re doing great.</div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jesse</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-168330</link>
		<dc:creator>Jesse</dc:creator>
		<pubDate>Wed, 06 Apr 2011 02:12:10 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-168330</guid>
		<description>Richard! this is Jesse.  I knew you from enclave... i was friends with jamie woodward.  I found your site googling something.  anyway, crazy script.  genius.</description>
		<content:encoded><![CDATA[<div class='microid-f65ea28795ce46bb334bcc04995f97ba9dd00b2c'>Richard! this is Jesse.  I knew you from enclave&#8230; i was friends with jamie woodward.  I found your site googling something.  anyway, crazy script.  genius.</div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Richard K Miller</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-155105</link>
		<dc:creator>Richard K Miller</dc:creator>
		<pubDate>Tue, 16 Feb 2010 23:12:13 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-155105</guid>
		<description>Hey thanks, Jimmy. I actually just used Ruby Mechanize for the first time a few weeks ago. Great tool. I created a script to open/close my router&#039;s DMZ, which I&#039;ll post in a couple days.</description>
		<content:encoded><![CDATA[<div class='microid-781638ceff08110e27cb6063c5cd105a241b5e67'>Hey thanks, Jimmy. I actually just used Ruby Mechanize for the first time a few weeks ago. Great tool. I created a script to open/close my router&#8217;s DMZ, which I&#8217;ll post in a couple days.</div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jimmy Zimmerman</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-155103</link>
		<dc:creator>Jimmy Zimmerman</dc:creator>
		<pubDate>Tue, 16 Feb 2010 22:53:33 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-155103</guid>
		<description>Cool script. You might want to also check out the &#039;mechanize&#039; gem. It acts as a browser in maintaining session cookies, setting the User Agent string to appear to be Safari/Firefox/IE, etc. It has CSS selector support for scraping information that you want from a page or submitting forms. It easily turns any website into a web service.

--
Jimmy</description>
		<content:encoded><![CDATA[<div class='microid-511ec28010a82e20e63514fd22985dac08012531'>Cool script. You might want to also check out the &#8216;mechanize&#8217; gem. It acts as a browser in maintaining session cookies, setting the User Agent string to appear to be Safari/Firefox/IE, etc. It has CSS selector support for scraping information that you want from a page or submitting forms. It easily turns any website into a web service.</p>
<p>&#8211;<br />
Jimmy</p></div>
]]></content:encoded>
	</item>
	<item>
		<title>By: Richard K Miller</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-153606</link>
		<dc:creator>Richard K Miller</dc:creator>
		<pubDate>Wed, 14 Oct 2009 07:52:20 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-153606</guid>
		<description>@D. Jamison: Awesome thoughts and comments. Thanks. I see it&#039;s not a lone interest of mine.

Your idea of giving people software they can run on their own computer seems on the money. Wesabe.com does this (with personal finances) - they offer a desktop app for logging into your bank account and pulling in your transactions. Your bank password stays safely on your machine. I imagine this should nearly eradicate concerns about liability, anonymizing traffic, randomizing requests, etc.</description>
		<content:encoded><![CDATA[<div class='microid-781638ceff08110e27cb6063c5cd105a241b5e67'>@D. Jamison: Awesome thoughts and comments. Thanks. I see it&#8217;s not a lone interest of mine.</p>
<p>Your idea of giving people software they can run on their own computer seems on the money. Wesabe.com does this (with personal finances) &#8211; they offer a desktop app for logging into your bank account and pulling in your transactions. Your bank password stays safely on your machine. I imagine this should nearly eradicate concerns about liability, anonymizing traffic, randomizing requests, etc.</p></div>
]]></content:encoded>
	</item>
	<item>
		<title>By: D. Jamison</title>
		<link>http://richardkmiller.com/780/reminiscing-about-provo411-com-and-scraping-the-course-catalog/comment-page-1#comment-153555</link>
		<dc:creator>D. Jamison</dc:creator>
		<pubDate>Thu, 08 Oct 2009 21:19:24 +0000</pubDate>
		<guid isPermaLink="false">http://richardkmiller.com/?p=780#comment-153555</guid>
		<description>I&#039;m glad to see that others recognize the need for a third party to assist in course registration.  This seems to be a common problem and I don&#039;t expect the schools have any interest in addressing it. I had written something like what you have there for undergrad.  I went a couple steps further: mine registered me for the class and sent me a text when I was in.

I let a couple friends use my code but I otherwise forgot about the idea until law school.  Something about those first-year grades really gives everyone a lot of anxiety.  And we all wanted to have our results the instant they came out.  I was even checking constantly on the weekends, knowing that they wouldn&#039;t be released on a Saturday.  I finally put together a website that automated the task.  I had some friends help test it.  It worked as advertised, sending alerts when new grades arrived.

I realized, like you did, that the school may not appreciate being hammered by such a system.  I came up with a couple solutions.  The first is to add some intelligence to the checking.  You can aggregate requests for a particular course (whether it be for grades or for a spot in registration) and then your requests become O(num. of courses) instead of O(num. of users) (depending on how the school&#039;s system works, of course).  I did some research and discovered that many schools use the same 3rd parties for their web/database systems.  So expanding to several schools seemed like it would be easy: customizing, when necessary, for proprietary systems.

But that only works if students want to be notified when grades are available - not for alerting the students to their individual grade.  In order to do that, the student has to authenticate to the site.  Sure, aggregation could reduce the polling for updates and then the individuals could be logged in to retrieve corresponding data (or register them).  It just seemed to snowball into lots of traffic.  More than that - we&#039;d be requiring the credentials of the students in order to do their F5ing for them.  There are privacy/security issues there and I&#039;m sure it violates the policies of every school.

The solution was to put the tools at the edge: let the users run the software.  And this is where I abandoned the project - I didn&#039;t care about my grades any longer.  But the idea was this: give users some app that they can run on their machine.  The credentials remain with the student, and the traffic hitting the school is now distributed.  The traffic could even be made to look like human browsing (with a fake user-agent and random delays between requests).

Anyway, I think the demand is there.  Students who desperately need a class to graduate will pay for this and I think overzealous grade checkers will too.  The money comes in through sales of apps that expire and use the right licenses.  Maybe there&#039;s a web-based solution that can be platform-independent, but I don&#039;t know what it is.</description>
		<content:encoded><![CDATA[<div class='microid-07cc751b63f822659abe54b6baae43d3d0f84502'>I&#8217;m glad to see that others recognize the need for a third party to assist in course registration.  This seems to be a common problem and I don&#8217;t expect the schools have any interest in addressing it. I had written something like what you have there for undergrad.  I went a couple steps further: mine registered me for the class and sent me a text when I was in.</p>
<p>I let a couple friends use my code but I otherwise forgot about the idea until law school.  Something about those first-year grades really gives everyone a lot of anxiety.  And we all wanted to have our results the instant they came out.  I was even checking constantly on the weekends, knowing that they wouldn&#8217;t be released on a Saturday.  I finally put together a website that automated the task.  I had some friends help test it.  It worked as advertised, sending alerts when new grades arrived.</p>
<p>I realized, like you did, that the school may not appreciate being hammered by such a system.  I came up with a couple solutions.  The first is to add some intelligence to the checking.  You can aggregate requests for a particular course (whether it be for grades or for a spot in registration) and then your requests become O(num. of courses) instead of O(num. of users) (depending on how the school&#8217;s system works, of course).  I did some research and discovered that many schools use the same 3rd parties for their web/database systems.  So expanding to several schools seemed like it would be easy: customizing, when necessary, for proprietary systems.</p>
<p>But that only works if students want to be notified when grades are available &#8211; not for alerting the students to their individual grade.  In order to do that, the student has to authenticate to the site.  Sure, aggregation could reduce the polling for updates and then the individuals could be logged in to retrieve corresponding data (or register them).  It just seemed to snowball into lots of traffic.  More than that &#8211; we&#8217;d be requiring the credentials of the students in order to do their F5ing for them.  There are privacy/security issues there and I&#8217;m sure it violates the policies of every school.</p>
<p>The solution was to put the tools at the edge: let the users run the software.  And this is where I abandoned the project &#8211; I didn&#8217;t care about my grades any longer.  But the idea was this: give users some app that they can run on their machine.  The credentials remain with the student, and the traffic hitting the school is now distributed.  The traffic could even be made to look like human browsing (with a fake user-agent and random delays between requests).</p>
<p>Anyway, I think the demand is there.  Students who desperately need a class to graduate will pay for this and I think overzealous grade checkers will too.  The money comes in through sales of apps that expire and use the right licenses.  Maybe there&#8217;s a web-based solution that can be platform-independent, but I don&#8217;t know what it is.</p></div>
]]></content:encoded>
	</item>
</channel>
</rss>

