Sometimes you’re up late, browsing around on the web, and something hits you. You get immediate inspiration– you’re not sure how you can achieve what you’ve just thought of, it might be impossible, and it might not be worth your time, but it’s too late: you’re going to spend the next 6 hours trying to figure it out.

This is what happened to me last night, after I read a very simple and straightforward tweet from my good friend Clint:

Shit. There goes my time.

Kopp’s Frozen Custard is, if you don’t know already, a frozen custard & burger place whose home resides in Wisconsin. Their food is pretty incredible albeit probably pretty terrible for you, but hey– it’s Wisconsin. I love it so much that I made an extremely dorky YouTube video about it.

Anyway, it was time to create a twitter account for the frozen custard restaurant.

Step 1. Scrape the HTML

The flavors of the day for Kopp’s are displayed on their front page, embedded in an iFrame. It’s just a PHP script that updates every day with the new flavors.

Of course, at this point, Kopp’s could’ve easily made an RSS feed for their flavors and I wouldn’t've wasted any of my time. But, it also means I couldn’t've done this relatively fun project, so I’m a little unsure of my stance on the lack of a feed.

So, since Kopp’s didn’t offer a feed, I had to make one. After a good amount of searching on the web, I found a service that makes feeds from basic HTML sites, an “HTML scraper”. It’s called Feed43.

On Feed43, I was given the ability to input a website (the flavor list) and then given the option to narrow down variables within the HTML. I successfully retrieved both flavors from the list and embedded them into an XML format that would look good on Twitter.. Essentially, it was like this:

Flavors of the Day for [Date]: [Flavor 1] & [Flavor 2]

Pretty simple, huh?

So that (or, so I thought) concluded the scraping part of the process.

Step 2. RSS to Twitter

A quick Google search for “RSS to Twitter” comes up with quite a few responses. A lot of the services have been shut down, but one seemed to fit the ticket quite nicely: twitterfeed.

“Feed your blog to twitter,” they explain on the site. If only it were that easy.

Quickly making a Kopp’s twitter account then inputing the information into the twitterfeed preferences page, I was sadly greeted by an error:

We couldn’t find valid date/time stamps on items. Please make sure your feed contains valid pubDate entries for each post.

And this is where it all went downhill. Twitterfeed requires a pubDate on each feed post because when it checks every given interval, it compares the date of the most recent tweet it’s posted to the most recent feed post. If there’s a newer post, it tweets. Simple enough, right?

Well, the feed from Feed43 didn’t include pubDate, as you could probably gather. Why twitterfeed requires pubDate, I’m not so sure, because Twitter automatically rejects duplicate tweets. I’m assuming it’s just a nice thing to do so that twitterfeed doesn’t bring down Twitter with countless pings (although, maybe it would be a good plan considering what they could do to finally get money out of the project).

So, it wasn’t going to be this easy.

Step 3. Trying (& Failing) to Add pubDate to Feed

Basically, at this point, I had to find a way to somehow inject pubDate into the newly-created Kopp’s FlavorFeed™. It didn’t matter what time or date it injected (because of what I explained earlier about Twitter blocking duplicate tweets), it just needed to be there to make twitterfeed happy.

Yahoo Pipes came into mind right away, so I set off to try and use that.

What a pain in the ass. After dicking around for a good 2 hours or so, trying to insert pubDate into the feed using modules and connections and doohickeys, I had nothing to show for myself besides a mangled RSS feed that changed every time I refreshed it. Twitterfeed threw out every version I could throw at it. I don’t blame it.

Step 4. Taking a Step Backwards

I gave up on trying to toss pubDate into the feed, so I tried to find some other HTML scraper that would give me a feed back with a date in it.

Here’s where Dapper came into play. Dapper seemed to be a Feed43 for those who didn’t know what they were doing (an easier version). In some ways, this was great– it was painless to isolate the variables I needed– but then it also was a huge pain in the ass in itself. It didn’t give me nearly the same editing power that Feed43 gave me. Instead of being able to output what I had wanted before (Flavor of the Day for [Date]: [Flavor 1] & [Flavor 2]), all it could give me was [Date]: [FLAVOR 1], [FLAVOR 2].

But you know what? I decided to take Dapper up on its offer and not stress too much about losing an ampersand and a prefix. Twitterfeed let me add a prefix of my own, limited to 20 characters, which I REALLY don’t understand, meaning I had to do with “Flavor for [Date]” instead.

However, in the end, it worked and now I have a twitter feed that updates daily with Kopps’ Flavor of the Day. Hooray– I finally allowed myself to get to sleep. At 8AM.

Kopp’s Frozen Custard FlavorFeed™ on Twitter (@kopps)

What I’ve Learned & What I Can Do Better

I made a lot of compromises to get the Kopp’s Twitter FlavorFeed™ to work. I ended up having to use multiple web applications which could easily change in a day to completely mess up the feed and then the tweets. I was unable to style it in the way that I wanted. In a perfect world, I would’ve written some code all on my own that would automatically parse the information from the Kopp’s website and then update the Twitter account– what I have now is definitely a rickety, hacked up version of what could’ve been.

Actually, scratch that. In a perfect world, Kopp’s would have an RSS feed for their Flavors of the Day. Lesson learned.

But they still make some damned good frozen custard.