Web scraping workshop success—we’re doing it again in October!

Henare from OpenAustralia Foundation with the wonderful attendees of the first Introduction to Web Scraping Workshop, September 2015

Earlier this month we ran our first Introduction to Web Scraping Workshop and it was a big success! We had 6 wonderful attendees. By the afternoon everyone had written a scraper to collect data from the web.

You can read more about how it went on information graphics designer Kelly Tall’s blog. We’re looking forward to seeing what everyone does with their new scraping skills!

We owe a huge thanks to the Media, Entertainment, & Arts Alliance for providing a great venue and to Fleetwood Macciatto for a delicious lunch.

We were really pleased and impressed with how much everyone learned—so we’re doing it all again.

If you want to learn how to scrape structured data for your projects then…

The next Introduction to Web Scraping Workshop is Sunday, 25th October, near Central in Sydney.

Register here

There are 10 places available.

If have any questions please email (contact@oaf.org.au) or contact us on twitter (@OpenAustralia).

Things we learned from our first workshop

Pairing works well

At the OpenAustralia Foundation we’ve found Pair programming to be one of the best ways to learn together and transfer knowledge between people.

In planning the workshop we thought pairing would be a good way for people to learn and problem solve, and also that it would be useful to get some experience working in this style.

We paired attendees up to write their scrapers and it was a big success. The pairs had fun working through problems together and were able to remind each other of the techniques they’d been shown when necessary.

We got very positive feedback on this format, so we’ll definitely do it again.

A reference guide would have been useful

We didn’t provide hand-outs with a step by step guide, but rather focused on a detailed, live walk-through of writing a real scraper.

While the live programming was an engaging and realistic intro to scraping, a simple list of methods and techniques could have been a useful aid during and after the workshop. We’ve since had feedback that a resource like this would help avoid easy mistakes after the workshop, such as missing the bundle exec command before running irb or ruby scraper.rb. We’ll prepare something like this for the next event.

Luke drafted a detailed guide to writing a first scraper based on Henare’s demonstration in the workshop—hopefully we can draw on this to create a simpler list of steps and techniques for attendees.

Setup is tricky

Getting your laptop set up for scraping is not straightforward. Only 3 of the 6 attendees managed to get Ruby and the necessary scraping libraries installed for the workshop. This wasn’t a problem on the day because we had already planned to be working in pairs, but it would be better if everyone went home with their laptop ready to run the scraper they had written.

We’re not sure of the best way to handle this problem. We want to keep the workshop focused on scraping rather than get bogged down installing Ruby.

We’re very open to ideas on this. We’ll start by recommending that attendees try a free Ruby on Rails Installfest event that aims to help people get past this often confusing step.

These workshop events are still an experiment for the OpenAustralia Foundation so we’re very open to ideas and keen for feedback.

Remember to get in contact if you want to attend the next workshop in late October as space is limited. You can email us at contact@oaf.org.au.