Digital Library Services for Democracy

One Sunday morning, a bit over two weeks ago, Kat, busy putting the finishing touches to her article on transparency for the Centre for Policy Development‘s group submission to the government 2.0 taskforce “Upgrading Democracy” leans over to me and says:

“How about we submit a Perl app to the taskforce as an essay in transparency?”

I say, “Okay… but I’ll write it in Ruby”

A perfect half serious, half tongue-in-cheek example of what we’re about. If you have the choice between talking about something and doing something, “do” wins every time.

So, I started writing a little Ruby script to screen scrape data from the Federal Register of Lobbyists which is one of those things that I’d been meaning to take a look at for a long time. Now you might reasonably ask, why do that?

The Federal Register of Lobbyists “… is a public document that contains information about lobbyists who make representations to Government on behalf of their clients”

Look at how the Federal Register of Lobbyists is organised. You can get a list of all the lobbyists on the register and for each of those you can follow the link which leads you to the list of clients they represent.

This is useful if you know the name of a lobbying firm or individual but not a whole lot of use if you know the name of a client and want to find out who lobbies for them on their behalf.

So the purpose of writing the little Ruby script was simply to download the information from the whole register by downloading 270 or so web pages, extract all the relevant information from the web pages and save it as a local file in a format that can then be easily loaded into any spreadsheet program for easy analysis.

This program demonstrates the value of transparency on a dataset which is about transparency. It essentially “unlocks” the data that is trapped on the web pages in a form which makes it hard to analyse. Many people have talked about this as the importance of data and API’s. I won’t try to repeat or even summarise what they’ve said.

Anyway, I submit the program to the taskforce with a small explanatory email. In the email I say that I had originally intended to submit the resulting data with the submission but did not because the website that hosts the register crashed and the copyright on the website did not allow for republishing without permission.

So, all I was submitting was the source code for a small Ruby script which is copyright OpenAustralia Foundation and I licensed under the GPL, so that anyone can use and alter the script freely within the terms of license.

And then I mostly forgot about it…

Some time passed…

Lots of submissions were appearing on the taskforce website but mine wasn’t there. What was going on? I emailed the taskforce to find out, received a reply from the secretariat explaining that the submission would be up early next week as they were processing many submissions.

Perfectly fair. The following week rolls round, more submissions appear on the taskforce website, but mine is still not there. That’s strange. On the Thursday I email again, asking what’s happened, I even mention it on Twitter, but no luck.

Finally, yesterday, more than two weeks after I emailed the taskforce my submission I send another email outlining the dates of all my correspondence and asking again when my submission would appear.

Then, today I receive this email

Good afternoon Matthew,

Apologies for the delay in dealing with your submission and correspondence. Putting submissions online is an ongoing process that is still being worked through, particularly after the redesign of the Submissions page. We are working to have the remaining submissions online by the end of this week.

We have been analysing security issues about putting your submission online. The Taskforce blog is hosted on a Department of Finance and Deregulation server, and as such is subject to security policies which prevent us from offering your program as a direct download to users.

However we have placed the emailed text of your submission on the Submissions page in whole, including the link to the page where users will still be able to access the program. Please see http://gov2.net.au/submissions/matthew-landauer-openaustralia-foundation/

We are also asking all submission authors if they would like the WordPress comment function switched on for their submission. Should we turn it on for yours?

Regards,

Government 2.0 Taskforce Secretariat
Australian Government Information Management Office
Department of Finance and Deregulation

“Security issues” eh? What does that mean?

Thanks for responding. I greatly appreciate it.

I’m a little surprised, to put it mildly, to hear you’re saying that
there are security issues with putting my submission online. What
exactly are those security concerns? Maybe I can help put any fear to
rest by answering some questions?

As for comments on my submission please go ahead and switch them on.

A small point of clarification on my end – the emailed text (without
the program) is not my submission. That you have put the text up I
think is a good first step. The emailed text was a little further
explanation but the core of my submission is still the Ruby program.
So, I hope that we can work through whatever security concerns you
might have so that my submission in full can be put up on the
taskforce blog along with those of everyone else.

All the best,
Matthew

And shortly thereafter, I get the following response

Hi Matthew,

To be more specific – the security policies that stop us from putting up your program for download aren’t about the nature of the program. They are restrictions about offering externally-created code for download on a departmentally-hosted website.

We acknowledge that you didn’t consider the email to be your submission, and are able to replace/update the text we have put up on the Submissions page if you want to make changes to what is online now. And of course we will retain the link to your Github page so that blog users will still have access to your program, albeit indirectly.

Regards,

Government 2.0 Taskforce Secretariat

And then my response is

I obviously don’t know what the departments guidelines are
specifically, but I would imagine (applying common-sense to this) that
there are restrictions on the distribution of program binaries via the
website.

However, what I’m supplying is source code which by itself is of no
security threat. It is not a binary program where someone clicks on a
link and then it starts up a nasty program which does something
terrible. This is a bit of source code that by itself does nothing. It
is no different then a piece of text, of which there is an enormous
amount on the taskforce website.

In light of this, I would ask you to reconsider publishing the program
in full on your website.

All the best,
Matthew

Let’s hope that this can get resolved quickly. I’m sure that nobody could have anticipated that someone would come along and do an apparently silly thing like submit a Ruby program as a submission to a government taskforce, but in this instance it made good sense, as a practical way of demonstrating many facets of what it takes to do government 2.0 in a practical way.

I took care not to infringe any copyright of the original data, licensed the script under a liberal license so that anyone could reuse it and still there was a problem…