Matthew’s back: Improving the stability and scalability of the morph.io platform

I’m very happy to be back at the Foundation for a little stint now to get my hands dirty on some coding, something I haven’t had any opportunity to do over the last year. It’s funny how you can miss these things!

As you might have noticed, I’ve been away from working on morph.io and being part of OpenAustralia Foundation for the last 12 months, with 10 of those with me working at the Digital Transformation Office (DTO). It’s been an amazing experience working at the DTO, a crazy rollercoaster ride, and I’ve learned heaps. It was an extremely difficult decision to move on but ultimately I think the right one for me.

Morph.io, OpenAustralia Foundation’s scraping platform, used by civic technology organisations from around the world and thousands of individuals to scrape around a million web pages per week, started as an experiment.

We didn’t really know whether it would work. Would people want to use it? Would it attract a community? Would members of the community be willing and able to support the platform financially?

We knew we needed something like morph.io to help us build and maintain planningalerts.org.au but we really didn’t know if it was worth making it something that a much broader group of people could use.

So, when I started working on morph.io I built it using the tech equivalent of double-sided sticky tape, brown paper and a pair of scissors. Build it in the simplest possible way to get going and prove (or disprove) the concept as quickly as possible. This is a cornerstone of lean.

Over the last year, morph.io has definitely experienced growing pains. More and more people have been joining, running more and more scrapers. This is a good problem to have! However, when things go wrong on the platform they tend to go pretty badly wrong. All at once the job queue gets backed up, the io and cpu of the server gets pegged and we all too easily run out of disk space. It takes a lot of effort to recover from those situations. It doesn’t impact users beyond their scrapers taking a long time to start or run or in the worst situations the site running slowly, but it doesn’t help people trust the platform.

All the time that we’re firefighting problems on the morph.io server, the less time we have for developing new features.

So, with that in mind I’m spending the next month or two focusing on improving the underlying stability and scalability of the morph.io platform. Here’s the list (brought together from existing issues on Github) of what I’m planning to work on.

It’s going to be a busy month or two for morph.io, so please stay tuned for updates.

If you have stability or scalability things you particularly would like to see addressed soon then get in touch.