morph.io is stuck in 2016. Help us get it out.

morph.io has been quietly running scrapers for the civic tech world for over a decade. Over 11,100 public scrapers, 43 million rows of data, and one server that’s still running Ubuntu 16.04. Journalists, researchers, council watchers and open data people use it to get structured data out of websites that never wanted to give it up.

It isn’t a side project for us either. Every Planning Alerts scraper runs on morph.io, and Planning Alerts pulls its development application data straight out of the morph.io API. When morph.io has a bad day, people stop hearing about what’s being built near them.

It’s also, right now, in trouble. Not in a dramatic way. In the ordinary way that open source infrastructure gets into trouble. The stack aged and the maintainers got thin on the ground. The pile of things held back by other things held back by other things got tall enough that nobody could climb it in a weekend.

We’d rather say that out loud than let it rot politely. So here’s the honest state of it, starting at the bottom of the stack, and what we need help with.

The server: one box, still on Ubuntu 16.04

morph.io runs on a single server, running an operating system we can’t upgrade in place.

Our Terraform configuration for it is public, and it contains this:

# We can't set the image below because it would force replacement
# image            = "linode/ubuntu16.04lts"

Ubuntu 16.04. Standard security support for it ended on 30 April 2021. The extended security maintenance window that followed ran out in April 2026.

That commented-out line is the whole problem in miniature. Terraform can’t manage the image, because changing it would destroy and rebuild the instance. So the box has stayed as it is. A decade of software choices have piled up on top of a foundation nobody can move without planning it properly.

This is why the Ruby version is where it is. It isn’t that nobody noticed Ruby 2.7 went out of support. It’s that the platform underneath doesn’t comfortably run anything newer, and the upgrade has never had a clear owner with time to do it.

We do know how to do better than this, because we have. Planning Alerts has multiple servers to provide redunancy. We can do a major upgrade there, like a Ruby version bump, without downtime. morph.io has none of that. It’s one server, deployed with Capistrano, and it also happens to host the Discourse container behind help.morph.io. Its provisioning playbooks still live in the morph.io repo rather than our central infrastructure repo. There’s a FIXME in that repo describing the certificate copying as a temporary measure “once morph provisioning is moved to this repo”. It hasn’t moved yet.

The application: Ruby 2.7, Rails 6.0, and a Gemfile full of TODOs

Open the Gemfile and the rest of the story tells itself.

morph.io is pinned to Ruby 2.7.6. Ruby 2.7 stopped getting security patches on 30 March 2023, and 2.7.6 came out in April 2022. We’re two releases behind the last version of an interpreter that isn’t supported any more.

It runs Rails 6.0.6.1. Rails 6.0 is well past end of life. The 6.1 series that came after it finished maintenance in October 2024. We’re not one upgrade behind, we’re several.

Underneath that sit the compromises you’d expect. Each one is documented in the Gemfile by a past maintainer who knew exactly what they were leaving behind:

• psych is pinned below 4, with a comment noting the fix landed in Rails 6.1. We can’t pick up security fixes until Rails moves.
• Sidekiq is held at version 5 because production still runs Redis 3.x. There’s a TODO: Upgrade sidekiq as soon as we can sitting right there in the file.
• render_sync is a fork we maintain of a gem that’s no longer supported, with a TODO: Move away from this unsupported gem beside it.
• jquery-ui-rails is held at version 5 because newer versions throw Sprockets::FileNotFound.
• annotaterb is capped at 4.15 because 4.16 wants Ruby 3.0, which we don’t have.
• The front end is CoffeeScript, Sprockets, Uglifier, jQuery, bootstrap-sass, d3 3.5 and, genuinely, zeroclipboard-rails.
• The search layer is Elasticsearch 7.

Then there’s the part that most affects people trying to actually run a scraper. Scrapers execute in Docker images built from openaustralia/buildstep, and that repo still carries three stacks: cedar-14, heroku-18 and heroku-24. Cedar-14 means Ubuntu 14.04. We started the move onto the modern base image and haven’t finished it. Until we do, what language versions a scraper gets depends on history rather than intent.

There’s also a small mountain of operational toil that eats the time we do have. We rotate the mitmproxy CA certificate by hand, across two repos and a production server, every couple of years. We regenerate the Sorbet RBI files by hand every time a gem moves. The Docker Compose path that would make that easier is still marked BETA in the README.

None of this is anyone’s fault. It’s what happens when a small charity runs real infrastructure for a long time.

Where you can help

If you write Ruby or you do infrastructure, there’s a clear and unglamorous path here. Every step of it is useful on its own.

Get us onto a supported operating system. This is the one that unblocks everything else. It means building a new server from a current Ubuntu LTS with Ansible, working out what breaks, then moving the data and the running scrapers across. If you’ve done this kind of lift before and can tell us where the landmines are, that alone would help.

Move the provisioning into the infrastructure repo. morph.io is the last service still provisioned from its own repo instead of alongside everything else. Bringing it in makes it maintainable by more than one person, which is the actual goal.

Work through the upgrade chain. Ruby 2.7 to 3.x, Rails 6.0 to 6.1 to 7 and beyond, then Redis, then Sidekiq, then the psych pin falls out on its own. Some of this needs the new server first, but plenty doesn’t. A PR that gets the test suite green on Ruby 3.0 unblocks everything after it, and it doesn’t have to be one person in one heroic pull request.

Finish the buildstep migration. Get scrapers off cedar-14 and heroku-18 and onto the Ubuntu 24.04 image, and work out what breaks for existing scrapers when they move.

Kill the held-back dependencies. Replace the render_sync fork with something maintained. Work out why newer jquery-ui-rails breaks Sprockets. Retire CoffeeScript. Any one of these unblocks the next person.

Make it easier to start. Clone morph.io this week and it might take you three hours and four dead ends to get a local server running. If it does, the most useful thing you can contribute is a pull request to the README describing exactly what you hit. Getting the Docker Compose environment out of BETA would help every contributor who comes after you.

Pick up a small fix. We try to label issues that are easy fixes, and they’re a good way in.

The house rules are short. RuboCop must pass, RSpec must pass, write specs for new code, and branch names use the feature/, bugfix/, refactor/, docs/ or chore/ prefixes. It’s all in the README. The code is AGPL.

If you’d rather fund the work than do it

Pull requests aren’t the only way to help, and for some people they aren’t the realistic one.

The honest constraint here isn’t that the work is hard. Most of it is well understood, and a competent Rails developer with a clear run at it could get through a lot. The constraint is that we’re a small charity and nobody currently has that clear run.

So if you or your organisation can fund developer time on morph.io, we’d like to hear from you. That could fund a defined piece of work: moving the platform onto a supported operating system, getting it onto a supported Ruby and Rails, or finishing the buildstep migration. It could be sponsoring a contractor for a block of months. We’re open to talking about what a useful contribution looks like rather than handing you a fixed number.

Everything funded this way stays open source under the AGPL, on the public repo, usable by anyone.

If that’s a conversation worth having, email Ben Fairless, our CEO, at exec@oaf.org.au.

If a smaller amount is more your speed, that helps too. You can donate to the OpenAustralia Foundation directly.

What we can promise

We can promise real code review from people who know the system, and reasonably quick responses to pull requests. We can promise your work goes into production on a service people depend on, not a demo.

We can’t promise a large team behind you, because there isn’t one. That’s precisely the problem we’re asking you to help with.

Maybe you’ve used morph.io to get data out of a website that didn’t want to give it to you. Maybe you just want to keep a piece of civic infrastructure alive. Either way, the repo is at github.com/openaustralia/morph. Open an issue, pick up an easy fix, or tell us what broke when you tried to get it running. Or get in touch about funding someone to do the heavy lifting.

Any of those helps. Even the one where you just tell us what broke.