The story of a tiny, almost invisible change

I don’t really like technical blog posts.

Instead this is a story of a tiny, almost invisible change we made to PlanningAlerts, what led us to make this change and the surprising amount of under-the-hood work it took to get there.

We made a “tiny” change to how comments are shown on PlanningAlerts. “delivered to the planning authority” is now a link (it’s underlined).

When you click on the link it gives you some more information

This should give you extra confidence that the comment was definitely delivered to the local authority.

How it all started

A kind and considerate soul emailed us to let us know of a strange comment that had been posted on PlanningAlerts.

Does this post have any truth in it?

Is this post damaging the Planning Alerts site?

Was the CAPS really necessary?

Here is the comment in full

”DONT” NOT LEAVE COMMENT HERE IF IN FAVOUR OR AGAINST APPLICATION. MANY IF NOT ALL DO NOT MAKE IT TO COUNCIL MOST COMMENTS HERE NEVER MAKE IT TO THE RELEVANT DEPARTMENTAL AUTHORITY!!

Ive been let down by this page thinking mine and many others are being heard here. It makes a difference. It does not. After contacting council in past, they had no clue about my rejection of development or from others on this page. It seems Acting authorities never know about what written here. This site should be taken down as everyone presumes it has direct or indirect authority to distribute to relevant departments.

*** PLEASE CONTACT COUNCIL OR RELEVANT AUTHORITY DIRECTLY ***

There were similar comments made by the same person on three other applications.

My first emotional response was “how dare this person say this, how rude!”.

That person didn’t even contact us directly to ever let us know there was a problem. We don’t read all the comments. There’s just too many for a tiny team doing too much. And then to throw around those blanket accusations.

Ouch! It really hurt to hear that. We have put so much time and effort over the years into ensuring that emails from PlanningAlerts get successfully delivered. We went as far as developing a new open-source transactional email server, Cuttlefish, to give us visibility and improve deliverability across the board. We made PlanningAlerts to give people access to the democratic process and getting their comment delivered is pretty core to that objective.

Then to my horror I realised that what the rude comment was saying was partly true. For the last couple of months 15 or so comments were not delivered to that one particular council. This was terrible!

We are human and our software is written by humans.

In 10 years PlanningAlerts had an amazing track record. We actually only know of one instance when emails were not delivered to a council In that case, a council changed their email address and didn’t forward emails to their old address. As soon as we were notified, we manually readdressed the emails and worked with the council to get them delivered in time.

So, what was going on? Down the rabbit hole of email sending

(Here’s where we get technical)

What was super weird at first was that only some emails to that one council were bouncing. The bounce message mentioned “DMARC” which was not something I knew about at that stage. So why were only some messages getting bounced and not others when all the comment emails are generated the same way? Nothing in the content of the comments looked like spam, the formatting of each email is identical. I found it really confusing.

Then, I noticed a pattern. All the comment emails that were bouncing were made by people with “yahoo.com” or “yahoo.com.au” email addresses. A quick search of “yahoo DMARC” led me to an explanation.

Yahoo had an unusually strict DMARC policy which meant that any problems with emails that look to originate from yahoo domains would get hard bounced. So, we had a DMARC problem with our comment emails but it had been unknown to us up to that point because almost nobody was enforcing it so strictly.

At this point I’ll just quickly backtrack a little. For a long time we’ve been ensuring that our emails pass SPF and DKIM tests. Cuttlefish helps enforce this and does it all pretty much automatically. However DMARC, does two main things. It allows an owner of a domain (e.g. yahoo.com) to say how email servers should handle mail from that domain when SPF and DKIM tests fail. So, in our case SPF and DKIM was fine, or so we thought, so why were the yahoo.com emails failing to get delivered?

Well it turned out it was to do with the second part of what DMARC does. It also enforces domain alignment; the domain in the “from” address of the email needs to be the same domain as used in SPF and DKIM. This is the problem that got us. It turned out that our comment emails had headers of the following form:

From: <email address of the person making the comment>
Sender: contact@planningalerts.org.au
To: <email address of council>

The Sender header is used in circumstances where emails are being sent on behalf of someone else. It’s defined in the email RFCs but generally has not been particularly well supported.

The purpose of all that was to ensure that when someone makes a comment via email that they receive replies from the council rather than us.

It turns out that DMARC essentially stops the Sender from ever being used. It appears that this was a conscious choice made by the DMARC designers because the Sender field is not shown to users in email clients.

So, we switched the email headers to:

From: contact@planningalerts.org.au
Reply-to: <email address of the person making the comment>
To: <email address of council>

And this magically solved our problem.

We resent all the recent comments that had previously bounced because of this, and our job was done.

Actually no. It would have been easy to stop there but actually we wanted to make sure that if something similar to this happens in the future we don’t find out about it because someone emails us asking about a comment on the site. We should be the first ones to know there is a problem.

After we fixed the way that comment emails are sent we started working on a whole host of other changes to improve the visibility and the impact of any potential future hard bounces for comment emails.

Improving visibility and reducing the impact of problems

In explaining earlier that only emails from the “yahoo.com” didn’t get delivered I was over-simplifying. What actually happened was that a yahoo.com email would hard bounce and then the email address of the council would get added to a “deny list” in Cuttlefish. Once an address gets added to the “deny list” any further emails sent to that address will not be sent. Addresses then get cleared out from the “deny list” after one week. This is an automatic feature of cuttlefish designed to reduce the chances that we send out bad emails which improves the deliverability of our emails across the board.

So the case of sending emails to local authorities is different from sending transactional emails (email alerts, etc..) to “random” people on the internet. We should be really confident that the email address of the council is correct and that emails to them generally should not bounce.

So, next we added a feature to cuttlefish.io to allow PlanningAlerts when it’s sending the comment emails to say “please ignore the deny list when sending this email”. This means that if an email hard bounces to a council, that cuttlefish will still attempt delivery for any subsequent emails to the same address. So now if hard bounces do happen the impact will be reduced.

Then we started working on the “visibility” part. If a hard bounce happens for a comment email how can we get notified of this urgent problem? One option would be that we get an email. In recent years we’ve been trying hard to limit the number of emails generated automatically for notifications and instead send those notifications to Slack. So, why not just try sending a notification directly to Slack instead?

How to do this? We don’t want Cuttlefish to have the knowledge to notify us of certain problems with certain emails but not others. It makes much more sense for PlanningAlerts itself to do that because it’s the one that knows the difference between the different kinds of emails. This is where we need to get some of the delivery information from Cuttlefish back into PlanningAlerts.

This finally gave me the reason and motivation to add a feature to Cuttlefish that I’ve been wanting to do since I first built it – which is to add webhooks where every time there is a successful or failed email delivery a custom URL (on another site of your choosing) is hit with information about the email. In our case we’ll be making a new URL endpoint on PlanningAlerts to accept those webhook callbacks.

So how do we identify if the email corresponds to a comment email or an alert email and which comment email? I think the most elegant solution to that is to allow the sender of an email to attach custom metadata.

Here’s an example of a comment email as seen in Cuttlefish showing the attached metadata (in this case comment-id):

So then it was a relatively simple matter of hooking up a bit of logic in PlanningAlerts where on every “delivery event” it receives from cuttlefish, it checks whether it’s from a comment and whether it was a hard bounce and if so it sends a notification to our slack channel.

At the same time as we put this in we noticed another council was bouncing emails! This time for some reason they had just decided to specifically block our email server. Thanks for that. Thankfully they quickly fixed it when we asked them to and we again resent the emails.

Making it visible to people using PlanningAlerts

We’re confident now that all the changes we made to improve the delivery of emails to councils and to notify us of problems makes it unlikely that similar problems will reoccur in future.

However all this stuff is under the hood and invisible to users. Perhaps that’s how it should be?

However, it’s not uncommon that we get emails from people who just want to make sure that the council did in fact receive the comment they made through PlanningAlerts. The council tells them they never received it, so we check in Cuttlefish to find that the council absolutely received the email.

What if we could make this more clear for people?

Why don’t we do something similar to what is in RightToKnow.org.au where the message says whether it was successfully delivered?

So that’s what we did. The final piece in a long chain.