Does Your Council Do Open Data

I’ve been working at the ODI for just over three months, and, while I’m no longer working for a council (I used to work at Lichfield District Council), I still like to keep an eye on what my former colleagues in local government are up to, particulary around open data.

With this in mind, and with the tech team’s regular innovation weeks giving up the time and space to try out new ideas, I thought to myself “I wonder how many local authorities actually have an open data section?” and decided to use my innovation week to find out.

If you’re not of a technical persuasion, and are just interested in the headline results, just click to get to the headline results. If you’re interested, read on fellow techie…

The first place I looked was Chris Taggart’s Open data scoreboard on Openly Local. This has a list of councils and links to their open data sections. The only problem is this relies on councils to contact Chris so he can list them, there still must be a decent amount of councils that have an open data section, but haven’t told Chris about it.

I used the JSON version of the data as a starting point (which helpfully also had every other council listed, along with their website) and added them to a database.

I then queried the database for councils without an open data section and tried adding some common suffixes for open data sections to the council’s website domain, such as /data, /opendata, /open-data and /transparency, and seeing what the webserver returned. If any of the responses were 200 (that is, the webserver said the page existed), I added the URI to the database as an open data section for that council.

However, after picking some of my results at random, I found that a lot of these pages, while the webserver returned 200, they were actually ‘Page not found’ pages (which should have returned a 404). So I decided to run my queries again, but this time adding an obviously junk suffix to the domain first. If the junk one returned a 200, I knew that this council’s website wasn’t set up properly and I wouldn’t be able to guess an open data section from the authority.

I then decided it might be a good idea to see how many councils had open data sections that were accessible within two clicks of the homepage. This involved me scraping the homepage of every council website and seeing if I could find a link to the open data section. If not, I then scraped every page that was linked to from the homepage to see if I could find a link there.

The results of my survey were that 38% of councils had open data sections that were either listed in Openly Local, or that I could reasonably guess at the URI. As I also recorded the region the council was in, I could see that of this, the highest proportion of councils with an open data section was in London (56% of councils), and the lowest was in Northern Ireland (4% of councils).

Of every council that had an open data section, 11% had a link to their Open Data section from the homepage, and 16% had a link on the next level down. The overwhelming majority did not link to any of their open data sections from the top two levels.

I’ve visualised the data (using the magic of D3) below, so you can see for yourself, and the code I used to generate the data (as with all of our code) is available on Github. As we’re all about open data here, you can also download the raw data too.