To relieve my frustration at being stuck at home last week, instead of making a presentation about open data at Future-Democracy ‘10, I decided to try to see what my local council spent on gritting the roads during the last snow fall.

I understand the value of data, a significant part of my work in different organisations has been trying to get them to understand what data and information they have, to organise it better and to share it more effectively. However, I’m no geek and my job means I have limited time to find my way round the growing flood of government data.

As I embarked on my quest, borne of having a little time on my hands rather than having a genuine enquiry, I discovered how difficult it is for an amateur to find what they want. The csv file, essentially the excel spreadsheet of expenditure I found was for the wrong period (though given the last snow was in February before the government demanded all spending above £500 be published maybe I’m being unfair?). Although, even if it had covered February 2010 it doesn’t contain enough information for me to even begin to work out what the over 5000 line items were for. I also realised as I got further into my slightly frivolous quest that it was doomed from the start. The data that is being published by local councils is about payments to suppliers, it is not about internal spend. With gritting being an internal function the data I want isn’t even going to be there. My expectations had been raised by the publicity about open data and if I had genuinely wanted the information I think that this would have (unfairly probably) increased my frustration.

My experience confirmed one of the central premises of my presentation; that the move towards open data is very much to be welcomed, but that it is unlikely that there are a legion of armchair auditors with the time, energy and knowledge to scrutinise government effectively.

It seems to me that this comes down to understanding the difference between data and information. Data is a collection of discreet facts, while information is data that have been processed into a format that is understandable by its intended audience. In my presentation I drew the comparison between off-cuts of meat that many in the audience might be slightly unsure how to turn into something palatable, and sausages which most would be happy to cook and eat.

The key question for me is therefore who is processing the data, contextualising it so that it can be used and digested by citizens? The Lewes data on spending on suppliers contained £428.50 for sandwiches for Environmental Health. I’m in no position to determine whether this was a sensible spend or not; why did Environmental Health need sandwiches on five different occasions in the last seven months (but not in August and September)? Are they feeding people playing some sort of governance function? Is there a critical all day meeting they need to have once every month or so, and that getting sandwiches in buys greater efficiency than if they had to stop and go to get their own food? Or is the Director of Environmental Health claiming all of their lunches on my money? I am in no position to make any judgements about this because absolutely no context is provided for me to make that judgement.

The problem from my perspective is that the people with the time and motivation to burrow into the data are as likely to be special interest groups as they are individual armchair auditors. My worry is that interest groups, by their very nature have a particular point of view to push, or a world view that will colour their interpretation of the data. Councils that release raw data leave the field wide open for others to create the context. At a time of cuts this is a risky strategy; someone somewhere may well be working on an app that compares the Chief Executive’s salary to the number of libraries closed or some such false comparison.

Don’t get me wrong. It is positive that councils have to open up their books, but they are going to have to do a lot more than release csv files. One part of the solution is technical and probably expensive, from a council’s perspective; create the online app to go on their site that allows ordinary citizens to interrogate the data in a way that makes more sense and provides a proper context. But there are other things that councils can do too. Frontline staff are an amazing resource in this regard. Are councils developing plans for training staff about the data, what it means and helping them to facilitate conversations with the residents they work with and for? Are councils creating channels for these frontline workers to feedback into the council what the concerns about residents are, how they are reacting to the data that is being released, where they are misinterpreting it, and where they have genuine concerns?

If we really think that greater transparency is a good, and I do, then we also have to think very quickly about how to ensure that the conversations that result from the data drive positive change, rather than giving the floor to people who can shout loudest about their misinterpretation of the data that is handed to them.

(My presentation and this resulting blog draws its inspiration from a number of sources and conversations, but I’d particularly like to thank Richard at who helped me focus on the particular challenges faced by armchair auditors)