Published on April 22, 2015

Does Public Engagement Actually Change Anything?

By Reema Patel

Reema is a Policy Analyst at Involve, working on the Citizens and Science programme - including Sciencewise, the expert national resource centre for public dialogue input into science and technology policy.

DominoEffect

In one West Wing episode, an exasperated President Jed Bartlett looks around at his political team and says, ‘”twenty-seven lawyers in the room, anybody know Post Hoc, Ergo Propter Hoc?’’. After some pressing, a translation is supplied, ‘After it, therefore because of it.”

Quizzical looks all round.

“…It means,” explains the President, “one thing follows the other, therefore it was caused by the other. But it’s not always true. In fact, it’s hardly ever true.”

I explore this because public engagement processes are very much beset by this problem. How do we know when a public engagement process has influenced decisions and events? This is not a problem unique to public engagement but a problem characterising most fields that aim to influence decision-making. It is a problem which in fact besets politicians and policy-makers themselves. Campaign groups, lobbyists, social researchers and scientists all face precisely the problem in demonstrating the worth their work adds to both the policy making process and their stakeholders.

Recognising this is the case prevents public engagement practitioners from being crippled by the problem for too long. Jed Bartlett and his team, Latin translations aside, still need to get on with the business of effective decision-making with a decent grasp of what they think generally may have happened and a decent grasp of what they think is likely to influence what will happen amidst the myriad of pressures jostling for attention. They aren’t able to engage for too long in the navel-gazing philosophy of causation.

jedbartlett

Does Jed Bartlett’s problem (which is quite a bit older than the West Wing – philosopher Hume posed this issue back in the 1700s) mean we can’t measure the impact of public engagement at all, and does that mean we should just retreat from the decision-making process in a search for legitimacy, focusing as public engagement purists on creating the perfect engagement process? I think not. Most people would recognise that creating the ‘perfect process’ without some kind of focus on objectives is impossible – the question is what objectives we choose. Should we exclude the instrumental value public engagement has to strengthening policy-making? I’d hope not – that would prompt the creation of a widening gap between the practice of public engagement and the policy making process.

We might not be able to measure or recognise whether action a had effect b with a given initiative or project; but some of us might well be able to recognise or understand whether certain organisations and institutions have what has been termed by academics ‘policy resonance’– ‘a propensity to influence, as opposed to the achievements of a tangible and measurable outcome’ (pp.423 Maximising the Policy Impacts of Public Engagement: A European Study, by Emery, Mulder & Frewer, 2015). What seems to define policy resonance is a disposition towards producing impact, under certain conditions. Part of that may be because of the relationships they have cultivated, their legitimacy in the eyes of policy-makers, the quality of research produced, whether it’s owned or driven by policy-makers, the understanding public engagement practitioners have of how policy works, a history and track record they bring and – of course – a reputation for a commitment to quality and impartiality with both public and policy maker. This list could go on for quite a while and still not be exhaustive (as an aside, the Emery, Mulder and Frewer paper provides a great snapshot of what these considerations could be). It seems that the creation of policy resonance is probably a more fruitful endeavour than the mulling of engagement practitioners over causation issues – one for another blog post.

I return to Jed Bartlett’s point which is particularly valid when we’re recognising the myriad variables affecting policy-making. At the same time, I know that when I hit a snooker ball, the snooker ball’s going to move. Some things do follow other things and we can understand a causal connection exists between them without needing to root around for something in between other than an explanation.

So how do I know what I know about the snooker ball, and how do I know what I know about the effectiveness of a public engagement process under certain conditions? Perhaps it’s about finding the explanation – understanding the environment and teasing out what we think are legitimate causal connections within a particular socio-cultural context (‘policy resonance’ for the decision-maker). Our best just might or might not be good enough.

Photo 1 Credit: N08/6970861063

Photo 2 Credit: m3lbatoast

3 Responses to “Does Public Engagement Actually Change Anything?”

  1. April 23, 2015 at 7:21 am

    Well written blog and always good to hear Jed Bartlett quoted, particularly in a run up to an Election

    But you avoid discussing decades of evaluation (perhaps centuries? You can find quasi-experiments testing effectiveness of different farming and diets for the Hebrews!) and social science approaches to ‘causation’. The sciencewise/Involve model is well respected and mature enough now to start doing some much more rigorous evaluations to see if it works or not. I’m sure BIS would support it – as they do so in other areas of complex social, behavioural and policy interactions.

    So come on, draw up a plan to really experiment with your different techniques! Not just the usual weak programme evaluation – some nice chats with people you already know – with all the biases that go with such interviews – spliced with some glorified monitoring data (e.g. http://www.sciencewise-erc.org.uk/cms/assets/Uploads/Learning-Resources/SW-Evaluation-FR-230315.pdf).

    It looks like you have a good theory of change, so a strong place to start to think about moving beyond observational studies in isolation, and higher up the progression of standards of evidence – check out things like Nesta, Public Health England, Project Oracle, Dartington Social Research Unit or the What Works Centres evidence standards.

  2. April 24, 2015 at 10:27 am

    Nice post!

    I’ve been thinking about the same problem from a local government perspective.

    One related issue is that we don’t have a common way of describing the influence of public engagement. Here is my attempt that builds on Paul Burton’s ideas: http://localopolis.blogspot.co.uk/2012/02/evaluate-policy-outcomes-from-public.html

  3. April 24, 2015 at 10:42 am

    Nice article. I can never resist anything that credibly gets the West Wing in!

    I have had a row with a dialogue specialist recently about ‘market research’ vs ‘dialogue’. My view was that when a policy maker wants to know that the public thinks about something, with a view to making a decision- that is allowed. It’s also perhaps closer to market research than dialogue which is more free ranging. But obviously generating free ranging discussion is the entire point of all of these exercises, constraining views would be counter productive, so getting the balance is essential. My issue was that the information from which the free ranging discussion was anchored was too broad, simplistic and incorrect. But because it is complex it took much more time to get correct, with much more multi-stakeholder input, and creativity needed in the design of materials than had been allocated in the budget.

    One of the problems I have with ‘dialogue’ is that sometimes it could be more helpful and more anchored to policy makers real life decisions because without that it doesn’t have that policy resonance you describe. It is useful and interesting to know what the public things about x ‘ology’ or ‘y’ technology, but it only takes us so far.

    I would like to see much closer involvement of the public in decision making, which may require different research methodologies and perhaps a combination of a few to understand what people think in what context. There are many complaints about the expense of dialogue, which severely constrains its use, but perhaps it will look more like a good use of taxpayers money if it is more focused on the policy decisions which need to be made?

Leave a Reply