A hand holding a pair of scales, creative commons SouthernFried

Many people working in consultation have noted that those who are happy with a proposal are much less likely to make a submission than those who are negatively impacted. It is far easier to get a petition going opposing a development than it is supporting one and most referendums end up supporting the status quo for similar reasons. So if consultations don’t give a complete indication of how the community feels about the issue, they are disempowering the silent majority.

I vividly remember an example where a Councillor faced vocal opposition from constituents to a new super market development and as a result voted against it. At the time it seemed only a handful of residents were for it. However at election time when the Councillor went round door knocking, a considerable number of people were wondering what happened to the super market. Those who actually supported the development were not noticed because they lived further from the development, were less educated and poorer than the vocal opponents, and as a result they were not heard in the traditional consultation.

Given this ‘response bias’ in consultation someone recently suggested to me that consultation submissions could be weighted in some way to account for the over representation of negative views. I think there are many reasons why this is a dangerous route to take.

In my experience consultations are rarely just about the numbers of responses received. Consultations are about exploring ideas, concerns and unintended consequences of a proposed policy – it is not a vote. Indeed, one of the most important roles of the organiser is to make it clear that consultations are not referendums. Consultations can gather better information about what local people think as well as increase understanding of the likely impacts of the policy. In the latter case a single response providing a vital and overlooked piece of information may carry more weight than thousands of identical template responses.

Weighting can fill a vital role in quantitative consultation in order to balance different demographics response rates, to compensate for lacking recruitment – for example if fewer women than men respond (but only if there is a large enough sample size to begin with). Weighting works for quantitative engagement but not for qualitative responses. However if it is done, it must be done transparently and in a considered way.

Weighting is also useful where the impact of a policy on certain groups is more important than views from the overall population. For example, when consulting on young people’s services it may make sense to weight the responses from young people higher than those from adults. The responses from adults should not be dismissed outright however, as it is possible that some youth activities will have negative or positive impacts on the rest of the population.

While there is some guidance on analysing the answers to consultations it is hard to give clear rules in advance on how to deal with the complexities of consultations. Analysing consultations is an art and not a science. Each contribution has to be assessed based on:

  1. How it fits with the purpose of the consultation (is it to survey public opinion or to gather evidence?),
  2. The demographics of the target population (is the respondent part of the target audience, do they represent a group that is not well represented, are they part of an organised stakeholder group?),
  3. The content of the response (is the response a template campaign letter or personally written, does the response provide new information about the policy and its impacts, does the letter provide evidence to back up claims made?).

Is there a response bias in favour of negative views in consultations?

Yes, definitely.

Is weighting the responses the way to deal with this?

Probably not.

Weighting responses that are negative towards a government proposal would (rightly) be seen as undemocratic, unfair and as a manipulative way to skew the debate.

So what should we do?

The Councillor I mentioned earlier found that ward walks helped him to reach out to the silent majority who would not respond on their own. If you suspect that there is a large silent majority try to get their responses through methods such as door knocking, peer interviews, informal engagement or demographically sampled surveys, methods which get views from of the silent voices. Involve have produced a few useful guides to informal engagement that you might find useful in doing so: Not another Consultation and Say & Play.

The problem is that we have an imbalance in the supply of consultation responses. Rather than devaluing some responses to account for the imbalance, the solution is to gather balancing views. The job of the consultation professional is to ensure a level playing field and to seek out the quiet ‘easy to overlook’ voices. In short, amplify the quiet voices rather than trying to muffle the loud ones!

Image: MorgueFile Creative Commons southernfried