Blinded by science
The assessment of research is of enormous public importance – the findings of research determine the evidence base for a vast range of decisions which affect every aspect of our lives. The rise of bibliometrics as the primary means by which we gauge the quality of research is therefore something that the public should know more about, especially because it is deeply flawed.
For obvious reasons, this isn’t something that’s had much attention from the media (although the Guardian published an article relating to the topic last year). It might not be sexy, but it’s an issue which is critically important, with implications that touch more or less every aspect of our lives.
As an early career academic I had the mantra ‘publish or perish’ bashed into my consciousness on a more or less daily basis. To have a successful academic career, researchers need to publish as much of their work as possible, in the highest rated journals. If you publish a research article, and it gets referred to a lot by other academics, this is regarded as an indication of the research being ‘good.’ Even if the only reason your research is being cited all the time is because others are using it to illustrate what is wrong with it. There’s also this strange academic norm, in which researchers and theorists are expected to mainly cite research published recently, which is a bit bonkers, really. Old research can often be the most important or most relevant basis for new studies.
In order to publish, you need to provide a context for your work, citing key texts from the big players in your field (thereby driving up their citation indices). If you’re being natty about it and you have recent publications yourself, you’ll also make sure you always cite all of them (even if they’re not relevant), in order to drive up your own citation rating.
The whole process is mediated by journal editors who rely on peer review to ensure that what’s being published is of high quality. And, the main measure of quality on which journals are judged is how much they are cited by others. It’s a kind of crazy game, with the rules being set, refereed, and maintained by the players. One way of looking at it would be to call it an international popularity contest.
The general drive towards transparency and accountability in the academic world is a good thing. But, I believe to do this properly, we need to take a realist stance, capable of recognising the complexity of the world we are researching. Instead, we’ve created a culture of numbers, in which the whole academic edifice has come to believe that fair judgements of quality can be reached by using algorithms to analyse statistical data, even though those data are unable to measure quality in any meaningful sense. The use of citation indices as a proxy for quality is completely illusory, and yet, the system has taken hold amongst an international community which really ought to know better.
But, once you’ve invested into this system, and start to do well at it, it would be against your own self-interest to be critical of it. That’s not to say there have not been attempts to get the academic world to examine its own practices. In an excellent report from the Joint Committee on Quantitative Assessment of Research, a group of mathematicians and statisticians explain all of this in much greater detail and with a much more thorough grasp of the issues than I can offer.
One of the most crucial things they say is that although citation counts seem to be correlated with quality, the precise interpretation of rankings based on citation statistics is not very well understood. And, given that citation statistics play such a central role in research assessment, it is clear that authors, editors, publishers and institutions are becoming adept at finding ways to manipulate the system to their advantage.
The report concludes that the long term implications of this are unclear and unstudied. But, it doesn’t take a hugely active imagination to envisage some of the potential consequences. The report was published in 2008, so, by some standards is itself already out of date, and given that practices for assessing quality in academic research show no signs of changing, it seems, so far, to have fallen on deaf ears.
Everybody needs good neighbourhood programmes
“We want to be super-local, seriously neighbourhood-based and almost microscopically granular” so said Francis Maude, last year, on the government’s proposed Communities First Fund.
Indeed “the neighbourhood” is the location for a number of government initiatives including the proposed neighbourhood plans.
This government is not the first one to decide that they want programmes to be delivered at the neighbourhood level. What lessons can we learn from previous neighbourhood level government initiatives?
I was prompted to ask this question after meeting John Hitchin, who had worked on the EC1 New Deal for Communities (NDC) programme. He gave me a copy of their evaluation report. Unlike many evaluations it is an accessible and practical document that provides some food for thought.
So here are three lessons I would draw from previous neighbourhood level programmes;
- Participation should be broadly understood
Sometimes the idea of resident involvement, which was central to the New Deal for Communities (NDC) programmes, translated into creating ways for residents to be involved in the NDC itself, rather than participation in community life.
One of the problems with the state itself creating spaces for resident involvement is that the state can then, in turn, ignore the views of these residents. JRF’s work on participation in Haringey starkly illustrates this point. They looked at the different mechanisms that residents could be involved in decision making in the local authority, the primary care trust and in the police. They found that public officials were very sceptical of the ‘representativeness’ of any residents that got involved in these mechanisms and this enabled them to discount views that were challenging. There was little evidence that resident involvement had actually materially changed practice or policies.
Rather than seeing participation through the lens of public services and encouraging residents to become more involved with neighbourhood initiatives, future neighbourhood programmes could look at ways in which they can support people to be more involved in community life.
- Support community groups to be themselves
Sometimes the additional funding that comes with neighbourhood programmes means that local community groups change their behaviour in order to obtain money from these programmes.
This can be more or less subtle. Community groups can start to adopt the organisational culture of the public sector (KPIs and all), start running new programmes which are not their priorities but the priorities of their funder, or spend more of their time understanding the needs of their funder rather than the needs of the people who use their services.
Finding ways to support community groups without drastically altering their culture or behaviour is no easy trick. I have mentioned before the Grassroots Grants programme, which I think had some success in supporting small community groups that had not previously received government money. Looking closely at this programme could pay dividends for those who are designing new neighbourhood programmes.
- Don’t top up core funding
The temptation for programmes aimed at improving neighbourhoods is to spend money making the area cleaner, greener and safer, since these are invariably the priorities for residents.
There is a real danger that this will mean that those public services that are already responsible for these things will use this as an excuse to lower their levels of service.
More subtly, when a neighbourhood programme tops up existing public services it can make it harder to influence the way in which those services are delivered. Changing the culture of existing public services was one of the most notable achievements of some of the neighbourhood management pilots, and the idea of “bend the spend” should be maintained as a focus of neighbourhood programmes.
New governments want to make an impression. They want to make it clear that they are distinct from the previous government. That is understandable. This article is not an argument for preserving programmes or initiatives that went before. Rather, it is an argument that we should learn from what went before.
To AV or not to AV – what is the question?
There is a tremendous irony at the heart of the current debate on AV. Politicians of all persuasions are looking the electorate in the eye and telling us, as sincerely as they can, that we are the audience with the greatest stake in the issue of voting reform. But their internal debates suggest something rather different. Where does that leave us?
Those in favour of AV tell us how unrepresentative the current system is, how only about a third of MPs secure a majority of the votes in their constituency, how many MPs can effectively ignore the needs and votes of many of their constituents, how the existence of safe seats means some don’t even need to work hard to engage their core support, and how most voters are effectively disenfranchised by this situation.
Those in favour of FPTP invoke the ghoul of perpetual coalition government and compromise, the fact that second, third and fourth choices should not have the same weight as first choices, the relative complexity of AV, and even the fact that AV is too small a step and should be rejected in favour of something closer to PR. Oh, and the fact that the only countries in the world to use AV for national elections are Australia, Papua New Guinea and Fiji (which is apparently considering a change anyway).
These are all reasonable arguments, and positive reasons to engage with the question of electoral reform. They should in themselves persuade people that it matters to them and encourage them to think about it and vote on it.
And yet, when those same politicians look each other in the eye, the debate takes a rather different course. Those in favour of AV claim that FPTP is mainly being defended by politicians with a stake in it (mostly the Conservatives). Those in favour of FPTP claim AV is really being advanced as a mechanism for increasing the power of the Lib Dems.
In other words, the debate turns negative, and is about power for politicians, not power for the people. And politicians, let’s not forget, have the loudest voices on this issue, so people are well aware of their internal wranglings. The debate on Newsnight a couple of nights ago, in which four politicians spoke a lot and two non-politicians said less, was a case in point.
Isn’t this exactly what we don’t need? Our voting system is at the heart of the relationship between Parliament and the electorate, and all politicians acknowledge that something needs to be done to engage with us again (even if they can’t agree on how). Surely an extended debate on the future of that system should be taken as an opportunity to re-engage people, and to persuade them of the importance of their part in the electoral relationship? It would be a shame if it actually reinforces the disconnect between us.
It comes down to the question of what are we voting for in May – something that matters to us, or something that really concerns the fortunes of 650 (for now) MPs?
When soundbites bite back
‘We are all in this together’ and ‘Your country needs you’ – two soundbites that sum up much of government policy at the moment. The thinking and logic behind both is clear; the situations they are invoked to address are familiar and important to us all; and the sentiments behind them are, on the face of it at least, motivating and ‘very British’. Yet neither seems to have caught the public imagination, and indeed both are facing considerable backlash at the moment.
Why should this be? Perhaps part of the reason goes beyond reactions to the perceived fairness and necessity of individual policies and measures. Perhaps we in the UK are predisposed to resist and even to fight against messages like these. Perhaps while they may have been ‘very British’ in the past, they are not any more.
An excellent and potentially very important new report from the WWF, Common Cause, certainly suggests this may be the case. Although primarily concerned with how to motivate individual action to address environmental and human problems on a global scale, it’s not too much of a leap to apply its conclusions to the UK scene as well.
The ‘traditional’ approach to motivating individual action and behaviour change on issues such as global poverty and climate change involves putting the facts in front of people, on the basis that ‘if only they knew’ about the scale of the problem, they would do something to help. Common Cause argues that this approach to what it calls ‘bigger-than-self’ problems is fundamentally flawed because the way in which people respond to facts is determined by their underlying values.
It seems that ‘individuals are often predisposed to reject information when accepting it would challenge their identity and values’, and that for these individuals such information ‘may simply serve to harden resistance to accepting new government policies or adopting new private sphere behaviours’. This sounds relevant to the UK situation, so what are today’s British values, and how do they help or hinder acceptance of the government’s messages regarding the country’s own ‘bigger-than-self’ problems?
Common Cause classifies values as extrinsic or intrinsic, the former being associated with image, status and self-advancement, and the latter relating to the importance of relationships, community and self-development. Importantly, these values are not innate, but rather a product of culture and experience, generated and strengthened by the media, the services, the policies, the attitudes and all of the other influences that individuals are exposed to in daily life. The two sets also act in opposition to each other, with strong extrinsic values making people less likely to value community and relationships, and strong intrinsic values making worldly success seem less important.
Values are formed by experience, but they are underpinned by what cognitive scientists refer to as ‘deep frames’ – long-held, stable conceptual structures that contain particular values. Once established, deep frames (and the values they espouse) can be ‘activated’ very easily by mentioning key terms and phrases, and a frame (and its associated values) is strengthened every time it is activated. This makes established frames durable and difficult to shift, but not unchangeable over the long term.
Common Cause gives the examples of ‘War of Terror’ and ‘tax relief’ as phrases that instantly activate frames regarding security issues and the proper role of government respectively. As soon as you hear them, your views on the underlying issues are brought to the surface, and the nature of those views depends on the deep frames to which you have unconsciously subscribed.
I’d say that ‘We are all in this together’ and ‘Your country needs you’ are two more examples of soundbites that activate deep frames and bring to the fore values that will influence the way they are received. These messages are inherently associated with intrinsic values, and on the basis of the Common Cause’s argument will be well received by people who hold such values and resisted by people with extrinsic values.
It may be that 50 years ago, the dominant frames in the UK were intrinsic – certainly the experience of the two World Wars is likely to have been an influence in this direction. In that context, the soundbites we are considering, and a more general appeal to the ‘British sense of fair play’, would have gone down well – indeed, ‘Your country needs you’ went down very well when accompanied by Kitchener’s face and pointing finger.
The trouble is that, as George Monbiot has pointed out in a recent Guardian piece, in the past two decades or more extrinsic values have been continually activated and reinforced by the media, advertising and government policy, and as a result the frames that espouse these values have come to dominate in UK society. The focus on celebrity, rich lists, competition and market forces has promoted a deep-seated emphasis on the self at the expense of the community.
Fast-forward to the 21st century, then, and the dominant frames have changed, so it is perhaps unsurprising that messages which appeal to intrinsic values and community feeling are now not only falling on deaf ears, but are being counterproductive and actively hardening many people’s resistance to issues on which action is vital.
Common Cause’s solution to this situation is for campaigns to acknowledge the importance of frames and to try to activate and strengthen intrinsic values, rather than fighting against extrinsic attitudes with facts and soundbites. The report presents a powerful argument for this, but it is a long-term solution. The problems these soundbites are trying to address are all too evident in the here and now. Perhaps a different approach to motivating change in people’s attitudes and behaviours is needed?
Where’s the Big Society when you need it?
Last night Newsnight told us what most of us knew already, that local councils are being forced to play slash and burn with services in a bid to cut around 25% out of their soon-to-be-shrunk budgets, in anticipation of the Comprehensive Spending Review this Autumn.
This is why I was less than surprised when a charity worker told me he was being forced to make around 70 of his core staff redundant today, and this is why: government pays for a great deal of what the charity sector does. This is especially so in the case of unpopular but essential services, such as homeless halfway homes, adult education centres and ex-offender programmes, these are programmes charities would not be able to get funding for elsewhere. With local councils cutting back on how much they will spend on such services, the charities providing them are having to make people redundant. Whether the government intended for this to happen or not, local councils have heard the message “cuts, cuts, cuts” and now they are doing so.
Wasn’t the Big Society supposed to stop this from happening?
Well, the concept of the Big Society, as far as I understand it, goes something like this: government pulls back on some of the services it provides and works with charities, volunteers and businesses that step in, form partnerships and help tailor and co-produce locally envisaged, cost effective solutions. This has the potential to be exciting stuff.
Indeed, think tanks and government are already coming up with ideas of how the Big Society could work on the ground. We have had Matthew Taylor appear on Newsnight to talk about the RSA’s approach to the subject (and big up the work of Projects in general, which can be viewed here), our Connected Communities programme has published a report on the part social networks will play in making the Big Society real, there are ideas for a Big Society Bank and National Citizen Service and significant research has been made on the potential of co-produced local services.
But ideas take time. The ink has barely dried on the speeches outlining some of these plans. The pilots for the National Citizen Service are not due to start until mid 2011. The comprehensive spending review will be with us in November.
Some of these ideas really are quite exciting, and plausible too. Take co-production for example. The Nacro Preston Restorative Justice project already enables local people to work with the police and other government agencies to rehabilitate criminals locally, with ordinary people staging mediations between offenders and their victims and helping to design restorative justice programmes themselves.
The lesson we can draw from examples like these is that the Big Society can work, but that it takes more than just a smattering of time, effort and investment (which is not what you want to hear when you have been told you live in an age of economic austerity). Significant funds have been channelled into recruiting and training the volunteers who take part in the aforementioned Nacro project, with the commitment required by volunteers not insignificant. This makes the Big Society timeline fairly incompatible with that of ‘cut and cut now’. With the idea of the Big Society still something of a stick figure drawing, as opposed to a finely articulated policy, I fear time has already run out for those losing their jobs as we speak.
The politics of TINA
TINA has been responsible for some of the most far-reaching political decisions of the last two decades or more. You’ve never heard of her? TINA is the popular abbreviation of ‘There Is No Alternative.’ A political catchphrase made famous by Margaret Thatcher when explaining economic liberalism, which was picked up almost two decades later by Tony Blair when justifying military action in Iraq. As Polly Toynbee has recently pointed out, TINA is being used again – this time when outlining the government approach to the financial crisis. At the Liberal Democrat party conference yesterday, Nick Clegg invoked TINA when trying to ingratiate party members to huge spending cuts.
Whatever your opinion of Thatcher, the Iraq war or the coalition government’s economic plans, we should be suspicious of people who tell us that TINA is justification enough for any political decision. There are always alternatives. There are alternatives to free market liberalism, to war and to coalition, whether or not you agree with them. When Labour introduced a minimum wage in 1999, many businesses argued this would force redundancies and that there was no alternative to painfully low wages. But there was an alternative, and in this case a very effective one, which business has now come to embrace.
TINA is too often used as a means of sidelining real debate and avoiding difficult conversations. But avoiding these conversations does a disservice to citizens and prevents us from getting to heart of the social, political and economic problems we face.
Take spending cuts as an example. Whilst there is unquestionably a need for action in the face of a £148 billion shortfall between government expenditure and government revenue, it is not the case that there is no long-term alternative to tackling our spending habits by cutting services. As is often quoted, the British public want “Swedish” welfare for “American” taxes. At the moment, the government has decided to get rid of Swedish welfare. But there is a long-term alternative in challenging our American rates of tax and opening up a conversation with the public about how much they are willing to pay for the services they want.
This short post is not necessarily an argument in favour of the alternatives, but it is an argument in favour of acknowledging them. Our society and communities are faced with a range of alternative futures at the moment: now is not the time to avert our gaze.
Civic Geeks
What does “civic behaviour” look like? Voting springs to mind, as does volunteering, with perhaps starting a charity or social enterprise towards the black-belt end of being an active citizen. Debugging a page of code in the evenings is not something many of us would immediately point towards. But this particular example of civic behaviour, hidden to many of us, is going on across the country.
It’s become much more visible to those with an interest in technology through the example of pioneers like mySociety, who presciently argued for public sector data to be freely available in helpful formats to everybody at the same time as demonstrating how it could be put to social use through sites like TheyWorkForYou – created entirely by volunteers. And while the slowly-turning machinery of government chewed the idea over (now manifest in data.gov.uk), ingeniously came up with their own solutions of scraping it from the Government’s very web 1.0 sites and making it available to others.
Other enterprising groups have established their own community websites, which pull local residents around their neighbourhood, achieving in a Big Society-ish way some of what local government would like to do, while hacking events like those run by Rewired State (“Geeks meet Government”) bring people together to make useful and open applications from public data.
Rory Cellan-Jones broke the news today that many government websites could be cut, after a review from the government that highlights some of their soaring cost. This review seems in sympathy with a report the RSA published earlier this year that heard a variety of stories around the depressingly wasteful cost of public sector IT and argued for a more parsimonious approach to technology in a cold economic climate.
When the RSA was founded it aimed to “embolden enterprise, to enlarge Science, to refine Art, to improve our Manufactures, and extend our Commerce”, and offered premiums or awards “for any and every work of distinguished ingenuity”. William Shipley, a drawing master, felt deeply about the importance to Britain of the skill of drawing. One of the first premiums given is recorded in the minutes of the RSA’s very first meeting on 22nd March 1754:
“It was likewise proposed, to consider of giving Rewards for the Encouragement of Boys and Girls in the Art of Drawing; and it being the Opinion of all present that the art of Drawing is absolutely necessary in many Employments Trades & Manufactures, and that the Encouragement thereof may prove of great Utility to the public, it was resolved to bestow Premiums on a certain number of Boys or Girls under the age of sixteen…”
Drawing was a key skill in the eighteenth century, but in the twenty-first, it seems to me as though developing computer code is also important to solving some of the real problems we face. Developing code that helps people to feel attached to their neighbourhood, strengthens community, helps keep the government accountable, and reduces the burden on public money is of course a civic behaviour.
Changing the culture of civic life
A government report last year on co-production showed citizen involvement in public services to be higher in the UK than in many other European countries considered to be beacons of public service innovation, including Denmark and Germany.
But this is not translating into greater numbers of people feeling able to influence community-based decision-making.
Feeling unable to make a difference locally not only limits citizen participation in community life, it also weakens civic health and the forms of innovation and attachment between people it creates.
A forthcoming RSA Citizen Power report on civic health shows that attempts to strengthen civic health, from community-asset transfer schemes to participatory budgeting, have been undermined by a narrow focus on ‘consumer power’ as the key driver of public service reform.
Our “age of austerity”, signalled by forthcoming cuts to local government, has the potential to substantially weaken the civic health and wellbeing of our society, particularly in the most deprived parts of the country where public services, from Sure Start centres to Drug Action Teams, are a matter of necessity, not choice.
But it also opens up a space for us to re-evaluate what we want from public services and the values they should embody.
This is an extract from an article published in The Guardian. Read the complete article
From Data to Behaviour
Matthew Taylor’s thesis that a more engaged citizen (as opposed to a consumer) is required to reform government, one that will understand the need to make trade-offs and one that takes personal responsibility for their actions, is perfectly exemplified by many of today’s activist geeks. Individuals like Tom Steinberg and MySociety, a voluntary organisation of technical experts donate their time to the challenge of scraping data from public sector websites (like Hansard records) and re-published it in websites that are far more engaging, allowing others in turn to become more engaged.
Today, principally because of their example, and reports like the Power of Information review, both the present government and the opposition fully recognise the value of making such data available online, where communities of people linked by the internet can ‘get excited and make things‘. This appears to be causing the enormous pent-up enthusiasm to be released; indicated by the membership of the new http://data.gov.uk/ site’s discussion group reaching 1,464 (at time of writing) with some extremely active discussions.
Stephen Timms recently spoke at the RSA, highlighting the success of the government’s effort to open up more data, and the efforts made are likely to remain, with the Conservative party also showing enthusiasm for the idea. When it comes to opening up government data, the most ambitious example I’ve seen so far is http://www.recovery.gov/, the website for the US Government’s American Recovery and Reinvestment Act – Obama’s stimulus package.
I often write about ”persuasive technology” on this blog, which has, as a pretty foundational tenet, that allowing people to see the effect of their actions, or “self-monitoring” can enable and encourage them to change their behaviour (a la real-time energy displays). Freeing public sector data is simply this on a much more ambitious scale.
Responding to young adults in the CJS
This morning the 2020 Public Services Trust hosted a seminar for the Barrow Cadbury Trust’s new report from their Transition 2 Adulthood programme.
Chatham House Rule forbids me from naming names but the usual suspects were present as you’d expect, and they were saying the same old things, having the same old discussions, welcoming the same old recommendations (albeit written in a new way) and challenging the politicians to make the changes happen.
The report itself makes some good recommendations and follows much of the common sense models of change that the Prison Learning Network is working on. But for me, in continuing to separate individuals into age groups, the T2A programme is missing a beat.
While other fields and public services are successfully moving towards co-production and personalisation in order to meet the needs of individuals, the CJS will find itself limited by tailoring responses to these groups. As someone commented earlier, the CJS always seems to be behind other government departments. Can it afford to continue to do so?
Of course, understanding the trends of characteristics and needs of particular groups over particular times in their life course, like between childhood and adulthood, will be valuable but should not solely provide the foundation for policy development or allocation of resources.
Maybe what’s needed is a cross Government personalised plan for everyone. This might catch the people before they enter the CJS and will certainly provide the holistic response needed when we seek to address individuals’ criminal activities, lifestyles and attitudes.
Maybe it’s not just about the Government?
RSA Animate
RSA Mobile
