The Pupil Premium: time for a Plan C?
Schools have multiple purposes, some universally agreed upon, many contested. Judging from Nick Clegg’s re-announcement about the Pupil Premium on Monday, schools have one new purpose: to demonstrate the impact of the Liberal Democrats on Coalition policy.
As Conor Ryan’s blog explains in detail, the Pupil Premium is essentially a small addition to a school funding system which already has significant weighting for disadvantage and other factors. Its existence may actually stall progress towards more radical changes to school funding – ones which would differentiate more for social class, but less for age. Given what we know about the importance of the early years, it is difficult to justify why secondary pupils have so much more money spent on them than primary children. Such changes would create genuine losers, mainly sharp-elbowed secondary schools in more affluent constituencies.
But let’s welcome the Pupil Premium’s existence, and its totemic as well as cash value, and ask ourselves two further questions. What should schools do with this additional funding? And how should they account for this spend?
Schools which are successfully closing the achievement gap – and the Education Endowment Foundation has found a significant number of them – are likely to be taking a whole-school and whole-budget approach. Yes, within their strategies will be a number of smaller initiatives, but at their foundations lie high quality teaching for all, and forensically targeted teaching interventions for a few, all underpinned by excellent pastoral and extra-curricular support. Any school that sees the pupil premium as the answer to their achievement gap woes (and there won’t be many of them) probably won’t succeed.
There is thus no clear rationale for the DfE requirement for schools to account separately for their Pupil Premium spend. And unless schools use this funding to develop a highly controlled, bounded intervention, it is unlikely that we will ever know the specific impact of this injection of funding. Correlation will be difficult, causation almost impossible.
There are three orthodoxies around the use of this funding which need challenging:
First, unlike yesterday’s announcement about personalised SEN budgets, there is an overwhelming consensus that it should be schools themselves, rather than individual pupils or parents, who should determine how the Pupil Premium should be spent. This might well be justified, but could a few schools create a model where where pupils, parents and school co-commissioned additional support, learning from the disaster of Individual Learning Accounts and the quiet success of Pupil Learning Credits.
Second, schools are not being encouraged to think more expansively about how the Premium could lever additional match funding from other sources. If funding was pooled with other schools or other funding streams that are dealing with a similar client group it could be spent collaboratively to achieve far more, and develop more robust evaluation methodologies, adopting EEF methodology. Again, some schools may be taking this approach, but competitive and accounting pressures mean that collaborative use of this funding is unlikely to be the norm.
Third, Clegg has placed the Pupil Premium at the centre of his strategy to increase social mobility. Anna Vignoles’ blog demonstrates the tenuous link between such spending initiatives and social mobility, and the RSA’s Louise Thomas has usefully questioned many assumptions in this debate. The truth is that, despite the distracting headline figures around Oxbridge entrance, it is too early to know the full impact of the previous government’s policies on social mobility. If social mobility has stalled in the last twenty years, it will be largely as a result of what happened in the twenty years before that. The pupil premium may help deliver outcomes which provide stronger foundations for social mobility to increase, but it would be a mistake to promise any more from this short term initiative.
Later this week the RSA publishes its Plan C for economic recovery: ‘coping with long term slow growth’. My own plan C for the pupil premium (which would need no re-announcement) consists of three Cs: co-commissioning, collaboration, and coping with premature evaluations from a government in a hurry to prove its impact.
Resilience: often necessary, occasionally evil
Yesterday, straight from an energising discussion with our Projects team about future RSA approaches to public services issues, I rushed to deal with something more current and tangible. My twelve year daughter has a long term health condition, which means regular appointments and occasional bouts of hospitalisation. After twelve years navigating a Victorian monolith, we now have the airy complexity of a brand new PFI building. We’ve gone straight from Dickens to Huxley.
My daughter has always been intense and feisty – most people who spend a few hours with her need to come up for air at some point – but in her regular interactions with medical people and places, this is amplified. And adolescence is now adding to the mix. Yesterday, she refused to answer questions that weren’t using the correct medical terms on the piece of paper in front of the physiotherapist. She asked irritating questions, gave cryptic answers, and her body language was moody, sullen and horizontally sprawled – she looked like she was on our sofa watching something excruciatingly boring on TV.
Like any parent would, I often plead for her to be more polite to a group of people that definitely want her to be as well as possible. At the same time, I know that her assertive games are a form of resilience – a way of coping with loss, setbacks and change, and steeling herself for future battles and disappointments. She is an expert patient now, and her attitude in some ways ensures that the system treats her as such.
I remember Maria Balshaw, now Director of Manchester City Galleries, arguing that ‘arsiness’ was a key attribute of creativity, so should possibly be taught in schools. I doubt if this idea will catch on, but we do need to accept the need to develop qualities in our young people that aren’t always pleasant. Whether it’s the liberal perspective on social and emotional learning, or the more traditional approach through character education, both emphasise qualities and attitudes that, in essence, make children easier for us adults to deal with. Just be nice. Even our Opening Minds framework, which includes ‘coping with change’ as a key aspect of the ‘managing situations’ competency, might not be quite ready to develop and assess approaches which elicit and celebrate the nasty.
This links to an emerging idea for a broader RSA project: can we harness new insights into the teenage brain and other research to ask how can schools and society relish rather than fear the teenage years? What kinds of behaviour change do we need to promote, in both teenagers and the adults and institutions which deal with them, to ensure a happy, productive adolescence?
Beyond Big Society Capital: a future shaped by ‘socialised investment’?
Many have spoken of the potential value of social investment, or investments that include both financial and social returns. Building on initiatives from the previous government, the current government has articulated a vision and strategy for growing the social investment market. The Big Society Bank (now renamed Big Society Capital) forms a core part of this, but the underlying aim is to provide the infrastructure and support necessary for the social investment market to overcome various barriers to its growth. While social investment has been growing, there are several problems that limit its potential scale, such as a lack of investment readiness on the part of social ventures, lack of market information, and cultural and behavioural barriers among both those running social ventures and potential investors. This is why much of the debate on the feasibility of social investment has revolved around the importance of ‘social market intermediaries’ that are able to provide the expertise and infrastructure required to connect funds to appropriate social ventures, and to enhance the market readiness of social ventures.
But it might also be argued that an even more fundamental shortcoming is the lack of a focus on giving citizens a greater stake in social investment. Social investors are predominantly rich individuals (such as philanthropists) or large organisations, and social investment is typically associated with social enterprises. In comparison, citizens have very limited opportunities to become social investors, which isn’t helped by the lack of engagement with social investment by institutional investors such as pension funds.
This absence of a democratic space for citizens to develop a greater stake in social investment has led Bristol City Council to pursue its forward-looking Building a Better Bristol programme (which 2020PSH will be assisting), which is working on developing credible investment products and governance structures to support ‘socialised investment’ – getting the wider public, businesses and institutional investors involved in investing in the social and economic development of the city while ensuring financial feasibility and investor confidence.
At a time of austerity and debates about how public services and economic and social development can become more sustainable, productive, and locally relevant, ‘socialised investment’ raises some very interesting long-term opportunities for local government, including:
- A new role for local government. As localism becomes more embedded in statutory and policy frameworks, local government can carve up an enabling local leadership role (possibly having a ‘social market intermediary’ role of its own) to catalyse social change and public service reform.
- Public money can be combined innovatively with pooled socialised investment funds and leveraged to collaboratively finance services and developments, which could also help relieve pressure on cash-strapped local authorities.
- The commissioning and delivery of services will theoretically be more intertwined with what citizens want from their services. While tax is a legal obligation, social investors will expect not just a financial but also social return – and this will need to be negotiated with citizens. Expectations of social returns will also have implications for the degree to which commissioning is based on the generation of social value. It will also have the potential to reshape how we should think about productivity in public services – something 2020PSH will be exploring soon.
- Greater local accountability and transparency, and greater local distinctiveness.
2020PSH has long argued for social productivity as an organising principle for public services – that is, public services should be judged according to the degree to which they help citizens, society and local communities achieve the social outcomes they desire. If it works, and if it is able to successfully scale out across the country in the long term, ‘socialised investment’ could be a fundamental part of socially productive public services. While it certainly is at its very infancy, the potential is also great.
Nothing to lose but your chains?
Contradicting my previous call for reflectivity, last week I wrote a rapid-reaction article for the Local Government Chronicle about the future educational role of Local Authorities. I’ll expect a bit of heat since I suggested that councils needed to embrace academisation, and also that:
“In the near future, some academies may realise that total independence comes with pitfalls, or that relationships with their academy provider have become excessively autocratic or bureaucratic – that they have nothing to lose but their chains.”
The constellation of RSA activities I am getting my head round here, from the Academies Commission to work on the middle tier to the creation of our own family of academies model, all connect with broader issues around freedom, automony and control. I’ve also been inspired by Adam Lent’s recent RSA Journal article on freedom, and how an individualistic, consumerist model of freedom has come to dominate and overwhelm older, more collective concepts.
Beyond the blustering rhetoric from all sides in education debates around trusting teachers and freeing up schools, is there space for a more sophisticated, nuanced and evidence-based conversation about freedom in our education system?
The general New Public Management mantra was that central government should define the what, giving localities, and professionals the freedom, within a tight accountability regime, to do the how in any way they saw fit. This should be challenged on two fronts. First, individual institutions and communities should have significant influence over defining the ‘what’. This has to lie at the heart of localism. But second, where there is overwhelming evidence that one particular approach delivers the best outcomes, the Centre should reserve the right, and may even have the duty, to mandate this approach. It’s not about favouring prescription or autonomy – it’s about developing a clear rationale for when each approach is employed.
What might this mean in practice? On the what, the government is quite rightly slimming down the national curriculum, partly to give space for a local whole curriculum to flourish. This commitment needs to be reinforced by an ‘intelligent accountability’ that values the whole curriculum, rather than only the nationally-prescribed elements.
On the how, given the universal consensus about the importance of literacy, if the synthetic phonics approach is as effective as advocates claim (and the jury between my ears is still out), then the government is quite right to enforce this method in any way it can. In areas of practice where the evidence is mixed, schools should be encouraged to adapt and innovate, partly to play a role in improving the evidence base.
Last week, I participated in an OECD conference on local governance, part of a project on Governing Complex Education Systems. The UK* was seen by participants as an outlier in the way that we had hollowed out the local, giving both more freedom to schools and more power to the centre than any other OECD country. Whether we are outliers, trailblazers or just a nation of educational guinea pigs, the experiments taking place in our schools right now may, if properly reflected upon, shed new light on the theory and practice of freedom, both in education and across public services, and what kinds of freedoms lead to the best possible outcomes for all.
* Despite my attempts to explain the wonders of devolution, the OECD participants did not distinguish between England and the devolved nations.
Double Dip but No Strategy?
Blog by Henry Kippin, 2020 Public Services Hub at the RSA
Confirmation that the UK is now officially back into recession must be pretty shattering news for a government that has based a big part of its political strategy on avoiding a double-dip. Statistical confirmation merely supports what many in the construction, manufacturing and high street sectors already knew – that we are now in a sustained period of low-to-no growth. The Guardian today talks of ‘little sign of recovery before 2014’.
The blame game is in full swing. Opposition blame the Coalition for a rapid adjustment policy that has stripped demand and stunted recovery. Coalition blames Labour for the inheritance of a huge debt burden and an economy in decline. Both blame the Eurozone economies – but at different times, and for different reasons. Both are stuck between fingering blame on the financial sector and hoping it will drive economic recovery, and the green shoots of a more sustainable and collaborative growth model risk being trampled by the urge to create quick and effective economic incentives.
Buried under the economy and Leveson this week was a report from the Public Administration Select Committee , which called for an annual ‘statement of national strategy’ in the absence of a ‘critical and unfulfilled’ strategic direction for government. The implication being that – in the rush to decentralise, liberalise and retreat from clunking fists and deliverology, there is little coherent thinking on the long term direction for the economy and public services.
There has certainly been a lack of coherence (from both government and opposition) in this area. As we write in a forthcoming report, “public services need to be reformed to meet the demands of the long-term, under the fiscal constraints of today. Economic growth needs to be re-kindled – but in a more sustainable form that carries public confidence and delivers fairer returns. These are the two big policy challenges of our time. Yet they are not being considered together, and risk pulling in conflicting directions.”
In a new 2020PSH report (launched on 10th May at the RSA) we suggest a few ways of pulling these agendas together. But a vital part of this is taking a view of the public sector and public services as an intrinsic part of the growth strategy – not as a structural barrier to be peeled back. This means thinking hard about public service productivity: what we mean by it, and how we can use public spending to drive sustainable growth and employment in future. We think our social productivity approach (which emphasises strong public-private-social relationships) is a way in to this, and will be picking this up in a new programme on public service productivity in the summer.
Henry Kippin @h_kippin
Learning how to bow out gracefully
How often do we ask ourselves: is what I’m doing truly working? It’s a simple question and one which makes intuitive sense to ask if we ever intend to learn from our mistakes and improve the impact of our work. And yet despite this it is something we tend to ignore time and again. Both at an individual and institutional level, many of us are reticent to the idea of evaluating the policies and practices that shape the effectiveness of our public services and which dictate their value for money. Indeed, these days it is quite rare to hear or see the term ‘evidence-based policymaking’.
For youth services at least, Project Oracle is attempting to change all of this. The initiative was set up by the Mayor of London’s office to help smaller organisations that are working with young people evaluate their programmes using ‘rigorous and internationally recognised’ standards. The way it works is that once organisations have signed up they are provided with free advice and support on how to assess their work, and are guided through different ‘levels’ of evaluation that gradually become more sophisticated. The Project has the added benefit of creating a sound structure for collecting and disseminating cross-comparable data that everybody in the sector will find meaningful – at least in the capital.
While attending a recent seminar at NESTA to learn more about the project, I heard a number of interesting points being raised about the obstacles to undertaking evaluation schemes and the subsequent difficulties of making use of the data once it gets collected. Many of those attending, for instance, said they feel as though there are cultural differences between people working in the voluntary sector and those in the academic/policy world; academics may insist on gathering quantitative data but service practitioners find anecdotal evidence far more useful. Another key issue raised was that although many funders are willing to pay for the evaluation of an organisation’s operations, only ever a handful actually commit to providing the resources for the implementation of recommendations that emerge from the research.
While all of this is no doubt interesting and useful, it felt as though the conversation side-stepped one of the biggest impediments to the initiation, the quality and the utility of evaluation schemes. This is the simple fact that many of us have difficulty in accepting defeat and apportioning fault. Whether a frontline practitioner or a senior manager, taking part in an evaluation process may open up a Pandora’s Box of knock-effects, which at best may lead to the radical restructuring of the organisation and at worst the termination of projects and ultimately job losses. Vested interests aside, there is also the challenge to service users and colleagues who may find themselves in the uncomfortable position of saying, albeit honestly, that someone’s efforts and practices are ineffective. It is one thing to acknowledge failings in our own work, but to highlight the caveats in someone else’s takes some courage.
The reason why this is doubly important is because there has rarely been a greater time when we have had to identify failure in our work and be open to new approaches. Whereas in previous years public service innovation was characterised by the sharing and adoption of universally recognised ‘best-practice’ approaches at home and abroad, the next stage is arguably going to be an era of localised, radical experimentation. In other words, it is likely that organisations providing public services will be encouraged to become their own ‘innovation labs’, testing different methods and practices until they land on the thing that works best for them. In practice, this could mean a school experimenting with different ways of teaching maths, or it could mean a GP consortium trying out innovative new health treatments with their patients.
Wherever this new wave of experimentation and rapid evaluation takes places, it will demand that service users, practitioners and those in senior management have a certain type of mindset which is comfortable with ambiguity and not afraid of failure.
It could be said that in the future there will be two sides to the coin when it comes to public service transformation. The first is that success depends on learning what works and adopting these approaches; the second is that we learn what doesn’t and ensure that these styles gracefully bow out. To date, it seems we have focused too much on the former at the expense of the latter.
Has the Department of Health been misusing behavioural economics?
It seems that as soon as Andrew Lansley announced plans to “nudge” the population towards healthier and more socially responsible behaviour, scathing criticisms have been levelled at the alleged naivety – and to some the danger – of his policy backing for voluntary pledges, grounded in nudge theory. The Commons health select committee criticised the Department of Health’s (DH) Public Health Responsibility Deals as ineffective for tackling health and social problems such as smoking, excessive drinking and obesity – warning that they could widen health inequalities; that self-interested companies should not be allowed to set the agenda; and that the government should use legislation if “nudging” fails. The House of Lords Science and Technology Committee similarly questioned the efficacy of using voluntary mechanisms without sufficient regulatory measures to achieve behaviour change. Even internally within government the idea has had little traction; a National Audit Office report last year noted government departments were not consulting the Behavioural Insight Team on issues around alternatives to regulation at the assessment stage. Oliver Letwin even acknowledged that it was “open to question whether any of this would have any effect whatsoever.” The Behavioural Insights Team itself cautioned that “’nudge’ approaches should not be seen as a hard alternative to other policy approaches, but a useful complement or additional tool.”
The recent government decision on minimum alcohol pricing has brought the issue back into sharp focus. David Cameron has effectively clashed with the DH (and the drinks industry) on the issue and has agreed with health experts. The tensions were evident in the decision by the Department of Health to issue a press release about recent achievements of the responsibility deal just as the minimum pricing decision was close to completion. Just as in the past, the Department’s voluntary pledges have come under attack and have been unsettled by evidence that conflicts with its core assumptions by experts and research – including a Sheffield University study that shows raising prices cuts drinking. The consumer watchdog Which? has also conducted an audit of the Government’s use of voluntary pledges to tackle obesity and found that many food companies have failed to commit, and that legislation needs to replace “doomed” voluntary schemes.
It appears that the Department of Health is committing the error that the government’s own Behavioural Insights Team warned against; using “nudge” as a “hard tool”, rather than a complementary addition to other policy approaches such as legislation and regulation. Behavioural economics offers great insights into how pro-social behaviour can be promoted without always relying on ‘hard’ state intervention, and the RSA has demonstrated this in numerous publications as part of its Social Brain project. Similarly, as Matthew Taylor has argued, businesses can use their expertise on (effectively) manipulating our cognitive systems (which they have used for their highly sophisticated marketing strategies) to achieve “downstream” social gains – and therefore they have a part to play in promoting positive behaviour change. But evidence indicates that in highly complex issues that have a huge impact on society – such as how to deal with obesity, drinking or smoking – “nudging” cannot be the centrepiece, but rather a complementary part of a larger, more comprehensive package. The DH’s failure to acknowledge this has led to critics deriding the responsibility deals as coups for big business – an impression reinforced by experts, consumer groups and civil society groups walking out on previous negotiations. A distinct lack of support and collaboration is unlikely to lead to the development of effective policy for meeting complex challenges.
And this is a key lesson we have learned at the 2020 Public Services Hub while researching collaborative and innovative forms of regulation for our Zero Carbon Hub case study. In zero carbon new homes policy, we found that voluntary schemes alone were insufficient for achieving significant carbon reductions for new homes, and that it was a collaborative form of regulation that explained the major achievements towards zero carbon targets. The careful establishment of the Zero Carbon Hub enabled key stakeholders (including Government, business and civil society groups) to collectively produce solutions, facilitated by the ‘shared space’ provided by the Hub and supported by its clear long-term policy vision, without needing to resort to lobbying government to influence policy. This is in clear contrast to the DH responsibility deals, where the debate is highly polarised and different stakeholders show little interest in engaging one another. Had the Department followed the example of the ZCH, where collaborative regulation – rather than state withdrawal and voluntary nudging – is the organising principle of policy, a significantly more comprehensive and effective approach to tackling public health issues may have emerged. It may be time to see tools such as nudge as valuable parts of a collaborative policy framework that includes regulation – but a new, smarter form that is able to convince industry, government, experts and civil society to participate in ‘shared spaces’; transcending both traditional, top-down state intervention and the ineffectual volunteerism approach promoted by sections of the right. Some on the right see behavioural economics as an opportunity for state disengagement (claiming that a “nudging state” is a better alternative to a “nanny state”) – but if it is to effectively inform policy, behavioural economics should not be (mis)used in this way.
Sunderland City Council’s Community Leadership Programme: Big Society, but with the politics?
It is now stating the patently obvious to say that the public policy and public services landscape – especially at a local level – is undergoing a transformative change. Austerity, fiscal pressures, political shifts in Whitehall and a raft of social, political and demographic changes and pressures are forcing decisionmakers and public managers to rethink the way services are designed and delivered. “Public service innovation” has become a buzz-term, and councils are at the forefront of attempts to re-design services in the context of massive fiscal squeezes, ageing populations and rising demand for services. This is why 38% of all local authorities in England and Wales applied to be a part of the LG Group and NESTA’s Creative Councils programme last year – with the hope of implementing radical innovations to meet these challenges.
Public policy is also changing: the localism bill and associated policies and drives for greater decentralisation (or, according to critics, attempts to shift responsibility – and blame – from Whitehall to local government) – along with overarching narratives such as ‘the Big Society’ – are heralding in a future where ‘more with less’ is set to become the central organising principle. The Big Society vision has been used to provide a great deal of the moral case for change: the government says it wants to see greater power in the hands of local communities and citizens, and wants to see an active and engaged citizenry as a core part of a new era of politics and public services.
The difficulties and contradictions are abundantly clear. Some argue councils simply can’t deliver ‘big society’ initiatives at the same time as they are forced to absorb massive cuts in funding – this is why Liverpool City Council withdrew from the government big society pilots last year. Nevertheless, several other councils are making strong attempts despite recognising the pressures of austerity and disagreeing with current Government policy. For examples, thirteen Labour councils – including Sunderland – have joined the Labour Co-operative Councils network, which seeks to develop co-operative models for running local services – putting citizens at the centre.
Much of the scepticism about the ‘big society’ comes from many regarding it as a rhetorical device, and arguing that it relies too much on volunteerism without a solid political structure behind it. It is clear that for truly ‘big society’ politics to take shape, the concept needs to be less associated with volunteerism (although this is important) and more grounded in a new political economy – a point Philip Blond has repeatedly made.
In this respect, Sunderland City Council’s Community Leadership Programme (CLP), which is a key strand of the council’s ‘Sunderland Way of Working’, provides a good example of a local model that is moving towards a political ‘big society’ approach- even though the Labour-run council would eschew the ‘big society’ as a political term. As the 2020 Public Services Hub’s latest report – an evaluation of Sunderland’s CLP – shows, the CLP encompasses multiple strategic layers. This includes engaging elected members more effectively as community leaders and creating the processes and structures necessary to empower them at the community front-line. The second layer is about reconfiguring public services so that they are locally responsive and foster new forms of delivery and accountability in partnership with citizens. The third layer harnesses the power of people, place and council to achieve sustainable growth at a time of political and economic flux. While Sunderland’s CLP certainly faces challenges and has space for improvement, the 2020PSH report shows that it provides valuable lessons for localities across the UK.
At the roundtable marking the launch of the report (on Wednesday 8th February), there was also a general broad agreement by participants on different sides of the political fence about the importance of locally responsible and citizen-centric services supported by various forms of community leadership. Participants at the roundtable also raised some of the challenges that face local politics in practice. For example, Christina Dykes (of the Conservative Next Generation Project) spoke of the need for a culture change at the local level, which Government needs to be proactive in helping. Conservative councillor for Hammersmith and Fulham (and former Leader) Stephen Greenhalgh also said there needs to be greater effort in achieving mass engagement and communication to make local politics relevant to local people. Representing civil society, Neil Jameson (CEX of Citizens UK) and Lucy de Groot (CSV) also argued that civil society is the missing ingredient in many approaches to reinvigorating local democratic politics – civil society has the greatest engagement and contact with citizens, and yet it is often the weakest political actor in the local mix, and so it needs to have greater support. All participants generally agreed there needs to be a politics behind local democracy, and the state at various levels has a key role to play in making narratives such as ‘big society’ viable in practice.
To read the 2020PSH report, click here.
Blinded by science
The assessment of research is of enormous public importance – the findings of research determine the evidence base for a vast range of decisions which affect every aspect of our lives. The rise of bibliometrics as the primary means by which we gauge the quality of research is therefore something that the public should know more about, especially because it is deeply flawed.
For obvious reasons, this isn’t something that’s had much attention from the media (although the Guardian published an article relating to the topic last year). It might not be sexy, but it’s an issue which is critically important, with implications that touch more or less every aspect of our lives.
As an early career academic I had the mantra ‘publish or perish’ bashed into my consciousness on a more or less daily basis. To have a successful academic career, researchers need to publish as much of their work as possible, in the highest rated journals. If you publish a research article, and it gets referred to a lot by other academics, this is regarded as an indication of the research being ‘good.’ Even if the only reason your research is being cited all the time is because others are using it to illustrate what is wrong with it. There’s also this strange academic norm, in which researchers and theorists are expected to mainly cite research published recently, which is a bit bonkers, really. Old research can often be the most important or most relevant basis for new studies.
In order to publish, you need to provide a context for your work, citing key texts from the big players in your field (thereby driving up their citation indices). If you’re being natty about it and you have recent publications yourself, you’ll also make sure you always cite all of them (even if they’re not relevant), in order to drive up your own citation rating.
The whole process is mediated by journal editors who rely on peer review to ensure that what’s being published is of high quality. And, the main measure of quality on which journals are judged is how much they are cited by others. It’s a kind of crazy game, with the rules being set, refereed, and maintained by the players. One way of looking at it would be to call it an international popularity contest.
The general drive towards transparency and accountability in the academic world is a good thing. But, I believe to do this properly, we need to take a realist stance, capable of recognising the complexity of the world we are researching. Instead, we’ve created a culture of numbers, in which the whole academic edifice has come to believe that fair judgements of quality can be reached by using algorithms to analyse statistical data, even though those data are unable to measure quality in any meaningful sense. The use of citation indices as a proxy for quality is completely illusory, and yet, the system has taken hold amongst an international community which really ought to know better.
But, once you’ve invested into this system, and start to do well at it, it would be against your own self-interest to be critical of it. That’s not to say there have not been attempts to get the academic world to examine its own practices. In an excellent report from the Joint Committee on Quantitative Assessment of Research, a group of mathematicians and statisticians explain all of this in much greater detail and with a much more thorough grasp of the issues than I can offer.
One of the most crucial things they say is that although citation counts seem to be correlated with quality, the precise interpretation of rankings based on citation statistics is not very well understood. And, given that citation statistics play such a central role in research assessment, it is clear that authors, editors, publishers and institutions are becoming adept at finding ways to manipulate the system to their advantage.
The report concludes that the long term implications of this are unclear and unstudied. But, it doesn’t take a hugely active imagination to envisage some of the potential consequences. The report was published in 2008, so, by some standards is itself already out of date, and given that practices for assessing quality in academic research show no signs of changing, it seems, so far, to have fallen on deaf ears.
Realism in a time of crisis
The idea of ‘being a realist’ is rather slippery. Realism in everyday parlance is taken for granted as a sort of common sense perspicacity. We can all relate to being told to ‘get real,’ or ‘be realistic,’ but what that actually means, when you stop and think about it, could be all sorts of different things to different people.
In academic circles, realism is even more slippery, with a cascade of categories you need a lot of patience to pick a path through. At the top: philosophical realism, scientific realism, political realism, artistic realism. Beneath each of those, more specifics: critical realism, transcendental realism, naïve realism, so on and so on. There’s a veritable spaghetti junction of realism out there.
In my research training, I studied ontology and epistemology – basically what there is to know, and how we know it. Lots of people think this stuff is boring, but for me, it is in these questions that the real power to effect change lies. In the world of academic research, the paradigm within which you conduct your work is probably the number one defining influence on what you do, even if you don’t say so yourself.
For a long time there was a tug of war between quantitative and qualitative paradigms; positivism versus interpretivism. It doesn’t get much popular press, but in the relatively recent past, profoundly significant advances in the philosophy of science have brought us to a point where we have a real opportunity to transform our world by adopting a different stance to what we know about it. And, it’s realism.
Personally, I’m hugely influenced by Roy Bhaskar’s Critical Realism, but I have one major criticism, which is that it can be bloody difficult to understand. And, what’s the use in world-changing philosophy, if it isn’t going to make sense to enough people for it to take hold? I find Ray Pawson’s version of realism a little easier to digest, probably because his work is so firmly rooted in in the type of real world research I’ve done myself, like evaluating educational and public health interventions.
So, it’s rather bold, perhaps, but I’ll risk sounding ridiculous and assert that only realism can save us from the state of crisis we are in. Adam Lent wrote yesterday about ethical revolutions. Whether we are indeed on the cusp of a time of ethical revolution and not further descent into crisis depends on changes happening on many levels. So many of our power structures are built on evidence bases which have non-realist foundations which by their nature try to eliminate complexity and identify clean causal relationships. With this explicit rejection of the reality of the complex world we occupy, we’re shooting ourselves repeatedly in the feet.
I’ll try to illustrate. In public services, we are constantly trying to do things within complex social systems, and in order to know whether the things we’re doing are effective, we need to measure and evaluate. With something like an educational intervention with the aim of, say, reducing teenage pregnancy, its effectiveness might be measured by counting the number of teenagers who become pregnant after receiving the intervention and comparing that with the number of pregnancies amongst those who didn’t get the lesson.
The research will try to ‘control’ out all of the social factors which influence whether or not teenagers become pregnant, and make a claim as to the causal relationship between the intervention and the outcome. We want to be able to conclude,’ if the kids have the sex education class, they won’t get pregnant’. But of course, the world is messier than that. The best sex education in the world doesn’t stop condoms breaking every now and then, or emergency contraception failing.
The positivist view of the world relies on the exclusion of critical but unpredictable factors. This type of research might lead to nice, neatly analysable statistics, but whether it’s useful or not, or answers the most important questions is another matter. Realist evaluation, by contrast, assumes that in order to infer a causal relationship between two events (i.e. sex education and teenage pregnancy), it is necessary to understand the underlying mechanism that connects them AND the context in which the relationship occurs. Rather than asking simply ‘what works?’, a realist evaluator asks ‘what is it about this intervention that works, for whom, in what circumstances?’ More complex, yes, but more useful also.
Even if you’re with me this far, you might well be thinking, ‘fair enough, but how is this going to save the world, exactly?’ I have a feeling it’s going to take more than blog post for me to fully unpack this and show why it matters so much. I promise I shall try to avoid becoming terribly dull…
RSA Animate
RSA Mobile
