Category Archives: social science

Complexity and evaluation

Here is the abstract of my latest article on complexity and evaluation, which you can find here:

This article offers a critical review of the way in which some scholars have taken up the complexity sciences in evaluation scholarship. I argue that there is a tendency either to over claim or under-claim their importance because scholars are not always careful about which of the manifestations of the complexity sciences they are appealing to, nor do they demonstrate how they understand them in social terms. The effect is to render ‘complexity’ just another volitional tool in the evaluator’s toolbox subsumed under the dominant understanding of evaluation, as a logical, rational activity based on systems thinking and design. As an alternative I argue for a radical interpretation of the complexity sciences, which understands human interaction as always complex and emergent. The interweaving of intentions in human activity will always bring about outcomes that no one has intended including in the activity of evaluation itself.

Advertisements

Complexity and ideology

If you can prevent yourself following the footnotes to the end of the post, try and guess who offered this critique of scientific method when applied to the social:

“Yet the confidence in the unlimited power of science is only too often based on a false belief that the scientific method consists in the application of a ready-made technique, or in imitating the form rather than the substance of scientific procedure, as if one needed only to follow some cooking recipes to solve all social problems. It sometimes almost seems as if the techniques of science were more easily learnt than the thinking that shows us what the problems are and how to approach them.”[1]

Perhaps this is a quotation from a post-Marxist sociologist, or a post-modern relativist worthy of being mocked by natural scientists such as Alan Sokal?

How about this quotation from the same person on the limitations of modeling social phenomena using statistical methods:

“Statistics of limited use because it proceeds on the basis of reducing complexity: it deliberately ignores the structure into which the individual elements are organized. We can talk in generalities, if, all things being equal, certain patterns will occur. We should have developed beyond the understanding that we are in search of simple regularities which will help us with predicting events. The idea that to be scientific we have to produce laws has proved very harmful.”[2]

Maybe these are the thoughts of a famous social anthropologist or a critical management scholar?

Or lastly, the observations of our eminent mystery guest on social complexity:

“Since a spontaneous order results from the individual elements adapting themselves to circumstances which directly affect only some of them, and which in their totality need not be known by anyone, it may extend to circumstances so complex that no mind can comprehend them all. Consequently, the concept becomes particularly important when we turn from mechanical to such ‘more highly organized’ or essentially complex phenomena as we encounter in the realms of life, mind and society. Here we have to deal with ‘grown’ structures with a degree of complexity which they have assumed, and could assume only because they were produced by spontaneous ordering forces.”[3]

Perhaps this is a contemporary of Norbert Elias, another process sociologist? Or perhaps another pragmatist arguing that ‘mind and culture developed concurrently rather than successively’[4]? Continue reading

Meeting the universe half way

In her book Meeting the Universe Half Way the theoretical physicist Karen Barad (2007) draws on quantum theory and the philosophy of the Nobel prize-winning physicist Nils Bohr, to develop her thinking about the paradoxical relationship between the knower and the known and the sense we can make of the world through our engagement with it. She argues that Bohr’s philosophical reflections on his work in physics provides opportunities for linking the natural and social worlds in the sense that we are part of the natural world we seek to understand. She accepts that both Bohr’s views (he was regarded as too philosophical for a physicist!), and her own interpretation of them are contested, but I will explore them nonetheless because both perspectives are interesting and helpful in the context of the discussion on this blog about systematic ways of comprehending the social. Her ideas are interesting in terms of furthering the discussion about what it means to be scientific. Continue reading

On the complexity of stability and change

Long before theories of complexity became established in the natural sciences, the sociologist Norbert Elias wrote about social development as the complex evolution of ‘blindly operating’ processes. Greater interdependence in increasingly highly differentiated societies has led to longer and longer chains of people who are functionally interdependent with others. In other words, and without drawing on complex adaptive systems models, Elias noted how we are formed by, and at the same time we are forming the social processes of which we are part. It is not adequate to ascribe social change to the actions of highly charismatic individuals, on the one hand, or to mystical descriptions of emerging ‘wholes’ realising some kind of archetypal order, on the other. Instead, he argues, society evolves through the interweaving of intentions, a patterning which simply produces more patterning. Our plans and strategies form a tissue, an intermeshing web of actions and reactions, which are very difficult to interpret and to predict. There are trends in the patterning of social relations, and these tend in a particular direction. But the direction is not always forwards, and the consequences not always good. Development, or developments, are not always positive but are likely to both create and destroy. Continue reading

Payment by results: research methods and disciplinary power

I was sitting in a meeting with a social development organisation listening to the kinds of requirements that have been placed upon it by a governmental body in order to trigger the full funding for a grant that they had succesfully bid for. 10% of the grant is ‘performance related’. In other words, and on a sliding scale of reward for performance, the social development organisation has to prove that it has helped educate a certain number of girls in a developing country to a predicted level of attainment, and that these girls will have stayed in school for the three year duration of the project and not dropped out. Additionally money is released against the achievement of pre-reflected project milestones. ‘Results’ are validated by ‘rigorous research methods’ which turned out to mean quasi-experimental methods. In other words, the rubric insists that the project sites be compared with communities where there has been no such intervention, and which are ‘similar in every way’. The organisation will only be fully rewarded if it achieves exactly what it said it would, and precisely to the timetable it set out in the proposal.

This particular social development organisation I am visiting is one amongst a dozen or so others which have received similar or much bigger grants, some of which amount to the low tens of millions. All of them have proposed highly complex interventions in very different developing countries involving the girls themselves, their families, teachers, head teachers, community groups, religious and community leaders, sometimes even boys. As with most social development these days the intervention is highly ambitious and leaves the impression that the organisation, working through a local social development organisation in the country concerned, will be intervening in particular communities at breakfast, lunch and dinner and in a variety of different and incalculable ways. This combination of interventions may be necessary, but the extent and range of them makes the question of causality extremely problematic, experimental methods or no.

The other thing that struck me is that the dozen or so social development organisations receiving this money all have to use the same project management tools and frameworks so that the government department can aggregate progress and results across all countries and all projects. Quantification and standardisation is necessary, then, in order to render the projects commensurable, and in order to make a claim that the government has made a quantifiable contribution to the Millennium Development Goals (MDGs) which they can ‘prove’. The kind of assertion that the government would like to make is that it has improved X tens of thousands of girls’ education to Y degree through its funding of a variety of organisations. These results, the claim will continue, will have been rigorously demonstrated through scientific methods and will therefore be uncontestable. Continue reading

Complex, but not quite complex enough II

Evaluation scholars abstract to varying degrees from the social programmes they are invited to evaluate. Perhaps the highest degree of abstraction is demonstrated by those evaluators using experimental methods who are concerned to draw statistical distinctions between a ‘treatment group’ and a comparator group which is randomly selected. Experimentalists are generally disinterested in social theory and think of causality in terms of independent and dependent variables. Meanwhile, adherents of Theories of Change (ToCs) made popular by the Aspen Institute (1997), draw on propositional logic and represent social change in the form of entity-based logic models showing the linear development of social interventions towards their conclusions. Additionally, however, they will often point to the importance of participation and involvement of the target population of programmes to inspire motivation. In this sense TOCs are a hybrid of functionalism and emancipatory social theory, which encourages participants in social programme to be active in the change process.

Less abstract still are ‘realist’ evaluators who claim to be interested in ‘generative’ theories of causality, i.e. ones which open up the ‘black box’ of what people actually do to make social programmes work or not.  Realistic evaluation draws on Bhaskar’s critical realism (1978) as taken up and developed by Pawson and Tilley (1997) and Pawson (2006) and is the theory most often linked to the complexity sciences, particularly complex adaptive systems theory (CAS).  In trying to reconcile realistic evaluation and CAS they adopt a functionalist, systems-based understanding as a default position and argue that interactions between human beings take place as ‘mechanisms’ and have an effect at different ‘levels’ of reality.The conceptual link between  CAS and realistic evaluation is that they both have an understanding that stability and change does not arise because of ‘variables’, the staple of experimental methods, nor does it proceed with propositional logic as in ToC, but as a result of what people are doing in their local interactions with other people. CAS are relational models demonstrating how patterns emerge over time because of ensembles of interacting agents. So from a realistic perspective and in the words of Pawson and Tilley:

Realists do not conceive that programmes ‘work’, rather it is the action of stakeholders that makes them work, and the causal potential of an initiative takes the form of providing reasons and resources to enable programme participants to change. (1997: 215)

So both CAS and realist evaluators are most interested in local interaction as the basis for developing more general observations about the success or otherwise of social interventions. Realistic evaluators argue that interventions do or do not achieve what they set out to because of a combination of context, mechanism and outcomes (CMO). The perspective is concerned with finding what works for whom and in what circumstances and then extrapolating a detailed and evolving explanation to other contexts. In Pawson’s words it is predicated on the ‘steady accretion of explanation’ (2006: 176) about a reality which exists independent of the evaluators who are enquiring into it.  Continue reading

Complex, but not quite complex enough

During the last 10-15 years there have been repeated appeals to the complexity sciences to inform evaluative practice in books and journals about evaluation. This partly reflects the increased ambition of many social development and health programmes which are configured with multiple objectives and outcomes and the perceived inadequacy of linear approaches to evaluating them. It could also be understood as a further evolution of the methods vs theories debate  which has led to theory-based approaches becoming much more widely taken up in the evaluative practice. It is now very hard to avoid using a ‘theory of change’ both in programme development and evaluation. What kind of theory informs a theory of change, however?

Although the discussion over paradigms has clearly not gone away, the turn to the complexity sciences as a resource domain for evaluative insight could be seen as another development in producing richer theories better to understand, and make judgements about, complex reality. However, some evaluators are understandably nervous about the challenge of what they perceive as being the more radical implications of assuming that non-linear interactions in social life may be the norm, rather than the exception. In a variety of ways they try to subsume them under traditional evaluative orthodoxies, which is just as one might expect any thought collective to respond. Continue reading