Rediscovering the power of words to generate knowledge - Qualitative methods, robustness and accountability - towards a paradigm shift in MEAL

24 July 2023

This blog post follows the publication of the "Qualitative Data Analysis Toolbox" in April 2023. This toolbox was jointly written by CartONG and Terre des hommes (Tdh), to equip NGOs in their use of qualitative data.

One day, I was talking to a colleague about our project to promote qualitative methods and he said - a little mockingly but nonetheless somewhat seriously: "can words really be analysed?". Since then, I've come to realise that this question is neither trivial nor isolated; that in our field - especially in emergency - there is some scepticism about participative and qualitative approaches: they may be seen as important and nice, but are also seen as difficult to implement. Qualitative research is often seen as subjective, anecdotal, and biased, and is often practised in a less than rigorous way, ultimately only used to provide a few illustrative quotes in a report. It is true, as ALNAP points out, that "While there is sometimes a tendency to see quantitative approaches to evidence generation as ‘hard’ and qualitative approaches as somehow ‘soft’, the use of qualitative approaches should not be an excuse for lack of rigour. Unfortunately, many humanitarian evaluations do not use academically recognized qualitative methods, and fail to meet basic quality standards related to accuracy, representativeness, and relevance."

The temptation to be 'quantitative-only' and the challenges for quality

The MEAL sector is professionalising in a fast-changing environment, with major opportunities and challenges in terms of data culture. There is a growing emphasis on topics such as 'evidence generation; digitisation of data collection and analysis - which, with a good degree of mastery, represent invaluable advantages for teams; globalisation, standardisation of methods and tools, sectoral specialisation.

One of the challenges is that these changes are taking place against a backdrop of contradictory demands, where accountability is sometimes confused with "compliance", and MEAL and program teams are being asked to work much faster, produce more data and feed standardised indicators using model tools often developed at HQ level. The teams have to navigate in the midst of paradoxical injunctions: to do more and better, to address complexity but also in a simpler and more accessible way, to apply adaptive, participative, ethical and accountable management models while responding to ever-more demanding bureaucratic rules and procedures. And this, often with fewer resources, in ever-shortening funding cycles and in a context where the commitments of the Grand Bargain are slow to materialise. In particular, this has led to an increasing tendency towards 'quantification' in MEAL work, reducing the place for qualitative methods and imposing major constraints in terms of data quality. The more widespread use of mobile data collection tools, coupled with the development of standard questionnaires (often 'structured' - or based almost exclusively on closed-ended questions with pre-determined response options) has accentuated the tendency to collect primarily quantitative data. This has also been associated with a tendency to prioritise certain skills when recruiting MEAL staff (mathematics, statistics, IT, information system management) rather than others (social sciences, communication, social work, psychology).

This trend means that in some teams with too little diversity in terms of background education, there may be an inadequate understanding and use of qualitative data collection methods, for example formulating focus group guides that are conducted as "structured group surveys" (where yes/no answers are counted and reported in quantitative terms). In addition, teams are often faced with operational constraints which prevent them from analysing qualitative data systematically, as recommended in the guides - which are often more suited to the academic or research world and are difficult to apply in the humanitarian and development field. Excellent guides exist for the development sector, but they should be adapted and promoted more widely.

Uphill, time pressure and competition for resources often mean that there is a lack of preparation for identifying information needs, preparing an analysis plan and designing a triangulation strategy based on mixed methods. Added to this is the specialisation of sectors, which can lead to a tendency for teams to work in silos, with data collection left to the primary responsibility of the MEAL teams and not enough joint reflection on the most appropriate methodological approaches for the studies.

The potential harmful consequences for individuals

The high demand for data to conduct rapid needs assessments for the purposes of drafting project proposals, or to feed indicators, can lead to surveys being carried out too frequently or on the basis of an unsuitable methodology, generating over-solicitation and "assessment fatigue" among the people surveyed (and within the MEAL teams, program and surveyors too). Ultimately this can also cause unmet expectations and more social desirability bias.

Using structured and standardised data collection tools without contextualising them can lead to unadapted questionnaires:

  • The questionnaires are too long, with questions that are leading, too technical and difficult to understand.
  • Intrusive questions to consult people about difficult, complex and emotional experiences during needs assessments, which are sometimes multisectoral on top of things, can generate discomfort or disenchantment in people, or even real revictimisation.
  • Structured questionnaires leave little room for people affected by crises to express their opinions and experiences in their own way, in their own words, saying what they want to say and keeping quiet about what they don't want to share.

In surveys, it is often the heads of household who are targeted, and often the same people who are excluded (children, people with disabilities, women). This makes it difficult to really understand everyone's situation, with the risk that inequalities cannot be identified and minimised. In some contexts, it is so difficult to reach and consult certain population groups that they have to be approached very carefully, in line with socio-cultural norms; in this case, household surveys and probabilistic sampling methods can definitely produce significant biases.

Finally, exclusive reliance on structured questionnaires can limit our ability to understand trends, to explore the 'why' and 'how' a situation emerges or evolves, to understand gender and diversity dynamics.  This can mean that we are less able to analyse and use data to make informed and effective decisions, readjust our intervention. It may bring us to focus on what seems more important (or what is more important to us) rather than understanding what is more important to communities.

This is not only due to the focus on quantitative data, but is the result of a combination of multiple causes (including the way in which we practice quantitative data collection and analysis). Promoting a more rigorous and transparent use of qualitative methods and giving our teams the means to apply them in an effective and balanced way is one of the ingredients that contributes to improving data culture at Tdh- knowing that we are working on several fronts at the same time to achieve this.

Improving quality and accountability - in particular through (well used) qualitative methods

Qualitative methods are important in humanitarian work because of their relevance when exploring a new situation, measuring a change in behaviour, perception or psychosocial status, and above all because they allow us to be more accountable. If carried out properly, they allow us to:

  • Adopt a stance of listening, openness and respect for the voice and opinions of others. This can lead if the work is well done - to engaging in conversations in a less dominant way by considering the humanitarian affected population power dynamics.
  • Understand children's points of view, opinions and experiences, as they are better suited to their evolving cognitive and emotional capacities, and more relevant to potentially complex subjects.   
  • Improve the dynamics and impact of participatory processes, because qualitative data collection methods are more engaging and often more natural for communities.
  • Generate the rich, nuanced data needed for triangulation and robust analysis (explanatory and interpretative analysis), to understand people's experiences and opinions and to better inform decisionmaking.
  • Finally, and based on my personal experience, a truly rigorous and transparent qualitative analysis process done in a group of peers involves a truly listening ear to what the respondents are saying, how they are saying it, and by extension questioning our own biases as humanitarians (confirmation bias, institutional bias, cultural bias, etc.).

Although qualitative methods can contribute to improve our accountability, they are only part of the solution. It is about going beyond the posture of the humanitarian researcher and technician who focuses on the needs of the population rather than on the adaptation strategies in place, asking questions and analysing the answers. The aim is to generate potentially difficult spaces for discussion, and to encourage the proactive involvement of the community with a humble everyday attitude that fosters trust.

Everyone seems to agree that significant changes must be made, and this is repeated in all the major humanitarian forums (e.g. Humanitarian Network and Partnership Week), but these are slow to materialise globally. As MEAL professionals, we can do our bit by changing our methodological practices, by aiming for quality while shedding technical attitudes and refusing to engage in practices which are not ethical or methodologically sound.

Special thanks to Catherine Dixon, Elise Shea and Maeve de France for their guidance.