How useful are composite indicators in thinking about population ageing? What I have in mind here are not the kinds of composite measure that are used for various aspects of individual ageing like frailty or social isolation or physical fitness. These tell us about the performance or condition of individuals, about how they are ‘getting on’ relative to each other. Composite indicators that aim to tell us about the performance of countries are quite different, and their construction poses a different set of problems.
It was the perceived need for a full discussion of these methodological problems that prompted the OECD to publish a methodological handbook in 2008 – at a time when the number of composite indicators in current use to compare country performance in different policy domains was estimated at a mere 160 (400+ by 2011 and who’s still counting?). Thirteen years on, and there are still lively arguments about the meaningfulness of the scores and rankings produced by many of the composite indicators used for policy monitoring and public communication. Many of them are now well-established features in the world of international policy analysis, but they are not all entirely free of suspicion. Their proliferation though seems inevitable; it goes hand in hand with the increase in the quantity of data ready and available for synthesis and aggregation. And synthesis or aggregation is where some of the main problems are to be found. As one review commented, they are “fraught with normative assumptions in variable selection and weighting” – so much so that media and stakeholders should be warned about the dangers of taking them at face value.
Some indices, such as the Human Development Index, are relatively simple (but not immune to criticism) – in that they require data for only a handful of component indicators – life expectancy, years of schooling and income per capita. Keeping it simple has various advantages. It helps with conceptual transparency. It also makes the task of data collection and synthesis relatively easy, and when the data are regularly collected by national statistical agencies, as with the HDI, they are reliable as well as readily available.
Some composite measures can run into difficulties because they require a lot of data, and for many countries it may be unreliable or unavailable. This is what happened to the Global AgeWatch Index – now reconstituted as Global AgeWatch Insights – which means that there is no longer an index. The index was started in 2012 by HelpAge International as a multidimensional measure of older people’s wellbeing. The aim was to develop a tool “for monitoring governments' progress in eradicating poverty, improving health and advancing the inclusion of older people”. The index assigned a score and a ranking to individual countries. As the press release announcing the change of approach (i.e. no more index) stated, the country ranking tables attracted wide media attention and they were widely used by advocacy groups based in poorly performing countries to start discussions about the situation of older people. One of the important features of composite indicators like this is that they lose their value if they are not regularly updated, which may be challenging as well as expensive, and it wasn’t long before data problems emerged: too many countries (98 in fact) couldn’t be included in the index because of issues with data quality or comparability. Other problems “included the revision of datasets [used] to calculate the index, the year-on-year volatility of subjective data from respondent surveys, and inconsistencies in age cohorts and years of observation for survey findings used to measure the indicators”. The 2017 press release acknowledged that “the index methodology must be more robust to mitigate these limitations”.
The story with the Active Ageing Index is somewhat different. This started off in 2010 as a European venture, a collaboration between the EU and the UN Economic Commission for Europe. From the outset it combined data collected by national statistical agencies with multi-country survey data, especially the European Social Survey and SHARE As the name suggests, the focus for the index was different from that of AgeWatch – not wellbeing, but activity – understood here as active participation in society and the economy plus the assets and resources (like good health) needed to be active. In other words, it combined measures of activity (broadly understood) with measures of capability. The aim was to “produce a robust evidence base on active ageing by measuring to what extent the potential of older people is realized”. As with AgeWatch, the AAI was conceived as a tool for policy monitoring (i.e. how well are these countries doing at enhancing and making use of the potential of older people?). Between 2010 and 2018, which is when the index was last updated, there was a shift of interest away from European cross-country comparisons towards (i) the development of versions of the index for non-European countries, along with specially tailed single country indexes that allow for subnational comparisons and (ii) the development of an active ageing measure that can be applied to individuals rather than countries (which is the approach taken by Age UK’s Index of Wellbeing in Later Life).
A more recent development than either the AgeWatch Index or the AAI is the Ageing Society Index, launched in 2018 by the US-based Research Network on an Ageing Society. The developers were of course aware of both AgeWatch and AAI, and set out to do something different. The Ageing Society Index is presented (and defended) as a measure of countries’ adaptation to societal ageing. The members of the Research Network were asked to identify the characteristics of a successful ageing society. What they decided (in an avowedly normative exercise) was that such a society would cohesive, with minimal tension and competition between generations and major sex or racial subgroups, productive with opportunities for effective engagement within and outside the workforce, healthy, equitable and secure. Hence the five dimensions of the index: productivity and engagement; well-being; equity; cohesion; security. Fine as far as it goes, but consider what is missing – nothing about sustainability e.g. of pensions. Although it would be wrong to say that this should be part of any satisfactory measure of adaptation to societal ageing, the omission does make it clear that there are other ways of trying to measure it.
The developers were also less ambitious in their country coverage – 18 countries only, all of them OECD members, with only 2 non-European countries, the USA and Japan. This may seem a bit arbitrary (where’s Canada or Australia?) until we realise that the index was originally designed to help pinpoint policy sensitive areas where the USA do better. In other words, the selection was based on useful comparator countries for the US.
What stands out is how well the USA performs. Better certainly than it did in the 2015 Global AgeWatch Index – better indeed than Denmark or Austria or Germany or the Netherlands. Norway and Sweden are strong when it comes to social support, but older Americans “are more adaptable to the changes and risks of extended lifespans than their counterparts in other industrial countries”. The selection and weighting of indicators is of course crucial in leading to these results – and the developers did check to see how scores and rankings were affected by other preference weightings. Not much it seems. The UK by comparison does rather badly – in the bottom half of the rankings, mainly because of its poor performance in the security dimension (which includes income and cost of long-term care).
The usefulness, anyway, is clear – it helped the developers make recommendations in what they call ‘policy sensitive’ areas. Though policy solutions may be hard to find when it comes to some of the areas they highlight: big differences between “haves and have nots” or the relatively low percentage of older people who say they lack family, friends or neighbours they can count on.
And the Index ‘did its job’ for me also, by making me look into the reasons for the UK’s relatively poor performance.
About the Authors:
Kenneth Howse is a Senior Research Fellow at the Oxford Institute of Population Ageing. He is also a key member of The Oxford Programme on Fertility, Education and the Environment (OxFEE).
Opinions of the blogger is their own and not endorsed by the Institute
Comments Welcome: We welcome your comments on this or any of the Institute's blog posts. Please feel free to email comments to be posted on your behalf to administrator@ageing.ox.ac.uk or use the Disqus facility linked below.
