Whitehall Monitor 2024: Methodology

We set out below our methodology behind various pieces of analysis that have contributed to this year’s Whitehall Monitor. For questions on the below, or regarding any of our findings, please get in touch via email at enquiries@instituteforgovernment.org.uk.

We refer to figures following the numbering system used in the PDF version of the report.

Open access data 

Our charts draw principally from publicly available government data, which in most cases we have further processed and analysed. We have collated the final data inputs for each of our charts and published them on our website alongside this report. 

If you have any questions about the analysis of this data, please get in touch via email at the address listed above. 

Throughout the report 

Defining departments 

Where possible, we categorise bodies into ‘departmental groups’ according to where ministerial responsibility lies, even when these are reported under a separate departmental heading in government data. For example, we group Ofsted with the DfE departmental group, and not as a separate department, as it is reported in the original ONS Public Sector Employment data. 

In such cases where source data reports organisations as independent from core departments, we have identified the departmental group to which those organisations belong by using the ‘sponsor department’ identified by the most recent (2020) public bodies report published by the Cabinet Office, or by government statements – such as on machinery of government changes – to parliament. 

Unless otherwise indicated, the figures used throughout the report are for departmental groups, and therefore include the civil servants who work in the ‘core’ departments as well as the agencies and non-ministerial departments they oversee.

A table listing the departments and their associated organisations is found overleaf. We have not included organisations that no longer exist – for example, because they have been merged with other bodies or renamed. However, historic organisations are counted in our figures for change over time, and details of those used in our analysis is available upon request. 

Machinery of government (MoG) changes

We reflect the new departmental structures created following February 2023’s MoG changes only where it is possible to do so using publicly available data. For example, the latest size of each departmental group reflects the existence of DBT, DSIT and DESNZ, whereas charts showing change over multiple years exclude the new departments and reflect, where possible, the final size of their predecessors. In each case, we explain why we have used which version of departmental structures in the figure sources.

List of departments and associated organisations

Defining civil servants 

We define civil servants as politically impartial, appointed officials of the UK Home Civil Service, which supports the work of the UK’s central government departments. This includes agencies that employ civil servants such as executive agencies, non-ministerial departments and some non-departmental public bodies.

Our definition includes staff of the three Whitehall-based territorial offices that manage the UK’s relationship with Scotland, Wales and Northern Ireland. And we include the civil servants employed by the devolved governments of Wales and Scotland, but not the staff of the Northern Ireland Civil Service, which is administratively distinct.

In this way, civil servants are defined more narrowly than public sector workers: police, teachers, NHS staff, members of the armed forces or local government officers are not counted as civil servants. Nor do we include the UK’s diplomatic service in our analysis since it, too, is administratively separate from the UK Home Civil Service. Civil service grades Broadly, there are five civil service job grades: 

1. Administrative officer/administrative assistant (AO/AA) – the most junior civil service grade. These roles tend to comprise administrative support and operational delivery roles, such as prison officers and caterers. 
2. Executive officer (EO) – civil servants in this grade offer business and policy support and include roles such as executive assistants, finance, HR, IT and communications specialists. 
3. Senior executive officer/higher executive officer (SEO/HEO) – includes policy officers and officials with specific policy responsibilities. 
4. Grades 6 and 7 – these civil servants tend to be experienced officials with significant policy responsibilities. 
5. Senior civil service (SCS) – the most senior grade of the civil service made up of the senior management team. Generally, directors are ultimately responsible for the policy work of their team and directors general oversee directors and work closely with the department’s ministers. Each department also has a permanent secretary as part of the SCS, who supports the minister at the head of the department, acts as the accounting officer and is responsible for the day-to-day running of the department.

Inflation/real-terms figures 

For government spending information that spans multiple years, we use the GDP deflator to present numbers in consistent prices. The GDP deflator is a measure of economy-wide inflation and so is appropriate for considering changes in government spending. We use the GDP deflator published alongside the November 2023 autumn statement. For information on pay, we use the Consumer Price Index (CPI) to present numbers in consistent prices as this is the relevant measure to understand how much pay packets are worth to households.

Part 1 

Size and turnover: Civil service staff numbers

To analyse civil service staff numbers, we use Table 9 from the ONS’s quarterly Public Sector Employment series, which contains staff numbers (headcount and full-time equivalent, FTE) in all public organisations that employ civil servants. Unless stated otherwise, we use FTE figures, which count part-time staff according to the time they work (e.g. a person working two days a week as 0.4 FTE); this is more accurate than headcount, which does not distinguish between full-time and part-time employees.

Staff numbers are generally reported to the nearest 10. The ONS and the Cabinet Office (for its Civil Service Statistics) report staff numbers lower than five as “..”. We have rounded any staff numbers lower than five to three.

Our figures exclude temporary census field staff, and for operational security reasons staff from Central Government Security (formerly Security and Intelligence Services) have been excluded from civil service statistics published since Q2 2016. We adjust for this by manually excluding Central Government Security staff from our datasets before this date, too.

To analyse other characteristics of the civil service, such as age, pay, profession, location and ethnicity, we use the Cabinet Office’s annual Civil Service Statistics release. To analyse the engagement and experience of civil servants, we use the Cabinet Office’s annual Civil Service People Survey.

Our analysis of change over time for each departmental group takes account of the reclassification of staff between organsiations, as relevant. We do this to more precisely portray the changing size of departments, rather than changes to the way in which officials are formally counted and reported, where it is more accurate to do so. Reclassificaitons are usually noted by the ONS in footnotes to the data tables. 

The Foreign and Commonwealth Office (FCO) and the Department for International Development (DfID) were merged to form the Foreign, Commonwealth and Development Office (FCDO) in Q3 2020. Our calculated rates of change and size of FCDO over time are assumed to be equivalent to the sum of the figures for the two component departments for quarters prior to Q3 2020; the same applies for BEIS before the merger of BIS and DECC. We do not do the same for the new DBT, DSIT or DESNZ because those MoG changes were not the result of simple mergers between departments.

Professions

Our analysis of the civil service professions uses Table 8A in the Cabinet Office’s annual Civil Service Statistics. All civil servants belong to a profession. The professions in the civil service have changed over time, and the current professions are listed below. For all of our analysis, we group very similar professions into ‘profession groups’, and it is these groups that we use and refer to as ‘professions’ throughout our analysis.

For some analysis we also group professions into three categories. The operational delivery profession occupies its own category. We group all other professions into ‘cross-departmental specialisms’ (professions such as policy, which span multiple departments), and ‘departmental specialisms’ (professions such tax, which are overwhelmingly concentrated in one department).

The list of professions below reflects how we group them into cross-departmental specialisms and departmental specialisms. For the years in which they existed, now-defunct professions are also grouped into the categories below. Note that operational delivery is not included in the list below.

Cross-departmental specialisms (Institute parent category)

• Analytics (Institute parent category)
  • Economics
  • Operational research
  • Social research
  • Statistics
• Communications
• Digital, data and technology
• Finance (Institute parent category)
  • Corporate finance
  • Finance
• Human resources
• Internal audit
• Policy
• Project delivery
• Property
• Knowledge and information management
• Legal

Departmental specialism (Institute parent category)

• Counter fraud
• Geography
• Inspector of Education and Training
• Intelligence analysis
• International trade
• Medicine
• Planning (Institute parent category)
  • Planning
  • Planning inspectors
• Psychology
• Science and engineering
• Security
• Tax
• Veterinarian

In Figure 1.10, there are eight departments shown. The four departments on the top row of the figure (MoJ, HO, BEIS and MoD) are the four that experienced the greatest absolute growth in staff numbers (FTE) between Q2 2016 and Q3 2023. The four departments shown on the bottom row are the four that experienced the greatest proportional growth in staff numbers over the same period (DIT, DCMS, DfE and Defra).

For each department, the five profession groups highlighted are those that experienced the greatest growth in absolute numbers over the period. If this would have included either of the ‘not reported’ or ‘other’ categories, we have disregarded them. We have also disregarded a very small number of cases where the 2016 figure was implausibly low (or 0), resulting in an artificially high change. In the cases of MoJ and DIT, where data is shown from 2017, we have analysed the five profession groups with the greatest growth between 2017 and 2023.

It should be noted that we have used separate datasets to identify the departments to include in this chart (the ONS’s quarterly Public Sector Employment data) and to analyse the change in professions over time (the Cabinet Office’s Civil Service Statistics). These two government datasets give slightly different numbers for the total number of civil servants in a department at any given time. 

As we discuss in the text, the civil service’s professions data has historically been inconsistent. We have avoided correcting for this, except for in one instance. DWP did not report the professions of over 95% of its staff in 2018, 2020 and 2021. Given the size of the department, this would significantly skew the trend data, particularly for the operational delivery profession. For this reason, we interpolated DWP’s professions numbers for these years based on its reported numbers in 2017, 2019 and 2022. 

Throughout our analysis, we combine the figures stated in the source data under “not reported” and “other”. It should be noted that the civil service relies on self-reporting of professions by civil servants for this data. 

Turnover of civil servants 

Data on civil service staff turnover is derived from the annual Cabinet Office Civil Service Statistics dataset (Tables 20, 42 and 43). We use headcount rather than fulltime equivalent staff for all staff turnover calculations. Figures relate to the core department and do not include agencies within the departmental group. 

External staff turnover is calculated as the number of civil servants who left the civil service entirely over the course of a given year, as a percentage of the average civil service headcount during that period. Average civil service headcount is calculated as the mean of civil service headcount at the beginning and end of the interval (for instance, headcount in March 2021 and March 2022 for the period 2021–22). We use average headcount to account for the fact that the number of civil servants changes over the course of the year. 

Internal staff turnover is calculated as the number of civil servants who transferred to another department over the course of a given year, as a percentage of the average civil service headcount during that period.

Total staff turnover is calculated as the number of civil servants who either left the civil service entirely or transferred to another department over the course of a given year, as a percentage of the average civil service headcount during that period. This is an underestimate of real internal turnover in the civil service because it does not include civil servants who transferred to another role within the same department. Unfortunately, data on staff transfers within departments is not publicly available. 

Structure and location 

Public bodies 

Throughout this report we use ‘public bodies’, which includes public corporations, unclassified bodies and parliamentary bodies, to encompass the widest possible group of organisations. However, in a number of charts we are relying on Cabinet Office data that is limited to a narrower subset, ‘arm’s-length bodies’. This category only encompasses non-departmental public bodies, executive agencies and nonministerial departments. 

The figures on the number of bodies that have been reviewed as part of the public body review programme (Figure 1.15) were provided by the Cabinet Office, and are accurate as of January 2024. 

To calculate the number of arm’s-length bodies in 2022, we looked at the last public update to the Cabinet Office dataset in 2020, and adjusted for public announcements on the establishment or abolition of bodies according to GOV.UK, the body’s own official website, or the National Archives. 

Figures for the number of arm’s-length bodies in 2023 (Figure. 1.16) were provided directly by the Cabinet Office and are accurate as of September 2023. These figures do not include the Supreme Court as a non-ministerial department and, although it is unclear whether this is a substantive reclassification, we have reflected the absence of the Supreme Court in our data. 

Location of civil servants 

In December 2023, the Cabinet Office published guidance on the government’s Places for Growth initiative, which clarified that the government’s relocation targets excluded Scottish and Welsh government civil servants. 

Because of the way the Civil Service Statistics are reported, some data we analyse and charts we produce are able to exclude civil servants who work for specific departments; others cannot. Where possible to do so, we exclude Scottish and Welsh government civil servants from our charts on the location of staff, to reflect the government’s methodology. This means that Figure 1.17, Figure 1.18 and Figure 1.19 all exclude Scottish and Welsh government civil servants, while Figure 1.20 and Figure 1.21 include those working for the Scottish and Welsh governments. 

All charts exclude civil servants based overseas.

Figure 1.19 displays a forecast of the percentage of senior civil servants based in London from 2024 to 2030. This forecast is a simple linear extrapolation of data supplied from 2021 Q1 and 2023 Q2. 

Budgets and major projects 

Spend by public bodies 

Our analysis uses the Online System for Central Accounting and Reporting (OSCAR) II annual data taken from three releases (2022 for 2017/18–2021/22, 2017 for 2015/16 and 2016/17 and 2015 for 2010/11–2014/15) to assemble financial returns from 2010 through to 2022. For the 2023 figure, due to delays in the publication of finalised full-year data, we have used an interim ‘in-year’ dataset for the 2022/23 financial year. 

We used final outturn figures and excluded so-called ‘non-budget’ and ‘non-voted’ expenses, as these are more likely to include double-counted figures or revaluations. This means that the data does not include spending granted directly to the devolved administrations or local authorities. 

This analysis is necessarily imperfect, and some spending by public bodies (particularly executive agencies that are closely integrated into their departments) is categorised as departmental spend. For instance, the OSCAR dataset suggests that most spending by the Ministry of Justice is directly spent by the department. This is incorrect, as the MoJ’s annual reports consistently show most money is spent by its arm’s-length bodies, such as HM Courts and Tribunals Service and HM Prison and Probation Service. Spending by non-ministerial departments such as the Food Standards Agency is also categorised as departmental, despite them being a type of ALB. As a result, our analysis underestimates the proportion of spending by ALBs, and overestimates the proportion of spending by departments.

This dataset also underestimates the total scale of spending by public bodies as it does not consistently include public corporations, or includes only their net rather than gross spending. 

Major projects 

Analysis of the government major projects portfolio (GMPP) comes from data collected and published by the Infrastructure and Projects Authority (IPA). The latest IPA dataset (2022–23) contains data on whole-life costings of individual projects unless where exempt, as in accordance with the government’s transparency policy, on grounds such as national security or commercial sensitivity. However, total whole-life costings of projects delivered by individual departments include the costs of exempt projects. Our analysis for Figure 1.27 includes costings of exempt projects while Figures 1.28, 1.29 and 1.30 do not include projects with exempt costings or projects that are exempt from providing a risk rating.

Whole-life costs of the portfolio have been adjusted to present numbers in consistent prices. The GDP deflator is a measure of economy-wide inflation and so is appropriate for considering changes in government spending. We use the GDP deflator published alongside the November 2023 autumn statement, adjusting the figure for 2020/21 to be the midpoint of the 2019/20 and 2021/22 values as the pandemic affected the deflator’s measurement. Our calculations use the total value of individual projects published in the IPA’s data which may vary from the headlines in the IPA annual report due to rounding. 

Real-terms changes to planned departmental administrative spending were taken from Public Expenditure Statistical Analyses, 2023 and adjusted using the GDP deflator.

Spending on consultants and temporary labour 

The data in this section has been taken from departments’ annual reports and accounts, which display consultancy spend and contingent or temporary labour for each financial year. DHSC had not released its 2022/23 annual reports and accounts at the time of writing, so Figures 1.31 and 1.32 do not display DHSC’s consultancy and temporary labour spend for 2022/23. The increase in total consultancy spend from 2018/19 to 2022/23, therefore, excludes DHSC. We have not analysed the territorial offices or HMRC. 

Other data points are unavailable because departments have not reported consultancy or temporary labour spend in their annual reports and accounts. BEIS and DCMS have not reported temporary labour spend for any of the years between 2018/19 and 2022/23. Defra has not reported its temporary labour spend for 2020/21, which we have estimated using an average of Defra’s 2019/20 and 2021/22 spend. For years prior to the merger of FCO and DfID, FCDO’s spend is plotted as the sum of DfID and FCO. Some departments do not report core departmental spend separately, and instead report core and executive agency spending. These departments include the Cabinet Office, MoJ and HMT. BEIS core consultancy spending was calculated by subtracting ALB and executive agency spending from departmental group spending. 

Morale, pay and industrial relations 

Civil service pay 

Figure 1.39 uses the median permanent secretary and director general pay as per the Senior Salaries Review Body reports, 2014-23. The SSRB gives the median permanent secretary salary as a range between 2014–17; we use the mean of the range to represent the median salary. The data is presented in real terms and is adjusted by 2023 prices, using the OBR’s November 2023 estimates. 

Figure 1.40 was created using data from the annual reports of the Bank of England and Financial Conduct Authority (FCA) for 2022/23. HM Treasury salary information was sourced from its ‘organogram of staff roles and salaries’, published in the same financial year (on 29 December 2022). The values in the chart represent the mean of all reported salaries at each respective grade. Where salaries were provided as a range rather than a point (as with the Treasury and some FCA salaries), salaries were assumed to be the midpoint of the range. 

Civil service paybill scenarios 

We calculated estimates for the actual current civil service paybill and for two hypothetical scenarios. 

To estimate the actual current civil service paybill we combined statistics on the number of civil servants at each grade (FTE) and the mean salary for each grade. 

In the first scenario, where the civil service in 2023 has the same grade composition as in 2010, we estimated the number of civil servants at each grade by multiplying (i) the proportion of civil servants at each respective grade in 2010 and (ii) total headcount in 2023. The paybill for this scenario was then estimated by multiplying these estimates for headcount by grade with the mean salary by grade in 2023. 

In the second scenario the civil service in 2023 again has the same grade composition as in 2010 but wages rise in line with the private sector. Mean civil service salary by grade is grown in line with wage growth in the whole private sector from 2010 (specifically, the private sector gross annual earnings measure in Office for National Statistics, Annual Survey of Hours and Earnings, 2010–23. Again, the paybill was estimated by multiplying these estimates with actual headcount by grade for 2023. 

Those with unreported grades were excluded from all calculations due to limited data availability, meaning that the estimated total paybill is a slight underestimate in each scenario. 

For Figure 1.23, we take staff costs in administration day-to-day spending from HM Treasury, Public Expenditure Statistical Analyses, 2023, and calculate the total civil service paybill as the number of civil servants multiplied by mean wage. We calculate the paybill for a financial year by averaging the two calendar year values, and to forecast the total paybill we assume it exhibits the same percentage change as staff costs in administration spending. We also show staff costs in administration RDEL, which is based on departments’ reported planned spending as set out in HM Treasury, Public Sector Expenditure Analysis, July 2023. How departments spend their administration budgets is subject to change, so these figures are only provisional. 

We deflate figures using the GDP deflator from the autumn statement 2023. 

Relationship between low pay and overall morale 

We calculated correlation coefficients between the employee engagement index of all civil servants and the nine people survey theme scores for the period 2010–2022, using both median and mean scores. Scores for the ‘pay and benefits’ theme are much less correlated with overall engagement than any of the other themes. 

We note that ‘pay and benefits’ theme scores are, however, considerably more correlated with scores for the questions ‘I want to stay working for [my organisation]’/’I want to leave [my organisation]’ (positively and negatively, respectively) than any other theme scores over the same time frame. 

Diversity 

Female, minority ethnic and disabled staff 

For the proportion of female staff in the civil service, the population benchmark is taken from the ONS’s Labour Force Survey (Table A02). The number of economically active women aged 16–64 is averaged over the most recent four quarters, and divided by figures for the whole economically active population, treated in the same way. The latest quarter available in this dataset is May–July 2023. 

The same methodology is used to reach the population benchmark for minority ethnic representation – using Table A09 – and for disabled representation, using Table A08. In both of these cases, the latest quarter available is April–June 2023.

Sexual orientation 

For Figure 1.44, the population benchmark is calculated from the ONS’s Sexual Orientation, UK dataset, based on the Annual Population Survey. The latest figure available, and the one used for the benchmark, is for 2022. 

Socio-economic background 

In the absence of robust workforce statistics, our analysis of civil servants’ socioeconomic background uses the annual Civil Service People Survey. The Cabinet Office publishes the results of the people survey by socio-economic background. This data breaks down responses by officials in the national statistics socio-economic classification categories ‘never worked’, ‘routine’, ‘intermediate’ and ‘high’, based on a series of questions about the main income earner in their household when they were 14 years old. Under each of these categories, the data states the number of survey responses received by officials in each grade. 

We have combined the number of survey responses received under the ‘never worked’ and ‘routine’ categories into the ‘low’ socio-economic background category, in line with the terms used by the Social Mobility Commission, and used the resulting data to estimate the socio-economic background of each grade. 

Educational background of permanent secretaries and directors general 

We conducted a desk research exercise in August 2023 to gather information on the educational background of all permanent secretaries and directors general. Information was gathered from a range of public sources, including GOV.UK and LinkedIn. 

Fast Stream diversity 

The data on the Civil Service Fast Stream diversity was taken from written parliamentary questions UIN 174514, 174515, 174516 and 174517. The benchmark for the economically active working-age UK population that is from a minority ethnic background is calculated from the ONS ‘A09: Labour market status by ethnicity’ dataset. It is the economically active population aged 16–64 that self-identified as of ‘Mixed’, ‘Indian’, ‘Pakistani’, ‘Bangladeshi’, ‘Chinese’, ‘Black/African/ Caribbean’ or ‘Other’ ethnicity, as a percentage of the total economically active population aged 16–64. To adjust for seasonal variations in employment data, we calculate the average for each of these benchmarks over the last four quarters up to 31 March 2022 (to match the publication date of the 2023 Civil Service Statistics dataset, which is the source of our data on civil service diversity). 

The population benchmark for those who are eligible for free school meals is taken from the DfE’s ‘Schools, pupils and their characteristics’ 2021/22. It is calculated as the percentage of pupils eligible for free school meals at January 2022. It includes all state-funded primary, secondary and special schools, non-maintained special schools and state-funded alternative provision schools. 

The population benchmark for those who attended state school is also taken from the DfE dataset ‘Schools, pupils and their characteristics’ 2021/22. It is calculated as the percentage of 11–15-year-olds attending a state-funded school.

Part 2

Ministerial resignations

Our tally of resignations includes only those outside of reshuffles; those during reshuffles are often difficult to discern from sackings (although we do include Jonathan Aitken’s resignation in 1995, when he resigned to fight a libel action).

We do not count those who announced in advance that they would step down at the next planned reshuffle or those who were sacked. This means that our numbers may differ very slightly from others’.