9.3 Administrative sources

9.3 Administrative sources#

The United Nations’ Fundamental Principles of Official Statistics (UNFPOS) state that “Data for statistical purposes may be drawn from all types of sources, be they statistical surveys or administrative records.”

The term administrative data refers here to data collected by a government ministry, department or agency primarily for administrative (not research or statistical) purposes. These administrative purposes are related to the corresponding executive or lawful functions such as authorisations, registrations, permits, payments, sanctions, control etc. Administrative data may include both data in administrative registers and data in other administrative sources.

Using administrative data for statistical purposes is not a recent phenomenon – there are examples of compiling statistics from data on the number of births and deaths dating from the early 17th century. However, it has become increasingly widespread during the last two decades as progress in technology and increased computing capacity have permitted statistical agencies to overcome many of the limitations previously associated with the processing of large administrative datasets. This together with advances in data linking methods has provided NSOs with the opportunity to make better use of administrative data in the production of official statistics, both in replacing existing data collection methods, in supplementing statistical survey data and in the creation of new statistical products.

In 2011, the UNECE published a handbook entitled Using Administrative and Secondary Sources for Official Statistics – A Handbook of Principles and Practices (🔗). It gives an overview of administrative data sources and data collection issues, and it is used as a basis for this section. The rapid development during the last decade has increased the common knowledge of administrative sources among statisticians. Further information on the latest experience can be found on NSOs’ and international statistical organizations’ websites, joint projects and various statistical conferences.

The use of administrative data and the production of register-based statistics are covered in Chapter 13.2 — Register-based statistics.

!

Links to guidelines, best practices and examples:

Statistics Bhutan Guideline for Assessing Quality of Administrative Data for Producing Official Statistics (🔗)

9.3.1 Types of administrative data#

Public authorities in all countries collect a large amount of data as a part of their ongoing operations. Administrative sources cover a wide range of activities such as collecting taxes, social security and health care, employment and unemployment policies, and registration systems for civil events (births, deaths, marriages etc.), businesses, properties and vehicles. This administrative data has become increasingly available to NSOs, and in consequence, there has been a significant impact on how data are collected, and official statistics are compiled.

Administrative data come from many different sources depending on the structure of the public sector in the country. The most common way to utilise administrative data in statistics production is to combine data from different administrative sources with or without data from statistical surveys. In this data integration process, some administrative data sets have a principal role and are widely used by many NSOs. Below, administrative data sources are grouped according to their coverage and content to illustrate their importance and usefulness for the production of official statistics.

Large and complex administrative data systems contain both registration data and data of transactions. Typical examples are data from population and business registration systems, social security and health care systems, taxation systems, customs systems and building and property registration systems. Usually, these data are complete, exhaustive by nature covering all citizens, enterprises, or the entire stock of properties and buildings. Basic information in the administrative registers in this group may contain identification code, name, address, registration date and other identification and classification information. The authorities responsible for these registers have created systems for continuous (often on-line) updating of the basic content of registers. The authorities keeping these large administrative registers may also have other data systems to carry out their main administrative functions. For example, tax authorities may have basic registers of people and enterprises liable to taxation and separate systems for personal and enterprises’ income tax, value-added tax and property tax. Data from the producers of these large data systems are usually used as source material in many different statistical systems varying from population and social statistics to business, economic, and environment statistics.
Administrative data with a more specific scope include typically registers and data from transport and traffic authorities, justice and electoral authorities as well as education and school systems. Often this data is an important, or only, source, e.g., in production of transport statistics, statistics on justice and crime, statistics on general elections and statistics on education. Administrative data from this group is also used as an additional source for business and economic statistics as well as population and social statistics. This group also belongs to a vast amount of specific administrative data useful in compilation of environment, energy and waste statistics, and administrative data for public sector activities and finances. Some administrative data have the scope and content which may be difficult to obtain through surveys.

9.3.2 Advantages in collecting administrative data#

The data collection arrangements and the potential advantages vary from country to country depending on the national circumstance and on the extent to which an NSO plans to use administrative registers and data. The most commonly discussed advantages of using administrative sources in statistics production are presented below.

Cost-effectiveness: an important advantage in using administrative data to compile statistics is that the cost of data collection is relatively small in relation to the costs that are incurred in conducting censuses, creating and maintaining statistical base registers and conducting direct surveys as essentially the costs of data collection have already been covered by the administrative process itself. Use of administrative data, however, is not cost-free to NSOs. There may be a need to invest in ICT- and production systems, coordination mechanisms, and statistical methods and new competences. The NSO may also be obliged to pay data transfer and transmission costs to administrations even though data itself is free of charge.
Reduced response burden: respondents may react negatively to a survey if they feel they have already provided similar information to public authorities. The increasing reluctance of both firms and individuals to participate in statistical surveys, not least surveys of small and medium-sized businesses, may threaten the quality of statistics. Use of administrative data leads to reduced response burden and at the same time may solve the problem of growing non-response rates and increasing the quality of incoming source data and statistics.
Timeliness and frequency: when governments are developing their ICT-systems and moving towards on-line data collection and mobile services, administrative registers are continuously updated, and administrative data are available relatively quickly. Consequently, statistics based on administrative data can be prepared more quickly and can be released earlier than from data collected through statistical censuses and surveys. Due to the continuous on-line or other updating systems of administrative registers, administrative data may also increase the frequency with which statistics are traditionally compiled and published.
Coverage and completeness: administrative sources often give complete coverage of the target population, whereas sample surveys often directly cover only a relatively small proportion of it. Use of administrative data sources may diminish or eliminate the errors due to non-response and other typical errors of sample surveys. Administrative registers and data offer good sources for creating and maintaining statistical base registers, better coverage of target populations for sample surveys, and can make statistics more accurate. Use of administrative data instead of surveys can also improve regional and small-area data and more detailed information.
Relevance: an NSO can improve its ability to respond to new data needs and increase the relevance of statistics by seeking and taking into production new administrative data sources. Administrative registers and data combined with survey data can improve the flexibility of the NSO to react quickly to new statistics demands. Administrative data can play an important role, e.g., in filling data gaps needed to measure progress on the Sustainable Development Goals (SDGs) of the 2030 Agenda for Sustainable Development that cannot be satisfied through traditional methods alone.

9.3.3 Challenges and issues in collecting administrative data#

Although there are many good reasons to use administrative data, several challenges and problems are associated with data collection. These relate to the data collection arrangements, the preparedness of the data providers and the NSO, and the quality of the data itself. The national statistical legislation usually gives a sound basis to NSOs to collect and process statistical surveys and publish statistics. As to the acquisition of administrative data, legislation is often inadequate and may cause many problems or even prevent acquisition.

The challenges are related to the administrative culture and traditions. Administrative authorities may be reluctant to grant access to “their data” for statistical purposes. This may happen even in a country with appropriate statistical legislation in place. Administrative authorities may have a strong proprietorship regarding administrative registers and data for which they are responsible. In some cases, legal restrictions or confidentiality provisions may be imposed that restrict access to administrative data. In some countries, the desire and proposals of the NSO for minor changes to the administrative data collection arrangements that would improve the usefulness of the data may be impossible to implement in practice.

To meet the common challenges and be better prepared for data acquisition from administrative sources Chapter 10.3.4 — Requirements for access to and acquisition of administrative data summarises the basic requirements based on the current knowledge and practice of the countries.

Appropriate data collection arrangements make it easier to assess and respond to the problems relating to the quality of administrative data and the use of these data in statistics production. In each country, the quality of administrative data varies across the sources. Before collecting and using any administrative data set, it is necessary to conduct a thorough quality assessment and plan corrective measures. Often this quality assessment requires that the NSO get access to record-level administrative data. Poor quality administrative data should not be used in the production of statistics.

Box 5: Collaborative on the use of Administrative Data

Administrative data collected by governments and service providers in the course of their day-to-day business is an increasingly important source for the production of official statistics. There is an urgent need to strengthen the capacity of national statistical systems to leverage administrative data for statistical purposes to fill gaps in the data available to policy and decision-makers to monitor progress and implement the 2030 Agenda and address challenges such as the COVID-19 pandemic.

Responding to this need for increased use of administrative data for statistical purposes, the United Nations Statistics Division (🔗) and the Global Partnership for Sustainable Development Data (🔗) have jointly convened a multi-stakeholder collaborative of more than 25 countries and regional and international agencies.

A key objective of the Collaborative is to strengthen the capacity of countries to use administrative data sources for statistical purposes across thematic areas and throughout the business process. Members of the Collaborative are working together in a coherent and cross-cutting manner to address both urgent and longer-term needs around the access and use of administrative data for statistical purposes, building on advances +made in various sectors and by different partners. Next to building on these advances, the Collaborative aims to fill knowledge gaps and develop tools and other learning resources.

Rooted in the Collaborative’s Guiding Principles (🔗), the initiative aims to provide a platform for sharing resources, tools, best practices and experiences. It also contributes to raising awareness among all members of national statistical systems about the benefits of sharing and combining administrative sources to enhance the quality, timeliness, coverage, and level of disaggregation of statistical data.

Among the tangible outputs developed within the last year are an Inventory of Resources on Administrative Data (🔗), four (4) expert clinics on labour statistics (🔗), quality (🔗), interoperability (🔗) and cooperation with data owners (🔗), a self-assessment tool for statistical legal frameworks and a guideline for Memorandums of Understanding. Developing e-learning resources for ministries, departments and agencies that collect and own administrative data is in progress.

Below are listed often mentioned quality problems, doubts and perceived attitudes towards collecting and using administrative data in statistics production together with some solutions used in current practice.

Differences in units, concepts and definitions of variables: units, concepts and definitions of variables used in administrative data often differ from the statistical ones. This is especially a big problem in the field of business and economic statistics. There are many case-by-case studies on how these problems can be solved made by NSOs and international organizations. Sometimes good proxies can be used; sometimes, additional survey data are needed. Differences and possible correction methods should be published together with published statistics.
Differences in classifications: often the classifications used in administrative sources differ from the statistical standards. This may confuse users of statistics. By statistical law, the NSO decides and confirms classifications to be used in official statistics. In some countries, the national classification of economic activities, based on the international statistical standard and confirmed by the NSO, is given an official national standard to be also used by all administrative data producers. This may not entirely prevent wrong coding as the codes used for units may be erroneous. Creating and using link tables is an often-used method to adjust for differences in classifications even though it adds to the NSO’s workload.
Inadequate data quality: data included in administrative registers and data files may be incomplete or inaccurate, or data can be missing. In some cases, the staff of an authority responsible for the administrative data may not be properly trained, or methodology for recording appropriate information may be flawed. The corrective measures may include good quality control systems and correction methods of incoming data, training of staff and discussions of quality issues with the administrative authority.
Lack of statistical know-how: the authorities providing administrative data do not generally have the same levels of statistical knowledge and capacities as an NSO. As a result, inconsistent data may occur. For example, some of the data may not be accurately or carefully recorded and could be incomplete or inaccurate for items of secondary interest or value to the primary administrative purposes. In these cases, close co-operation between the NSO and the producers of data in question and training activities of the NSO may help.
Need for additional data checking: depending on the source, administrative data require comprehensive validation checks. Furthermore, a reconciliation process between different data sources may be necessary. This can involve considerable effort and costs to the NSO.

9.3.4 Requirements for access to and acquisition of administrative data#

Challenges relating to data collection arrangements are related to the general administrative and legal structure of the country. Access to administrative data, the preconditions for data collection and the possibilities to use these data vary from country to country. Various frameworks are needed to facilitate the access of the NSO to administrative data. These frameworks include legal, policy, organizational and technical dimensions.

All these dimensions should be considered when the NSO is planning data collection from administrative sources. Short descriptions of the main requirements for successful data collection are briefly summarised below.

Legal frameworks

Use of administrative data in the compilation of official statistics requires a firm legal basis. A mandate ensuring the NSO access to administrative data is seen as one of the key issues in modernising statistical legislation. Depending on the national legislation structure, it is important to note that the mandate of the NSO to collect data from administrative sources may also require changes in other legislation. Chapter 3.4 — Legislative frameworks gives further guidance on sound statistical legislation. The principal issues are highlighted in the following paragraphs.

The legislation must provide the NSO with the right to acquire the administrative data free of charge and ensure that the NSO provides the appropriate level of data safeguards such as data protection and confidentiality.

This is vital in gaining and maintaining the public approval of the NSO’s ability to properly manage administrative data and for the public to have confidence in the official statistics disseminated.

Simply to be allowed to collect data from an administrative source, or to have access to them, is a rather vague/rough provision and needs more precise interpretation. Ideally, the NSO should have access to record-level administrative data and relevant metadata, it should have the right to combine administrative data with survey and other data, and it should be informed about the administrative data sources and changes in them in due time.

Access and use of administrative data in statistics production have raised important questions regarding whether this increases the probability of violating privacy or breaching business secrets or breaking statistical confidentiality rules. Therefore, it is important to synchronize other legislation with the statistics law. Legislation must ensure the implementation of watertight data security, protection of privacy and statistical confidentiality. These rules should be made known to the administrative authorities and the society at large. It should also be guaranteed that when the NSO acquires administrative data, it is handled strictly according to the statistical confidentiality principles. Data flows go only in one direction, i.e., from administrative authority to the NSO.

Public approval for the government use of administrative data

Even if an NSO is an autonomous organization, it is at the same time part of the governmental sector.

The NSO cannot operate alone in its efforts to use administrative data in statistics production. Instead, it needs support from other authorities, political masters, the general public, and the society.

Public opinion about sharing data between different government departments varies from country to country. It may be in favour due to the increasing efficiency of administration, and it may be even hostile in fear of diminished privacy and ‘Big Brother Syndrome’. It is important to make clear that the use of administrative data for statistical purposes does not mean sharing data within the governmental sector. The NSOs should be active in explaining the protective measures taken in accordance with the confidentiality principles and the law. Open discussion and debate, explaining the rationale and benefits of using administrative data in statistics production should be a key principle of the NSO.

One of the most important users of official statistics is the government sector with its growing needs for new and relevant, high-quality statistics. At the same time in many countries, there is growing pressure for budget cuts. In this situation, the governments may be willing to support NSO’s efforts to develop sound infrastructure, lower the costs of statistics and increase conditions to meet the new data needs. In return, NSOs should advocate for getting access to administrative data for statistical purposes as it reduces the costs of the production of statistics without altering the quality of the final products.

Technical frameworks

Technical frameworks refer to the mechanisms by which data and metadata are transferred. The data transfer mechanisms can vary between countries and sometimes between administrative authorities of the same country. This depends mainly on the maturity and sophistication of the ICT-systems of the data providers, on one side, and the NSO on the other side. Data providers may send the data files to the NSO, or the NSO may extract the data directly from the administrative source database. Machine-to-machine access for Big Data sets and on-line access are becoming more common. Broader open data initiatives at the government level may further support direct access in the future.

From a technical perspective, it is becoming easier for NSOs to access data from administrative sources when governmental organizations are developing IT-systems and digitalizing their processes.

Further, providers of administrative data may also use international statistical standards for data and metadata transmission. In addition, the transmission of data to the NSO may benefit from common national standards applied across the entire government sector.

Cooperation with administrative authorities that are data providers

The use of administrative data for the production of official statistics creates a strong link, or sometimes even interdependency, between the NSO and administrative data providers. Therefore, the NSO needs to follow up on changes in the administrative structures, in legislation relating to administrative data and any e-government initiatives.

Good co-operation between the NSO and providers of administrative data is needed to ensure that the administrative registers and data are suitable also in the long run as sources of data for the production of official statistics.

In return, administrations can benefit from the know-how and expertise of the NSO in collecting, processing and analysing data and eventually improve the timeliness and coverage of their administrative data. In some countries, NSOs have set up working groups to enhance collaboration and regular dialogue with administrative data providers. These collaborations mechanisms are often governed by memoranda of understanding between the statistical office and the administrative data providers which foresee that the NSO would be informed and consulted well in advance about any changes on the structure, coverage and timeliness of the administrative data set used for statistical purposes.

Such policies and agreements can lead to positive outcomes on both sides. The NSOs can build relationships with administrative data providers by offering expertise in collecting, editing, and storing data by promoting statistical standards and providing guidance in quality issues. These measures, in return, may improve the quality of administrative data. However, the borderline between the producer of official statistics, typically the NSO, and the administrative data providers has to remain clear; in particular when it comes to the principles of confidentiality and professional independence as presented in Chapter 4.2.2 — Legal frameworks, obligations, and restrictions.

Preparation and facilities of the NSO

Acquisition from administrative sources needs to be carefully planned and monitored to detect any potential hindrances and issues; in particular when such data is going to replace or has replaced data collection through surveys conducted by the NSO. When administrative data is used for the time, NSO should carefully assess its impact and, when relevant, adapt its internal production processes. These changes may take some time as they require adequate staff, infrastructure and IT technology.

The NSO with established use of several administrative sources may consider creating a specific functional unit for administrative data in the data collection department through which all data coming from administrative sources should go before they are processed.

This unit would be responsible for checking that incoming data files are acceptable and conducting the first validation and quality checks. The unit may also be responsible for managing the system of data collection agreements (ToR) between the data providers and the NSO. The aims and development of this kind of unit are described in the paper ‘The system of collecting administrative data and how it responses to the quality guidelines of the code of practice and the peer review’ (🔗) published in 2017.

9.3.5 Processing administrative data#

The Generic Statistical Business Process Model (GSBPM) is designed to be applicable regardless of the data source. It can be used for the description and quality assessment of processes also based on administrative data. GSBPM is discussed in Chapter 6.3.1 — Administrative structure and finance of the national statistical office and Chapter 15.4.3 — Generic Statistical Business Process Model (GSBPM). Before planning the use of administrative data for statistical purposes, it is recommended to take a closer look at this model which covers all phases of the statistical production process.

At a general level, an NSO should have a clear understanding of what specific administrative data are needed and for which statistical purposes. It is important that for each administrative data set the underlying administrative process and relating legislation have been carefully analysed. A thorough understanding of the content of administrative data, including definitions of units, concepts and variables as well as updating systems is equally of great importance.

There are many variations regarding the use of administrative data in the production of statistics. For example, these data can be utilised to substitute for or supplement survey data, construct statistical registers, generate and update sampling frames, and create integrated statistics such as national accounts and as a part of register-based statistics. Some administrative sources, e.g., the administrative population register, can be used simultaneously for many statistical purposes.

Processing of administrative data is not a specific part or a separate sub-process of the GSBPM, but it is embedded in all process phases. Even though the GSBPM identifies possible steps in the statistical process, it does not require any strict order in which these steps or sub-processes are to be conducted.

It is important when processing administrative data as one data source for statistics that all necessary steps and sub-processes have been considered and taken into account in the process planning.

An illustrative example of the processing of administrative data in the framework of GSBPM is described in the paper: Methodologies for an Integrated Use of Administrative Data in the Statistical Process (🔗).

Processing administrative data in the context of register-based statistics is further discussed in Chapter 12.2 — Register-based statistics.

!