8.3 Quality assurance frameworks, guidelines, and tools#
Chapters 8.3.1 – 8.3.3 describe generic quality assurance frameworks, quality guidelines and other quality tools developed by international and supranational statistical organizations for use by NSOs.
Chapter 8.3.1 summarises the generic UN National Quality Assurance Framework (NQAF), which is a core feature of the UN Quality Assurance Frameworks Manual for Official Statistics developed under the guidance of the UN Statistical Division.
Chapter 8.3.2 summarises the quality-related standards and tools developed within the European Statistical System (ESS).
Chapter 8.3.3 includes quality standards and codes of statistical practice developed by other international and supranational organizations, including the IMF and the UN statistical commissions.
Chapter 8.4 describes some specific quality assurance frameworks, guidelines and tools that have been developed by individual NSOs. Obviously, not all NSO quality frameworks and tools can be included. Those described are ones that are well documented, that are readily accessible via the Internet, and that can be seen as representing good practice.
8.3.1 United Nations National Quality Assurance Frameworks Manual#
8.3.1.1 Objectives, structure and content#
The United Nations National Quality Assurance Frameworks Manual for Official Statistics (UN NQAF Manual) (🔗), including recommendations, framework and implementation guidance, was adopted by the UNSC in 2019. It is directed at assuring the quality of official statistics throughout an entire NSS. It builds on and replaces the generic United Nations National Quality Assurance Framework (UN-NQAF) template and guidelines adopted by the UNSC in 2012.
The Manual is the single most useful guidance document for an NSO that does not have a quality assurance framework and would like to develop one, or for an NSO that wants to revise and improve its framework.
It does not aim to replace existing statistical quality assurance frameworks and guidelines. Producers of official statistics that are already fully engaged in quality assurance in accordance with existing quality frameworks may view the Manual as a reference that supports what they are already doing, and as a source of information on the application of quality assurance in different situations.
There has been a significant uptake of ideas from the NQAF by developed and developing NSOs alike, including, for example, Lithuania, Poland, Palestine and Jamaica.
The structure of the Manual is shown in Figure 8.
Fig. 8 Structure of the UN NQAF Manual for Official Statistics#
8.3.1.2 UN NQAF principles, requirements and elements#
Chapter 3 of the Manual presents 19 UN NQAF principles, and the requirements for their implementation, organized in four levels, as indicated below. Each requirement is complemented in the annex to the Manual by a detailed list of elements to be assured.
Level A: Managing the statistical system: Coordination of the NSS and managing relations with all stakeholders is a precondition for the quality and efficient production of official statistics. Ensuring the use of common statistical standards throughout the system is an important aspect of this management.
Principle 1 - Coordinating the NSS: coordination of the work of the members of the NSS is essential for improving and maintaining the quality of official statistics.
Principle 2 - Managing relationships with data users, data providers and other stakeholders: the statistical agencies should build and sustain good relationships with all their key stakeholders, including users, data providers, funding agencies, senior government officials, relevant community organizations, academia and the media. The statistical agencies should have access to all data necessary to satisfy the information needs of society in an effective and efficient way.
Principle 3 - Managing statistical standards: standards refer to the full set of statistical concepts, definitions, classifications and models, methods and procedures used to achieve the harmonised treatment within and across processes and across time and space. The use of standards promotes the consistency and efficiency of statistical systems at all levels.
Level B: Managing the institutional environment: The quality of the institutional environment affects the quality of the processes it embraces and the outputs they produce.
Principle 4 - Assuring professional independence: NSOs should develop, produce and disseminate statistics without any political or other interference or pressure from other government agencies or policy, regulatory or administrative departments and bodies, the private sector or any other persons or entities. Such professional independence and freedom from inappropriate influence ensure the credibility of official statistics. This applies to the national statistical office and to other producers of official statistics.
Principle 5 - Assuring impartiality and objectivity: statistical agencies should develop, produce and disseminate statistics respecting scientific independence and in a way that is professional, impartial and unbiased, and in which all users are treated equitably.
Principle 6 - Assuring transparency: NSOs’ policies and management practices, and the terms and conditions under which their statistics are developed, produced and disseminated and, if applicable, subsequently revised (including the legal basis and purposes for which the data are required), are documented and available to users, respondents, owners of source data and the public.
Principle 7 - Assuring statistical confidentiality and data security: NSOs should guarantee that the privacy of data providers (persons, households, enterprises and other data providers) will be protected and that the information they provide will be kept confidential, will not be able to be accessed by unauthorized internal or external users and will be used for statistical purposes only.
Principle 8 - Assuring commitment to quality: NSOs should be dedicated to assuring quality in their work, and systematically and regularly identify strengths and weaknesses to continuously improve the process and product quality.
Principle 9 - Assuring adequacy of resources: the financial, human, and technological resources available to NSOs should be adequate both in magnitude and quality, and sufficient to meet their needs regarding the development, production and dissemination of statistics.
Level C: Managing statistical processes: International standards, guidelines and good practices are fully observed in the statistical processes the NSO uses to develop, produce and disseminate official statistics. The credibility of the statistics is enhanced by a reputation for good management and efficiency.
Principle 10 - Assuring methodological soundness: in developing and producing statistics, NSOs should use sound statistical methodologies based on internationally agreed standards, guidelines or best practices.
Principle 11 - Assuring cost-effectiveness: NSOs should assure that resources are effectively and efficiently used. They should be able to explain to what extent set objectives were attained, that the results were achieved at a reasonable cost and are consistent with the principal purposes of the statistics.
Principle 12 - Assuring appropriate statistical procedures: effective and efficient statistical procedures, underpin quality and should be implemented throughout the statistical production chain.
Principle 13 - Managing the respondent burden: individuals, households or businesses that provide the data upon which statistical products are based are fundamental contributors to the quality of outputs. The requirement to collect data should be balanced against production costs and the burden placed on respondents. Mechanisms to maintain good relationships with providers of data and to proactively manage the respondent burden are essential to improving quality.
Level D: Managing statistical outputs: Statistics serve the needs of national governments, research institutions, businesses, the general public and the international community.
Principle 14 - Assuring relevance: statistical information should meet the current and/or emerging needs and expectations of its users. Without relevance, there is no quality. However, relevance is subjective and depends upon the varying needs of users. The NSOs challenge is to weigh and balance the conflicting needs of current and potential users to produce statistics that satisfy the most important and highest priority needs within the given resource constraints.
Principle 15 - Assuring accuracy and reliability: NSOs should develop, produce and disseminate statistics that accurately and reliably portray reality. The accuracy of statistical information reflects the degree to which the information correctly describes the phenomena it was designed to measure, namely, the degree of closeness of estimates to true values.
Principle 16 - Assuring timeliness and punctuality: NSOs should minimize the delays in making statistics available. Timeliness refers to how quickly after the reference date or the end of the reference period, the outputs are made available to users. Punctuality refers to whether outputs are delivered on the promised, advertised or announced dates.
Principle 17 - Assuring accessibility and clarity: NSOs should ensure that the statistics they develop, produce and disseminate can be found and obtained without difficulty, are presented clearly and in such a way that they can be understood, and are available and accessible to all users on an impartial and equal basis in various convenient formats in line with open data standards. Provision should be made for allowing access to microdata for research purposes, in accordance with an established policy that ensures statistical confidentiality.
Principle 18 - Assuring coherence and comparability: NSOs should develop, produce and disseminate statistics that are consistent, meaning it should be possible to combine and make joint use of related data, including data from different sources. Furthermore, statistics should be comparable over time and between areas.
Principle 19 - Managing metadata: NSOs should provide information covering the underlying concepts and definitions of the data collected and statistics produced, the variables and classifications used, the methodology of data collection and processing, and indications of the quality of the statistical information—in general, sufficient information to enable the user to understand all of the attributes of the statistics, including their limitations.
8.3.1.3 NQAF implementation#
Chapters 4 to 8 of the UN NQAF Manual deal with all aspects of implementation.
Chapter 4 lists the various tools and instruments for quality assessment, including a section on risk management;
Chapter 5 is concerned with the development and implementation of an NQAF at an NSO or another statistical organization;
Chapter 6 discusses the role of NSS-wide bodies in the implementation of an NQAF throughout the NSS;
Chapter 7 approaches quality assurance from the perspective of the data source being used, which is particularly pertinent in the discussion of new data sources; and
Chapter 8 introduces quality assurance for statisticians involved in the compilation of SDG indicators.
8.3.1.4 Quality assurance in the global system#
Chapter 9 of the UN NQAF Manual provides reference materials for statisticians who are interested in the links between quality assurance at the national and global level.
It discusses collaboration within the global statistical system in assuring data quality at the global level, taking into consideration the need for international comparability of data, especially in the context of the compilation of the indicators for monitoring progress towards national, regional and global goals and targets of the 2030 Agenda for Sustainable Development.
8.3.2 European Statistical System – quality management standards, guidelines and tools#
The European Statistical System (ESS) comprises the NSOs of 27 European Union (EU) Member States, four EFTA countries, and Eurostat.
The ESS is a prolific source of quality-related regulations, standards, guidelines and tools, as summarised in the following sections.
For an NSO in an EU Member State or EFTA country, these documents are the natural starting point and basis for developing a quality assurance framework.
European legislation and regulations relating to quality
The Amended Regulation (EC) No 223/2009 on European Statistics (🔗) includes two articles on quality:
Article 11: The ES CoP shall aim at ensuring public trust in European statistics by establishing how European statistics are to be developed, produced and disseminated in conformity with the statistical principles as set out in Article 2(1) and best international statistical practice. The CoP shall be reviewed and updated as necessary by the ESS Committee.
Article 12: To guarantee the quality of results, European statistics shall be developed, produced and disseminated on the basis of uniform standards and harmonised methods. In this respect, the following quality criteria shall apply: i) relevance, ii) accuracy, iii) timeliness, iv) punctuality, v) accessibility and clarity, vi) comparability, and vii) coherence.
Specific quality requirements, such as target values and minimum standards for the statistical production, may be laid down in sectoral legislation.
The Member States shall provide the Commission (Eurostat) with reports on the quality of the data transmitted. The Commission (Eurostat) shall assess the quality of data transmitted and shall prepare and publish reports on the quality of European statistics.
The Inventory of regulations in the field of statistics containing provisions on quality and quality reporting (2023) (🔗) comprises a list of domain-specific regulations, each of which contains a quality management reference or references. The list is currently being updated.
European Statistics Code of Practice (ES CoP)
The ES CoP has been, arguably, the most influential statistical quality-related document in the last 20 years.
It was most recently updated in November 2017. It is the cornerstone of ESS quality management. It is a self-regulatory instrument. It includes the ESS Quality Declaration and 16 statistical quality principles.
ESS Quality Declaration: “The European Statistical System is a partnership in which Eurostat and the national statistical authorities of each EU Member State and EFTA country cooperate. Together, our mission is to provide independent, high-quality statistical information at European, national and regional levels and to make this information available to everyone for decision-making, research and debate…
We operate under a strict legal regime supplemented by a robust, world-class and self-regulatory quality framework, the backbone of which is the European Statistics Code of Practice. Our compliance with the Code of Practice is periodically assessed by means of review mechanisms and strict follow-up of the improvement actions identified.
We see quality as the basis of our competitive advantage in a world experiencing a growing trend of instant information which often lacks the necessary proof of quality…
We are committed to statistical excellence by systematically identifying our strengths and weaknesses, as well as related risks which we duly take into account by the continuous development of our common quality framework…”
ES CoP Principles: The 16 ES CoP principles, reproduced below, are closely related to the 19 UN NQAF principles. Some are essentially identical. Collectively they cover the same ground. This is not at all surprising as the initial (2012) version of the UN NQAF drew on the ES CoP (2011), and neither set of principles has been greatly changed in subsequent revisions.
The 16 principles are organized into three groups, covering the a) institutional environment, b) statistical processes, and c) statistical output. These groups are closely related to the four levels of the UN NQAF, the main difference being that the ES CoP institutional environment group includes the UN NQAF statistical system and institutional environment levels.
Each ES CoP principle is exemplified by indicators of best practices and standards (not included here) that provide guidance and reference material for reviewing ES CoP implementation.
Institutional Environment
Principle 1: Professional Independence
Principle 1bis: Coordination and cooperation
Principle 2: Mandate for Data Collection and Access to Data
Principle 3: Adequacy of Resources
Principle 4: Commitment to Quality
Principle 5: Statistical Confidentiality and Data Protection
Principle 6: Impartiality and Objectivity
Statistical Processes
Principle 7: Sound Methodology
Principle 8: Appropriate Statistical Procedures
Principle 9: Non-excessive Burden on Respondents
Principle 10: Cost-Effectiveness
Statistical Output
Principle 11: Relevance.
Principle 12: Accuracy and Reliability
Principle 13: Timeliness and Punctuality
Principle 14: Coherence and Comparability
Principle 15: Accessibility and Clarity
For EU Member States and EFTA countries, the ES CoP is the cornerstone of the quality assurance framework. It may be used without change, or it may be incorporated in modified form, in a national statistical code of practice, as, for example, by the NSOs of the UK, Ireland and Hungary.
Quality Assurance Framework of the European Statistical System
The ESS quality assurance framework (🔗) accompanies and complements the ES CoP. It identifies possible activities, methods and tools that provide guidance and evidence for the implementation of the ES CoP.
European Central Bank - Statistics Quality Framework
The European Central Bank Statistics Quality Framework (ECB SQF) was developed in 2008. It is compatible with the ES CoP. It sets out the main quality principles and elements guiding the production of ECB statistics. It is a statement of intent, not a standard. However, most elements are fully reflected in current practices.
The quality assurance procedures included in the ECB SQF cover programming activities and development of new statistics, confidentiality protection, data collection, compilation and statistical analysis, data accessibility and dissemination policy, monitoring and reporting, and monitoring and reinforcing the satisfaction of key stakeholders.
ESS Handbook for Quality and Metadata Reporting
The latest version of the ESS Handbook for Quality and Metadata Reports (EHQMR) was disseminated in 2020. It superseded the ESS Handbook for Quality Reports, 2014 and the Single Integrated Metadata Structure and its Technical Manual, 2014 (🔗).
The EHQMR is based on the Single Integrated Metadata Structure (SIMS) v 2.0, which provides definitions and reporting guidelines for all ESS quality and reference metadata concepts. SIMS includes a two-component reporting structure:
the Euro-SDMX Metadata Structure (ESMS) for user reports; and
the ESS Standard for Quality Reports Structure (ESQRS) for quality reports.
SIMS streamlines and harmonises metadata and quality reporting in the ESS. It minimises the reporting burden on NSOs by facilitating once for all purposes reporting, whereby the concepts covered in both quality and user reports are reported upon only once. It facilitates storage of all reports in a single database.
The EHQMR incorporates the ESS Quality and Performance Indicators (🔗). These indicators are accompanied by guidelines which, for each indicator, give the definition, applicability, calculation formulae, target value, aggregation levels, interpretation, and references.
Other ESS Quality Management Tools
The ESS Quality Glossary (🔗), first published in 2003, covers many technical terms in ESS quality documentation, providing a short definition of each term and indicating the source of the definition. It was transferred to the Concepts and Definitions Database (CODED), where it is now available as a theme. Other glossaries containing quality-related terms are also available as themes in CODED, namely the SDMX Glossary Version 1.0 (2016) (🔗) and General Statistical Terminology (🔗).
The Quality Guidelines for Multi-source Statistics (🔗) provide practical support for the design and implementation of multisource statistics within a comprehensive quality framework. For each ES CoP output quality dimension, the guidelines are developed around three objectives:
error prevention;
monitoring/correction/adjustment of possible errors during the statistical production process; and
assessment of the impact of the errors on the final estimates.
The Handbook on Data Quality - Assessment Methods and Tools (🔗) details the full range of methods for assessing process and output quality and the tools that support them. The Handbook on improving quality by analysis of process variables (🔗) describes a general approach and useful tools for identifying, measuring and analysing key variables associated with a statistical process. The European Self-Assessment Checklist for Survey Managers (DESAP) enables the conduct of quick but systematic and comprehensive quality assessments of a statistical process (survey, census or administrative process) and its outputs and identification of potential improvements. The documents are also available in electronic form: Electronic DESAP-E checklist (🔗) and Electronic DESAP user guide (🔗) and an abbreviated version is available as DESAP condensed (🔗).
8.3.3 Other internationally developed quality assurance frameworks#
International Monetary Fund - Data Quality Assessment Framework
The International Monetary Fund (IMF) first developed its Data Quality Assessment Framework (DQAF) in 2001. Its aim is to complement the quality dimensions of the IMF Special Data Dissemination Standard (SDDS) and the Enhanced General Data Dissemination System (eGDDS) and to underpin the assessment of the quality of the data provided by countries as background for IMF Reports on the Observance of Standards and Codes (ROSC). The SDDS, its new version (SDDS Plus), and the eGDDS provide guidance to member countries on the provision and relevance of their economic and financial statistics.
The DQAF is designed for use by IMF staff and NSOs in assessing the quality of specific types of national datasets.
It covers the national accounts, the consumer price index, the producer price index, government financial statistics, monetary statistics, the balance of payments and external debt. It has been very widely used by the IMF and by the NSOs with which the IMF has been involved in ROSC activities. It has also influenced quality assurance framework developments in other countries such as Italy, Netherlands and Finland.
It is a process-oriented quality assessment tool. It provides a structure for comparing existing practices against best practices using five dimensions of data quality: integrity, methodological soundness, accuracy and reliability, serviceability and accessibility, in addition to the so-called prerequisites for data quality. It identifies three to five elements of good practice for each dimension and several indicators for each element. Furthermore, in the form of a multilevel framework, it enables datasets to be assessed concretely and in detail through focal issues and key points. The first three levels of the framework (dimensions, elements and indicators) are generic, that is, applicable to all datasets, the lower levels are specific to each type of dataset.
The DQAF dimensions, elements and indicators are rather different from the quality dimensions and indicators in the UN NQAF and the ES CoP. Although mappings between the DQAF, ES CoP and UN NQAF dimensions and indicators have been prepared, an NSO cannot readily design a quality assurance framework incorporating both. In essence, it has to make a choice between a UN NQAF/ES CoP approach and a DQAF approach.
African Charter on Statistics
The African Charter on Statistics (🔗) was adopted in 2009 and entered into force in 2015. It presents six quality principles expressed in the form of 25 quality statements covering most of the quality principles in the ES CoP but reorganized and tailored to the African situation. The principles are:
Professional independence – comprising scientific independence, impartiality, responsibility and transparency;
Quality – comprising relevance, sustainability, data sources, accuracy and reliability, continuity, coherence and comparability, timeliness, topicality, African specificities, and awareness building;
Mandate for data collection and resources – comprising mandate, resource adequacy and cost-effectiveness;
Dissemination – comprising accessibility, dialogue with users, clarity and understanding, simultaneity, and correction;
Protection of individual data, information sources and respondents – comprising confidentiality, giving assurances to data providers, use for statistical purposes, and rationality; and
Coordination and cooperation – comprising coordination and cooperation.
ECLAC – Code of Good Practice in Statistics
The Code of Good Practice in Statistics for Latin America and the Caribbean (🔗) was developed with support from the Economic Commission for Latin America and the Caribbean (ECLAC) and Eurostat by a working group of ECLAC countries in 2011. It was modelled on the European Statistical Code of Practice (2008) and extended to include coordination of the NSS as a whole (as subsequently incorporated in the 2017 ES CoP).
Caribbean Community – Statistics Code of Practice
The Caribbean Community (CARICOM) developed its Statistics Code of Practice (🔗) with support from the European Union. It is based on the ES CoP and has 15 principles and 78 indicators.
ASEAN Community Statistical System Code of Practice
ASEAN Community Statistical System (ACSS) adopted its Code of Practice (🔗) in September 2012. It is modelled on the ES CoP (2008) but with fewer principles and indicators.
UNECE Quality Indicators for the Generic Statistical Business Process Model
As detailed in Chapter 15.4.3 — Generic Statistical Business Process Model (GSBPM), the Generic Statistical Business Process Model (GSBPM v5.1) provides the standard template for describing surveys and administrative data collections in terms of 8 phases and 44 subprocesses. Quality Indicators for the GSBPM Version 2.0 - For Statistics derived from Surveys and Administrative Data Sources (🔗) provides a set of indicators to monitor the quality of the production processes for each phase.
United Nations Statistical Quality Assurance Framework
At its first meeting, the Committee for the chief statisticians of the United Nations System (CCS-UNS) decided that a generic quality assurance framework would be developed for use by United Nations agencies in managing their statistical data. Several UN agencies already had a quality framework or a code of practice of some sort and, while these differed from one another, there was a high degree of overlap.
The resulting United Nations Statistical Quality Assurance Framework (UN SQAF) was developed by a UN Task Team and adopted by the CCS-UNS in March 2018. It is based on a broad concept of quality that incorporates institutional, process and output dimensions. It is not prescriptive. It provides a template and guidelines that can be adapted by a UN agency to suit its circumstances. It is expected that UN agencies without a quality framework will adapt this generic version to the situation of their agency.
Whilst the UN SQAF is designed for international statistical organizations, the ideas it incorporates are also informative for NSOs.