9.1 Introduction

Although a chief statistician does not need to know all the details of every statistical activity, a general understanding of statistical concepts and statistical processes is vital for effective decision-making. The starting point is the Generic Statistical Business Process Model (GSBPM) described in Chapter 6 - The National Statistical Office and Chapter 15.4.3 — Generic Statistical Business Process Model (GSBPM). The GSBPM describes a statistical process in terms of eight phases, namely, Specify Needs, Design, Build, Collect, Process, Analyse, Disseminate and Evaluate. Chapter 6 covers Specify Needs and this chapter deals with the next four phases, i.e., Design, Build, Collect, and Process, leaving chapters 7 and 9 to discuss the Analyse and Evaluate phases, respectively.

For the purpose of the chapter, statistical processes are divided into four groups according to the input data source, namely:

Surveys and censuses are the traditional data source and are still very much used. Chapter 9.2 — Sample surveys and censuses, the longest section in the chapter, details all aspects of the design, build, collect and process phases. It also includes subsections on respondent relations and staff training, which cross phases.Although well established and precisely targeted to address specified needs, surveys and censuses are time-consuming, labour-intensive and expensive. At the same time, NSOs are having to deal with shrinking budgets and emerging requests for more timely and disaggregated statistics and for indicators covering new domains, in particular in relation to monitoring the 2030 Sustainable Development Agenda and other regional and national development policies. To address these challenges, NSOs and other producers of official statistics are beginning to use the vast amounts of data that have become available in our digital society to supplement existing survey inputs. Indeed, the ‘datafication’ of society together with cost-efficient storage capacity and a rapid increase in computing performance have opened new opportunities for combining statistical surveys and censuses, and administrative records, and non-traditional data sources such as geospatial information and Big Data.

The most dramatic increase has been in the use of administrative data. Indeed, faced with a new data request, the first step an NSO should take is to check whether the need can be met by the use of an existing administrative data source. Chapter 9.3 — Administrative sources describes the particular features of the design, build and collect phases of processes using administrative data that distinguish them from surveys and censuses.

Geospatial data means data that have a geographic component. The most prolific source of such data is satellite imagery. Geospatial data can be used to greatly enrich other data sources. Chapter 9.4 — Geospatial data describes the types of geospatial data, their uses and the challenges faced in such uses.

Chapter 9.5 — Big Data deals with Big Data, the Internet-of-Things (IoT) sensors, wearables and mobile devices as tools that can complement traditional ways of collecting data due to their potentially high population coverage and use in daily life. The section draws on a vision paper by CBS Netherlands and Statistics Canada entitled Future advanced data collection (🔗), presented at the 62nd ISI World Statistics Congress 2019. It is crucial for NSOs to consider implementing advanced data collection capabilities to increase their value for the societies they serve. Data platforms and other infrastructure usually referred to as the Cloud could be valuable tools in bringing together and processing traditional and non-traditional data from public and private sources.

Box 4: New role for official statistics in Estonia

Estonia is well known for its innovative practices and the introduction of digital technologies and processes into its public administration. To facilitate more effective use of data sources and prevent duplication of information collected, Statistics Estonia (SE) proposed changes to its statistical law. Through this change, it has taken on the role of national data steward, tasked with coordinating data governance in Estonia. This function of SE will provide an overview, and the possibility to use a central database more efficiently for various statistical analyses since data providers (enterprises, institutions as well as and individuals) do not have to be burdened with responding to multiple questionnaires. The amended law requires all government authorities administering state databases to appoint persons responsible for data governance. It is expected that a data-sharing service will be added shortly to the statistical legislation, allowing SE to better coordinate data sharing of existing state data with other state authorities. This cross-usage of data between state authorities reduces duplication of data. It will enable SE to link data collected from different data sources and produce domain-specific data analyses and statistics while preserving statistical principles and confidentiality of data. SE provides this service only within the public sector and to research institutions. Link to Statistics Estonia, data science competence center (🔗).