8.6 Evaluation of a statistical process and its outputs#
Evaluation of a process and its outputs may be of several types, each of increasing complexity, with increasing demands on resources, as indicated in Figure 6 and described in the paragraphs below. However, all types of evaluation should result in a quality report detailing the findings, identifying quality and performance problems and making recommendations to senior management for quality and performance improvements that cannot be undertaken with the resources currently available to the process.
Abbreviated versions of evaluation reports may be made available to users. The ESS Handbook on Quality and Metadata Reporting (described in Chapter 8.3.2 — European Statistical System – quality management standards, guidelines and tools) indicates the quality and metadata that should be disseminated to users and outlines how they should be described.
8.6.1 Self-evaluation#
The objectives of self-evaluation are to help the manager and staff responsible for the process to assess its quality and the quality of its outputs, to identify structural weaknesses, and to propose quality improvements.
Self-evaluation is undertaken by the process manager and persons involved in the design and implementation of the process. Depending upon the process and the resources available, it may be conducted annually or biennially. It typically starts with the completion of a standard evaluation checklist that is cross-referenced to the quality principles and guidelines. During completion of the questionnaire, areas of concern that arise are further probed. A set of improvement action items is a primary output. These are typically divided into two groups: those that can be implemented immediately without additional resources; and those that require allocation of additional resources and that will be referred to senior management.
Supported self-evaluation is a form of self-evaluation in which a quality expert sits in on the self-evaluation process and helps guide it. This is the recommended approach for a first-time self-evaluation.
8.6.2 Internal peer-based evaluation#
Experience has shown that self-evaluation has its limitations. Those responsible for a process may be too close to see some of its deficiencies or possibilities for improvement. This is the reason for a peer-based or external evaluation.
The objectives of the peer-based evaluation are similar to those for self-evaluation but with the understanding that the evaluation is more penetrating, and the target audience includes senior management.
The procedures are also similar to self-evaluation but with the important distinction that the evaluation team contains members of staff having no connection to the process, the so-called peers, one of whom typically manages the evaluation.
The team is supported by the process manager and other experts involved in its design and implementation.
8.6.3 External evaluation#
External evaluation initiated by the NSO
The need to conduct periodic external evaluations of processes and their outputs may be built into an organization’s quality framework. Another trigger for external evaluation is significant criticism of its outputs by users or other stakeholders. In organizations that are receiving support from donors or partners, the need to evaluate the efficacy of a donor or partner intervention is also a common trigger.
The target of an external evaluation may be the process or a particular part of the process. The evaluation objectives are to provide the senior management and process manager with an objective view of the quality of the process or some part of it, and hence to identify structural weaknesses and to propose quality improvements for addressing them.
The procedures are similar to internal peer-based evaluation but with the important distinction that the evaluation team is largely or entirely from outside the organization.
The team is provided with information by the process manager and other experts involved in its design and implementation.
Given that there may be a need to identify and make use of evaluators from outside the organization, it is useful to establish relationships with other NSOs in the NSS and in other countries with the aim of exchanging evaluation services.
External evaluation initiated by a regional organization
An external evaluation may also arise in the context of a regional programme to check compliance with a regional quality standard, in which context it is often referred to as a peer review.
For example, external evaluations, termed peer reviews, form part of the European Statistical System (ESS) strategy to monitor implementation of the ESCoP in the EU and EFTA Member States (🔗), administered by Eurostat. The object of each review is to evaluate the compliance/alignment with the ESCoP in the Member State and to help the NSO and other producers of statistics comprising the NSS to further improve compliance. A first-round of peer reviews was carried out in 2006-2008 and a second-round 2013-2015. The third round of peer reviews is being prepared and will take place in 2021-2023. More details are provided in Peer reviews in the European Statistical System (🔗).
This type of peer review programme has been adopted by African countries, as described in Peer reviews of national statistical institutes and national statistical systems in African countries (🔗). The reviews are centred on compliance with the African Charter on Statistics, adopted in 2009, which provides a set of principles and standards for the functioning of African statistical systems.
8.6.4 User surveys and other user feedback#
User surveys and other user feedback are not another type of evaluation, rather they are essential inputs to all types of evaluation, especially in relation to the relevance of outputs, and to obtain user perceptions of other aspects of quality.
A user survey is a survey that assesses the satisfaction and/or the perceptions of the users. As regards its coverage, the UNECE’s Recommendations for Promoting, Measuring and Communicating the Value of Official Statistics (🔗) identify six groups of items:
General satisfaction with products: relevance, accessibility, access preferences and what characteristics of official statistics the user considers the most important.
General satisfaction with user support: level of current service and improvements to service.
Design, communication and metadata: general design of the official statistics website design of statistical warehouse, interpretability, navigation and visualization.
Relevance, responsiveness and innovation: how effectively official statistics inform the public debate on current issues; the importance of official statistics in helping to understand societal developments and use of new technologies, methods and data sources by the NSO.
Awareness of brand and message: level of trust in official statistics; understanding the mandate of the NSO; perceived presence or absence of political interference and overall satisfaction with the organization.
Satisfaction with specific products and services: knowledge, use and quality of a particular product and use of anonymized microdata.
Users are divided into groups according to their needs, as discussed in Chapter 7 - Users and their Needs. A user survey can be targeted to one or several groups, as appropriate.
A user survey can take different forms, for example, using mail-out – mail-back questionnaires, personal interviews or web-based surveys. The choice depends on the nature of the information being sought, and on the resources available.
The ISO 10004:2018 Quality management – Customer satisfaction (🔗) standard provides further ideas, and the UNECE’s Recommendation cited above include a generic user survey.
Other sources of user feedback include sector committees set up by the organization and comprising experts in the various subject matter areas, and focus groups convened to address particular issues.
8.6.5 Labelling#
The labelling of a statistical output follows from and depends upon a favourable evaluation. It is intended to convey a general message to users about its quality. This is important in a world of multiple suppliers. It is a means of distinguishing official statistics from other sources (see also Chapter 3.5 — Certification and branding of official statistics). It enhances the visibility and credibility of the output. The label may be brief, for example, official statistics, or experimental statistics. It should always be accompanied by an explanation of its meaning.
A label can also be given to an NSO as a whole, reflecting its capacity to produce high-quality statistics, as demonstrated, for example, by certification in terms of a generic quality management systems standard.