Model Evaluation

Quick links: FAIRMODE UK-US collaboration Model intercomparisons

Model evaluation is a necessary part of both model development and application. The usual method for evaluation of air flow models such as FLOWSTAR-Energy and dispersion models such as the ADMS suite is to compare the model predictions with measured values. Some model outputs are straightforward to evaluate because there is an abundance of measurement data available, for example NO₂ and PM₁₀ pollutant concentrations within urban areas, which are continuously recorded for regulatory purposes. However, some model outputs are harder to evaluate due to lack of available measurement data, for instance dry and wet deposition fluxes, dispersion and decay of radioactive isotopes and the number and length of visible plumes. In cases where measurement data are unavailable or unsuitable for model validation purposes, model evaluation exercises may still be performed, for instance by comparing different models and/or checking numerical model output against theoretical solutions.

CERC are experts in air quality model evaluation. Routine model evaluation is performed as part of the majority of air quality management projects, as for instance described in an air quality model verification report for the City of London Corporation. New releases of the ADMS models are evaluated using measurement data from field campaigns, with the results being published on CERC’s model validation page. Wherever possible, new model features are developed using available measurement datasets for validation purposes, for example the road tunnels module within ADMS-Urban, ADMS-Roads and ADMS-Airport.

CERC have developed the freely available Model Evaluation Toolkit specifically for evaluating local and regional air quality model performance in terms of both ambient air quality objectives and air quality forecasting skill. The tool is easy to use and produces a series of graphs and statistics that allow detailed inspection of model performance.

The Forum for air quality modelling in Europe: FAIRMODE

FAIRMODE was launched in 2007 and is currently chaired by the Joint Research Centre (JRC). Its aim is to bring together air quality model users in order to promote and support the harmonized use of models by EU Member States, with emphasis on model application under the European Air Quality Directives.

CERC have taken an active role in the series of FAIRMODE technical meetings since 2010, primarily within the assessment working group, WG1. The DELTA Tool has been developed by JRC within the FAIRMODE initiative and CERC have contributed to its development in terms of providing feedback relating to the application of the tool and revisions to the associated guidance document. CERC have also contributed to FAIRMODE’s EU Composite Mapping Exercise.

A selection of CERC presentations from recent FAIRMODE meetings is given below.

• 2017: Model's fitness-for-purpose in the context of exceedance modelling & Evaluation of DELTA Forecasting MQO v5.5: forecasting system evaluation project challenges
• 2016: Evaluation of Delta Forecasting MQO v5.4 for London & Statistics relating to ‘exceedance situations’ for London
• 2015: Evaluation of Delta Forecasting MQO v5.1 for London
• 2014: Evaluation of DELTA forecast functionality & Feedback on the NO2 MQO with 1-hr and 3-hr averages

UK-US collaboration on evaluation of road source modelling tools

The US Environmental Protection Agency, UK Environment Agency and Defra signed an agreement to cooperate on Air Quality Modelling and Exposure Science in 2008, holding annual meetings alternately in the UK and US. CERC are active participants in these meetings, in particular focusing on aspects of local dispersion modelling. This has led to productive collaborations with staff from the USEPA Office of Research and Development on evaluation of road source modelling tools, using experimental data ranging from an idealised line source in a wind tunnel or at full scale to high-resolution measurements in urban areas. The effects of roadside noise barriers on dispersion from road sources have also been investigated. These evaluation studies have contributed to improvements in road source modelling in the ADMS-Urban and ADMS-Roads models.

Model intercomparison exercises

In addition to model evaluation using measurements from field campaigns and monitoring networks, it is often informative to perform model intercomparison exercises that highlight the strengths and weaknesses of different models. CERC have lead and been involved with a number of model intercomparison projects.

Examples where model predictions from ADMS 6 (or earlier versions thereof) are assessed alongside those from the US EPA’s steady-state plume model AERMOD:

• Particular features of the complex terrain modules within ADMS 5 and AERMOD are compared in Carruthers et al. (2011).
• The UK Atmospheric Dispersion Modelling Liaison Committee (ADMLC) project on ‘A Review of the Limitations and Uncertainties of Modelling Pollutant Dispersion from Non-point Sources’ compared ADMS 5 and AERMOD results for low-level agricultural sources.
• ADMS 3.1 and AERMOD are evaluated alongside the US EPA models HPDM and ISCST3 using three atmospheric dispersion field studies: Prairie Grass, Kincaid and Indianapolis in Irwin et al. (2003).

Results from ADMS-Urban and ADMS-Roads have been assessed alongside both road source and regional models:

• Data from two US field campaigns (the Idaho Falls and Caltrans Highway 99 tracer studies) have been used to assess ADMS-Roads alongside RLINE, AERMOD and CALINE in Heist et al. (2013).
• ADMS-Urban was evaluated alongside other UK urban air dispersion models (including PCM) in a model intercomparison exercise lead by the UK Department for Environment Food & Rural Affairs (Defra). Reports presenting the results from Phase I and Phase II of this exercise are available online.
• As part of the evaluation of the ADMS-Urban Regional Model Link, street-scale resolution model predictions have been compared to the coarser resolution predictions from the CAMx regional model during an air quality modelling project in Hong Kong; details are given in Stocker et al. (2014).

ADMS-Airport model predictions have been extensively evaluated alongside other model predictions and air quality measurements, for example during the Project for the Sustainable Development of Heathrow, DfT's expert panel on modelling was in 'full agreement in the recommendation of the CERC model ADMS-Airport for future modelling work at Heathrow', following an intercomparison exercise involving ADMS-Airport, EDMS, LASPORT and ALAQS. The ADMS-Airport validation work was also published as a journal article.