README.Rmd

---
output: github_document
---

<## Relating forecast performance to methodology: the influence of model structure and target specificity on the performance of European COVID-19 forecasts

Kath Sherratt (1), Rok Grah (2), Bastian Prasse (2), The European COVID-19 Forecast Hub, Sam Abbott (1), Sebastian Funk (1)

(1) London School of Hygiene & Tropical Medicine
(2) European Centre for Disease Prevention and Control

### Abstract

Background: The performance of forecasts in capturing observed data often varies in time and space with no overall "best" forecast. Two varying features of forecasting methods that could help explain this are: the forecaster’s approach to model structure; and whether the forecaster specifically tunes their model to each target. We investigated short-term forecasts of weekly incident deaths for 32 countries submitted to the European COVID-19 Forecast Hub between March 2021 and March 2023. 

Methods: We categorised 39 models by their structure (mechanistic, semi-mechanistic, statistical), and by their specificity (the number of locations each team targeted; and whether the target location was the same as the modeller’s institutional location, as a proxy for model adaptation to local conditions). We assessed forecast performance using the weighted interval score. First, we compared performance relatively against a median ensemble of all models. Next, we used a generalised additive model to explore performance among different model structures and specificity, while accounting for the level and trend of observed incidence, the forecast horizon, and random variation among models. 

Results: We evaluated forecasts of COVID-19 deaths up to 4 weeks ahead for up to 32 countries over 104 weeks. No one model structure consistently outperformed the ensemble. Mechanistic models saw the widest range in performance, with the range of scores including both beating the ensemble and performing up to three times worse. Models that targeted only one or two countries appeared to perform on average better and more consistently compared to multi-country models. 

Interpretation: Based on this study we would recommend that mechanistic models should be part of predictive ensembles, with an emphasis on using information from the local context of where they are applied. Multi-model comparisons should encourage methodological diversity while ensuring that detailed information on methodology is collated alongside the numerical predictions.

Current working draft:
- <https://docs.google.com/document/d/1OOVUHR_BGWcviSNxvaHvbXD16Bb3Y_zhw--7gAGBqMk/edit#>

### Environment

Packages are managed using [renv](https://rstudio.github.io/renv/articles/renv.html).
In order to install all the required packages, install the `renv` package and run

```{r renv, eval = FALSE}
renv::restore()
```

### Data

All the data used in the analysis is stored in the `data/` directory.
It has been obtained from public sources.
In order to re-download the data in the `data/` directory, use

```{r get_data, eval = FALSE}
## Get metadata from googlesheet; save to data/
source(here("R", "get-metadata.R"))

## Get observed data and all Hub forecasts; exclude forecasts; save to data/
source(here("R", "import-data.R"))
```

### Analyses

In order to re-generate the forecast scores, use

```{r generate_scores, eval = FALSE}
## Score forecasts & ensembles on the log and natural scales; save to data/
source(here("R", "score.R"))
```

In order to run the GAM on the scores, use

```{r gam, eval = FALSE}
## Model the weighted interval score; save to data/
source(here("R", "model-wis.R"))
```

### Results

View results:

- [Results](./report/results.pdf)
- [Supplement](./report/supplement/supplement.pdf)

Re-generate results pdf:

- [results.Rmd](./report/results.Rmd)
- [supplement.Rmd](./report/supplement/supplement.Rmd)