August 28, 2018 | News | Interview

Forecasting Fertility: Should we care about the method?

Five questions to Christina Bohk-Ewald on her new study evaluating cohort fertility forecasting methods.

Forecasts of completed fertility (CF) predict how many children will be born on average by women over their entire reproductive lifetime. They answer questions such as: What will be the completed level of fertility of women who are now, for example, aged 30? Such forecasts are of high relevance for policy making and planning, and for understanding overall fertility trends.


In a new PNAS study, Christina Bohk-Ewald, Peng Li, and Mikko Myrskylä show that the choice of the projection method doesn't really matter much in terms of accuracy and uncertainty, at least in the group of the top performing methods. The researchers assess 20 major methods for completed fertility prediction, with 162 variants. Each method aims at completing lifetime fertility of women who have not yet reached their last reproductive age at the time the forecast is made.

They compared the projection results of each method by using fertility data of the complete Human Fertility Database (covering 1,096 birth cohorts from 29 countries) and the globally representative World Population Prospects database of the United Nations (covering 8,241 birth cohorts from 201 countries). They find only four methods that consistently outperform the naïve freeze rates method (that keeps current age-specific fertility rates constant), and only two methods produce uncertainty estimates that are not severely downward biased.

1. If almost no method is significantly better than the freeze rates method, why should we care about the method at all and not just use freeze rates?

Christina Bohk-Ewald:

Yes, the naïve freeze rates method is consistently outperformed by only four of 19 other methods. But some of the best-performing methods forecast cohort fertility up to 40% more accurately than freeze rates!

The main message of our findings is that while method complexity does not necessarily increase forecast accuracy, it is still true that freeze rates indeed can be outperformed by both simple extrapolation methods and complex Bayesian approaches.

Interestingly, comparing the best-performing methods, we find that the complex Bayesian approaches, which are demanding in terms of high-quality data, statistical methods, and computing power, do not consistently outperform simple extrapolation methods at all truncation ages.

2. What is it that makes those four top methods better than freeze rates?

Christina Bohk-Ewald:

The group of the top performing methods includes two simple extrapolation methods and two complex Bayesian approaches. Any of these four methods could be in the lead depending on the way we measure the quality of the forecasts (i.e., the metrics we apply).

Their methodological key features – or breakthroughs — appear to be feeding the forecast with information about past fertility trends in the country of interest and additionally with the fertility experiences of many other countries.

3. In what way does the development of methods to forecast cohort fertility (CF) still make sense? Is there any methodological enhancement that would be capable of considerable improvements?

Christina Bohk-Ewald:

Although the best methods forecast cohort fertility already very accurately, our study uncovers forecast situations that are still challenging, even for the best-performing methods.

All but two of the eight methods that provide prediction intervals underestimate forecast uncertainty, meaning their prediction intervals are too narrow and do not accurately reflect the uncertainty in the forecasts.

Another shortcoming is, not surprisingly, when fertility developments deviate from continuous trends or from the experience of other countries. For example, the best-performing methods produce relatively large forecast errors for Eastern Germany because fertility abruptly changed its long-term trend and fell to unprecedentedly low levels after German reunification.

Solving the quantification of forecast uncertainty and forecasting strong fertility declines may be attainable; solving the problem of abrupt trend changes is perhaps the holy grail of all forecasting and as such less likely to happen very soon.

Consequently, quantifying forecast uncertainty and forecasting strong fertility declines are one of the unresolved issues in cohort fertility forecasting that need to be addressed when developing new approaches.

4. Do you have any plans to address these unresolved issues in CF forecasting?

Christina Bohk-Ewald:

Yes! Our study clearly motivates us to develop a new method that addresses the unresolved issues, and it already indicates what ingredients might be useful in a new formula.

For example, next to the past fertility experiences in countries covered by the Human Fertility Database and the United Nations World Population Prospects, we may want to make use of the forecast errors derived in our validation study as a third unique data source. This might be helpful since our study shows us in what forecast situations which method type produces small or large forecast errors and for which fertility levels and trends they are particularly suitable.

The future will show if we can exploit this unique validation data source to the advantage of fertility forecasts.

5. Are researchers actually aware of the quality of the forecasting methods they use – or newly develop?

Christina Bohk-Ewald:

They should be! Our study demonstrates how important it is to carefully validate the performance of fertility forecast methods in terms of forecast accuracy and uncertainty estimates. This is a core area often neglected.

Since forecast accuracy did not improve continuously with the publication of new methods over time, we suggest that, as a standard procedure, researchers assess whether the new methods proposed outperform at least freeze rates or simple extrapolation methods.

To facilitate such benchmarking, we provide R functions (including scripts for the forecast methods as well as for the validation procedure) on GitHub. They are available at: https://github.com/fertility-forecasting/validate-forecast-methods
 

Read the full paper here:
Forecast accuracy hardly improves with method complexity when completing cohort fertility
Christina Bohk-Ewald, Peng Li, and Mikko Myrskylä