:orphan: 

.. _diagnostics:

Diagnostics
==========================

After fitting a model to data, it is important to quantify the goodness-of-fit (GoF) to the data in some way. Summary statistics, often known as diagnostics, are useful in comparing models and visualising goodness-of-fit to data in a way that can steer the modeller toward better models.

Residuals
---------------------------------
Residuals - the difference between a predicted quantity, such as concentration, and its observed value - indicate the error (or noise) model on the observations, for which there are three popular choices.

.. list-table:: Residuals vs IPRED for the additive (left), proportional (centre) and additive+proportional (right) noise models
    :class: image_table

    * - .. thumbnail:: /_autogen/indiv_examples/error_models/ao_tut/images/fit_sim_grph_outputs/RES(DV_CENTRAL)_wrt_IPRED(DV_CENTRAL)/pop_scatter.*
    
        Additive

      - .. thumbnail:: /_autogen/indiv_examples/error_models/po_tut/images/fit_sim_grph_outputs/RES(DV_CENTRAL)_wrt_IPRED(DV_CENTRAL)/pop_scatter.*
      
        Proportional

      - .. thumbnail:: /_autogen/indiv_examples/error_models/pa_tut/images/fit_sim_grph_outputs/RES(DV_CENTRAL)_wrt_IPRED(DV_CENTRAL)/pop_scatter.*
      
        Additive+Proportional
    
In the Additive Noise model, noise has a fixed standard deviation (or variance) that is independent of the predicted value.

In the Proportional Noise model, the level (variance) of noise increases with the magnitude of the predicted variable.

In the Additive+Proportional Noise model, there is an element of both: additive noise that ensures valid likelihoods when the predicted variable is zero; and a proportional component that better reflects a prediction-dependent noise level.


Weighted Residuals (WRES)
---------------------------------

To achieve a more intuitive and easy-to interpret diagnostic, we can scale residuals by the variance we expect given the predicted value.

.. list-table:: Weighted Residuals vs PRED for the additive (left), proportional (centre) and additive+proportional noise models
    :class: image_table

    * - .. thumbnail:: /_autogen/indiv_examples/error_models/ao_tut/images/fit_sim_grph_outputs/WRES(DV_CENTRAL)_wrt_PPRED(DV_CENTRAL)/pop_scatter.*
    
        Additive

      - .. thumbnail:: /_autogen/indiv_examples/error_models/po_tut/images/fit_sim_grph_outputs/WRES(DV_CENTRAL)_wrt_PPRED(DV_CENTRAL)/pop_scatter.*
      
        Proportional

      - .. thumbnail:: /_autogen/indiv_examples/error_models/pa_tut/images/fit_sim_grph_outputs/WRES(DV_CENTRAL)_wrt_PPRED(DV_CENTRAL)/pop_scatter.*
      
        Additive+Proportional

Residuals should have a mean of zero and a variance of one when normalized in this way, and if they do not follow the standard normal distribution this could be an indication that the model is misspecified. (Higher order moments of the distribution such as skew and kurtosis should also be close to their standardized values of zero and three, respectively.)


Conditional Weighted Residuals
---------------------------------
It was noted by Hooker et al. [HookerEtAl2007]_, however, that WRES can sometimes imply that a misspecified model is better than a correctly specified one. This problem arises because (in Nonmem) the WRES are computed from the FO statistics even when FOCE was used to fit the model to the data.

Here we reproduce Hooker's example model in which the response of the disease to the drug is limited through an Emax equation, generating data for a population of 200 individuals with fixed observation time points and inter-subject variability in Emax and C50:

.. literalinclude:: /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/tut_sections/GEN_EFFECTS.pyml
    :language: pyml

In order to compare the input concentration with the Emax response for fixed drug concentrations, we simply treat the observation times as if they were drug concentrations in the PREDICTIONS block and assign the corresponding Emax response to the output column, c[Effect]:

.. literalinclude:: /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/tut_sections/PREDICTIONS.pyml
    :language: pyml

We then apply a |PoPy| fit for two models: one with a correctly specified Hill coefficient of 4.5 (`sum_link_cwres_emax_correct_model_tut`),

.. literalinclude:: /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/tut_sections/FIT_EFFECTS.pyml
    :language: pyml

and one with a misspecified Hill coefficient of 1.0 (`sum_link_cwres_emax_misspecified_model_tut`).

.. literalinclude:: /_autogen/pop_examples/diagnostics/cwres_emax_misspecified_model/tut_sections/FIT_EFFECTS.pyml
    :language: pyml

.. list-table:: Fitted predictions for the correctly specified (left) and misspecified (right) models
    :class: image_table

    * - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/images/fit_grph_outputs/allOBS_vs_TIME/comb_spag.*
    
        Correctly specified model
    
      - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_misspecified_model/images/fit_grph_outputs/allOBS_vs_TIME/comb_spag.*
      
        Misspecified model

The figure above shows the predictions fitted to the data, where it is clear that the correctly specified model has achieved a better fit to the data.

As noted by Hooker, however, the popular WRES diagnostic for the misspecified model has a distribution whose parameters are closer to the standard normal (mean and skew of zero, standard deviation of one, kurtosis of three) compared with the correctly specified model after fitting with FOCE:

.. list-table:: WRES vs. PRED for the correctly specified (left) and misspecified (right) models
    :class: image_table

    * - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/images/fit_grph_outputs/WRES(Effect)_wrt_PPRED(Drug)/pop_scatter.*
    
        Correctly specified model
    
      - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_misspecified_model/images/fit_grph_outputs/WRES(Effect)_wrt_PPRED(Drug)/pop_scatter.*
      
        Misspecified model

To address this anomaly, Hooker suggested a new diagnostic - the Conditional Weighted Residual (CWRES) - that translated and scaled residuals with values more suited to an FOCE-fitted model.

When using this diagnostic rather than WRES, the moments of the error distribution suggest that the correctly specified model is a better fit to the data than the misspecified mode (with the exception of Standard Deviation which is close to one in both cases):

.. list-table:: CWRES vs IPRED for the correctly specified (left) and misspecified (right) models
    :class: image_table

    * - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/images/fit_grph_outputs/CWRES(Effect)_wrt_IPRED(Drug)/pop_scatter.*
    
        Correctly specified model
    
      - .. thumbnail::  /_autogen/pop_examples/diagnostics/cwres_emax_misspecified_model/images/fit_grph_outputs/CWRES(Effect)_wrt_IPRED(Drug)/pop_scatter.*
      
        Misspecified model

We therefore suggest using CWRES rather than WRES for diagnostics by requesting this plot ('{CWRES}_vs_IPRED') in the OUTPUT_SCRIPTS block of the model file:

.. literalinclude:: /_autogen/pop_examples/diagnostics/cwres_emax_correct_model/tut_sections/OUTPUT_SCRIPTS.pyml
    :language: pyml

.. comment 

  VPC corrections
  ==========================

      Bergstrand et al. [Bergstrand2011]_

.. comment

  IEEE citation format:

  Author’s initials. Author’s Surname. (Year, Month). “Title of article.” Journal Title [type of medium]. volume number, issue number, page numbers if given. Available: URL

.. [HookerEtAl2007] Hooker AC, Staatz CE, Karlsson MO. "Conditional weighted residuals (CWRES): a model diagnostic for the FOCE method." *Pharm Res.* 2007 Dec; 24(12):2187-97. doi: 10.1007/s11095-007-9361-x. Epub 2007 Jul 6. PMID: 17612795.