.. _builtin_mtut_example:


Generate multiple data sets and Fit using a Two Compartment Model
###################################################################

The previous example showed how to :ref:`builtin_tut_example`, which generates data from a model then fits the model to the synthetic data.

In |popy| it is also possible to iterate over a :ref:`tut_script`, using what we call a :ref:`mtut_script` or multi-tutorial script and apply the generate/fit process multiple times. In this example we will demonstrate a :ref:`mtut_script` using the same two compartment model with absorption and bolus dosing, as used in the Complex |poppk| :ref:`tut_script` example, see :numref:`fig_two_comp_depot_diagram_mtut`:-

.. _fig_two_comp_depot_diagram_mtut:

.. figure:: /_autogen/quick_start/builtin_tut_example/compartment_diagram.*
    :width: 80%
    :align: center
    
    Two compartment model with depot dosing for :ref:`mtut_script`.
     
An :ref:`mtut_script` is a tool for investigating identifiability of |pkpd| models, but gives you more information than a :ref:`tut_script` because the synthetic population data is sampled multiple times, which gives a spread of fitted |fx| results for each model parameter.

.. _running_a_mtut_script:
 
Running the MTut Script
============================

.. 
    duplication of text w.r.t. docs for mtut_example1

This multi tutorial example make use of the file ``c:\PoPy\examples\builtin_mtut_example.pyml``.
                            
:ref:`open_a_popy_command_prompt` to setup the |popy| environment in the folder ``c:\PoPy\examples\`` and, with the |popy| environment enabled, 
call :ref:`popy_run` on the :ref:`mtut_script` from the command line:-

.. code-block:: console

    popy run builtin_mtut_example.pyml
    
Running an mtut script can take a considerable amount of time, as it is equivalent to running a :ref:`tut_script` multiple times.
However in this toy example we only run the fit/gen cycle 30 times and limit the |fx| parameters that are estimated.

.. _syntax_mtut_script_builtin:

Syntax of MTut Script
========================

The :ref:`mtut_script` is very similar to the :ref:`tut_script` in :ref:`syntax_tut_script_builtin`, the primary difference is that the :ref:`mtut_script` specifies the number of populations to sample as follows:-

.. code-block:: pyml

    OUTPUT_OPTIONS: {n_pop_samples: 30}

Like a :ref:`tut_script`, the :ref:`mtut_script` encodes **both** the generation and fitting effects in the |gen_effects| and |fit_effects| sections. In this example the |gen_effects| and |fit_effects| sections are slightly different from :ref:`syntax_tut_script_builtin`, as follows:-

.. literalinclude:: builtin_mtut_gen_effects.pyml
    :language: pyml    

.. literalinclude:: builtin_mtut_fit_effects.pyml
    :language: pyml     
    
The GEN_EFFECTS->POP level defines the following |fx|:-

.. code-block:: pyml

    f[KA] ~ unif(0.1,0.3)
    f[CL] ~ unif(7.0, 13.0)
    f[V1] ~ unif(30.0,70.0)
    f[Q] ~ unif(0.5,1.5)
    f[V2] = 80
    f[PNOISE] ~ unif(0.1, 0.2)
    
This means that when running the :ref:`mgen_script` the |fka|, |fcl|, |fv1|, |fq| and |fpnoise| are sampled 30 times to create 30 different synthetic data sets with different true |fx| values. The |fv2| parameters in contrast always has the constant value 80. Note the |fx_isv_mat_builtin| matrix is also constant here.
    
Whereas the FIT_EFFECTS->POP section tries to estimate the following |fx|:-

.. code-block:: pyml

    f[KA] ~ P0.2
    f[CL] ~ P10.0
    f[V1] ~ P50.0
    f[Q] ~ P1.0
    f[V2] = 80
    f[PNOISE] ~ P0.15
    
These entries mean that when running the :ref:`mfit_script` the |fka|, |fcl|, |fv1|, |fq| and |fpnoise| are estimated for each of the 30 synthetic data sets created by the :ref:`mgen_script`. The starting values for the parameter fitting are here set to the middle of the sampling region used in the **gen_params**. For example |fka| has initial value 0.2, whereas the true value lies in the region [0.1,0.3] for each synthetic data set. |fv2| is not estimated at all, it is merely set to the true value |ie| 80. Similarly |fx_isv_mat_builtin| is set to the true constant value here. The 'P' in the **fit_params** forces the fitting process to estimate positive values.

Restricting the estimation to 5 |fx| parameters greatly speeds up the run time of :ref:`mfit_script`, which makes this toy example much faster to run. The :ref:`mtut_script` is only sampling 30 synthetic data sets, which is a relatively small number. You could increase this if you wish to obtain more detailed scatter plots (see below). Most of the run time is taken up by :ref:`mfit_script`, the :ref:`mgen_script` is relatively quick to run, as it only has to evaluate the |odes| in the |derivatives| block once per synthetic data set, where as the :ref:`mfit_script` must evaluate the |odes| multiple times to estimate |fx| for each synthetic data set. 

.. _summary_of_builtin_mtut_results:

Summary of MTut Results
=============================

The :ref:`mtut_script` should generate an output folder containing three new scripts:-

.. code-block::

    builtin_mtut_example.pyml_output/
        builtin_mtut_example_mgen.pyml
        builtin_mtut_example_mfit.pyml
        builtin_mtut_example_mcomp.pyml
                                              
The purpose of each of theses scripts is as follows:-

.. _table_mtut_output_scripts:

.. list-table:: Scripts output by a multi tutorial script
    :header-rows: 1
    
    * - Script
      - Purpose
      - Documentation

    * - \*_mgen.pyml
      - Generate multiple synthetic data sets from model
      - :ref:`mgen_script`

    * - \*_mfit.pyml
      - Fit model to multiple synthetic data sets
      - :ref:`mfit_script`

    * - \*_mcomp.pyml
      - Compare gen model and fit model |fx|
      - :ref:`mcomp_script`
 
The :ref:`mgen_script` is very similar to the :ref:`gen_script` described in :ref:`builtin_gen_example` and the :ref:`mfit_script` is very similar to the :ref:`fit_script` described in :ref:`builtin_fit_example`. See :ref:`files_generated_by_mtut_script` for more info. 
   
Therefore here we mainly discuss the :ref:`mcomp_script` outputs, which processes the results of :ref:`mgen_script` and :ref:`mfit_script`. The simplest output is a visual comparison of the true and fitted |fx| values as shown in :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl` and :numref:`table_true_vs_mfit_fx_plots_builtin_f_q1_f_v1_f_pnoise`.

.. _table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl:

.. list-table:: Fitted model vs true scatter plots for |fka| and |fcl|
    :class: image_table
    
    * - .. thumbnail:: /_autogen/quick_start/builtin_mtut_example/fx_scatter/fitted_vs_true_for_f_ka_.*
      - .. thumbnail:: /_autogen/quick_start/builtin_mtut_example/fx_scatter/fitted_vs_true_for_f_cl_.*
      
In :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl` the blue dots are a scatter plot of fitted |fx| vs true |fx|. The green dots are initial |fx| vs true |fx|. For example in the case of |fcl| the true values are sampled as follows:-

.. code-block:: pyml

    f[CL] ~ unif(7.0, 13.0)
    
|ie| the true values are uniformly sampled in the range [7.0,13.0]. The fitted |fcl| parameters are specified as follows:-
    
.. code-block:: pyml

    f[CL] ~ P10.0
 
The initial values for |fcl| are always 10.0. see green dots in a horizontal line on the right graph in :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl`. The 'P' specifies that the fitting value of |fcl| is restricted to positive numbers. The final fitting values are the blue dots on the right graph in :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl`. For |fcl| the blue dots are clustered along the black 45 degree line. Hence fitting for |fcl| works well, this agrees with the initial findings in :ref:`builtin_fit_example`. The |fka| fitted values (blue dots) on the left graph in :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl` are less obviously grouped around the 45 degree line. Especially the outlier when the fitted value is estimated at 0.48 compared to a true value of 0.21.
      
.. _table_true_vs_mfit_fx_plots_builtin_f_q1_f_v1_f_pnoise:

.. list-table:: Fitted model vs true scatter plots for |fq| and |fv1| and |fpnoise|
    :class: image_table
    
    * - .. thumbnail:: /_autogen/quick_start/builtin_mtut_example/fx_scatter/fitted_vs_true_for_f_q_.*
      - .. thumbnail:: /_autogen/quick_start/builtin_mtut_example/fx_scatter/fitted_vs_true_for_f_v1_.*
      - .. thumbnail:: /_autogen/quick_start/builtin_mtut_example/fx_scatter/fitted_vs_true_for_f_pnoise_.*
     
In :numref:`table_true_vs_mfit_fx_plots_builtin_f_q1_f_v1_f_pnoise` (left graph) the |fq| rate results for the |peripheral| compartment are reasonably well estimated, apart from one outlier. The |fv1| parameter (middle graph) is really badly estimated, there does |not| seem to any correlation between true |fv1| values and fitted |fv1| values at all. The |fpnoise| parameter (right graph) is reasonably well estimated, with the results mostly lying along the 45 degree line.

In this case, the |fv1| parameter (the |vol| of the |central| compartment) is shown to be hard to estimate, compared to |fcl| and |fq| |clears|. The |fka| estimate is a bit unstable, possibly due to this parameter requiring time point data shortly after the dose, which does not always exist. |fpnoise| is reasonably well estimated, but it's likely that higher values of |fpnoise| make the other parameters harder to identify. 

.. comment
    See :ref:`identifiability` for more discussion.

Given the relatively small amount of data generated (50 data points spread across 10 individuals), the results in :numref:`table_true_vs_mfit_fx_plots_builtin_f_ka_f_cl` and :numref:`table_true_vs_mfit_fx_plots_builtin_f_q1_f_v1_f_pnoise` are reasonably good.


.. _rerun_mtut:
      
Re-run the MTut Script
========================

You can familiarise yourself with |popy|'s various features by tweaking the multi-tutorial example and re-running. 
A simple way of avoiding overwriting previous results is take a copy:-

.. code-block:: console
    
    copy builtin_mtut_example.pyml builtin_mtut_example_v2.pyml

and modifying it using your chosen text editor. When you are happy with the edited file, do:-

.. code-block:: console

    popy run builtin_mtut_example_v2.pyml

to run the script.

For example, you can adjust the amount of data generated, in this section:-

.. code-block:: pyml

    EFFECTS:
        ID: |
            c[ID] = sequential(10)
            t[DOSE] = 2.0
            t[OBS] ~ unif(1.0, 50.0; 5)
            # t[OBS] = range(1.0, 50.0; 5)
                
Note the 'range' function can sample time points evenly (instead of randomly). This usually makes the model fitting easier as the data points span the time range better. You can experiment with the number of doses or make a random sample of dose times for each individual.

You could also change the initial fitting |fx| values, to make the |fx| estimation process start further away from the true |fx| values. You could also attempt to estimate more (or fewer) parameters, for example try estimating the |fx_isv_mat_builtin| covariance matrix.

You can also edit the underlying model or compartment structure, see the |model_params| or |derivatives| sections.

.. _convert_tut_to_mtut_script: 
      
Convert Tut to MTut 
======================

A :ref:`mtut_script` is essentially a :ref:`tut_script` with an extra loop. Therefore it's fairly simple to convert an existing tutorial script to a multi tutorial script. First change the script type from tut->mtut:-
    
.. code-block:: pyml

    METHOD_OPTIONS: {py_module: mtut}
    
Add the mtut |output_options| section:-
    
.. code-block:: pyml

    OUTPUT_OPTIONS: {n_pop_samples: 30}
    
Change the |output_scripts| section to this:-
    
.. code-block:: pyml

    OUTPUT_SCRIPTS: 
        MGEN: {output_mode: run}
        MFIT: {output_mode: run}
        MCOMP: {output_mode: run, dot_size: 12} 
    
You should then be able to run your new mtut script.


