.. _release_notes:


Release Notes
###############

This page lists the various |popy| releases to date:-

|popy| v1.2.4
================
Release date = TBC


|popy| v1.2.3
================
Release date = 2 Apr 2025

Improvements
-------------
* Option to specify the delimiter of the data file (e.g., a semicolon).
* Option to display a progress bar when iterating over individuals.
* Shortcut functions for the PREPROCESS block, making it easier to read Nonmem data files and to debug.
* Access to alternative Levenberg-Marquardt methods (e.g., the Trust Region Reflective algorithm) that may be more stable.
* Templating facility for config files whereby you can use ${ENVVAR} and PoPy will automatically substitute environment variable ENVVAR into the script.
* New "PRED_vs_TIME" and "{PRED}_vs_TIME" plot types that plot only predictions, without observation points.
* Faster "leastsq" r[X] optimization.
* A fail() function to the PREPROCESS to stop PoPy on demand during preprocessing.
* Access to initial states within the DERIVATIVES block via the s0[X] notation.
* Added output of random effect covariances as a correlation matrix.
* The \*\= operator is now accepted in DERIVATIVES.

Bug Fixes
----------
* Initializing a state within a condition without an "else:" clause no longer leaves the state uninitialized (and unrecognized entirely).
* A zero bioavailability no longer causes the code to crash when there are multiple doses.
* The \*\= compound operator is now accepted in the DERIVATIVES for a single compartment (but not for directed flows between two compartments).
* TABLE now has access to the c[X] input data and supports the PREPROCESS block so that non-standard input data doesn't fail.
* FITSUM now processes non-standard input data without needing a PREPROCESS block.
* Expressions evaluating to log(0) now give an error to the user rather than generating unrunnable code.
* Bug in Solution file's "fx_opt_types_path" corrected so Solutions can be copied directly between folders.
* Closing the console window in Windows 7 now forcibly terminates all processes in the current PoPy run.
* You can now have an empty list for some covariates (e.g., t[DOSE]).
* Pseudocode that evaluates to log(0) now raises an exception rather then generating unrunnable code.
* Data files with "." or "-" as aliases for zero are now converted much earlier to avoid problems downstream.
* PREPROCESS is now propagated to all scripts where it might be necessary/useful.

|popy| v1.2.2
================
Release date = 21 Feb 2024

Improvements
-------------
* Better checking of scripts and reporting of errors found during checking.
* Better handling of long file names/paths.
* Better error reporting when trying to write to a file that is open elsewhere (e.g., Excel).

Bug Fixes
----------
* Correct initialization of some random variables (e.g., Bernoulli)
* Copied DATA_FIELDS to solutions such that non-standard columns are recognized rather than looking for default column names.
* Passed PREPROCESS code to Grph scripts so that preprocessed input observations/covariates are loaded correctly.
* Removed erroneous point at dose events in spaghetti plots.
* Corrected the way that string columns are processed in auto-generated code so that conditions on string fields are evaluated correctly.

|popy| v1.2.1
================
Release date = 15 Dec 2023

Improvements
-------------
* Simplified the launch interface - e.g., to "popy run ..." rather than "popy_run" - to reduce the number of additional scripts bundled with PoPy.
* Retired the "imconv" and "edit" tools, trusting the user to use their preferred tools for image conversion and text file editing.
* Retired the CPPODE solver in favour of the superior CPPLSODA solver.

Bug Fixes
----------
* Added measures to make PoPy more tolerant of long path names that exceed the Windows limit of 260 characters.
* Fixed errors when a single epoch contains more than one occasion without a reset separating occasions.

|popy| v1.2.0
================
Release date = 6 Nov 2023

Improvements
-------------
* Scatter plots implemented for rapid diagnostics of model fits.
* Tabular outputs now available (similar to Nonmem's $TABLE directive).
* Conditional Weighted Residuals added as a diagnostics measure.
* Sequential variables can now be defined as an arbitrary list.
* Likelihoods exported on a per-row basis for detailed analysis.
* Improvements to efficiency and reliability when using parallel processing.
* Simplifications to the output data structures, with fewer subfolders and better consistency in file locations.
* Added option to specify parallelism by % of CPU load rather than an absolute number of processors.

Bug Fixes
----------
* Random effects with a fixed variance of zero are now correctly treated as constants, and any associated fixed effects treated as unassociated fixed effects.
* Tabulated fixed effects were previously incorrect at times, but are now outputted correctly.
* Fixed an indexing bug when updating individuals' parameter values.
* Clearer numbering of populations in multi-population simulations to avoid confusion by both users and computers.
* Fixed bug in normal distribution objects after reseeding with a known value.
* Corrected the way that flows of the form "d[X] -= x + y" are evaluated.
* Improved the way variable substitutions are handled in the PREDICTIONS block.
* Fixed bugs arising when trying to write files to folders that don't (yet) exist.
* Corrected the way that analytic solvers deal with parameter sensitivity.

|popy| v1.1.2
================
Release date = 11 Mar 2022

Improvements
-------------
* Faster processing for models with lots (tens of thousands) of observation rows that have a normally distributed error model.
* Parameterized compartment definitions so that many compartments can be defined in a single line.
* Better feedback when calculations result in a not-a-number (NaN) that derails the fitting process.
* Assumes that any uninitialized d[X] are initialized to zero, so you can add to flows without having to explicitly initialize them first.
* Faster evaluation of normally-distributed likelihoods.
* Extended sequential() distribution now allows arbitrary splitting of tree levels, e.g., c[AMT] = sequential([0, 100, 200]).
* Better code parsing flags up errors in the script earlier in the process.

Bug Fixes
----------
* Covariance matrix (OMEGA) values that were defined as being constant were being modified by the JOE fitter.
* Models containing fixed effects that are constant would sometimes crash due to an indexing bug in the subset of all fixed effects.
* Incorrect calculation in ds/dm when using mixed models (with both a closed-form compartmental model and ODEs).
* IMP under parallel processing tried to recompute proposal distributions on every process rather than just on the master process.
* IMP used an approximate Lambda when it should have been using an exact one (and vice versa).
* Parsing the PREPROCESS block would sometimes alter its contents, causing faults downstream.
* Failure due to long filenames avoided by using symbolic links and raising warnings rather than errors under recoverable conditions.
* Covariances (OMEGAs) were being updated when they were defined as constants in some methods (e.g. JOE).
* Constant fixed effects were causing indexing problems in some fitting algorithms.
* Output folders created whenever they are needed (which wasn't always the case).
* Sim failed when a solution file had a different DATA_FIELDS content from the corresponding Gen file.
* Fixed effects "associated" with an r[X] that is effectively constant (e.g., mean=0 and variance=0) were not being updated at all, rather than being updated using alternative (e.g., gradient-based) methods.
* Incorrect covariance matrix when the model included no random effects.
* Tabulated f[X] values were all the same.
* Tutsum failed to copy the compartment diagram from Gensum.

|popy| v1.1.1
================
Release date = 22 Oct 2021

Improvements
-------------
* More stable estimation when using forward sensitivity equations.
* Improvements to VPC generation
* Some improvements in speed during fitting, especially for situations where there are lots of bolus doses per subject.
* Better memory management when fitting in parallel.
* Quotes in column headers of the data file are now permitted.

Bug Fixes
----------
* Fixed various bugs with fitting in parallel

|popy| v1.1.0
================
Release date = 22 Sep 2021

Improvements
-------------
* Reduced RAM usage when running |popy| in parallel using |mpi|. Previous version shared all subject data on all processors. New parallel code spreads subject data across all processors and avoids unnecessary duplication. Improves performance on large multi-core machines.
* Increased number of parameters that the |nd| fitting method and |bobyqa| optimiser can process, previously restricted to 35, now increased to 200.
* Change the colour of the dosing lines in individual subject plots to yellow (formerly black), this enables more informative graphs for subjects who have a lot of separate doses.
* Improved the speed and accuracy of the |rx| optimisation using more analytic gradient results.
* Added ability to create prediction corrected (pred_corr) and prediction variance corrected (predvar_corr) |vpc| output via the "norm_method" parameter of a |vpc_script|. 
* Added ability to create stratified |vpc| output via the "split_field" parameter of a |vpc_script|. A separate vpc is output for each unique value in a specified |cx| data variable.
* Made the |msim_script| and |vpc_script| code more RAM efficient.
* Made models with a large number of dosing events per subject run more efficiently.

Bug Fixes
----------
* Fixed bug when passing string data into |cython| functions.

|popy| v1.0.5
================
Release date = 13 May 2021

Improvements
-------------
* Added new |nd| fitting method, the latest and most robust |popy| estimation approach.
* Improved fitting and lower |objv| using |nd| method.
* Upgraded base |python| distribution of |popy| from version 3.5 to version 3.8. This allows us access to more up-to-date |python| packages and functionality.
* Improved automatic compartment diagram generation with "in" and "out" nodes.
* Improved automatic checking of input scripts.

Bug Fixes
----------
* Improvements in error catching and reporting when running |popy| in parallel using |mpi|.
* Fixed bug when using backslashes in |derivatives| section.

|popy| v1.0.4
================
Release date = 23 Dec 2020

Improvements
-------------
* Added |cpplsoda| |cpp| based implementation of |lsoda| |ode_solver|, which runs faster than |scipy_odeint| within |popy| and returns similar results.
* Improved |parallel| performance of |foce| algorithm.
* Improved fitting of |foce| method to achieve lower |objv| in many models.
* Revised documentation.

Bug Fixes
----------
* Bugs in gradient computation due to variable |cx| within individual data fixed.
* Fixed bugs in :ref:`standard error<uncertainty>` computation, now more reliable.

|popy| v1.0.3
================
Release date = 05 Sep 2020

Improvements
-------------
* Added implementation of an :ref:`fit_methods_foce` to complement the previous :ref:`fit_methods_joe`. Note we now recommend running the |joe| fitter, then the |foce| fitter in serial to get the best parameter fitting results in |popy|.
* New |rectnorm_dist| for fitting to |blq| data, using |joe| or |foce| methods without using the more computationally expensive |laplace| objective value. See :ref:`blq_example`. 
* Also new |blq| related |cennorm_dist| and |truncnorm_dist|.
* New |bin_dist| likelihood.
* Improved the speed and quality of the noise |fx| parameter estimates for |joe| fitting.
* New plots show the final |fx| population fit (same for all individuals, with |rx| zero) and the individual fit (|ie| empirical bayes estimates for |rx| optimised for each individual). Previously we plotted only the initial unoptimised |fx| fit and the final |fx| individual fits.
* The temporary functions generated by |popy| (based on the config file) are now written into a single module on disk. This single module compiles quicker using |cython|, compared to having each temporary function in it's own module.
* Added |preprocess| section to multiple scripts, for example a :ref:`fit_script`. This allows the user to easily filter and alter the data set, using simple |python| code, before fitting a model, without changing the |data_file| on disk.
* Added |postprocess| section for multiple scripts, for example a :ref:`gen_script`. This allows the user to easily alter or filter a synthetically generated data set, before saving to disk.

Bug Fixes
----------
* Fixed bug in |predictions| section that did not handle multiple observations correctly in some instances, when some rows should be ignored due to |python| 'if' statements.
* Fixed interpretation of '+inf' and '-inf' labels in |effects|.

|popy| v1.0.2
================
Release date = 18 Sep 2019

Improvements
-------------
* Added ability to process scripts in :ref:`parallel`. This speeds up computation time for models with a large number of individuals and takes advantage of multi-core computers.
* Created :ref:`postprocess` section to allow filtering of data create by a |popy| model. For example with a :ref:`gen_script`
* Added ability to process multiple files in a directory with one command using '*' syntax, see :ref:`popy_run_multi_scripts`, :ref:`popy_check_multi_scripts`, :ref:`popy_format_multi_scripts`, :ref:`popy_edit_multi_scripts`,  :ref:`popy_view_multi_html`
* Speed improvements to core |joe| algorithm, :ref:`standard_errors` computation and plotting using :ref:`grph_script`.
* :ref:`mfit_script` now outputs a summary of the fitting results for all populations to a single .csv file.  
* Displaying the diagonal elements of |fx| variance matrices at each iteration. The off diagonal |fx| elements are only displayed at the end of fitting, due to space considerations.
* Simplified the structure of old 'LEVEL_PARAMS', replaced with :ref:`effects` which removes the redundant 'split_field' and 'split_dict' attributes.
* Renamed the 'output_mode' values in :ref:`output_scripts`, using shorter names 'gen_script_and_run' -> 'run', 'gen_script_only' -> 'create', 'no_output' -> 'none'.
* Renamed the 'FINITE_DIFF' method in :ref:`covariance` to 'SANDWICH'.

Note scripts in v1.0.1 and v1.0.0 format can be automatically updated to v1.0.2 using the :ref:`popy_format` tool.

Bug Fixes
----------
* Made :ref:`method_options_float_format_spec` change format of all float values in output scripts and summary html outputs. Only applied to subset before.
* Fixed bug in iteration number when displaying |objv| over time :ref:`obj_vs_time_fit_example1`

|popy| v1.0.1
================
Release date = 28 Jun 2019

This is mainly a bug fix release of |popy| 

Improvements
-------------
* The :ref:`standard_errors` are now output in an easy to read format, instead of the whole hessian matrix.
* Changed the configuration file entry required to compute standard errors see |covariance|.
* Log files apart from the main log file are now stored in a '_log' sub directory. 
* The :ref:`popy_run` tool now asks the user if they want to proceed if the main log file already exist for a given script. Previously only the existence of the output folder was checked. The can still skip this check using the '-o' option.

Bug Fixes
----------
* Licence manager was giving a date error in North American time zones.
* Bug in configuration file parser for |derivatives| section when encountering round brackets (as opposed to expected curly brackets) in :ref:`dosing_functions`. 
* Documentation contents page indexing in PDF was incorrect.

|popy| v1.0.0
===============
Release date = 24 May 2019

This is the first version of |popy|, so there are no bug fixes or new features
only the initial implementation.

Or in other words, all the features and bugs are new!