Release v0.4.2 #1298
Merged
Release v0.4.2 #1298
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
PaulJonasJost
commented
Jan 30, 2024
•
PaulJonasJost
commented
- General
- Stabilize tests (Select: reduce test failure frequency #1240, Fix test.util.load_amici_objective / AmiciVersionError #1254)
- Update type annotations and documentations (Doc: Update/add/fix type annotations in ensemble+predict #1239, Doc: Fix links, typos, grammer, punctuation in doc/example/amici.ipynb #1248, Doc: Prettify petab.select docs #1255, Doc: Update SacessOptimizer #1258, Doc: enable linking to matplotlib & petab-select docs #1251, Doc: Update sphinx/rtd config to enable re-executing notebooks #1268, Doc: Clear amici and conversion reaction notebook output #1275)
- GHA/Codeowner changes (GHA: Clear PR cache after merge #1260, GHA: Reduce cache size #1261, GHA: Set mpl backend #1259, GHA: Fix install_deps.sh #1262, Set only active maintainers as default codeowners #1285)
- Update utility functions (delete_nan_inf can handle list input type #1243)
- Refactor progress bars (Refactor progress bars #1272)
- Clear Notebook output(Execute all notebooks / clear notebook output #1246, Doc: Use clang on RTD #1277, Doc: Clear hierarchical optimization notebook output #1274, Doc: Clear petab notebook output #1271, Doc: Clear getting-started notebook output #1276, Doc: Clear notebook output (fixed parameters, prior definition, store) #1278,)
- Optimize
- (Sac)ESSOptimizer: History of best objective values ( (Sac)ESSOptimizer: Create history of best objective values #1212)
- Fix missing fixed parameters in scatter search results (Fix missing fixed parameters in scatter search results #1265)
- Fix TypeError in pypesto.result.optimize.OptimizerResult.summary if x0 is None (Fix potential TypeError in pypesto.result.optimize.OptimizerResult.summary if x0 is None #1266)
- ESSOptimizer: Include results for local searches in OptimizeResult (ESSOptimizer: Include results for local searches in OptimizeResult #1270)
- New Feature: Spline Approximation (Feature spline regularization #1222)
- Select
- Allow for hierarchical problems (Add option for hierarchical models #1241)
- custom minimize method (Select: custom minimize method #1264)
- Hierarchical
- Log space startpoint sampling (Hierarchical log space startpoint sampling #1242)
- Support for box constraints on offset and scaling parameters (Hierarchical optimization: Support for box constraints on offset and scaling parameters #1238)
- restructuring and add relative to InnerCalculatorCollector (Hierarchical: restructuring and add relative to InnerCalculatorCollector #1245)
- History
- Support pathlib.Path for result/history files (Support pathlib.Path for result/history files #1247)
- Extended Amici history (Extended Amici history #1263)
- Visualize
- Fix time trajectories for hierarchical problems (Fix time trajectory simulation for hierarchical solver #1213)
- Fix hierarchical parameter plotting for all optimizers (Fix hierarchical parameter plotting for all optimizers #1244)
- Sacess history plot (Vis: Sacess history plot #1250)
- Objective
- Fix PEtab.jl version to before 2.5.0 (temporarily) (PEtab.jl makes problems since 2.5.1 #1256)
- PEtab
- Enable Importer passing verbose to create_model (petab: Enable Importer passing verbose to create_model #1269)
- PetabImporter: version-specific amici model directories (PetabImporter: version-specific amici model directories #1283)
- Problem
- Problem: add inner problem names, bounds and hierarchical flag (Problem: add inner problem names, bounds and hierarchical flag #1282)
- Use warnings.warn instead of logging.warn when loading Problem from HDF5 without an Objective (Use warnings.warn instead of logging.warn when loading Problem from H… #1253)
- Ensemble
- EnsemblePrediction: remove "no predictor" warning (EnsemblePrediction: remove "no predictor" warning #1293)
Store traces of the best values/parameters for the ESSOptimizer and the best values of each worker for the SacessOptimizer.
Adding type annotations, fixing some typos. Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>
* Initial, missing regularization grad Initial commit with spline regularization. The implementation of the gradient wrt theta is not completed (swapping to develop for some debugging of censored). Additionally, small fixes of censored gradient (sigma related, not important usually). * Complete gradient Implemented the complete gradient, including the regularization term for the ds_dtheta * Fix par plot, add reg to spline plot FIxed the parameter plot issue (will be in a separate PR). Added the linear regularization to the spline visualization. * Some random changes TODO Changes that I might have to revert for the Pull request. The ensemble one might've already been pushed by Polina. The solver one completely circumvents dsdtheta gradient calculation due to some "array cannot be inf or NaN" error which pops sometimes, but it's mostly always 0 so removing it makes it easier to deal with for now. This is a big TODO, do not forget. * Fix par_sim_idx in spline solver Fixed the par_sim_idx in the spline solver calculate gradient function. For all conditions, the rdata form is not managed by par_sim_idx, but with par_edata_indices. This is done such that if some parameters are not used in the condition (observable or noise parameters for instance) then the sensitivities wrt. them don't have to be calculated * Fix par_sim_index, pass edata_indices Added the passing of the edata indices from the calculator to the solver gradient calculation. Additionally, fixed the inner calculator collector quantitative calculation with nansum instead of a regular one (sometimes it gave only nan values because of this...) * Remove ds_dtheta_term calculation Since I've proven that the ds_dtheta gradient contribution is always 0, we can remove all of the code that was calculating it * Remove redundancy * Small cleanup * Remove a mistake * Add regularization test * Quality test fix * Spline tests fix + obj fun fix * Notebook update - Updated the notebook for non-linear semi-quantitative data - fixed the max_sensi_order problem - changed default min_diff_factor - fix quality * Daniel review changes * Dilan&Fabian review changes * Improve test coverage Added tests for untested lines in calculators --------- Co-authored-by: Stephan Grein <stephan.grein@uni-bonn.de>
* Update misc.py Add option for hierarchical models in model selection * Update misc.py Fix typo * Update misc.py import pandas * Update pypesto/select/misc.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update pypesto/select/misc.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update pypesto/select/misc.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update pypesto/select/misc.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update pypesto/select/misc.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update misc.py * Update misc.py * Update misc.py --------- Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>
* test * test2-changed * Add simulation_edatas argument into the AmiciCalculator and Objective * Implement set_simulation_edatas for the hierarchical calculator * Update test for simulation edatas. * Update set simulation edatas. * Fix additional argument inserted before. * Enable check of the simulation_edatas in the objective call. * Update pypesto/hierarchical/problem.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Update pypesto/hierarchical/calculator.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Fix type checking * Undo changes in objective and calculator. Instead fix it temporarily in the visualization routine. * Update model_fit.py Disable sensitivities to improve computation time. * Outsource definition of simulation rdatas to a separate helper function. * Update comments. * Move helper function into model_fit script. --------- Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com>
* log space startpoint sampling * Some updates * Update pypesto/hierarchical/solver.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Update pypesto/hierarchical/solver.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Update solver.py * Update pypesto/hierarchical/solver.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> * Update pypesto/hierarchical/solver.py Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> --------- Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>
…scaling parameters (#1238) * Initial working Implemented scaling and offset bounds for the analytical and numerical solver of the hierarchical problem. * Fix hierarchical test * Add test_constrained_inner_solver * Add tests for non coupled parameters * intentional failure * intentional failure * intentional failure * Test fix It seems L-BFGS-B would not converge to optimum in some cases if it was provided with the big bounds [-1e20, 1e20]. It seems like it is a scipy issue? Not sure. In any case, the numerical optimizer is better if the optimization is not provided with these dummy bounds. But in this case, we need to sample the bounds outside of the pypesto minimize function and provide it as x_guesses. * Add notebook changes * Update C.py * Revert "Update C.py" This reverts commit f099a2b. * Fix sampling + test * Cleanup * Some updates * Some more small updates * Add inner bounds to parameter plot * Fix import issues * Fix hier test * Fix parameter plot test * Remove figures * DIlan&Daniel review changes * Update parameters.py * Update solver.py
Fix return type documentation, cross references, typo, deprecated type annotations.
* Doc: enable linking to matplotlib docs * petab-select * link
#1248) * Doc: Fix links, typos, grammer, punctuation in doc/example/amici.ipynb * doc/example/petab_import.ipynb * doc/example/getting_started.ipynb
So far, after updating amici, test.util.load_amici_objective would fail with AmiciVersionError. In that case, we just want to re-import the model. That's what's achieved here.
* Hier. par. plotting for fides * hierarchical 'hess' bug * Fix storing of inner parameters in result Inner parameters were not being stored when the result was stored. This was due to a problem with putting a dictionary into an HDF5 format. I've transformed it into 2 lists now: a INNER_PARAMETERS_VALUES and INNER_PARAMETER_NAMES list. * adjusted documentation of hierarichal opt. * Fix some type issues * Small fix * Save only values, not names * Extend to all optimizers * Remove redundant * Small update * Update parameters.py * Small typo * Fix some bugs, use lists for saving instead * Fix tests * Update hierarchical parameter plot test * Daniel&Dilan review changes * Small change * More Dilan Review changes * Fix testing * Fix base test * Move inner pars to decorator * Collect decorators for minimize * Maren review changes --------- Co-authored-by: Lea@Mac <leaseep@gmail.com>
Might resolve frequent cache misses for Julia tests. Co-authored-by: Doresic <85789271+Doresic@users.noreply.github.com>
We have limit cache size. Only cache what's really relevant.
Avoid such issues:
```
fig = plt.figure()
.tox\windows\Lib\site-packages\matplotlib\pyplot.py:934: in figure
manager = new_figure_manager(
.tox\windows\Lib\site-packages\matplotlib\pyplot.py:465: in new_figure_manager
return _get_backend_mod().new_figure_manager(*args, **kwargs)
.tox\windows\Lib\site-packages\matplotlib\backend_bases.py:3402: in new_figure_manager
return cls.new_figure_manager_given_figure(num, fig)
.tox\windows\Lib\site-packages\matplotlib\backend_bases.py:3407: in new_figure_manager_given_figure
return cls.FigureCanvas.new_manager(figure, num)
.tox\windows\Lib\site-packages\matplotlib\backend_bases.py:1785: in new_manager
return cls.manager_class.create_with_canvas(cls, figure, num)
.tox\windows\Lib\site-packages\matplotlib\backends\_backend_tk.py:486: in create_with_canvas
window = tk.Tk(className="matplotlib")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tkinter.Tk object .>, screenName = None, baseName = 'pytest.EXE'
className = 'matplotlib', useTk = True, sync = False, use = None
def __init__(self, screenName=None, baseName=None, className='Tk',
useTk=True, sync=False, use=None):
"""Return a new top level widget on screen SCREENNAME. A new Tcl interpreter will
be created. BASENAME will be used for the identification of the profile file (see
readprofile).
It is constructed from sys.argv[0] without extensions if None is given. CLASSNAME
is the name of the widget class."""
self.master = None
self.children = {}
self._tkloaded = False
# to avoid recursions in the getattr code in case of failure, we
# ensure that self.tk is always _something_.
self.tk = None
if baseName is None:
import os
baseName = os.path.basename(sys.argv[0])
baseName, ext = os.path.splitext(baseName)
if ext not in ('.py', '.pyc'):
baseName = baseName + ext
interactive = False
> self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
E _tkinter.TclError: Can't find a usable tk.tcl in the following directories:
E C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tcl8.6/tk8.6 C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6 C:/hostedtoolcache/windows/Python/3.11.7/lib/tk8.6 C:/hostedtoolcache/windows/Python/3.11.7/lib/tk8.6 C:/hostedtoolcache/windows/Python/lib/tk8.6 C:/hostedtoolcache/windows/Python/3.11.7/library
E
E C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/tk.tcl: couldn't read file "C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/ttk/scrollbar.tcl": no such file or directory
E couldn't read file "C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/ttk/scrollbar.tcl": no such file or directory
E while executing
E "source -encoding utf-8 [file join $::ttk::library scrollbar.tcl]"
E (file "C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/ttk/ttk.tcl" line 102)
E invoked from within
E "source -encoding utf-8 C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/ttk/ttk.tcl"
E ("uplevel" body line 1)
E invoked from within
E "uplevel \#0 [list source -encoding utf-8 $::ttk::library/ttk.tcl]"
E (file "C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/tk.tcl" line 706)
E invoked from within
E "source C:/hostedtoolcache/windows/Python/3.11.7/x64/tcl/tk8.6/tk.tcl"
E ("uplevel" body line 1)
E invoked from within
E "uplevel #0 [list source $file]"
E
E
E This probably means that tk wasn't installed properly.
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\tkinter\__init__.py:2326: TclError
```
e.g. here https://github.com/ICB-DCM/pyPESTO/actions/runs/7261863194/job/19783894242?pr=1243
* delete_nan_inf can handle list input type * handle case of single opt result * get values as list per start * move delete nan inf to outer parameters function * update tests --------- Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com>
Fixes `.github/workflows/install_deps.sh: 13: [: Linux: unexpected operator` https://github.com/ICB-DCM/pyPESTO/actions/runs/7265048086/job/19793906617
So far, the OptimizerResults from Ess/Cess/SacessOptimizer (incorrectly) contained only the estimated parameter values. Now, fixed parameters are included as well, as done for the other Optimizers.
…0 is None (#1266) Previously, this would fail if `full=False` and `x0=None` (`TypeError: 'NoneType' object is not subscriptable`).
Co-authored-by: Daniel Weindl <daniel.weindl@helmholtz-munich.de> Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com>
Prerequisite for #1246: * Ubuntu 22.04 on RTD * Install all relevant deps on RTD * GHA: More time for documentation job
Added `pypesto.visualize.optimizer_history.sacess_history` to visualize SacessOptimizer histories. In principle, they could be plotted the same as any other history, but here a step plot is usually preferable, and we show the global best in addition to all workers.
* Enable create_objective to pass verbose to create_model * Add verbose as argument Added verbose as argument in all importer method which use `create_model` and passed it onto it.
tqdm progress bars are used in a couple of places. They don't play so well in non-interactive jobs, testing, ... . They also cause trouble with nbspinx (#1246, #1271). Progress bars can be disabled for specific tasks, but not globally (or at least not very conveniently). Since recently, tqdm can be controlled via environment variables (e.g., disabling all progress bars or changing update frequency). However, this works by changing the argument defaults, so it only works if we don't pass explicit `disable=...`. Therefore, this PR introduces some wrapper that checks whether the user explicitly enabled/disabled progress bars. If not, we go with the tqdm default, which means showing all progress bars unless globally disabled. An additional `enabled` argument is added for convenience. --------- Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>
A subset of #1246. Co-authored-by: Doresic <domagoj.doresic@gmail.com>
Update SacessOptimizer documentation --------- Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>
…1270) Include results for all local searches in scatter search `OptimizeResult`. Those might be useful for creating ensembles or waterfall plots.
* add extended history for AmiciObjective * convert ms to s * no amici in pypesto.optimize * Update test/base/test_history.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Update pypesto/optimize/optimizer.py Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> * Update pypesto/history/amici.py Co-authored-by: Fabian Fröhlich <fabian.frohlich@crick.ac.uk> * more compact code --------- Co-authored-by: Maren Philipps <55318391+m-philipps@users.noreply.github.com> Co-authored-by: Maren Philipps <maren.philipps@uni-bonn.de> Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com> Co-authored-by: Fabian Fröhlich <fabian.frohlich@crick.ac.uk>
Use different directories for models from different amici versions
(e.g., `amici_models/0.20.0/{model_name}` instead of `amici_models/{model_name}`).
Allows switching between amici versions without version errors or re-importing the model.
* Add hierarchical to Problem * Change to HierarchicalProblem class * Add to importer * Daniel review changes * Daniel review change * Paul review changes
…DF5 without an Objective (#1253) Allows using warnings filter to get rid of this message. ```python import warnings warnings.filterwarnings("ignore", category=UserWarning, message=".*You are loading a problem.*", module="pypesto.store.read_from_hdf5") ```
It's unclear why `EnsemblePrediction.predictor` would be needed. This PR removes a seemingly useless log message.
Codecov ReportAttention:
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #1298 +/- ##
==========================================
+ Coverage 84.24% 84.40% +0.15%
==========================================
Files 148 153 +5
Lines 11865 12456 +591
==========================================
+ Hits 9996 10513 +517
- Misses 1869 1943 +74 ☔ View full report in Codecov by Sentry. |
* Updated version and Changelog. * Added other commits. * Small typo --------- Co-authored-by: Doresic <85789271+Doresic@users.noreply.github.com>
PaulJonasJost
requested review from
stephanmg,
m-philipps,
dilpath,
dweindl,
Doresic,
vwiela,
FFroehlich and
a team
as code owners
FFroehlich
approved these changes
Feb 5, 2024
dilpath
approved these changes
Feb 5, 2024
stephanmg
approved these changes
Feb 5, 2024
dweindl
approved these changes
Feb 5, 2024
vwiela
approved these changes
Feb 5, 2024
Doresic
approved these changes
Feb 5, 2024
* Remove flake8-black due to no updates and running black separately already. * Solved B038 error on flake8 * Added to CHANGELOG.rst * Updated Changelog
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.