Profile: improve profiling code (#1447)

* Initial working update

Implement two targets: higher and lower. Choose which one to go to depending on first guess.
Different calculation of next_obj_target

TODO: change 1.5 to magic factor

* Add TODOs

* Introduce adaptive max and min steps

Implemented adaptive max and min steps in profiling. If the optimization during profiling fails (results in inf value), the algorithm will first try to iteratively decrease `max_step_size` to be closer to the last point that had a successful optimization. If that doesn't work (if we reduce max_step_size below min_step_size), then max_step_size is set back to the default and we try to increase min_step_size to "jump over" the problematic area.

Resampling random points and start from those is only the last resort and will be done if these two do not work. The idea is that we want to stay as close as we can to the last profiling point.

TODO: Put the adaptive reduction/increase of max_step_size/min_step_size into options

* Fix flags for exiting the "trust area"

* Bugfixes, Robustness, Logging, Better plotting

- BUGFIX: absolute value in objective targets at last_delta_fval
- BUGFIX: extrapolation explosions if we start at boundary
- Feature: Trust region on extrapolation
- Added y ticks back into the plot, sometimes the range is completely different.
- Added points to the plotting of profiles (in case of one result and one profile list id)
- Added color change to plotting of profiles (in case of one result and one profile list id)
- LOGGING: added logging.INFO with informations of steps made and successful optimizations.

* Fix default value for color_path

* Fix color value issues -- failing tests

* Add tuple to isinstance list

* No color_path plotting if color requested

* Fix if statements fixed_method

* We're making more steps then before

Test were testing whether the profiling method was making a lot of steps. Now we're making a lot more steps due to higher robustness.

* Change default magic and max values

* Change default method, remove TODOs

* Update quality colors

* Fix failing test

* Fix test and docstring

* Rewrite some too long if statements

* Some more if statements cleanup

* Change color if no

* Correct y-axis in obj.fun plotting

* Paul review changes

* More Paul review changes

* Fix if-while infinite loop bug

* Correct comment variable name

Co-authored-by: Maren Philipps <55318391+m-philipps@users.noreply.github.com>

* Change i_color to color_i

* Change docstring of color in lowlevel

* Expand colors docstring

* Fix color format checking and update docstring

---------

Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>
Co-authored-by: Maren Philipps <55318391+m-philipps@users.noreply.github.com>

pypesto/profile/options.py

@@ -32,9 +32,9 @@ class ProfileOptions(dict):
     reg_order:
         Maximum degree of regression polynomial used in regression based
         adaptive profile points proposal.
-    magic_factor_obj_value:
-        There is this magic factor in the old profiling code which slows down
-        profiling at small ratios (must be >= 0 and < 1).
+    adaptive_target_scaling_factor:
+        The scaling factor of the next_obj_target in next guess generation.
+        Larger values result in larger next_guess step size (must be > 1).
     whole_path:
         Whether to profile the whole bounds or only till we get below the
         ratio.
@@ -44,13 +44,13 @@ def __init__(
         self,
         default_step_size: float = 0.01,
         min_step_size: float = 0.001,
-        max_step_size: float = 1.0,
+        max_step_size: float = 0.1,
         step_size_factor: float = 1.25,
         delta_ratio_max: float = 0.1,
         ratio_min: float = 0.145,
         reg_points: int = 10,
         reg_order: int = 4,
-        magic_factor_obj_value: float = 0.5,
+        adaptive_target_scaling_factor: float = 1.5,
         whole_path: bool = False,
     ):
         super().__init__()
@@ -63,7 +63,7 @@ def __init__(
         self.delta_ratio_max = delta_ratio_max
         self.reg_points = reg_points
         self.reg_order = reg_order
-        self.magic_factor_obj_value = magic_factor_obj_value
+        self.adaptive_target_scaling_factor = adaptive_target_scaling_factor
         self.whole_path = whole_path
 
         self.validate()
@@ -112,5 +112,5 @@ def validate(self):
         if self.default_step_size < self.min_step_size:
             raise ValueError("default_step_size must be >= min_step_size.")
 
-        if self.magic_factor_obj_value < 0 or self.magic_factor_obj_value >= 1:
-            raise ValueError("magic_factor_obj_value must be >= 0 and < 1.")
+        if self.adaptive_target_scaling_factor < 1:
+            raise ValueError("adaptive_target_scaling_factor must be > 1.")

pypesto/profile/profile.py

@@ -24,7 +24,7 @@ def parameter_profile(
     profile_index: Iterable[int] = None,
     profile_list: int = None,
     result_index: int = 0,
-    next_guess_method: Union[Callable, str] = "adaptive_step_regression",
+    next_guess_method: Union[Callable, str] = "adaptive_step_order_1",
     profile_options: ProfileOptions = None,
     progress_bar: bool = None,
     filename: Union[str, Callable, None] = None,
@@ -93,7 +93,9 @@ def parameter_profile(
     profile_options = ProfileOptions.create_instance(profile_options)
     profile_options.validate()
 
-    # create a function handle that will be called later to get the next point
+    # Create a function handle that will be called later to get the next point.
+    # This function will be used to generate the initial points of optimization
+    # steps in profiling in `walk_along_profile.py`
     if isinstance(next_guess_method, str):
 
         def create_next_guess(
@@ -104,6 +106,8 @@ def create_next_guess(
             current_profile_,
             problem_,
             global_opt_,
+            min_step_increase_factor_,
+            max_step_reduce_factor_,
         ):
             return next_guess(
                 x,
@@ -114,6 +118,8 @@ def create_next_guess(
                 current_profile_,
                 problem_,
                 global_opt_,
+                min_step_increase_factor_,
+                max_step_reduce_factor_,
             )
 
     elif callable(next_guess_method):