RecursionError in SKMLPRunner due to infinite recursion in _mlp_loss_grad_lbfgs_intercept

[knakamura13]

Issue Summary
A RecursionError Exception in nn_examples.ipynb causes the notebook to be unusable.

The error indicates an infinite recursion in SKMLPRunner, specifically within the _mlp_loss_grad_lbfgs_intercept method.

The issue requires debugging of SKMLPRunner and possibly its inherited classes to resolve the problematic implementation.

Steps to Reproduce

• Innn_examples.ipynb, a RecursionError exception occurs on the following line of code:
  • run_stats_df, curves_df, cv_results_df, cx_sr = cx_skr.run()

Expected Behavior
This Exception should not occur. It indicates a problematic implementation in runners.SKMLPRunner, which requires further investigation.

Observed Behavior
The full Traceback for this exception is shared below:

ValueError: 
All the 5 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
5 fits failed with the following error:
Traceback (most recent call last):
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/sklearn/model_selection/_validation.py", line 888, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/Users/kylenakamura/documents-local/development-local/side-projects/mlrose-ky/src/mlrose_ky/runners/skmlp_runner.py", line 77, in fit
    return self.mlp.fit(x_train, y_train)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/sklearn/base.py", line 1473, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/sklearn/neural_network/_multilayer_perceptron.py", line 751, in fit
    return self._fit(X, y, incremental=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/sklearn/neural_network/_multilayer_perceptron.py", line 488, in _fit
    self._fit_lbfgs(
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/sklearn/neural_network/_multilayer_perceptron.py", line 532, in _fit_lbfgs
    opt_res = scipy.optimize.minimize(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_minimize.py", line 713, in minimize
    res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_lbfgsb_py.py", line 347, in _minimize_lbfgsb
    sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_optimize.py", line 288, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py", line 166, in __init__
    self._update_fun()
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py", line 262, in _update_fun
    self._update_fun_impl()
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py", line 163, in update_fun
    self.f = fun_wrapped(self.x)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py", line 145, in fun_wrapped
    fx = fun(np.copy(x), *args)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_optimize.py", line 79, in __call__
    self._compute_if_needed(x, *args)
  File "/Users/kylenakamura/anaconda3/envs/machine-learning/lib/python3.11/site-packages/scipy/optimize/_optimize.py", line 73, in _compute_if_needed
    fg = self.fun(x, *args)
         ^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/documents-local/development-local/side-projects/mlrose-ky/src/mlrose_ky/runners/skmlp_runner.py", line 94, in _loss_grad_lbfgs_intercept
    f, g = self._mlp_loss_grad_lbfgs(packed_coef_inter, x, y, activations, deltas, coef_grads, intercept_grads)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/documents-local/development-local/side-projects/mlrose-ky/src/mlrose_ky/runners/skmlp_runner.py", line 94, in _loss_grad_lbfgs_intercept
    f, g = self._mlp_loss_grad_lbfgs(packed_coef_inter, x, y, activations, deltas, coef_grads, intercept_grads)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kylenakamura/documents-local/development-local/side-projects/mlrose-ky/src/mlrose_ky/runners/skmlp_runner.py", line 94, in _loss_grad_lbfgs_intercept
    f, g = self._mlp_loss_grad_lbfgs(packed_coef_inter, x, y, activations, deltas, coef_grads, intercept_grads)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 966 more times]
RecursionError: maximum recursion depth exceeded

Additional Context

• This bug is present in the master branch in mlrose-hiive, meaning that it could not have been introduced by any commits created in the fork mlrose-ky.
• To resolve the issue, we will need to debug the entirety of SKMLPRunner and possibly its inherited classes _NNRunnerBase, _RunnerBase, and GridSearchMixin.

[knakamura13]

Possibly relevant PR in hiive: hiive#32

Though, if that PR fixes it, then why is the master branch on Hiive still broken?

[harrisonfloam]

@knakamura13 Just glancing through this repo and don't have time to dig into this today, but I wrote that PR in hiive/mlrose and remember dealing with this stuff. I think I just treated SKMLPRunner as completely broken when I was in ML last spring.

IIRC the issue is somehow related to sklearn.metrics, maybe something related to what I solved by adding a 'classes' attribute. Try setting error_score='raise' for more verbose output.

[knakamura13]

This issue might have been resolved with the release of version 1.1.6, though I don't remember the specific change that fixed it.