Merge pull request hyperspy#3498 from ericpre/RELEASE2.3.0

Release 2.3.0

.gitignore

@@ -29,29 +29,16 @@ desktop.ini
 dist
 /MANIFEST
 build/*
-bin/*.bat
-PortableInstall/*
 hyperspy.egg-info/*
 data/octave-core
 deb_dist/*
-doc/_build/*
-doc/.build/*
-doc/api/*
 doc/auto_examples
-doc/backreferences_dir
-doc/gh-pages
+doc/backreferences
+doc/_build
 doc/log.txt
-doc/user_guide/_build/*
-doc/user_guide/.build/*
 examples/io/*.msa
 examples/io/rgb*.jpg
 examples/region_of_interest/*.msa
-hyperspy/io_plugins/unbcf_fast.c
-hyperspy/misc/etc/test_compilers.obj
-hyperspy/tests/io/edax_files.zip
-hyperspy/tests/misc/cython/test_cython_integration.c
-setup/windows/requires/*
-setup/windows/reccomends/*
 
 ### Code ###
 # Visual Studio Code - https://code.visualstudio.com/

CHANGES.rst

@@ -14,6 +14,49 @@ https://hyperspy.readthedocs.io/en/latest/changes.html
 
 .. towncrier release notes start
 
+2.3.0 (2025-03-02)
+==================
+
+New features
+------------
+
+- Add :meth:`~.api.signals.BaseSignal.remove_spikes` method to remove spikes on :class:`~.api.signals.Signal1D` and :class:`~.api.signals.Signal2D` using local median. (`#3436 <https://github.com/hyperspy/hyperspy/issues/3436>`_)
+- Add :meth:`~.api.signals.Signal1D.remove_baseline` to remove baseline using `pybaselines <https://pybaselines.readthedocs.io>`_; see the :ref:`baseline removal <signal1D.remove_baseline>` section in the user guide. (`#3441 <https://github.com/hyperspy/hyperspy/issues/3441>`_)
+
+
+Enhancements
+------------
+
+- Fix and improve :meth:`~.model.BaseModel.as_signal`:
+
+  -  Fix lazy support which stops working from dask version 2024.12.0.
+  -  Add vectorised implementation using ``function_nd`` method of Components. For components not implementing ``function_nd`` (for example, in HyperSpy extensions, old or custom components), the old slow implementation is used. All :class:`~.api.model.components1D.Expression` based-components have the ``function_nd`` created automatically. (`#3476 <https://github.com/hyperspy/hyperspy/issues/3476>`_)
+- Improve setting chunking:
+
+  - add automatic chunking in :meth:`~.api.signals.BaseSignal.map` for non-lazy signal to optimize chunking for multiple core processing,
+  - :meth:`~.api.signals.BaseSignal.as_lazy` can now take the ``chunks`` argument,
+  - improve documentation of the ``rechunk`` argument. (`#3489 <https://github.com/hyperspy/hyperspy/issues/3489>`_)
+
+
+Bug Fixes
+---------
+
+- Fix plotting single spectra in :func:`~.api.plot.plot_spectra` with ``style="mosaic"``. (`#3483 <https://github.com/hyperspy/hyperspy/issues/3483>`_)
+
+
+Improved Documentation
+----------------------
+
+- Document implementation of model convolution for subclassing :class:`~hyperspy.models.model1d.Model1D` in the :ref:`extension guide <extension_components_label>`. (`#3494 <https://github.com/hyperspy/hyperspy/issues/3494>`_)
+
+
+Maintenance
+-----------
+
+- Add support for python 3.13. (`#3468 <https://github.com/hyperspy/hyperspy/issues/3468>`_)
+- Clean up dependencies, which are now used in eXSpy_ and RosettaSciIO_. (`#3482 <https://github.com/hyperspy/hyperspy/issues/3482>`_)
+
+
 2.2.0 (2024-11-08)
 ==================
 
@@ -1002,7 +1045,7 @@ Enhancements
 * Improve error message when file not found (`#2597 <https://github.com/hyperspy/hyperspy/pull/2597>`_)
 * Add update instructions to user guide (`#2621 <https://github.com/hyperspy/hyperspy/pull/2621>`_)
 * Improve plotting navigator of lazy signals, add ``navigator`` setter to lazy signals (`#2631 <https://github.com/hyperspy/hyperspy/pull/2631>`_)
-* Use ``'dask_auto'`` when rechunk=True in :py:meth:`~._signals.lazy.LazySignal.change_dtype` for lazy signal (`#2645 <https://github.com/hyperspy/hyperspy/pull/2645>`_)
+* Use ``'dask_auto'`` when rechunk=True in :py:meth:`~.api.signals.BaseSignal.change_dtype` for lazy signal (`#2645 <https://github.com/hyperspy/hyperspy/pull/2645>`_)
 * Use dask chunking when saving lazy signal instead of rechunking and leave the user to decide what is the suitable chunking (`#2629 <https://github.com/hyperspy/hyperspy/pull/2629>`_)
 * Added lazy reading support for FFT and DPC datasets in FEI emd datasets (`#2651 <https://github.com/hyperspy/hyperspy/pull/2651>`_).
 * Improve error message when initialising SpanROI with left >= right (`#2604 <https://github.com/hyperspy/hyperspy/pull/2604>`_)

conda_environment.yml

@@ -21,5 +21,5 @@ dependencies:
 - scikit-learn >=1.0.1
 - scipy >=1.6.0
 - sympy >=1.10
-- tqdm >=4.9.0
+- tqdm >=4.59.0
 - traits >=4.5.0

doc/Makefile

@@ -36,6 +36,7 @@ help:
 clean:
 	rm -rf $(BUILDDIR)/*
 	rm -rf auto_examples/
+	rm -rf backreferences/
 
 html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html

doc/conf.py

@@ -331,6 +331,7 @@
 intersphinx_mapping = {
     "cupy": ("https://docs.cupy.dev/en/stable", None),
     "dask": ("https://docs.dask.org/en/latest", None),
+    "dask_image": ("https://image.dask.org/en/latest", None),
     "exspy": ("https://exspy.readthedocs.io/en/latest", None),
     "h5py": ("https://docs.h5py.org/en/stable", None),
     "holospy": ("https://holospy.readthedocs.io/en/latest", None),
@@ -340,6 +341,7 @@
     "matplotlib": ("https://matplotlib.org/stable", None),
     "numpy": ("https://numpy.org/doc/stable", None),
     "pint": ("https://pint.readthedocs.io/en/stable", None),
+    "pybaselines": ("https://pybaselines.readthedocs.io/en/stable", None),
     "python": ("https://docs.python.org/3", None),
     "rsciio": ("https://hyperspy.org/rosettasciio/", None),
     "scipy": ("https://docs.scipy.org/doc/scipy", None),
@@ -425,7 +427,11 @@
 sphinx_gallery_conf = {
     "examples_dirs": "../examples",  # path to your example scripts
     "gallery_dirs": "auto_examples",  # path to where to save gallery generated output
-    "backreferences_dir": "backreferences_dir",  # directory where object granular galleries are stored
+    # directory where function/class granular galleries are stored
+    "backreferences_dir": "backreferences",
+    # Modules for which function/class level galleries are created. In
+    # this case hyperspy in a tuple of strings.
+    "doc_module": ("hyperspy",),
     "filename_pattern": ".py",  # pattern to define which will be executed
     "ignore_pattern": "_sgskip.py",  # pattern to define which will not be executed
     "compress_images": (

doc/make.bat

@@ -38,6 +38,7 @@ if "%1" == "clean" (
 	for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
 	del /q /s %BUILDDIR%\*
 	rmdir /q /s auto_examples
+	rmdir /q /s backreferences
 	goto end
 )
 

doc/user_guide/signal/generic_tools.rst

@@ -930,3 +930,65 @@ is preserved. If alpha is one, the Tukey window is equivalent to a Hann window.
 
 Apodization can be applied in place by setting keyword argument ``inplace`` to ``True``.
 In this case method will not return anything.
+
+
+Spike removal
+^^^^^^^^^^^^^
+
+Cosmic rays or X-rays can cause intense pixels in the data. To correct for these experimental
+artifacts, two approaches are implemented in HyperSpy to remove spikes:
+
+- using a median filter - see :meth:`~.api.signals.BaseSignal.remove_spikes`
+- using interpolation (Signal1D only) - see :meth:`~.api.signals.Signal1D.spikes_removal_tool`
+
+Using a median filter
+"""""""""""""""""""""
+
+The :meth:`~.api.signals.BaseSignal.remove_spikes` finds spikes using the following condition:
+
+.. math::
+
+    \mathrm{value} > \mathrm{median} + \sigma * \mathrm{threshold\_factor}
+
+where :math:`\sigma` is the standard deviation.
+
+Where the condition above is fullfilled, the values are replaced by their local median.
+The parameter ``threshold_factor`` can be specified by the user (default value is 5)
+and the axes along which the median and the standard deviation are calculated can be specified
+using the ``axes`` parameter.
+
+Using interpolation (Signal1D only)
+"""""""""""""""""""""""""""""""""""
+
+:meth:`~.api.signals.Signal1D.spikes_removal_tool` provides an user
+interface to remove spikes from spectra. The ``derivative histogram`` allows to
+identify the appropriate threshold. It is possible to use this tool
+on a specific interval of the data by :ref:`slicing the data
+<signal.indexing>`. For example, to use this tool in the signal between
+indices 8 and 17:
+
+.. code-block:: python
+
+   >>> s = hs.signals.Signal1D(np.arange(5*10*20).reshape((5, 10, 20)))
+   >>> s.isig[8:17].spikes_removal_tool() # doctest: +SKIP
+
+
+The options ``navigation_mask`` or ``signal_mask`` provide more flexibility in the
+selection of the data, but these require a mask (booleen array) as parameter, which needs
+to be created manually:
+
+.. code-block:: python
+
+   >>> s = hs.signals.Signal1D(np.arange(5*10*20).reshape((5, 10, 20)))
+   
+   To get a signal mask, get the mean over the navigation space
+
+   >>> s_mean = s.mean()
+   >>> mask = s_mean > 495
+   >>> s.spikes_removal_tool(signal_mask=mask) # doctest: +SKIP
+
+.. figure::  ../images/spikes_removal_tool.png
+   :align:   center
+   :width:   500
+
+   Spikes removal tool.

doc/user_guide/signal1d.rst

@@ -58,9 +58,11 @@ Background removal
     improvements.
 
 The :meth:`~.api.signals.Signal1D.remove_background` method provides
-background removal capabilities through both a CLI and a GUI. The GUI displays
-an interactive preview of the remainder after background subtraction. Currently,
-the following background types are supported: Doniach, Exponential, Gaussian,
+background removal capabilities through both a CLI and a GUI. Selected components
+from :mod:`~.api.model.components1D` are fitted in a given range to estimate the
+background and substract it from the signals.
+The GUI displays an interactive preview of the remainder after background subtraction.
+Currently, the following background types are supported: Doniach, Exponential, Gaussian,
 Lorentzian, Polynomial, Power law (default), Offset, Skew normal, Split Voigt
 and Voigt. By default, the background parameters are estimated using analytical
 approximations (keyword argument ``fast=True``). The fast option is not accurate
@@ -84,6 +86,34 @@ Example of usage:
    figure) click inside the axes of the figure and drag to the right
    without releasing the button.
 
+.. _signal1D.remove_baseline:
+
+Baseline removal
+----------------
+
+.. versionadded:: 2.3
+
+The :meth:`~.api.signals.Signal1D.remove_baseline` method provides baseline
+removal capabilities through both a CLI and a GUI. The baseline is estimated using
+methods implemented in `pybaselines <https://pybaselines.readthedocs.io>`_.
+
+.. code-block:: python
+
+    >>> s = hs.data.two_gaussians()
+    >>> s.remove_baseline()
+
+.. figure::  images/signal_1d_remove_baseline.png
+   :align:   center
+
+   Interactive baseline removal. The estimated baseline is shown as a blue dash
+   in the signal plot. The GUI allows selecting the method and the main parameters.
+
+
+.. minigallery::
+    :add-heading: Example using :meth:`~.api.signals.Signal1D.remove_baseline`
+
+    ../examples/processing/baseline_removal.py
+
 
 Calibration
 -----------
@@ -130,43 +160,6 @@ passed) can perform data smoothing with different algorithms:
 * :meth:`~.api.signals.Signal1D.smooth_savitzky_golay`
 
 
-Spike removal
---------------
-
-:meth:`~.api.signals.Signal1D.spikes_removal_tool` provides an user
-interface to remove spikes from spectra. The ``derivative histogram`` allows to
-identify the appropriate threshold. It is possible to use this tool
-on a specific interval of the data by :ref:`slicing the data
-<signal.indexing>`. For example, to use this tool in the signal between
-indices 8 and 17:
-
-.. code-block:: python
-
-   >>> s = hs.signals.Signal1D(np.arange(5*10*20).reshape((5, 10, 20)))
-   >>> s.isig[8:17].spikes_removal_tool() # doctest: +SKIP
-
-
-The options ``navigation_mask`` or ``signal_mask`` provide more flexibility in the
-selection of the data, but these require a mask (booleen array) as parameter, which needs
-to be created manually:
-
-.. code-block:: python
-
-   >>> s = hs.signals.Signal1D(np.arange(5*10*20).reshape((5, 10, 20)))
-   
-   To get a signal mask, get the mean over the navigation space
-
-   >>> s_mean = s.mean()
-   >>> mask = s_mean > 495
-   >>> s.spikes_removal_tool(signal_mask=mask) # doctest: +SKIP
-
-.. figure::  images/spikes_removal_tool.png
-   :align:   center
-   :width:   500
-
-   Spikes removal tool.
-
-
 Peak finding
 ------------
 
@@ -178,7 +171,7 @@ method.
 Estimate peak width
 -------------------
 
-For asymmetric peaks, `fitted functions <model.fitting>` may not provide
+For asymmetric peaks, :ref:`fitted functions <model.fitting>` may not provide
 an accurate description of the peak, in particular the peak width. The function
 :meth:`~.api.signals.Signal1D.estimate_peak_width`
 determines the width of a peak at a certain fraction of its maximum value.

examples/processing/README.rst

@@ -0,0 +1,4 @@
+Processing
+==========
+
+Below is a gallery of examples on signal processing.

examples/processing/baseline_removal.py

@@ -0,0 +1,23 @@
+"""
+Baseline Removal
+================
+
+This example shows how to remove a baseline from a 1D signal using the
+`pybaselines <https://pybaselines.readthedocs.io>`_ library.
+"""
+
+#%%
+# Create a signal
+import hyperspy.api as hs
+s = hs.data.two_gaussians()
+
+
+#%%
+# Remove baseline using :meth:`~.api.signals.Signal1D.remove_baseline`:
+s2 = s.remove_baseline(method="aspls", lam=1E7, inplace=False)
+
+#%%
+# Plot the signal and its baseline: 
+(s + (s-s2) * 1j).plot()
+# Choose the second figure as gallery thumbnail:
+# sphinx_gallery_thumbnail_number = 2

hyperspy/_signals/_signal1d_tool.py

@@ -0,0 +1,12 @@
+try:
+    from pybaselines import Baseline
+except ImportError:
+    pass
+
+
+def _remove_baseline(data, method, x, **kwargs):
+    baseline_fitter = getattr(
+        Baseline(x, check_finite=False, assume_sorted=True),
+        method,
+    )
+    return data - baseline_fitter(data, **kwargs)[0]