Memory debugging (#896)

* Fixed two memory leaks and added a FIXME for a third one:
  - Fixed leak in json_memload()
  - Fixed leak in global session when running with DLITE_ATEXIT_FREE set
  - Added a FIXME for a leak in mapping_create_rec() in src/dlite-mappings.c
  - Set DLITE_ATEXIT_FREE when running `make memcheck`
  - Ignore external leaks in pydantic and numpy random_bit_generator
* Moved pydantic suppression to valgrind-python.supp
* Documented how to do memory debugging with DLite.
* Updated Python version in documentation examples
* Updated tips and triks
* Updated test to avoid unexpected failure
* Fixed docstring formatting
* Do not test wheel for Python 3.11 instead of 3.12 for manylinux2014_i686

.github/workflows/ci_build_wheels.yml

@@ -22,10 +22,12 @@ jobs:
             system_type: ["manylinux", "2010"]
             arch: i686
             py_minors: 8,9
+            # Python 3.12 often fails with segfault during shutdown for
+            # manylinux2014_i686 - should be debugged
           - os: ubuntu-20.04
             system_type: ["manylinux", "2014"]
             arch: i686
-            py_minors: 8,12
+            py_minors: 8,11
             # Python 3.12 fails since cibuildwheel still depends on distutils
             # for musllinux
           - os: ubuntu-20.04

CMakeLists.txt

@@ -406,11 +406,24 @@ if(MEMORYCHECK_COMMAND)
   set(MEMORYCHECK_SUPPRESSIONS_FILE
     ${suppfile}
     CACHE FILEPATH "File that contains suppressions for the memory checker"
-    )
+  )
+
+  # Generate a wrapper for ctest that allows to passing arguments from the
+  # environment to ctest.  Useful for running a single test, like
+  #
+  #     CTEST_ARGS="-R test_compat" make memcheck
+  #
+  configure_file(
+    ${dlite_SOURCE_DIR}/cmake/ctest.sh
+    ${dlite_BINARY_DIR}/ctest.sh
+    @ONLY
+    #USE_SOURCE_PERMISSIONS  # available from cmake 3.20
+  )
 
   add_custom_target(memcheck
     COMMAND ${CMAKE_COMMAND}
-        -E env PYTHONMALLOC=malloc ${CMAKE_CTEST_COMMAND}
+        -E env PYTHONMALLOC=malloc DLITE_ATEXIT_FREE=
+            -- ${dlite_BINARY_DIR}/ctest.sh
             --exclude-regex static-code-analysis
             --force-new-ctest-process
             --output-on-failure

bindings/python/tests/test_dataset1_save.py

@@ -34,9 +34,9 @@
     get_unit_iri("Atom")
 
 with raises(MissingUnitError):
-    # Because prefixed units are not in EMMO by default
-    # They can be including by importing https://w3id.org/emmo/1.0.0-rc1/disciplines/units/prefixedunits
-    get_unit_iri("cm")
+    # Ångström is not included in EMMO by default. It can be including by
+    # importing https://w3id.org/emmo/1.0.0-rc1/disciplines/units/specialunits
+    get_unit_iri("Å")
 
 
 # To be fixed in issue https://github.com/SINTEF/dlite/issues/878

cmake/ctest.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+@CMAKE_CTEST_COMMAND@ $CTEST_ARGS $@

doc/contributors_guide/tips_and_tricks.md

@@ -3,7 +3,19 @@ Tips and Tricks
 
 Setting up a virtual Python environment for building dlite
 ----------------------------------------------------------
-See the [Build and install to a virtual Python environment] in the installation instructions.
+See [Build against Python environment] in the installation instructions.
+
+
+Debugging Python storage plugins
+--------------------------------
+Exceptions occurring inside Python storage plugins are not propagated to the calling interpreter, and will therefor not be shown.
+However, it is possible to write them to stderr by setting the `DLITE_PYDEBUG` environment variable.
+
+For example, run:
+
+    DLITE_PYDEBUG= python bindings/python/tests/test_storage.py 1>/dev/null
+
+to see tracebacks of exceptions occurring inside any of the tested storage plugins.
 
 
 Debugging tests failing inside docker on GitHub
@@ -15,38 +27,58 @@ Debugging tests failing inside docker on GitHub
         workon dlite
         pip install cibuildwheel
 
-3. To list all manylinux images for Python 3.7, do
+3. To list all manylinux images for Python 3.12, do
 
         cd dlite  # Root of DLite source directory
-        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux2014_x86_64:latest \
-        CIBW_BUILD=cp37-manylinux_* \
+        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux_2_28_x86_64:latest \
+        CIBW_BUILD=cp312-manylinux_* \
         python -m cibuildwheel \
         --print-build-identifiers \
         --platform linux \
         python
 
    This should write
 
-        cp37-manylinux_x86_64
-        cp37-manylinux_i686
+        cp312-manylinux_x86_64
+        cp312-manylinux_i686
 
-4. Run image.  For example, to run the image `cp37-manylinux_x86_64` do
+4. Run image.  For example, to run the image `cp312-manylinux_x86_64` do
 
-        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux2014_x86_64:latest \
-        CIBW_BUILD=cp37-manylinux_x86* \
+        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux_2_28_x86_64:latest \
+        CIBW_BUILD=cp312-manylinux_x86* \
         python -m cibuildwheel \
         --output-dir wheelhouse \
         --platform linux \
         python
 
    which should run the tests and hopefully fail at the same place as on
-   GitHub.  If that is the case, you can run the image again, but pause
+   GitHub.
+
+   If `pip wheel` fails with network errors, like `[Errno 101] Network is unreachable`, you may have create a network
+
+        docker network create -d bridge dlitenet  # Create network called dlitenet
+
+   and gets its IP with
+
+        ip=$(docker network inspect dlitenet | sed -n 's/ *"Gateway": "\(.*\)"$/\1/p')
+
+   you can then rerun `cibuildwheel` with the
+
+        CIBW_BUILD_FRONTEND="pip; args: --index-url http://$ip:3141/root/pypi/" \
+        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux_2_28_x86_64:latest \
+        CIBW_BUILD=cp312-manylinux_x86* \
+        python -m cibuildwheel \
+        --output-dir wheelhouse \
+        --platform linux \
+        python
+
+   If that is the case, you can run the image again, but pause
    it before running the tests by prepending `CIBW_BEFORE_TEST=cat` to
    the previous command:
 
         CIBW_BEFORE_TEST=cat \
-        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux2014_x86_64:latest \
-        CIBW_BUILD=cp37-manylinux_x86* \
+        CIBW_MANYLINUX_X86_64_IMAGE=ghcr.io/sintef/dlite-python-manylinux_2_28_x86_64:latest \
+        CIBW_BUILD=cp312-manylinux_x86* \
         python -m cibuildwheel \
         --output-dir wheelhouse \
         --platform linux \
@@ -80,14 +112,84 @@ Debugging tests failing inside docker on GitHub
 
         ls /tmp/cibuildwheel/repaired_wheel/DLite_Python-*.whl
 
-        pip install /tmp/cibuildwheel/repaired_wheel/DLite_Python-0.3.18-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+        pip install /tmp/cibuildwheel/repaired_wheel/DLite_Python-0.3.18-cp312-cp312m-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
 
 8. Now we can run the Python tests with
 
         python /project/bindings/python/tests/test_python_bindings.py
 
 
 
+Memory debugging
+----------------
+On Linux systems, one can use [valgrind] to check for and debug memory issues.
+
+
+### Finding memory issues
+To check for any memory issue, you can run valgrind on all tests with
+
+    make memcheck
+
+However, this usually takes a while. If you have localised an issue, and want
+to rerun valgrind on that issue, you can run
+
+    CTEST_ARGS="-R <test_name>" make memcheck
+
+Both of the above commands will write the output to `<BUILD_DIR>/Testing/Temporary/MemoryChecker.<#>.log`, where `<#>` is the test number.
+
+---
+
+Alternatively, you can run valgrind manually, with
+
+    valgrind <path/to/test-executable>
+
+if the test is written in C, or with
+
+    valgrind python <path/to/test_file.py>
+
+if the test is written in Python.
+This will write the output from valgrind to stderr.
+
+
+### Debugging segmentation faults
+If a test results in a segmentation fault, you can debug it with [gdb].
+See the [GDB Tutorial] for how to use `gdb`.
+
+Tests written in C can be debugged with
+
+    $ gdb <path/to/test-executable>
+    (gdb) run
+
+Test written in Python can be debugged with
+
+    $ gdb python
+    (gdb) run <path/to/test_file.py>
+
+Some useful gdb commands:
+- `where`: show the call stack when the test fails.
+- `up`, `down`: navigated up and down the call stack.
+- `print <expression>`: print the value of a variable or expression.
+
+
+### Debugging memory issues
+To debug other memory issues, invoke `gdb` as shown above, but set a breakpoint
+at the function before issuing the `run` command.  For example:
+
+    $ gdb python
+    (gdb) break <c_function_name>
+    (gdb) run <path/to/test_file.py>
+
+More useful gdb commands:
+- `break <function_name>`: set a break point at given function.
+- `break <file:lineno>`: set a break point at given source file position.
+- `continue`: continue to run the program after a break point.
+- `next`: execute next source line.
+- `step`: steps into the outermost subroutine on the next source line.
+  Behaves like `next` if the next source line has no subroutines.
+
 
 [virtualenvwrapper]: https://pypi.org/project/virtualenvwrapper/
 [Build against Python environment]: https://sintef.github.io/dlite/getting_started/build/build_against_python_env.html#build-against-python-environment
+[valgrind]: http://valgrind.org/
+[gdb]: https://sourceware.org/gdb/
+[GDB Tutorial]: https://www.gdbtutorial.com/

doc/getting_started/build/build.md

@@ -51,6 +51,8 @@ If you have [valgrind] installed, you can run the tests with memory checking tur
 
     cmake --build . --target memcheck
 
+For more info about checking and debugging memory issues, see [Memory debugging].
+
 
 Build documentation
 -------------------
@@ -138,3 +140,4 @@ Using [Visual Studio Code] (VS Code) it is possible to do development on the sys
 [configuration variables]: https://sintef.github.io/dlite/getting_started/build/cmake_variables.html
 [build against Python environment]: https://sintef.github.io/dlite/getting_started/build/build_against_python_env.html
 [build with Visual Studio]: https://sintef.github.io/dlite/getting_started/build/build_with_vs.html
+[Memory debugging]: https://sintef.github.io/dlite/contributors_guide/tips_and_tricks.html#memory-debugging

doc/getting_started/build/build_dependencies.md

@@ -50,6 +50,15 @@ On Ubuntu they can be installed with
     sudo apt install cppcheck valgrind
 
 
+Debugging (optional)
+--------------------
+Use [gdb] for debugging errors in the C code.  On Ubuntu it can be installed with
+
+    sudo apt install gdb
+
+See the [GDB Tutorial] for more information on how to use GDB.  Some DLite-specific tips can also be found in [Memory debugging].
+
+
 Building documentation (optional)
 ---------------------------------
 DLite uses [Sphinx] and [Doxygen] to generate documentation.
@@ -69,3 +78,6 @@ Ubuntu they can be installed with
 [Doxygen]: http://www.doxygen.org/
 [Sphinx]: https://www.sphinx-doc.org/
 [valgrind]: http://valgrind.org/
+[gdb]: https://sourceware.org/gdb/
+[GDB Tutorial]: https://www.gdbtutorial.com/
+[Memory debugging]: https://sintef.github.io/dlite/contributors_guidetips_and_tricks.html#memory-debugging

src/dlite-mapping.c

@@ -109,8 +109,11 @@ DLiteMapping *mapping_create_rec(const char *output_uri, Instances *inputs,
       to clear the cache.
     */
     if (!cheapest || cost < lowest_cost) {
+      //dlite_mapping_plugin_free(cheapest);
       cheapest = api;
       lowest_cost = cost;
+    } else {
+      //dlite_mapping_plugin_free(api);
     }
   }
   if (!(api = cheapest)) goto fail;

storages/python/python-storage-plugins/minio.py

@@ -18,9 +18,9 @@ def open(self, location: str, options=None):
             options: Supported options:
             - `bucket_name`: Name of bucket.  Defaults to "dlite".
             - `access_key`: Access key (aka user ID) of your account in S3
-               service.
+              service.
             - `secret_key`: Secret Key (aka password) of your account in S3
-               service.
+              service.
             - `session_token`: Session token of your account in S3 service.
             - `secure`: Whether to use secure (TLS) connection to S3 service.
             - `region`: Region name of buckets in S3 service.