Cannot update ADFun in parallel mode.

[khaiyichin]

Hi, I'm trying to use a class that contains a Cpp::ADFun<double> member in parallel mode. Following #197, I implemented a minimal example: an issue arises when I try to overwrite the ADFun member (with SetupADFun) in the parallel for loop.

#include <cppad/cppad.hpp>

using AD = CppAD::AD<double>;
namespace
{
    bool in_parallel(void)
    {
        return omp_in_parallel() != 0;
    }
    size_t thread_number(void)
    {
        return static_cast<size_t>(omp_get_thread_num());
    }
}

class Dummy
{
public:
    Dummy() { SetupADFun(); }

    std::vector<AD> Fun(const std::vector<AD> &x, const std::vector<AD> &params)
    {
        return std::vector<AD>{params[0] * x[0] + params[1] * x[1]};
    }

    void SetupADFun()
    {
        std::vector<AD> x_ad(2), y_ad(2);

        CppAD::Independent(x_ad);
        y_ad = Fun(x_ad, params_);
        ad_fun_obj_.Dependent(x_ad, y_ad);
    }

    void UpdateParams(const std::vector<double> &params)
    {
        // modify params_
        params_ = ConvertToCppAD(params);
    }

    std::vector<AD> ConvertToCppAD(const std::vector<double> &double_vec)
    {
        std::vector<AD> output(double_vec.size());
        for (int i = 0; i < double_vec.size(); ++i)
        {
            output[i] = CppAD::Var2Par(CppAD::AD<double>(double_vec[i]));
        }
        return output;
    }

    CppAD::ADFun<double> ad_fun_obj_;
    std::vector<AD> params_{0, 0};
};

int main()
{

    // setup for using CppAD in parallel
    assert(!in_parallel());
    const int int_num_threads = omp_get_max_threads();
    size_t num_threads = size_t(int_num_threads);
    CppAD::thread_alloc::parallel_setup(
        num_threads, in_parallel, thread_number);
    CppAD::thread_alloc::hold_memory(true);
    CppAD::parallel_ad<double>();

    // parallel_ad does not make this call and perhaps it should
    // (it setups up CppAD::vector but not std::vector).
    CppAD::CheckSimpleVector<double, std::vector<double>>();

    std::vector<Dummy> d_obj_vec(5); // create 5 dummy objects

    std::vector<double> input{1.0, 2.0}; // create an arbitrary test input

    std::vector<double> outputs{0, 0, 0, 0, 0}; // create output container

    for (auto loop = 0u; loop < 100u; ++loop)
    {
        std::cout << "loop: " << loop << std::endl;
#pragma omp parallel for
        for (int i = 0; i < 5; ++i)
        {
            outputs[i] = d_obj_vec[i].ad_fun_obj_.Forward(0, input)[0];
            d_obj_vec[i].UpdateParams({-2.0, -5.0}); // update with arbitrary parameters'
            d_obj_vec[i].SetupADFun();
        }
    }

    std::cout << "computed=";
    for (auto out : outputs)
    {
        std::cout << out;
    }
    std::cout << std::endl;
}

Specifically, the error messages are of the Attempt to return memory for a different thread while in parallel mode kind, with different executions having different amounts of those errors. The code runs in sequential mode.

[khaiyichin]

Here's an instance of output errors:

loop: 0
cppad-20240602 error from a known source:
Attempt to return memory for a different thread while in parallel mode
Error detected by false result for
    thread == thread_num() || (! in_parallel())
at line 941 in the file 
    /usr/local/include/cppad/utility/thread_alloc.hpp
cppad-20240602 error from a known source:
Attempt to return memory for a different thread while in parallel mode
Error detected by false result for
    thread == thread_num() || (! in_parallel())
at line 941 in the file 
    /usr/local/include/cppad/utility/thread_alloc.hpp
cppad-20240602 error from a known source:
Attempt to return memory for a different thread while in parallel mode
Error detected by false result for
    thread == thread_num() || (! in_parallel())
at line 941 in the file 
    /usr/local/include/cppad/utility/thread_alloc.hpp
cppad-20240602 error from a known source:
Attempt to return memory for a different thread while in parallel mode
Error detected by false result for
    thread == thread_num() || (! in_parallel())
at line 941 in the file 
    /usr/local/include/cppad/utility/thread_alloc.hpp
example: /usr/local/include/cppad/utility/error_handler.hpp:205: static void CppAD::ErrorHandler::Default(bool, int, const char*, const char*, const char*): Assertion `false' failed.
example: /usr/local/include/cppad/utility/error_handler.hpp:205: static void CppAD::ErrorHandler::Default(bool, int, const char*, const char*, const char*): Assertion `false' failed.
example: /usr/local/include/cppad/utility/error_handler.hpp:205: static void CppAD::ErrorHandler::Default(bool, int, const char*, const char*, const char*): Assertion `false' failed.
Aborted (core dumped)

[bradbell]

The problem is that the default constructor

Dummy() { SetupADFun(); }

Is getting called by thread zero at line

 std::vector<Dummy> d_obj_vec(5); // create 5 dummy objects

But the corresponding destructors are getting called inside the omp parallel for loop.
The Code will work if you change the default constructor to

Dummy() { }

And move the line

d_obj_vec[i].SetupADFun();

to the beginning of the parallel for loop.

You may run into a problem if the same parallel index 'i' uses different threads for different values of the 'loop' index.

[khaiyichin]

That makes a lot of sense, I really appreciate the quick response, thanks!