Optimizer and SPMD: Error using optimizer/subsref (line 358)

[davidwozabal]

I would like to solve many SDP problems in parallel (using MOSEK). However, parallelization in MATLAB seems to rely on serialization of objects, which does not work in YALMIP.

I gathered from remarks in previous posts in the YAMIP forum that it would be a good idea to generate an optimizer object beforehand (which seems to be a low level description of the problem that can be serialized?) and then use this in an SPMD block. As a result, my code is roughly of the following structure as the simple example below:

sdpvar a x;
Constraints = [a+1 <= x];
Objective = x^2;
P = optimizer(Constraints,Objective,[],a,x);

spmd
    decision = P(labindex:0.01:labindex+1);
end

However, while the code above works fine and circumvents the problems with serialization, I get the following error for my (unfortunately slightly more complicated) example:

Error using optimizer/subsref (line 358)
Number of elements must not change. Use [] as one of the size inputs to automatically calculate the appropriate size for that dimension.

If I remove the SPMD block and solve the problems serially (manually assigning labindices), the problems solve fine. Unfortunately, I did not manage to produce a minimal example like the one above, which replicates the error.

Is this a bug in the optimizer or I am using YALMIP in a wrong way?

[johanlofberg]

I guess you have more than 1 parameter in your non-revealed failing example

P([a b c]) is equivalent to [P(a) P(b) P(c)] which means your trial example will work for scalar parameter but likely will fail if you have more than 1 parameter but still send 1 x N argument

parfor appears to work in 2020 and later or something like that (and should definitely work with optimizer)

[davidwozabal]

You are correct, there several arguments in our code (some are vectors, some are matrices). However, I am passing the arguments not as a list but as a cell array as in the example below, which again works fine.

a = sdpvar(2,1); b = sdpvar(2,1); C = sdpvar(2,2,'full');
x = sdpvar(2,1); y = sdpvar(2,1);
Constraints = [a+1 <= x, b+1 <= y, C * x >= 2*a];
Objective = sum((x+y).^2);
P = optimizer(Constraints,Objective,[],{a,b,C},{x,y});

spmd
    decision = P({ kron(labindex:0.01:labindex+1, ones(2,1)), kron(labindex:0.01:labindex+1, ones(2,1)), kron(ones(1, 101), rand(2,2)) });
end

I tried reproducing the error in the original post by messing with the dimensions of the arguments. However, when I did that, I always got the (different) error:

Error using optimizer/subsref (line 208)
The dimension on a supplied parameter value does not match the definition.

What I also find strange is that the error only appears if I solve the problems in parallel.

Regarding PARFOR: We thought to rather not use parfor, since we have a large pool of N = C*n problems, and C cores. We want to assign n problems to each of the cores, and then solve them using the optimizer on every core (passing n sets of parameters). If we would use parfor, we would have to call the optimizer N times instead of C times. It was our impression that this would be significantly slower (is this at all correct?).

[johanlofberg]

Without the actual problematic code nothing can be said.

[davidwozabal]

We managed to find the following minimal example that reproduces the error and also gives a rough idea what the problem might be.

num_problems = 2;

% problem definition
parPhi = sdpvar(1,1);
tau = sdpvar(1,1);
h = parPhi'*tau;

cons = [tau>=0,sum(tau)==1];
options = sdpsettings('solver','linprog','verbose',0);
opt = optimizer(cons,-h,options,parPhi,h);

phi = 1:num_problems;

% works
result = opt(phi(1,1));

% does not work
parfor i=1:2
    result = opt(phi(1,i));
end

The parfor-loop produces the error mentioned in the original post. The problem seems to be that the objective h is one of the desired return values (as is also the case in one of the examples in the documentation of the optimizer). If the definition of the optimizer above is changed to

opt = optimizer(cons,-h,options,parPhi,tau);

everything works fine.

While the equivalent transformation in our problem would probably be possible, it would be rather inconvenient, since we would have to return a large number of decision variables.

[johanlofberg]

Looks like thy haven't made parfor completely safe to use in cases when the objects use advanced functionalities

When you use an expression as an output, the optimizer object will work with that as sdpvar expression and revert back to using sdpvar value oprators etc, and that does not work. Hence, use a simple variable as output and that more complicated/expensive strategy is not use (in you case, either use tauand map manually to objective, or introduce a new variable z with constraint z == h)

[davidwozabal]

Thanks a lot! The z == h option, seems reasonably simple and inexpensive.