Jacobian with different number of column and rows

[HassanLH]

Hi, first of all thanks for your work !

I am trying to use theseus essentialy for the pytorch implementation of Levenberg-Marquardt. I know this implementation is different but I am trying to use it in the same context as scipy.optimize.least_squares. I have a residuals function which returns errors and a jacobian one both written in pytorch. I followed tutorials (especially 3) to create "theseus warpers" for my functions.

I have one cost function for my whole optimization problem which warpsmy residuals and jacobian function.

I encounter a problem during optimization in dense_linearization.py : "self.A[:, row_slice, col_slice] = var_jacobian
RuntimeError: The expanded size of the tensor (563) must match the existing size (3936) at non-singleton dimension 2. Target sizes: [1, 563, 563]. Tensor sizes: [563, 3936]" (so here I have 563 parameters and 3936 residuals).

By looking in detail, I have those shapes in _linearize_jacobian_impl :
VAR JACOBIAN SHAPE : torch.Size([1, 563, 3936])
NUM COLS : 3936
NUM ROWS : 563
ROW_SLICE : slice(0, 563, None)
COL_SLICE : slice(0, 3936, None)
A SHAPE : torch.Size([1, 563, 563])

everything seems coherent except for A column number which seems to come from Linearization Class initialization :
`col_counter = 0
for var in ordering:
v_dim = var.dof()
self.var_start_cols.append(col_counter)
self.var_dims.append(v_dim)
col_counter += v_dim

self.num_cols = col_counter`

the number of column seems to come from the var dimension which are the optimization variables ? Shouldn't it be defined with respect to the returned error dimension ? I am probably understanding something wrong.

Is this behavior normal and I defined my cost function wrongly ?

Thanks for your help

[luisenp]

Hi @HassanLH. Thanks for your message! Our jacobians are expected to be of shape (batch_size, error_dim, var_dof), for each variable. It seems maybe you have dims 2 and 3 swapped?

[HassanLH]

Hi @luisenp, thanks for this fast response, I tried first with this shape but had the same kind of error. I thought it'd be the opposite (eg (batch_size, var_dof, error_dim)) as num_rows = cost_function.dim() and num_cols = var_jacobian.shape[2].
Does cost_function.dim() returns the errors dimension ?

By returning a jacobian of shape (batch_size, error_dim, var_dof) I get the same type of error message : "self.A[:, row_slice, col_slice] = var_jacobian
RuntimeError: The expanded size of the tensor (563) must match the existing size (3936) at non-singleton dimension 1. Target sizes: [1, 563, 563]. Tensor sizes: [3936, 563]"

and those shapes in _linearize_jacobian_impl :
VAR JACOBIAN SHAPE : torch.Size([1, 3936, 563])
NUM COLS : 563
NUM ROWS : 563
ROW_SLICE : slice(0, 563, None)
COL_SLICE : slice(0, 563, None)
A SHAPE : torch.Size([1, 563, 563])

[luisenp]

cost_function.dim() should return the error's dimension. If you have some minimal code to reproduce this error, I can help you debug.

[HassanLH]

Unfortunately I can't provide it to you but I'll look into it now that I know what to look for, I'll update here. Thanks !

[HassanLH]

That was the problem ! I didn't read with enough attention the tutorial and didnt change the dim method in my custom cost function. The optimization goes well now. Thanks again for your help

[luisenp]

Glad to know it's working, happy to help!