SiamMask to ONNX conversion error

[zeleniyslonik]

Hello,
Has anyone tried to convert SiamMask to ONNX?
I am trying to achieve this using shupinghu's code from #125 with minor editing, but I end up with the following error:
RuntimeError: Unsupported: ONNX export of Slice with dynamic inputs. DynamicSlice is a deprecated experimental op. Please use statically allocated variables or export to a higher opset version.

Here is the code.

import torch
import torchvision.models as models
import torch.nn as nn
import argparse
import torchvision
import torch.onnx

from collections import OrderedDict
from pysot.core.config import cfg
from pysot.models.model_builder import ModelBuilder

parser = argparse.ArgumentParser(description='trans demo')
parser.add_argument('--config', type=str, help='config file')
parser.add_argument('--snapshot', type=str, help='model name')
args = parser.parse_args()

cfg.merge_from_file(args.config)
cfg.CUDA = torch.cuda.is_available() and cfg.CUDA
device = torch.device('cuda' if cfg.CUDA else 'cpu')


class ConvertModel(nn.Module):
    def __init__(self, model):
        super(ConvertModel, self).__init__()
        self.model = model

    def forward(self, template, search):
        zf = self.model.backbone(template)
        xf = self.model.backbone(search)

        if cfg.MASK.MASK:
            zf = zf[-1]
            self.model.xf_refine = xf[:-1]
            xf = xf[-1]
        if cfg.ADJUST.ADJUST:
            zf = self.model.neck(zf)
            xf = self.model.neck(xf)
        mask, self.mask_corr_feature = self.model.mask_head(zf, xf)
        cls, loc = self.model.rpn_head(zf, xf)

        return cls, loc, mask

model0 = ModelBuilder()
model0.load_state_dict(torch.load(args.snapshot, map_location=lambda storage, loc: storage.cpu()))

model0.eval()
# print(model0)
model = ConvertModel(model0)

x = torch.randn(1, 3, 127, 127)
z = torch.randn(1, 3, 255, 255)

torch_out = torch.onnx._export(model, (x,z), "onnx-ized.onnx", export_params=True)

I get the params for dummy_input from the config file for that model.
Can anyone help me with that?

[zeleniyslonik]

Seems, that the problem was with the opset. Switching it to 11 was enough to solve this particular problem. My fault.
I`ve also encountered a need to split the network into two parts and transfer the mask_refine part of it into another file. However, during inference I get the following error:

2020-05-28 12:19:08.281828359 [E:onnxruntime:, sequential_executor.cc:183 Execute] Non-zero status code returned while running Conv node. Name:'Conv_149' Status Message: Invalid input shape: {0,0}
Traceback (most recent call last):
  File "/home/zeleniyslonik/PycharmProjects/onnx_siammask_tracker_python/track_on_video_onnx.py", line 230, in <module>
    processor.Execute()
  File "/home/zeleniyslonik/PycharmProjects/onnx_siammask_tracker_python/track_on_video_onnx.py", line 103, in Execute
    outputs = self.tracker.track(roiFrame)
  File "/home/zeleniyslonik/PycharmProjects/onnx_siammask_tracker_python/tracker/siammask_tracker.py", line 325, in track
    mask = self.sess_mask.run(None, {self.input_name_mask_1: xf_refine_1,
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/session.py", line 142, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Conv node. Name:'Conv_149' Status Message: Invalid input shape: {0,0}

The name of a layer and shape in the Status Message can vary. I believe, that this error can be linked to the TraceWarnings I get during conversion:

/home/zeleniyslonik/.local/lib/python3.8/site-packages/torch/tensor.py:464: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  warnings.warn('Iterating over a tensor might cause the trace to be incorrect. '
/home/zeleniyslonik/PycharmProjects/pysot_tracker_standalone/pysot/models/head/mask.py:77: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  p0 = F.pad(f[0], [16, 16, 16, 16])[:, :, 4*pos[0]:4*pos[0]+61, 4*pos[1]:4*pos[1]+61]
/home/zeleniyslonik/PycharmProjects/pysot_tracker_standalone/pysot/models/head/mask.py:78: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  p1 = F.pad(f[1], [8, 8, 8, 8])[:, :, 2*pos[0]:2*pos[0]+31, 2*pos[1]:2*pos[1]+31]
/home/zeleniyslonik/PycharmProjects/pysot_tracker_standalone/pysot/models/head/mask.py:79: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  p2 = F.pad(f[2], [4, 4, 4, 4])[:, :, pos[0]:pos[0]+15, pos[1]:pos[1]+15]

I use the following code for the conversion:

import torch.nn as nn
import torch.onnx

from pysot.core.config import cfg
from pysot.models.model_builder import ModelBuilder

config = '/home/zeleniyslonik/Desktop/pysot_tracker_standalone/siammask_r50_l3/config.yaml'
snapshot = '/home/zeleniyslonik/Desktop/pysot_tracker_standalone/siammask_r50_l3/model.pth'

cfg.merge_from_file(config)
cfg.CUDA = torch.cuda.is_available() and cfg.CUDA
device = torch.device('cuda' if cfg.CUDA else 'cpu')


class ConvertModel(nn.Module):
    def __init__(self, model):
        super(ConvertModel, self).__init__()
        self.model = model

    def forward(self, x1, x2, x3, mask_corr, pos):
        mask_refine = self.model.refine_head([x1, x2, x3], mask_corr, pos)
        return mask_refine

model0 = ModelBuilder()
model0.load_state_dict(torch.load(snapshot, map_location=lambda storage, loc: storage.cpu()))

model0.eval()
# print(model0)
model = ConvertModel(model0)

self_xf_dummy_0 = torch.randn(1, 64, 125, 125)
self_xf_dummy_1 = torch.randn(1, 256, 63, 63)
self_xf_dummy_2 = torch.randn(1, 512, 31, 31)
mask_corr_feature_dummy = torch.randn(1, 256, 25, 25)
pos_dummy = torch.randint(low=0, high=20, size=(2,))

torch_out = torch.onnx._export(model, (self_xf_dummy_0, self_xf_dummy_1, self_xf_dummy_2, mask_corr_feature_dummy, pos_dummy),
                                       "siammask_mask_refine.onnx",
                                       export_params=True, opset_version=11, verbose=True)

Should I ask it here or it is irrelevant to the project and I should ask it at ONNXRuntime page?
If not, how can it be solved? I am really stuck at it. Thanks in advance.

[GOBish]

@zeleniyslonik did you ever figure this out? Also, could you explain how you "split the model"?

[ShirGrinblat]

Hey @zeleniyslonik @GOBish did you find a way to convert SiamMask to ONNX that works? I'm looking for solution too, and I would love to get your help! Thanks!

[vjsrinivas]

If anyone else is still looking for an ONNX implementation, I was able to export it with some modifications to the original author's codebase: https://github.com/vjsrinivas/siammask_onnx

Also have the VOT18 three-branch version in ONNX file here: https://github.com/vjsrinivas/siammask_onnx/releases/tag/v1.0.0