Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AssertionError while training textsnake model #339

Open
mobassir94 opened this issue Jun 30, 2021 · 2 comments
Open

AssertionError while training textsnake model #339

mobassir94 opened this issue Jun 30, 2021 · 2 comments
Assignees
@HolyCrap96

Comments

@mobassir94
Copy link

mobassir94 commented Jun 30, 2021 •
edited
Loading

i was training textsnake and after train epoch i get assertion error,please check my model and train+error log attached :

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-apsisdev'
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-apsisdev'
Config:
optimizer = dict(type='SGD', lr=0.000125, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
    policy='poly', power=0.9, min_lr=1e-07, by_epoch=True, warmup=None)
total_epochs = 1200
checkpoint_config = dict(interval=1)
log_config = dict(interval=1, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
model = dict(
    type='TextSnake',
    pretrained='torchvision://resnet50',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=-1,
        norm_cfg=dict(type='BN', requires_grad=True),
        norm_eval=True,
        style='caffe'),
    neck=dict(
        type='FPN_UNet', in_channels=[256, 512, 1024, 2048], out_channels=32),
    bbox_head=dict(
        type='TextSnakeHead',
        in_channels=32,
        text_repr_type='poly',
        loss=dict(type='TextSnakeLoss')),
    train_cfg=None,
    test_cfg=None)
dataset_type = 'IcdarDataset'
data_root = '/home/apsisdev/IMPORTANT/totaltext'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadTextAnnotations',
        with_bbox=True,
        with_mask=True,
        poly2mask=False),
    dict(type='ColorJitter', brightness=0.12549019607843137, saturation=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(
        type='RandomCropPolyInstances',
        instance_key='gt_masks',
        crop_ratio=0.65,
        min_side_ratio=0.3),
    dict(
        type='RandomRotatePolyInstances',
        rotate_ratio=0.5,
        max_angle=20,
        pad_with_fixed_color=False),
    dict(
        type='ScaleAspectJitter',
        img_scale=[(3000, 736)],
        ratio_range=(0.7, 1.3),
        aspect_ratio_range=(0.9, 1.1),
        multiscale_mode='value',
        long_size_bound=800,
        short_size_bound=480,
        resize_type='long_short_bound',
        keep_ratio=False),
    dict(type='SquareResizePad', target_size=800, pad_ratio=0.6),
    dict(type='RandomFlip', flip_ratio=0.5, direction='horizontal'),
    dict(type='TextSnakeTargets'),
    dict(type='Pad', size_divisor=32),
    dict(
        type='CustomFormatBundle',
        keys=[
            'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
            'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
        ],
        visualize=dict(flag=False, boundary_key='gt_text_mask')),
    dict(
        type='Collect',
        keys=[
            'img', 'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
            'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
        ])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 736),
        flip=False,
        transforms=[
            dict(type='Resize', img_scale=(1333, 736), keep_ratio=True),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ])
]
data = dict(
    samples_per_gpu=12,
    workers_per_gpu=4,
    train=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_training.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadTextAnnotations',
                with_bbox=True,
                with_mask=True,
                poly2mask=False),
            dict(
                type='ColorJitter',
                brightness=0.12549019607843137,
                saturation=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(
                type='RandomCropPolyInstances',
                instance_key='gt_masks',
                crop_ratio=0.65,
                min_side_ratio=0.3),
            dict(
                type='RandomRotatePolyInstances',
                rotate_ratio=0.5,
                max_angle=20,
                pad_with_fixed_color=False),
            dict(
                type='ScaleAspectJitter',
                img_scale=[(3000, 736)],
                ratio_range=(0.7, 1.3),
                aspect_ratio_range=(0.9, 1.1),
                multiscale_mode='value',
                long_size_bound=800,
                short_size_bound=480,
                resize_type='long_short_bound',
                keep_ratio=False),
            dict(type='SquareResizePad', target_size=800, pad_ratio=0.6),
            dict(type='RandomFlip', flip_ratio=0.5, direction='horizontal'),
            dict(type='TextSnakeTargets'),
            dict(type='Pad', size_divisor=32),
            dict(
                type='CustomFormatBundle',
                keys=[
                    'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
                    'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
                ],
                visualize=dict(flag=False, boundary_key='gt_text_mask')),
            dict(
                type='Collect',
                keys=[
                    'img', 'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
                    'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
                ])
        ]),
    val=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_test.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 736),
                flip=False,
                transforms=[
                    dict(
                        type='Resize', img_scale=(1333, 736), keep_ratio=True),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]),
    test=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_test.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 736),
                flip=False,
                transforms=[
                    dict(
                        type='Resize', img_scale=(1333, 736), keep_ratio=True),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]))
evaluation = dict(interval=10, metric='hmean-iou')
work_dir = '../outputs/detection_model'
seed = 0
gpu_ids = range(0, 1)

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2021-06-30 13:21:39,966 - mmdet - INFO - load model from: torchvision://resnet50
2021-06-30 13:21:39,966 - mmdet - INFO - Use load_from_torchvision loader
2021-06-30 13:21:40,021 - mmdet - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias



-------------->>>>>>>>> Training FILE------------------->>>>>>>>>
 /home/apsisdev/IMPORTANT/mmocr/configs/textdet/textsnake/textsnake_r50_fpn_unet_1200e_ctw1500.py



/home/apsisdev/IMPORTANT/mmocr/mmocr/apis/train.py:74: UserWarning: config is now expected to have a `runner` section, please set `runner` in your config.
  warnings.warn(
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2021-06-30 13:21:42,119 - mmocr - INFO - Start running, host: apsisdev@ML, work_dir: /home/apsisdev/IMPORTANT/mmocr/outputs/detection_model
2021-06-30 13:21:42,119 - mmocr - INFO - workflow: [('train', 1)], max: 100 epochs
2021-06-30 13:21:46,941 - mmocr - INFO - Epoch [1][1/6]	lr: 1.250e-04, eta: 0:46:08, time: 4.623, data_time: 4.109, memory: 19933, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.3074, loss_cos: 0.6727, loss: 2.8666
2021-06-30 13:21:47,641 - mmocr - INFO - Epoch [1][2/6]	lr: 1.250e-04, eta: 0:26:54, time: 0.776, data_time: 0.207, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0280, loss_cos: 0.0010, loss: 1.9155
2021-06-30 13:21:48,416 - mmocr - INFO - Epoch [1][3/6]	lr: 1.250e-04, eta: 0:20:28, time: 0.775, data_time: 0.213, memory: 20230, loss_text: 0.6931, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0067, loss_cos: 0.0002, loss: 1.8932
2021-06-30 13:21:49,187 - mmocr - INFO - Epoch [1][4/6]	lr: 1.250e-04, eta: 0:17:14, time: 0.771, data_time: 0.213, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0135, loss_cos: 0.0014, loss: 1.9012
2021-06-30 13:21:49,956 - mmocr - INFO - Epoch [1][5/6]	lr: 1.250e-04, eta: 0:15:17, time: 0.769, data_time: 0.215, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0053, loss_cos: 0.0001, loss: 1.8918
2021-06-30 13:21:50,714 - mmocr - INFO - Epoch [1][6/6]	lr: 1.250e-04, eta: 0:13:58, time: 0.758, data_time: 0.214, memory: 20230, loss_text: 0.6932, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0158, loss_cos: 0.0003, loss: 1.9025
2021-06-30 13:21:50,780 - mmocr - INFO - Saving checkpoint at 1 epochs
2021-06-30 13:21:56,002 - mmocr - INFO - Epoch [2][1/6]	lr: 1.239e-04, eta: 0:18:35, time: 4.693, data_time: 4.122, memory: 20230, loss_text: 0.6931, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0105, loss_cos: 0.0003, loss: 1.8970
2021-06-30 13:21:56,683 - mmocr - INFO - Epoch [2][2/6]	lr: 1.239e-04, eta: 0:17:04, time: 0.681, data_time: 0.131, memory: 20230, loss_text: 0.6931, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0049, loss_cos: 0.0000, loss: 1.8911
2021-06-30 13:21:57,363 - mmocr - INFO - Epoch [2][3/6]	lr: 1.239e-04, eta: 0:15:53, time: 0.681, data_time: 0.130, memory: 20230, loss_text: 0.6930, loss_center: 0.6930, loss_radius: 0.5000, loss_sin: 0.0024, loss_cos: 0.0000, loss: 1.8885
2021-06-30 13:21:58,047 - mmocr - INFO - Epoch [2][4/6]	lr: 1.239e-04, eta: 0:14:57, time: 0.684, data_time: 0.130, memory: 20230, loss_text: 0.6930, loss_center: 0.6930, loss_radius: 0.5000, loss_sin: 0.0157, loss_cos: 0.0003, loss: 1.9020
2021-06-30 13:21:58,909 - mmocr - INFO - Epoch [2][5/6]	lr: 1.239e-04, eta: 0:14:20, time: 0.862, data_time: 0.304, memory: 20230, loss_text: 0.6929, loss_center: 0.6929, loss_radius: 0.5000, loss_sin: 0.0301, loss_cos: 0.0009, loss: 1.9169
2021-06-30 13:21:59,587 - mmocr - INFO - Epoch [2][6/6]	lr: 1.239e-04, eta: 0:13:40, time: 0.678, data_time: 0.130, memory: 20230, loss_text: 0.6929, loss_center: 0.6929, loss_radius: 0.5000, loss_sin: 0.0297, loss_cos: 0.0016, loss: 1.9172
2021-06-30 13:21:59,657 - mmocr - INFO - Saving checkpoint at 2 epochs
2021-06-30 13:22:04,842 - mmocr - INFO - Epoch [3][1/6]	lr: 1.227e-04, eta: 0:16:06, time: 4.655, data_time: 4.064, memory: 20230, loss_text: 0.6928, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0007, loss: 1.8980
2021-06-30 13:22:05,513 - mmocr - INFO - Epoch [3][2/6]	lr: 1.227e-04, eta: 0:15:23, time: 0.671, data_time: 0.131, memory: 20230, loss_text: 0.6929, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0115, loss_cos: 0.0002, loss: 1.8974
2021-06-30 13:22:06,275 - mmocr - INFO - Epoch [3][3/6]	lr: 1.227e-04, eta: 0:14:50, time: 0.763, data_time: 0.219, memory: 20230, loss_text: 0.6928, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0050, loss_cos: 0.0000, loss: 1.8906
2021-06-30 13:22:06,943 - mmocr - INFO - Epoch [3][4/6]	lr: 1.227e-04, eta: 0:14:17, time: 0.668, data_time: 0.130, memory: 20230, loss_text: 0.6928, loss_center: 0.6927, loss_radius: 0.5000, loss_sin: 0.0110, loss_cos: 0.0002, loss: 1.8967
2021-06-30 13:22:07,809 - mmocr - INFO - Epoch [3][5/6]	lr: 1.227e-04, eta: 0:13:55, time: 0.866, data_time: 0.307, memory: 20230, loss_text: 0.6926, loss_center: 0.6926, loss_radius: 0.5000, loss_sin: 0.0216, loss_cos: 0.0032, loss: 1.9100
2021-06-30 13:22:08,479 - mmocr - INFO - Epoch [3][6/6]	lr: 1.227e-04, eta: 0:13:29, time: 0.670, data_time: 0.130, memory: 20230, loss_text: 0.6926, loss_center: 0.6926, loss_radius: 0.5000, loss_sin: 0.0009, loss_cos: 0.0000, loss: 1.8861
2021-06-30 13:22:08,550 - mmocr - INFO - Saving checkpoint at 3 epochs
2021-06-30 13:22:13,483 - mmocr - INFO - Epoch [4][1/6]	lr: 1.216e-04, eta: 0:14:59, time: 4.376, data_time: 3.809, memory: 20230, loss_text: 0.6925, loss_center: 0.6925, loss_radius: 0.5000, loss_sin: 0.0132, loss_cos: 0.0005, loss: 1.8986
2021-06-30 13:22:14,296 - mmocr - INFO - Epoch [4][2/6]	lr: 1.216e-04, eta: 0:14:36, time: 0.813, data_time: 0.255, memory: 20230, loss_text: 0.6924, loss_center: 0.6924, loss_radius: 0.5000, loss_sin: 0.0053, loss_cos: 0.0001, loss: 1.8903
2021-06-30 13:22:15,142 - mmocr - INFO - Epoch [4][3/6]	lr: 1.216e-04, eta: 0:14:16, time: 0.847, data_time: 0.298, memory: 20230, loss_text: 0.6924, loss_center: 0.6924, loss_radius: 0.5000, loss_sin: 0.0304, loss_cos: 0.0008, loss: 1.9161
2021-06-30 13:22:15,831 - mmocr - INFO - Epoch [4][4/6]	lr: 1.216e-04, eta: 0:13:54, time: 0.689, data_time: 0.130, memory: 20230, loss_text: 0.6923, loss_center: 0.6923, loss_radius: 0.5000, loss_sin: 0.0034, loss_cos: 0.0001, loss: 1.8881
2021-06-30 13:22:16,683 - mmocr - INFO - Epoch [4][5/6]	lr: 1.216e-04, eta: 0:13:38, time: 0.851, data_time: 0.296, memory: 20230, loss_text: 0.6923, loss_center: 0.6923, loss_radius: 0.5000, loss_sin: 0.0051, loss_cos: 0.0001, loss: 1.8897
2021-06-30 13:22:17,362 - mmocr - INFO - Epoch [4][6/6]	lr: 1.216e-04, eta: 0:13:19, time: 0.679, data_time: 0.130, memory: 20230, loss_text: 0.6922, loss_center: 0.6922, loss_radius: 0.5000, loss_sin: 0.0333, loss_cos: 0.0014, loss: 1.9192
2021-06-30 13:22:17,456 - mmocr - INFO - Saving checkpoint at 4 epochs
2021-06-30 13:22:22,713 - mmocr - INFO - Epoch [5][1/6]	lr: 1.205e-04, eta: 0:14:34, time: 4.731, data_time: 4.160, memory: 20230, loss_text: 0.6921, loss_center: 0.6922, loss_radius: 0.5000, loss_sin: 0.0182, loss_cos: 0.0006, loss: 1.9031
2021-06-30 13:22:23,402 - mmocr - INFO - Epoch [5][2/6]	lr: 1.205e-04, eta: 0:14:14, time: 0.688, data_time: 0.132, memory: 20230, loss_text: 0.6921, loss_center: 0.6921, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0013, loss: 1.8972
2021-06-30 13:22:24,086 - mmocr - INFO - Epoch [5][3/6]	lr: 1.205e-04, eta: 0:13:56, time: 0.684, data_time: 0.131, memory: 20230, loss_text: 0.6920, loss_center: 0.6921, loss_radius: 0.5000, loss_sin: 0.0175, loss_cos: 0.0006, loss: 1.9021
2021-06-30 13:22:24,850 - mmocr - INFO - Epoch [5][4/6]	lr: 1.205e-04, eta: 0:13:40, time: 0.764, data_time: 0.222, memory: 20230, loss_text: 0.6919, loss_center: 0.6920, loss_radius: 0.5000, loss_sin: 0.0269, loss_cos: 0.0009, loss: 1.9117
2021-06-30 13:22:25,707 - mmocr - INFO - Epoch [5][5/6]	lr: 1.205e-04, eta: 0:13:27, time: 0.857, data_time: 0.301, memory: 20230, loss_text: 0.6918, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0158, loss_cos: 0.0008, loss: 1.9002
2021-06-30 13:22:26,383 - mmocr - INFO - Epoch [5][6/6]	lr: 1.205e-04, eta: 0:13:12, time: 0.677, data_time: 0.130, memory: 20230, loss_text: 0.6918, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0267, loss_cos: 0.0012, loss: 1.9115
2021-06-30 13:22:26,454 - mmocr - INFO - Saving checkpoint at 5 epochs
2021-06-30 13:22:31,640 - mmocr - INFO - Epoch [6][1/6]	lr: 1.194e-04, eta: 0:14:11, time: 4.683, data_time: 4.121, memory: 20230, loss_text: 0.6917, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0082, loss_cos: 0.0005, loss: 1.8922
2021-06-30 13:22:32,327 - mmocr - INFO - Epoch [6][2/6]	lr: 1.194e-04, eta: 0:13:55, time: 0.686, data_time: 0.131, memory: 20230, loss_text: 0.6916, loss_center: 0.6917, loss_radius: 0.5000, loss_sin: 0.0175, loss_cos: 0.0005, loss: 1.9013
2021-06-30 13:22:33,087 - mmocr - INFO - Epoch [6][3/6]	lr: 1.194e-04, eta: 0:13:41, time: 0.760, data_time: 0.215, memory: 20230, loss_text: 0.6916, loss_center: 0.6917, loss_radius: 0.5000, loss_sin: 0.0223, loss_cos: 0.0014, loss: 1.9070
2021-06-30 13:22:33,772 - mmocr - INFO - Epoch [6][4/6]	lr: 1.194e-04, eta: 0:13:27, time: 0.685, data_time: 0.130, memory: 20230, loss_text: 0.6915, loss_center: 0.6916, loss_radius: 0.5000, loss_sin: 0.0215, loss_cos: 0.0006, loss: 1.9052
2021-06-30 13:22:34,618 - mmocr - INFO - Epoch [6][5/6]	lr: 1.194e-04, eta: 0:13:16, time: 0.846, data_time: 0.298, memory: 20230, loss_text: 0.6914, loss_center: 0.6916, loss_radius: 0.5000, loss_sin: 0.0080, loss_cos: 0.0002, loss: 1.8911
2021-06-30 13:22:35,298 - mmocr - INFO - Epoch [6][6/6]	lr: 1.194e-04, eta: 0:13:03, time: 0.679, data_time: 0.130, memory: 20230, loss_text: 0.6914, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0208, loss_cos: 0.0007, loss: 1.9043
2021-06-30 13:22:35,369 - mmocr - INFO - Saving checkpoint at 6 epochs
2021-06-30 13:22:40,508 - mmocr - INFO - Epoch [7][1/6]	lr: 1.182e-04, eta: 0:13:51, time: 4.630, data_time: 4.057, memory: 20230, loss_text: 0.6912, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0140, loss_cos: 0.0033, loss: 1.9000
2021-06-30 13:22:41,204 - mmocr - INFO - Epoch [7][2/6]	lr: 1.182e-04, eta: 0:13:38, time: 0.696, data_time: 0.131, memory: 20230, loss_text: 0.6912, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0114, loss_cos: 0.0002, loss: 1.8942
2021-06-30 13:22:41,898 - mmocr - INFO - Epoch [7][3/6]	lr: 1.182e-04, eta: 0:13:26, time: 0.694, data_time: 0.131, memory: 20230, loss_text: 0.6911, loss_center: 0.6913, loss_radius: 0.5000, loss_sin: 0.0194, loss_cos: 0.0003, loss: 1.9021
2021-06-30 13:22:42,665 - mmocr - INFO - Epoch [7][4/6]	lr: 1.182e-04, eta: 0:13:15, time: 0.768, data_time: 0.219, memory: 20230, loss_text: 0.6911, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0135, loss_cos: 0.0003, loss: 1.8960
2021-06-30 13:22:43,505 - mmocr - INFO - Epoch [7][5/6]	lr: 1.182e-04, eta: 0:13:06, time: 0.840, data_time: 0.292, memory: 20230, loss_text: 0.6910, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0113, loss_cos: 0.0004, loss: 1.8939
2021-06-30 13:22:44,186 - mmocr - INFO - Epoch [7][6/6]	lr: 1.182e-04, eta: 0:12:55, time: 0.680, data_time: 0.131, memory: 20230, loss_text: 0.6909, loss_center: 0.6911, loss_radius: 0.5000, loss_sin: 0.0030, loss_cos: 0.0001, loss: 1.8851
2021-06-30 13:22:44,256 - mmocr - INFO - Saving checkpoint at 7 epochs
2021-06-30 13:22:49,429 - mmocr - INFO - Epoch [8][1/6]	lr: 1.171e-04, eta: 0:13:36, time: 4.653, data_time: 4.095, memory: 20230, loss_text: 0.6908, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0115, loss_cos: 0.0003, loss: 1.8938
2021-06-30 13:22:50,133 - mmocr - INFO - Epoch [8][2/6]	lr: 1.171e-04, eta: 0:13:24, time: 0.704, data_time: 0.135, memory: 20230, loss_text: 0.6907, loss_center: 0.6910, loss_radius: 0.5000, loss_sin: 0.0110, loss_cos: 0.0008, loss: 1.8936
2021-06-30 13:22:50,813 - mmocr - INFO - Epoch [8][3/6]	lr: 1.171e-04, eta: 0:13:14, time: 0.680, data_time: 0.131, memory: 20230, loss_text: 0.6907, loss_center: 0.6909, loss_radius: 0.5000, loss_sin: 0.0145, loss_cos: 0.0003, loss: 1.8964
2021-06-30 13:22:51,496 - mmocr - INFO - Epoch [8][4/6]	lr: 1.171e-04, eta: 0:13:03, time: 0.683, data_time: 0.130, memory: 20230, loss_text: 0.6906, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0002, loss: 1.8932
2021-06-30 13:22:52,351 - mmocr - INFO - Epoch [8][5/6]	lr: 1.171e-04, eta: 0:12:55, time: 0.855, data_time: 0.301, memory: 20230, loss_text: 0.6905, loss_center: 0.6909, loss_radius: 0.5000, loss_sin: 0.0344, loss_cos: 0.0010, loss: 1.9169
2021-06-30 13:22:53,023 - mmocr - INFO - Epoch [8][6/6]	lr: 1.171e-04, eta: 0:12:45, time: 0.673, data_time: 0.130, memory: 20230, loss_text: 0.6905, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0176, loss_cos: 0.0003, loss: 1.8991
2021-06-30 13:22:53,093 - mmocr - INFO - Saving checkpoint at 8 epochs
2021-06-30 13:22:58,334 - mmocr - INFO - Epoch [9][1/6]	lr: 1.160e-04, eta: 0:13:21, time: 4.680, data_time: 4.114, memory: 20230, loss_text: 0.6904, loss_center: 0.6909, loss_radius: 0.4999, loss_sin: 0.0072, loss_cos: 0.0001, loss: 1.8885
2021-06-30 13:22:59,036 - mmocr - INFO - Epoch [9][2/6]	lr: 1.160e-04, eta: 0:13:11, time: 0.701, data_time: 0.132, memory: 20230, loss_text: 0.6903, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0193, loss_cos: 0.0004, loss: 1.9007
2021-06-30 13:22:59,727 - mmocr - INFO - Epoch [9][3/6]	lr: 1.160e-04, eta: 0:13:02, time: 0.691, data_time: 0.131, memory: 20230, loss_text: 0.6903, loss_center: 0.6906, loss_radius: 0.5000, loss_sin: 0.0028, loss_cos: 0.0001, loss: 1.8836
2021-06-30 13:23:00,408 - mmocr - INFO - Epoch [9][4/6]	lr: 1.160e-04, eta: 0:12:52, time: 0.681, data_time: 0.131, memory: 20230, loss_text: 0.6902, loss_center: 0.6907, loss_radius: 0.4999, loss_sin: 0.0104, loss_cos: 0.0002, loss: 1.8914
2021-06-30 13:23:01,184 - mmocr - INFO - Epoch [9][5/6]	lr: 1.160e-04, eta: 0:12:45, time: 0.776, data_time: 0.221, memory: 20230, loss_text: 0.6901, loss_center: 0.6906, loss_radius: 0.4999, loss_sin: 0.0041, loss_cos: 0.0000, loss: 1.8849
2021-06-30 13:23:01,972 - mmocr - INFO - Epoch [9][6/6]	lr: 1.160e-04, eta: 0:12:37, time: 0.788, data_time: 0.221, memory: 20230, loss_text: 0.6900, loss_center: 0.6904, loss_radius: 0.4999, loss_sin: 0.0142, loss_cos: 0.0016, loss: 1.8962
2021-06-30 13:23:02,041 - mmocr - INFO - Saving checkpoint at 9 epochs
2021-06-30 13:23:07,660 - mmocr - INFO - Epoch [10][1/6]	lr: 1.148e-04, eta: 0:13:09, time: 4.756, data_time: 4.176, memory: 20230, loss_text: 0.6900, loss_center: 0.6902, loss_radius: 0.5000, loss_sin: 0.0257, loss_cos: 0.0028, loss: 1.9087
2021-06-30 13:23:08,343 - mmocr - INFO - Epoch [10][2/6]	lr: 1.148e-04, eta: 0:13:00, time: 0.684, data_time: 0.131, memory: 20230, loss_text: 0.6899, loss_center: 0.6904, loss_radius: 0.4999, loss_sin: 0.0077, loss_cos: 0.0001, loss: 1.8881
2021-06-30 13:23:09,020 - mmocr - INFO - Epoch [10][3/6]	lr: 1.148e-04, eta: 0:12:51, time: 0.676, data_time: 0.130, memory: 20230, loss_text: 0.6899, loss_center: 0.6901, loss_radius: 0.4999, loss_sin: 0.0162, loss_cos: 0.0006, loss: 1.8967
2021-06-30 13:23:09,691 - mmocr - INFO - Epoch [10][4/6]	lr: 1.148e-04, eta: 0:12:43, time: 0.671, data_time: 0.131, memory: 20230, loss_text: 0.6898, loss_center: 0.6903, loss_radius: 0.4999, loss_sin: 0.0231, loss_cos: 0.0006, loss: 1.9037
2021-06-30 13:23:10,526 - mmocr - INFO - Epoch [10][5/6]	lr: 1.148e-04, eta: 0:12:36, time: 0.835, data_time: 0.292, memory: 20230, loss_text: 0.6897, loss_center: 0.6903, loss_radius: 0.4999, loss_sin: 0.0067, loss_cos: 0.0001, loss: 1.8868
2021-06-30 13:23:11,202 - mmocr - INFO - Epoch [10][6/6]	lr: 1.148e-04, eta: 0:12:28, time: 0.676, data_time: 0.130, memory: 20230, loss_text: 0.6896, loss_center: 0.6902, loss_radius: 0.4999, loss_sin: 0.0035, loss_cos: 0.0001, loss: 1.8834
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6/6, 0.0 task/s, elapsed: 3535s, ETA:     0sTraceback (most recent call last):
  File "train_any_mmocr_detector.py", line 154, in <module>
    train_detector(model, datasets, cfg, distributed=False, validate=True)
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/apis/train.py", line 149, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/apsisdev/mmcv/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/apsisdev/mmcv/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/home/apsisdev/mmcv/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/apsisdev/anaconda3/envs/ocr/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 147, in after_train_epoch
    key_score = self.evaluate(runner, results)
  File "/home/apsisdev/anaconda3/envs/ocr/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 176, in evaluate
    eval_res = self.dataloader.dataset.evaluate(
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/datasets/icdar_dataset.py", line 149, in evaluate
    eval_results = eval_hmean(
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/core/evaluation/hmean.py", line 112, in eval_hmean
    assert utils.valid_boundary(texts[0], False)
AssertionError

i have 2 questions :

  1. how do i solve this AssertionError as shown above
  2. this step takes a lot of time : [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6/6, 0.0 task/s, elapsed: 3535s, ETA: 0sTraceback (most recent call last):
    how can i speed it up? seems like this step is working on cpu? i only have 7 image in test/validation set
@HolyCrap96
Copy link
Contributor

HolyCrap96 commented Jul 5, 2021 •
edited
Loading

@mobassir94 Hi, are you training textsnake on your own dataset?

@mobassir94
Copy link
Author

yes @HolyCrap96
i was trying to train textsnake on my own dataset which is similar to the toy dataset format provided by mmocr team in their documentation,my dataset is for bengali language

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@HolyCrap96 HolyCrap96

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mobassir94 @HolyCrap96