"Cleverer" data augmentation for object detection

[alex-mitrevski]

Problem description

Our current data augmentation script for object detection allows changing various object parameters - colour, brightness, noise, size, and orientation - but applies the same set of parameters to all object classes. This affects the realism of the augmented data, and in turn affects the generalisability of learned models to real-world objects.

For example, colour augmentation doesn't make sense for some objects (e.g. lemons) since the colour is a very distinguishing feature. For others, only certain colours should be allowed (for example, it doesn't make sense to allow a blue colour for apples). And then there are objects for which any colour can be allowed.

To resolve this issue, we need to be able to specify augmentation "constraints" for different object classes that preserve the realism of augmented data.

Proposed solution

I suggest modifying the augmentation config file (and obviously the script) so that different augmentation parameters can be specified for different object classes. This will allow inducing some prior knowledge about objects into the augmentation process, which would hopefully increase the generalisability of trained object detectors to objects in the wild.

The configuration file should, just as now, include a "default" section of parameters, but it should be possible to overwrite these for different object classes. An illustrative example of this is shown below (the example doesn't include all modifiable object parameters in the augmentation config file):

default:
    prob_rand_color: 0.5
    min_scale: 0.5
    max_scale: 1.5
lemon:
    prob_rand_color: 0.0
    allowed_color_ranges:
        r: [red_min, red_max]
        g: [green_min, green_max]
        b: [blue_min, blue_max]
    min_scale: 1.5
    max_scale: 2.5
plate:
    prob_rand_color: 0.8

[minhnh]

anyone wants to work on this specifically? or will you since you're currently using this?

[alex-mitrevski]

No one has started working on it yet. I was going to work on it myself (not immediately), but go for it if you're interested.