Mackerel Classification using Faster R-CNN + GLCC

PyTorch implementation for vgg16 Faster R-CNN with GLCC (Global and Local features Conjunct Classifier).

Original Paper >> https://ieeexplore.ieee.org/document/8717584

This is from faster r-cnn implemented with PyTorch (https://github.com/jwyang/faster-rcnn.pytorch).

This method is to classify mackerels into three category (blue mackerel, chub mackerle and hybrid (intermediate)).

This needs GPU environment because original faster-rcnn.pytorch needs, too. If you find the method to use in CPU only environment, please tell me via issues or pull request:)

1. Description (The Objective, Model )
2. Install
3. Result
4. GLCC train and test (we prepare pre-trained model)
5. Other model tips (we do not prepare pre-trained model)

1.Description

1-1.The objective

This is created under automatic mackerel categorization machine.

We have to classify mackerels to three species including blue mackerel, chub mackerel and hybrid(unknown or intermediate type).

While blue mackerels have dotted pattern abdomen, chub mackerel does not. Hybrids are difficult to distinguish because they have thin dotted pattern abdomen.

Blue mackerel	Chub mackerel	Hybrid

Other than above features, blue mackerels have round body shape, but chub mackerels have eclipse like one. So, we used these features for accurate recognition. To get body shape in image, we irradiated red laser from diagnoly above, and we could use body features as carvature of red laser on mackerel's abdomen.

Overall sorting machine	red laser mechanism
	red rectangle is global feature and green one is local feature.

Our final objective is detection and recognition.

Input	Output

1-2.Classification Algorythm

The overall architecture is depicted in below image.

This prposed method is composed in 2 stage, the one is Faster R-CNN and the other is GLCC(Global and Local features Conjunct Classifier).

This model can be trained end-to-end. In experiment, we trained Faster R-CNN (1-st stage) to detect each region, next we trained GLCC and former VGG16 backbone.

Algorythm

1. The model detected global region (= overall mackerel region) and local region (= abdomen region of mackerel) by Faster R-CNN
2. GLCC, followed neural network, predicts mackerel species using global and local regions.

(Warning)In original paper, we used 7class for detecting global and local features, but we used 3class in this implementation.

2.Install

Follow faster-rcnn.pytorch

$ conda create python=3.6 -n frcn_glcc_pytorch
$ source activate frcn_glcc_pytorch
$ cd glcc-frcn.pytorch
$ pip install -r requirements.txt
$ pip install torch==0.4 torchvision
$ cd lib
$ sh make.sh

3.Result

We evaluated methods using own test dataset of mackerel.

3-1.Feature

• Global ... using mackerel's overall feature called Global feature
• Local ... using mackerel's partial feature called Local feature
• GLCC ... using both overall and partial features

3-2.Evaluation Metric

• Accuracy
• Detection ... @IoU=0.5
• speed ... average processing time for one image

3-3. Result

• FRCN ... Faster R-CNN

Method	Backbone	Global feature	Local feature	Accuracy(%)	Detection(%)	speed[sec]
simple CNN	VGG16			67.98		0.01
FRCN-VGG16-Global	VGG16	✓		87.64	100	0.1144
FRCN-VGG16-Local	VGG16		✓	70.79	96.44	0.1166
FRCN-Res101-Global	ResNet101	✓		73.22	100	0.1120
FRCN-Res101-Local	ResNet101		✓	80.15	97.00	0.1142
FRCN-VGG16-GLCC	VGG16	✓	✓	89.33	-	0.1479
FRCN-Res101-GLCC	ResNet101	✓	✓	76.78	-	0.1452

Some result examples

prediction label is shown. Red is accurate, and blue is mistake. (all red-line prediction is blue, because we did not count them for accuracy)

Blue mackerel	Chub mackerel	Hybrid

4.GLCC

4-1.train

# VGG16
$ sh scripts/train_vgg_2class.sh
$ sh scripts/train_vgg_glcc_2class.sh

# Res101
$ sh scripts/train_res101_2class.sh
$ sh scripts/train_res101_glcc_2class.sh

4-2.test

VGG16 model

You can get pre-trained model by next command.

$ mkdir models
$ cd models
$ wget http://www.iic.ecei.tohoku.ac.jp/~tomo/glcc/glcc_1_100_833.pth
$ wget http://www.iic.ecei.tohoku.ac.jp/~tomo/glcc/glcc_faster_rcnn_1_100_833.pth
$ cd -

Test demonstration is below.

Detection and recognition result images will be stored in result_vgg16_glcc_2class (You can change directory path by changing result_dir in demo_glcc_2class.py).

$ python demo_glcc_2class.py --net vgg16

5.Other model tips

This is other back-bone model tips.

However, I do not prepare pre-trained model, sorry:(

5-1.GLCC + ResNet-101 model

Pre-trained model is not still prepared.

$ python demo_glcc_2class.py --net res101

5-2.Global feature

train

# VGG16
$ sh scripts/train_vgg_global.sh

# Res101
# sh scripts/train_res101_global.sh

test

Pre-trained model is not still prepared.

# VGG16
$ python demo_saba_global.py --net vgg16

# Res101
$ python demo_saba_global.py --net res101

5-3.Local feature

train

# VGG16
$ sh scripts/train_vgg_local.sh

# Res101
$ sh scripts/train_res101_local.sh

test

Pre-trained model is not still prepared.

# VGG16
$ python demo_saba_local.py --net vgg16

# Res101
$ python demo_saba_local.py --net res101