diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fde056..943eb59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,39 +1,98 @@ ARTOS Changelog =============== +Version 2.0-rc1 (2016-03-12) +---------------------------- + +- **[Feature]** Model Evaluation API for libartos and PyARTOS, including GUI support for evaluating models and plotting recall-precision graphs. +- **[Feature]** ARTOS now ships with `CaffeFeatureExtractor` which uses the *Caffe* library to extract image features from the layer of a CNN + and can be used as a drop-in replacement for the default HOG features. +- **[Improvement]** Completely refactored feature extraction: Any existing dependencies on HOG specifics have been removed from ARTOS and FFLD, + which are not strictly separated any more. A new feature extraction framework provides an abstraction layer which should make it easy + to implement arbitrary feature extractors. The type and parameters of the feature extractor used to learn a model will be saved with the model. + That means, the file format of model files has changed, as well as the format of background statistics, but ARTOS can still read the old format. + The GUI has been extended so that the feature extractor and its parameters can be changed at run-time in a convenient way. +- **[Improvement]** GUI controls for manipulating multiple models at once. +- **[Improvement]** GUI support for alternative image repository drivers. +- **[Improvement]** Implemented C++11 move semantics to reduce unnecessary temporary copies. +- **[Improvement]** Allow separate overlap parameters for non-maxima suppression and evaluation in `ModelEvaluator`. +- **[Improvement]** Model learner now returns informative error codes. +- **[Improvement]** Added some tools and examples which show how to use ARTOS from C/C++ if you wish to avoid the Python interface to libartos. +- **[Improvement]** Image features are now stored and processed using an optimized data container: `FeatureMatrix`. +- **[Change]** `MixedImageIterator` now stops after extracting all images. Before, it behaved like an infinite iterator: If it reached the end of a synset, + it would start over from the beginning. Now it will extract every image only once from each synset and `MixedImageIterator::ready()` will return false + when all images from all synsets have been extracted. +- **[Change]** Prefixed ARTOS-related CMake options with `ARTOS_`. +- **[Misc]** Added a boolean member variable `StationaryBackground::learnedAllOffsets` which indicates if the size of the images used for learning a + background covariance matrix was sufficient to learn covariances for all offsets. +- **[Fix]** Fixed division by zero when using many features. +- **[Fix]** Fixed a rare bug in the computation of background statistics: If the product of the maximum offset and the cell size of the feature + extractor exceeded the size of the images, the format of the computed statistics would have been invalid. + +### Incompatible changes ### + +#### General #### +- The format of model files and background statistics files created by ARTOS v2 differs from the file format used by previous versions. + +#### C API #### +- Removed `padding` paremeter from `create_detector`. The necessary amount of padding is now determined by the feature extractor automatically. + +#### C++ API #### +- Moved `FFLD::Intersector`, `FFLD::JPEGImage`, `FFLD::Mixture`, `FFLD::Model`, `FFLD::Object`, `FFLD::Patchwork`, `FFLD::Rectangle` and + `FFLD::Scene` to the `ARTOS` namespace. +- The interfaces of `FeatureExtractor`, `HOGFeatureExtractor` and `FeaturePyramid` have changed completely. +- Removed `padding` paremeter from `DPMDetection` constructor. The necessary amount of padding is now determined by the feature extractor automatically. +- Replaced `int Detection::l` (level) by `double Detection::scale`. +- Added `featureExtractor` parameter to constructor of `ModelLearnerBase`, `ModelLearner` and `ImageNetModelLearner`. +- Changed return type of `ModelLearnerBase::learn` and `ModelLearner::learn` from `bool` to `int`, while `0` indicates success. +- Changed parameters of `ModelEvaluator` constructors. +- Removed `StationaryBackground::StationaryBackground(unsigned int, unsigned int, unsigned int)`. + +### C API extensions ### + +- New `evaluator_*` functions for model evaluation. +- New functions for enumerating and changing feature extractors. +- New function: `num_feature_extractors_in_detector` +- New function: `get_image_repository_type` +- New function: `check_repository_directory` +- Additional return values for `add_model`. +- Additional return values for `learn_imagenet`, `learn_files_jpeg` and `learner_run`. +- Additional return values for `learn_bg`. + + Version 1.3 (2015-11-05) ------------------------ -- [Improvement] If there are more training images than RAM available, caching of the entire images containing positive samples may now be turned off +- **[Improvement]** If there are more training images than RAM available, caching of the entire images containing positive samples may now be turned off by setting the CMake option `CACHE_POSITIVES` to `OFF`. -- [Fix] Compatibility of PyARTOS with Pillow >= 2.0 -- [Fix] Fixed a minor bug with clustering +- **[Fix]** Compatibility of PyARTOS with Pillow >= 2.0 +- **[Fix]** Fixed a minor bug with clustering Version 1.2 (2015-11-02) ------------------------ -- [Improvement] Instead of the tar archives used by ImageNet, one may now store images and annotation files in plain directories by using the +- **[Improvement]** Instead of the tar archives used by ImageNet, one may now store images and annotation files in plain directories by using the new ImageDirectories interface as a drop-in replacement for ImageNet (see README.md for details). -- [Change] Added an abstraction layer to the model learning process: The new abstract base class ModelLearnerBase is agnostic of the concrete +- **[Change]** Added an abstraction layer to the model learning process: The new abstract base class ModelLearnerBase is agnostic of the concrete learning method, which may be the WHO method implemented by ModelLearner as well as any other linear classifier. -- [Misc] Added DPMDetection::detectMax(), which yields just the highest scoring detection on a given image (may be useful for classification-like tasks). -- [Misc] Support for legacy way of importing PIL. -- [Fix] Fixed a bug which broke AnnotationDialog if used with large images. -- [Fix] Fixed some memory leaks in the FFLD library shipped with ARTOS. -- [Fix] Minor fixes to the build process. +- **[Misc]** Added DPMDetection::detectMax(), which yields just the highest scoring detection on a given image (may be useful for classification-like tasks). +- **[Misc]** Support for legacy way of importing PIL. +- **[Fix]** Fixed a bug which broke AnnotationDialog if used with large images. +- **[Fix]** Fixed some memory leaks in the FFLD library shipped with ARTOS. +- **[Fix]** Minor fixes to the build process. Version 1.1 (2014-10-31) ------------------------ -- [Improvement] Efficient method for computing background statistics, which leverages the Fourier transform -- [Improvement] New background statistics (computed from 32k ImageNet samples) -- [Misc] Improved error message for incomplete ImageNet setup. -- [Misc] Made VOC annotations parser of FFLD compatible with non-VOC classes. -- [Fix] Fixed rare clustering bug. -- [Fix] Fixed a bug that occurred during aspect ratio clustering if the number of images was less than the number of clusters. -- [Fix] Fixed a bug that occurred during threshold optimization if there were no additional synsets available to take negative samples from. +- **[Improvement]** Efficient method for computing background statistics, which leverages the Fourier transform +- **[Improvement]** New background statistics (computed from 32k ImageNet samples) +- **[Misc]** Improved error message for incomplete ImageNet setup. +- **[Misc]** Made VOC annotations parser of FFLD compatible with non-VOC classes. +- **[Fix]** Fixed rare clustering bug. +- **[Fix]** Fixed a bug that occurred during aspect ratio clustering if the number of images was less than the number of clusters. +- **[Fix]** Fixed a bug that occurred during threshold optimization if there were no additional synsets available to take negative samples from. Version 1.0 (2014-07-09) diff --git a/docs/libartos/html/_caffe_feature_extractor_8cc.html b/docs/libartos/html/_caffe_feature_extractor_8cc.html new file mode 100644 index 0000000..28dcf0e --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8cc.html @@ -0,0 +1,140 @@ + + + + + + +ARTOS: CaffeFeatureExtractor.cc File Reference + + + + + + + + + + + + + +
+
+ + + + + + +
+
ARTOS +  2.0 +
+
Adaptive Real-Time Object Detection System
+
+
+ + + + + +
+
+ +
+
+
+ +
+ +
+ All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros
+ + +
+ +
+ +
+
+
CaffeFeatureExtractor.cc File Reference
+
+
+
#include "CaffeFeatureExtractor.h"
+#include <opencv2/highgui/highgui.hpp>
+#include <opencv2/imgproc/imgproc.hpp>
+#include <algorithm>
+#include <fstream>
+#include <sstream>
+#include <cmath>
+#include <cstdint>
+#include <cassert>
+#include "strutils.h"
+#include "portable_endian.h"
+
+Include dependency graph for CaffeFeatureExtractor.cc:
+
+
+ + +
+
+
+ + + + diff --git a/docs/libartos/html/_caffe_feature_extractor_8cc__incl.map b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.map new file mode 100644 index 0000000..99ab3a8 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.map @@ -0,0 +1,12 @@ + + + + + + + + + + + + diff --git a/docs/libartos/html/_caffe_feature_extractor_8cc__incl.md5 b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.md5 new file mode 100644 index 0000000..dce75df --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.md5 @@ -0,0 +1 @@ +6ae4c41110e685c5e13c27d324085e1b \ No newline at end of file diff --git a/docs/libartos/html/_caffe_feature_extractor_8cc__incl.png b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.png new file mode 100644 index 0000000..b780459 Binary files /dev/null and b/docs/libartos/html/_caffe_feature_extractor_8cc__incl.png differ diff --git a/docs/libartos/html/_caffe_feature_extractor_8h.html b/docs/libartos/html/_caffe_feature_extractor_8h.html new file mode 100644 index 0000000..570c07a --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h.html @@ -0,0 +1,158 @@ + + + + + + +ARTOS: CaffeFeatureExtractor.h File Reference + + + + + + + + + + + + + +
+
+ + + + + + +
+
ARTOS +  2.0 +
+
Adaptive Real-Time Object Detection System
+
+
+ + + + + +
+
+ +
+
+
+ +
+ +
+ All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros
+ + +
+ +
+ +
+ +
+
CaffeFeatureExtractor.h File Reference
+
+
+
#include "FeatureExtractor.h"
+#include <vector>
+#include <caffe/caffe.hpp>
+#include <opencv2/core/core.hpp>
+
+Include dependency graph for CaffeFeatureExtractor.h:
+
+
+ + +
+
+This graph shows which files directly or indirectly include this file:
+
+
+ + +
+
+

Go to the source code of this file.

+ + + + + + +

+Classes

class  ARTOS::CaffeFeatureExtractor
 
struct  ARTOS::CaffeFeatureExtractor::LayerParams
 
+ + + +

+Namespaces

 ARTOS
 
+
+
+ + + + diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.map b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.map new file mode 100644 index 0000000..e092e84 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.map @@ -0,0 +1,3 @@ + + + diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.md5 b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.md5 new file mode 100644 index 0000000..f0a4ab7 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.md5 @@ -0,0 +1 @@ +0040db3546a8dd28035436957265842c \ No newline at end of file diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.png b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.png new file mode 100644 index 0000000..b629c3d Binary files /dev/null and b/docs/libartos/html/_caffe_feature_extractor_8h__dep__incl.png differ diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__incl.map b/docs/libartos/html/_caffe_feature_extractor_8h__incl.map new file mode 100644 index 0000000..cd52175 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h__incl.map @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__incl.md5 b/docs/libartos/html/_caffe_feature_extractor_8h__incl.md5 new file mode 100644 index 0000000..5cae1e8 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h__incl.md5 @@ -0,0 +1 @@ +f84b88b6e85c76b86ce7747b02cda416 \ No newline at end of file diff --git a/docs/libartos/html/_caffe_feature_extractor_8h__incl.png b/docs/libartos/html/_caffe_feature_extractor_8h__incl.png new file mode 100644 index 0000000..f1c1136 Binary files /dev/null and b/docs/libartos/html/_caffe_feature_extractor_8h__incl.png differ diff --git a/docs/libartos/html/_caffe_feature_extractor_8h_source.html b/docs/libartos/html/_caffe_feature_extractor_8h_source.html new file mode 100644 index 0000000..4bb92a7 --- /dev/null +++ b/docs/libartos/html/_caffe_feature_extractor_8h_source.html @@ -0,0 +1,277 @@ + + + + + + +ARTOS: CaffeFeatureExtractor.h Source File + + + + + + + + + + + + + +
+
+ + + + + + +
+
ARTOS +  2.0 +
+
Adaptive Real-Time Object Detection System
+
+
+ + + + + +
+
+ +
+
+
+ +
+ +
+ All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros
+ + +
+ +
+ +
+
+
CaffeFeatureExtractor.h
+
+
+Go to the documentation of this file.
1 #ifndef ARTOS_CAFFEFEATUREEXTRACTOR_H
+
2 #define ARTOS_CAFFEFEATUREEXTRACTOR_H
+
3 
+
4 #include "FeatureExtractor.h"
+
5 #include <vector>
+
6 #include <caffe/caffe.hpp>
+
7 #include <opencv2/core/core.hpp>
+
8 
+
9 namespace ARTOS
+
10 {
+
11 
+ +
43 {
+
44 
+
45 public:
+
46 
+ +
52 
+
67  CaffeFeatureExtractor(const std::string & netFile, const std::string & weightsFile,
+
68  const std::string & meanFile = "", const std::string & layerName = "");
+
69 
+
70  virtual ~CaffeFeatureExtractor() {};
+
71 
+
76  virtual const char * type() const override { return "Caffe"; };
+
77 
+
81  virtual const char * name() const override;
+
82 
+
86  virtual int numFeatures() const override;
+
87 
+
91  virtual Size cellSize() const override;
+
92 
+
101  virtual Size borderSize() const override;
+
102 
+
108  virtual Size maxImageSize() const override;
+
109 
+
113  virtual bool supportsMultiThread() const override;
+
114 
+
122  virtual bool patchworkProcessing() const override;
+
123 
+
130  virtual Size patchworkPadding() const override;
+
131 
+
139  virtual Size cellsToPixels(const Size & cells) const override;
+
140 
+
148  virtual Size pixelsToCells(const Size & pixels) const override;
+
149 
+
160  virtual void extract(const JPEGImage & img, FeatureMatrix & feat) const override;
+
161 
+
173  virtual void setParam(const std::string & paramName, int32_t val) override;
+
174 
+
186  virtual void setParam(const std::string & paramName, const std::string & val) override;
+
187 
+
188 
+
189 protected:
+
190 
+
191  std::shared_ptr< caffe::Net<float> > m_net;
+
192  cv::Scalar m_mean;
+ + + + + + +
199  std::vector<int> m_layerIndices;
+
200  std::vector<Size> m_cellSize;
+
201  std::vector<Size> m_borderSize;
+
203  enum class LayerType { OTHER, CONV, POOL };
+
204 
+
205  struct LayerParams
+
206  {
+ + + + +
211  };
+
212 
+
219  virtual void loadNetwork();
+
220 
+
226  virtual void loadMean();
+
227 
+
234  virtual void loadScales();
+
235 
+
242  virtual void loadPCAParams();
+
243 
+
250  virtual void loadLayerInfo();
+
251 
+
261  virtual void getLayerParams(int layerIndex, LayerParams & params) const;
+
262 
+
266  void wrapInputLayers(std::vector<cv::Mat> & input_channels) const;
+
267 
+
276  virtual void preprocess(const JPEGImage & img, std::vector<cv::Mat> & input_channels) const;
+
277 
+
282  static std::map< std::pair<std::string, std::string>, std::weak_ptr< caffe::Net<float> > > netPool;
+
283 
+
284 
+
285 private:
+
286 
+
287  static bool initializedCaffe;
+
288 
+
289 };
+
290 
+
291 }
+
292 
+
293 #endif
+
Size kernelSize
Definition: CaffeFeatureExtractor.h:208
+
static std::map< std::pair< std::string, std::string >, std::weak_ptr< caffe::Net< float > > > netPool
Definition: CaffeFeatureExtractor.h:282
+ +
virtual void getLayerParams(int layerIndex, LayerParams &params) const
Definition: CaffeFeatureExtractor.cc:612
+
cv::Scalar m_mean
Definition: CaffeFeatureExtractor.h:192
+
LayerType layerType
Definition: CaffeFeatureExtractor.h:207
+ +
Definition: JPEGImage.h:38
+
virtual Size borderSize() const override
Definition: CaffeFeatureExtractor.cc:81
+
Definition: defs.h:19
+
virtual void loadNetwork()
Definition: CaffeFeatureExtractor.cc:369
+ +
Abstract base class for feature extractors.
Definition: FeatureExtractor.h:41
+
void wrapInputLayers(std::vector< cv::Mat > &input_channels) const
Definition: CaffeFeatureExtractor.cc:321
+ +
virtual void extract(const JPEGImage &img, FeatureMatrix &feat) const override
Definition: CaffeFeatureExtractor.cc:210
+
std::vector< Size > m_borderSize
Definition: CaffeFeatureExtractor.h:201
+
virtual const char * type() const override
Definition: CaffeFeatureExtractor.h:76
+
virtual Size cellSize() const override
Definition: CaffeFeatureExtractor.cc:72
+
FeatureCell m_scales
Definition: CaffeFeatureExtractor.h:193
+
virtual Size patchworkPadding() const override
Definition: CaffeFeatureExtractor.cc:108
+
virtual int numFeatures() const override
Definition: CaffeFeatureExtractor.cc:63
+
virtual Size cellsToPixels(const Size &cells) const override
Definition: CaffeFeatureExtractor.cc:114
+
virtual void loadLayerInfo()
Definition: CaffeFeatureExtractor.cc:563
+
int m_numChannels
Definition: CaffeFeatureExtractor.h:197
+
int m_lastLayer
Definition: CaffeFeatureExtractor.h:196
+
virtual void loadMean()
Definition: CaffeFeatureExtractor.cc:426
+
int m_numOutputChannels
Definition: CaffeFeatureExtractor.h:198
+
virtual void preprocess(const JPEGImage &img, std::vector< cv::Mat > &input_channels) const
Definition: CaffeFeatureExtractor.cc:334
+
Size stride
Definition: CaffeFeatureExtractor.h:210
+
virtual Size maxImageSize() const override
Definition: CaffeFeatureExtractor.cc:90
+
virtual const char * name() const override
Definition: CaffeFeatureExtractor.cc:57
+
FeatureMatrix::ScalarMatrix ScalarMatrix
Definition: FeatureMatrix.h:674
+
virtual bool supportsMultiThread() const override
Definition: CaffeFeatureExtractor.cc:96
+
virtual void loadScales()
Definition: CaffeFeatureExtractor.cc:480
+ +
virtual bool patchworkProcessing() const override
Definition: CaffeFeatureExtractor.cc:102
+
virtual void setParam(const std::string &paramName, int32_t val) override
+
Definition: CaffeFeatureExtractor.h:205
+
virtual ~CaffeFeatureExtractor()
Definition: CaffeFeatureExtractor.h:70
+
Definition: CaffeFeatureExtractor.h:42
+
FeatureMatrix::Cell FeatureCell
Definition: FeatureMatrix.h:673
+
std::vector< int > m_layerIndices
Definition: CaffeFeatureExtractor.h:199
+
std::vector< Size > m_cellSize
Definition: CaffeFeatureExtractor.h:200
+
virtual Size pixelsToCells(const Size &pixels) const override
Definition: CaffeFeatureExtractor.cc:136
+
virtual void loadPCAParams()
Definition: CaffeFeatureExtractor.cc:509
+
LayerType
Definition: CaffeFeatureExtractor.h:203
+
FeatureCell m_pcaMean
Definition: CaffeFeatureExtractor.h:194
+
ScalarMatrix m_pcaTransform
Definition: CaffeFeatureExtractor.h:195
+
CaffeFeatureExtractor()
Definition: CaffeFeatureExtractor.cc:22
+
std::shared_ptr< caffe::Net< float > > m_net
Definition: CaffeFeatureExtractor.h:191
+
Size padding
Definition: CaffeFeatureExtractor.h:209
+
+
+ + + + diff --git a/docs/libartos/html/_d_p_m_detection_8cc.html b/docs/libartos/html/_d_p_m_detection_8cc.html index 3078b3d..181e7f9 100644 --- a/docs/libartos/html/_d_p_m_detection_8cc.html +++ b/docs/libartos/html/_d_p_m_detection_8cc.html @@ -121,7 +121,7 @@
- +
@@ -129,7 +129,7 @@
This graph shows which files directly or indirectly include this file:
- +

Go to the source code of this file.

@@ -151,7 +151,7 @@