[Feature] Add scheduler visualization from mmpretrain to mmocr (#1866)

* 2023/04/18 vis_scheduler_transportation_v1 * lint_fix * Update docs/zh_cn/user_guides/useful_tools.md Co-authored-by: Tong Gao <gaotongxiao@gmail.com> * Update docs/zh_cn/user_guides/useful_tools.md Co-authored-by: Tong Gao <gaotongxiao@gmail.com> * Update docs/en/user_guides/useful_tools.md Co-authored-by: Tong Gao <gaotongxiao@gmail.com> * 2023/04/25 add -d 100 --------- Co-authored-by: Tong Gao <gaotongxiao@gmail.com>
open-mmlab · Apr 25, 2023 · 4eb3cc7 · 4eb3cc7
1 parent e9a31dd
commit 4eb3cc7
Show file tree

Hide file tree

Showing 4 changed files with 401 additions and 21 deletions.
diff --git a/docs/en/user_guides/useful_tools.md b/docs/en/user_guides/useful_tools.md
@@ -1,10 +1,10 @@
 # Useful Tools
 
-## Analysis Tools
+## Visualization Tools
 
 ### Dataset Visualization Tool
 
-MMOCR provides a dataset visualization tool `tools/analysis_tools/browse_datasets.py` to help users troubleshoot possible dataset-related problems. You just need to specify the path to the training config (usually stored in `configs/textdet/dbnet/xxx.py`) or the dataset config (usually stored in `configs/textdet/_base_/datasets/xxx.py`), and the tool will automatically plots the transformed (or original) images and labels.
+MMOCR provides a dataset visualization tool `tools/visualizations/browse_datasets.py` to help users troubleshoot possible dataset-related problems. You just need to specify the path to the training config (usually stored in `configs/textdet/dbnet/xxx.py`) or the dataset config (usually stored in `configs/textdet/_base_/datasets/xxx.py`), and the tool will automatically plots the transformed (or original) images and labels.
 
 #### Usage
 
@@ -25,19 +25,19 @@ python tools/visualizations/browse_dataset.py \
 | config              | str                                   | (required) Path to the config.                                                                                                                   |
 | -o, --output-dir    | str                                   | If GUI is not available, specifying an output path to save the visualization results.                                                            |
 | -p, --phase         | str                                   | Phase of dataset to visualize. Use "train", "test" or "val" if you just want to visualize the default split. It's also possible to be a dataset variable name, which might be useful when a dataset split has multiple variants in the config. |
-| -m, --mode          | `original`, `transformed`, `pipeline` | Display mode: display original pictures or transformed pictures or comparison pictures. `original` only visualizes the original dataset & annotations; `transformed` shows the resulting images processed through all the transforms; `pipeline` shows all the intermediate images. Defaults to "transformed". |
+| -m, --mode          | `original`, `transformed`, `pipeline` | Display mode: display original pictures or transformed pictures or comparison pictures.`original` only visualizes the original dataset & annotations; `transformed` shows the resulting images processed through all the transforms; `pipeline` shows all the intermediate images. Defaults to "transformed". |
 | -t, --task          | `auto`, `textdet`, `textrecog`        | Specify the task type of the dataset. If `auto`, the task type will be inferred from the config. If the script is unable to infer the task type, you need to specify it manually. Defaults to `auto`. |
 | -n, --show-number   | int                                   | The number of samples to visualized. If not specified, display all images in the dataset.                                                        |
 | -i, --show-interval | float                                 | Interval of visualization (s), defaults to 2.                                                                                                    |
-| --cfg-options       | float                                 | Override configs. [Example](./config.md#command-line-modification)                                                                               |
+| --cfg-options       | float                                 | Override configs.[Example](./config.md#command-line-modification)                                                                                |
 
 #### Examples
 
 The following example demonstrates how to use the tool to visualize the training data used by the "DBNet_R50_icdar2015" model.
 
 ```Bash
 # Example: Visualizing the training data used by dbnet_r50dcn_v2_fpnc_1200e_icadr2015 model
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
 ```
 
 By default, the visualization mode is "transformed", and you will see the images & annotations being transformed by the pipeline:
@@ -49,23 +49,23 @@ By default, the visualization mode is "transformed", and you will see the images
 If you just want to visualize the original dataset, simply set the mode to "original":
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m original
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m original
 ```
 
 <div align=center><img src="https://user-images.githubusercontent.com/22607038/206646570-382d0f26-908a-4ab4-b1a7-5cc31fa70c5f.jpg" style=" width: auto; height: 40%; "></div>
 
 Or, to visualize the entire pipeline:
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m pipeline
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m pipeline
 ```
 
 <div align=center><img src="https://user-images.githubusercontent.com/22607038/206637571-287640c0-1f55-453f-a2fc-9f9734b9593f.jpg" style=" width: auto; height: 40%; "></div>
 
 In addition, users can also visualize the original images and their corresponding labels of the dataset by specifying the path to the dataset config file, for example:
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py
+python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py
 ```
 
 Some datasets might have multiple variants. For example, the test split of `icdar2015` textrecog dataset has two variants, which the [base dataset config](/configs/textrecog/_base_/datasets/icdar2015.py) defines as follows:
@@ -85,11 +85,58 @@ icdar2015_1811_textrecog_test = dict(
 In this case, you can specify the variant name to visualize the corresponding dataset:
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py -p icdar2015_1811_textrecog_test
+python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py -p icdar2015_1811_textrecog_test
 ```
 
 Based on this tool, users can easily verify if the annotation of a custom dataset is correct.
 
+### Hyper-parameter Scheduler Visualization
+
+This tool aims to help the user to check the hyper-parameter scheduler of the optimizer (without training), which support the "learning rate" or "momentum"
+
+#### Introduce the scheduler visualization tool
+
+```bash
+python tools/visualizations/vis_scheduler.py \
+    ${CONFIG_FILE} \
+    [-p, --parameter ${PARAMETER_NAME}] \
+    [-d, --dataset-size ${DATASET_SIZE}] \
+    [-n, --ngpus ${NUM_GPUs}] \
+    [-s, --save-path ${SAVE_PATH}] \
+    [--title ${TITLE}] \
+    [--style ${STYLE}] \
+    [--window-size ${WINDOW_SIZE}] \
+    [--cfg-options]
+```
+
+**Description of all arguments**：
+
+- `config`: The path of a model config file.
+- **`-p, --parameter`**: The param to visualize its change curve, choose from "lr" and "momentum". Default to use "lr".
+- **`-d, --dataset-size`**: The size of the datasets. If set，`build_dataset` will be skipped and `${DATASET_SIZE}` will be used as the size. Default to use the function `build_dataset`.
+- **`-n, --ngpus`**: The number of GPUs used in training, default to be 1.
+- **`-s, --save-path`**: The learning rate curve plot save path, default not to save.
+- `--title`: Title of figure. If not set, default to be config file name.
+- `--style`: Style of plt. If not set, default to be `whitegrid`.
+- `--window-size`: The shape of the display window. If not specified, it will be set to `12*7`. If used, it must be in the format `'W*H'`.
+- `--cfg-options`: Modifications to the configuration file, refer to [Learn about Configs](../user_guides/config.md).
+
+```{note}
+Loading annotations maybe consume much time, you can directly specify the size of the dataset with `-d, dataset-size` to save time.
+```
+
+#### How to plot the learning rate curve without training
+
+You can use the following command to plot the step learning rate schedule used in the config `configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py`:
+
+```bash
+python tools/visualizations/vis_scheduler.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -d 100
+```
+
+<div align=center><img src="https://user-images.githubusercontent.com/43344034/232757392-b29b8e3a-77af-451c-8786-d3b4259ab388.png" style=" width: auto; height: 40%; "></div>
+
+## Analysis Tools
+
 ### Offline Evaluation Tool
 
 For saved prediction results, we provide an offline evaluation script `tools/analysis_tools/offline_eval.py`. The following example demonstrates how to use this tool to evaluate the output of the "PSENet" model offline.
@@ -110,11 +157,11 @@ python tools/analysis_tools/offline_eval.py configs/textdet/psenet/psenet_r50_fp
 
 In addition, based on this tool, users can also convert predictions obtained from other libraries into MMOCR-supported formats, then use MMOCR's built-in metrics to evaluate them.
 
-| ARGS          | Type  | Description                                                        |
-| ------------- | ----- | ------------------------------------------------------------------ |
-| config        | str   | (required) Path to the config.                                     |
-| pkl_results   | str   | (required) The saved predictions.                                  |
-| --cfg-options | float | Override configs. [Example](./config.md#command-line-modification) |
+| ARGS          | Type  | Description                                                       |
+| ------------- | ----- | ----------------------------------------------------------------- |
+| config        | str   | (required) Path to the config.                                    |
+| pkl_results   | str   | (required) The saved predictions.                                 |
+| --cfg-options | float | Override configs.[Example](./config.md#command-line-modification) |
 
 ### Calculate FLOPs and the Number of Parameters
 

diff --git a/docs/zh_cn/user_guides/useful_tools.md b/docs/zh_cn/user_guides/useful_tools.md
@@ -1,10 +1,10 @@
 # 常用工具
 
-## 分析工具
+## 可视化工具
 
 ### 数据集可视化工具
 
-MMOCR 提供了数据集可视化工具 `tools/analysis_tools/browse_datasets.py` 以辅助用户排查可能遇到的数据集相关的问题。用户只需要指定所使用的训练配置文件（通常存放在如 `configs/textdet/dbnet/xxx.py` 文件中）或数据集配置（通常存放在 `configs/textdet/_base_/datasets/xxx.py` 文件中）路径。该工具将依据输入的配置文件类型自动将经过数据流水线（data pipeline）处理过的图像及其对应的标签，或原始图片及其对应的标签绘制出来。
+MMOCR 提供了数据集可视化工具 `tools/visualizations/browse_datasets.py` 以辅助用户排查可能遇到的数据集相关的问题。用户只需要指定所使用的训练配置文件（通常存放在如 `configs/textdet/dbnet/xxx.py` 文件中）或数据集配置（通常存放在 `configs/textdet/_base_/datasets/xxx.py` 文件中）路径。该工具将依据输入的配置文件类型自动将经过数据流水线（data pipeline）处理过的图像及其对应的标签，或原始图片及其对应的标签绘制出来。
 
 #### 支持参数
 
@@ -37,7 +37,7 @@ python tools/visualizations/browse_dataset.py \
 
 ```Bash
 # 使用默认参数可视化 "dbnet_r50dcn_v2_fpnc_1200e_icadr2015" 模型的训练数据
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
 ```
 
 默认情况下，可视化模式为 "transformed"，您将看到经由数据流水线变换过后的图像和标注：
@@ -49,23 +49,23 @@ python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet
 如果您只想可视化原始数据集，只需将模式设置为 "original"：
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m original
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m original
 ```
 
 <div align=center><img src="https://user-images.githubusercontent.com/22607038/206646570-382d0f26-908a-4ab4-b1a7-5cc31fa70c5f.jpg" style=" width: auto; height: 40%; "></div>
 
 或者，您也可以使用 "pipeline" 模式来可视化整个数据流水线的中间结果：
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m pipeline
+python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m pipeline
 ```
 
 <div align=center><img src="https://user-images.githubusercontent.com/22607038/206637571-287640c0-1f55-453f-a2fc-9f9734b9593f.jpg" style=" width: auto; height: 40%; "></div>
 
 另外，用户还可以通过指定数据集配置文件的路径来可视化数据集的原始图像及其对应的标注，例如：
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py
+python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py
 ```
 
 部分数据集可能有多个变体。例如，`icdar2015` 文本识别数据集的[配置文件](/configs/textrecog/_base_/datasets/icdar2015.py)中包含两个测试集变体，分别为 `icdar2015_textrecog_test` 和 `icdar2015_1811_textrecog_test`，如下所示：
@@ -85,11 +85,58 @@ icdar2015_1811_textrecog_test = dict(
 在这种情况下，用户可以通过指定 `-p` 参数来可视化不同的变体，例如，使用以下命令可视化 `icdar2015_1811_textrecog_test` 变体：
 
 ```Bash
-python tools/analysis_tools/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py -p icdar2015_1811_textrecog_test
+python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py -p icdar2015_1811_textrecog_test
 ```
 
 基于该工具，用户可以轻松地查看数据集的原始图像及其对应的标注，以便于检查数据集的标注是否正确。
 
+### 优化器参数策略可视化工具
+
+MMOCR提供了优化器参数可视化工具 `tools/visualizations/vis_scheduler.py` 以辅助用户排查优化器的超参数调度器（无需训练），支持学习率（learning rate）和动量（momentum）。
+
+#### 工具简介
+
+```bash
+python tools/visualizations/vis_scheduler.py \
+    ${CONFIG_FILE} \
+    [-p, --parameter ${PARAMETER_NAME}] \
+    [-d, --dataset-size ${DATASET_SIZE}] \
+    [-n, --ngpus ${NUM_GPUs}] \
+    [-s, --save-path ${SAVE_PATH}] \
+    [--title ${TITLE}] \
+    [--style ${STYLE}] \
+    [--window-size ${WINDOW_SIZE}] \
+    [--cfg-options]
+```
+
+**所有参数的说明**：
+
+- `config` : 模型配置文件的路径。
+- **`-p, parameter`**: 可视化参数名，只能为 `["lr", "momentum"]` 之一， 默认为 `"lr"`.
+- **`-d, --dataset-size`**: 数据集的大小。如果指定，`build_dataset` 将被跳过并使用这个大小作为数据集大小，默认使用 `build_dataset` 所得数据集的大小。
+- **`-n, --ngpus`**: 使用 GPU 的数量, 默认为1。
+- **`-s, --save-path`**: 保存的可视化图片的路径，默认不保存。
+- `--title`: 可视化图片的标题，默认为配置文件名。
+- `--style`: 可视化图片的风格，默认为 `whitegrid`。
+- `--window-size`: 可视化窗口大小，如果没有指定，默认为 `12*7`。如果需要指定，按照格式 \`W\*H'。
+- `--cfg-options`: 对配置文件的修改，参考[学习配置文件](../user_guides/config.md)。
+
+```{note}
+部分数据集在解析标注阶段比较耗时，可直接将 `-d, dataset-size` 指定数据集的大小，以节约时间。
+```
+
+#### 如何在开始训练前可视化学习率曲线
+
+你可以使用如下命令来绘制配置文件 `configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py` 将会使用的变化率曲线：
+
+```bash
+python tools/visualizations/vis_scheduler.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -d 100
+```
+
+<div align=center><img src="https://user-images.githubusercontent.com/43344034/232755081-cad8fe62-349d-400a-bc38-7f5d17824011.png" style=" width: auto; height: 40%; "></div>
+
+## 分析工具
+
 ### 离线评测工具
 
 对于已保存的预测结果，我们提供了离线评测脚本 `tools/analysis_tools/offline_eval.py`。例如，以下代码演示了如何使用该工具对 "PSENet" 模型的输出结果进行离线评估：

diff --git a/tools/analysis_tools/browse_dataset.py → tools/visualizations/browse_dataset.py b/tools/analysis_tools/browse_dataset.py → tools/visualizations/browse_dataset.py