-
Notifications
You must be signed in to change notification settings - Fork 0
/
stderr.log
188 lines (186 loc) · 12.2 KB
/
stderr.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
nohup: ignoring input
wandb: Currently logged in as: peter-thomas-mchale. Use `wandb login --relogin` to force relogin
/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
wandb: Tracking run with wandb version 0.14.0
wandb: Run data is saved locally in /home/ec2-user/diffusion/wandb/run-20230326_140011-pf6bpay6
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run wobbly-gorge-1
wandb: ⭐️ View project at https://wandb.ai/peter-thomas-mchale/train-diffusion-model-on-cifar10
wandb: 🚀 View run at https://wandb.ai/peter-thomas-mchale/train-diffusion-model-on-cifar10/runs/pf6bpay6
0%| | 0/160 [00:00<?, ?it/s] 1%| | 1/160 [02:02<5:24:01, 122.27s/it] 1%|▏ | 2/160 [04:04<5:21:53, 122.23s/it] 2%|▏ | 3/160 [06:09<5:23:15, 123.54s/it] 2%|▎ | 4/160 [08:15<5:23:49, 124.55s/it] 3%|▎ | 5/160 [10:22<5:23:44, 125.32s/it] 4%|▍ | 6/160 [12:29<5:23:02, 125.86s/it] 4%|▍ | 7/160 [14:35<5:21:33, 126.10s/it] 5%|▌ | 8/160 [16:42<5:20:05, 126.35s/it] 6%|▌ | 9/160 [18:48<5:17:50, 126.30s/it] 6%|▋ | 10/160 [20:55<5:16:01, 126.41s/it] 7%|▋ | 11/160 [23:02<5:14:13, 126.53s/it] 8%|▊ | 12/160 [25:08<5:12:08, 126.55s/it] 8%|▊ | 13/160 [27:15<5:09:58, 126.52s/it] 9%|▉ | 14/160 [29:21<5:07:46, 126.48s/it] 9%|▉ | 15/160 [31:28<5:06:05, 126.66s/it] 10%|█ | 16/160 [33:35<5:03:47, 126.58s/it] 11%|█ | 17/160 [35:41<5:01:25, 126.47s/it] 11%|█▏ | 18/160 [37:47<4:59:02, 126.35s/it] 12%|█▏ | 19/160 [39:53<4:56:50, 126.31s/it] 12%|█▎ | 20/160 [42:00<4:54:52, 126.38s/it] 13%|█▎ | 21/160 [44:06<4:52:51, 126.41s/it] 14%|█▍ | 22/160 [46:13<4:51:09, 126.59s/it] 14%|█▍ | 23/160 [48:19<4:48:41, 126.43s/it] 15%|█▌ | 24/160 [50:26<4:46:42, 126.49s/it] 16%|█▌ | 25/160 [52:33<4:44:40, 126.52s/it] 16%|█▋ | 26/160 [54:40<4:42:52, 126.66s/it] 17%|█▋ | 27/160 [56:46<4:40:44, 126.65s/it] 18%|█▊ | 28/160 [58:53<4:38:47, 126.72s/it] 18%|█▊ | 29/160 [1:01:00<4:36:42, 126.73s/it] 19%|█▉ | 30/160 [1:03:06<4:34:22, 126.64s/it] 19%|█▉ | 31/160 [1:05:13<4:32:03, 126.54s/it] 20%|██ | 32/160 [1:07:19<4:29:50, 126.49s/it] 21%|██ | 33/160 [1:09:26<4:27:50, 126.54s/it] 21%|██▏ | 34/160 [1:11:32<4:25:47, 126.57s/it] 22%|██▏ | 35/160 [1:13:39<4:23:37, 126.54s/it] 22%|██▎ | 36/160 [1:15:45<4:21:15, 126.41s/it] 23%|██▎ | 37/160 [1:17:52<4:19:46, 126.72s/it] 24%|██▍ | 38/160 [1:19:59<4:17:39, 126.72s/it] 24%|██▍ | 39/160 [1:22:06<4:15:32, 126.71s/it] 25%|██▌ | 40/160 [1:24:12<4:13:13, 126.61s/it] 26%|██▌ | 41/160 [1:26:19<4:11:30, 126.81s/it] 26%|██▋ | 42/160 [1:28:25<4:08:58, 126.60s/it] 27%|██▋ | 43/160 [1:30:32<4:06:46, 126.55s/it] 28%|██▊ | 44/160 [1:32:39<4:04:43, 126.58s/it] 28%|██▊ | 45/160 [1:34:45<4:02:45, 126.65s/it] 29%|██▉ | 46/160 [1:36:52<4:00:28, 126.56s/it] 29%|██▉ | 47/160 [1:38:58<3:58:10, 126.47s/it] 30%|███ | 48/160 [1:41:04<3:56:00, 126.43s/it] 31%|███ | 49/160 [1:43:11<3:53:53, 126.43s/it] 31%|███▏ | 50/160 [1:45:17<3:51:42, 126.39s/it] 32%|███▏ | 51/160 [1:47:23<3:49:28, 126.31s/it] 32%|███▎ | 52/160 [1:49:30<3:47:24, 126.34s/it] 33%|███▎ | 53/160 [1:51:36<3:45:11, 126.28s/it] 34%|███▍ | 54/160 [1:53:42<3:43:12, 126.34s/it] 34%|███▍ | 55/160 [1:55:48<3:40:58, 126.28s/it] 35%|███▌ | 56/160 [1:57:55<3:39:12, 126.47s/it] 36%|███▌ | 57/160 [2:00:02<3:37:10, 126.51s/it] 36%|███▋ | 58/160 [2:02:08<3:35:02, 126.49s/it] 37%|███▋ | 59/160 [2:04:15<3:32:53, 126.47s/it] 38%|███▊ | 60/160 [2:06:21<3:30:34, 126.35s/it] 38%|███▊ | 61/160 [2:08:28<3:28:59, 126.66s/it] 39%|███▉ | 62/160 [2:10:34<3:26:40, 126.54s/it] 39%|███▉ | 63/160 [2:12:41<3:24:35, 126.55s/it] 40%|████ | 64/160 [2:14:48<3:22:31, 126.58s/it] 41%|████ | 65/160 [2:16:54<3:20:23, 126.56s/it] 41%|████▏ | 66/160 [2:19:01<3:18:23, 126.63s/it] 42%|████▏ | 67/160 [2:21:07<3:15:55, 126.40s/it] 42%|████▎ | 68/160 [2:23:13<3:13:36, 126.26s/it] 43%|████▎ | 69/160 [2:25:19<3:11:26, 126.23s/it] 44%|████▍ | 70/160 [2:27:25<3:09:05, 126.06s/it] 44%|████▍ | 71/160 [2:29:30<3:06:53, 125.99s/it] 45%|████▌ | 72/160 [2:31:37<3:04:59, 126.13s/it] 46%|████▌ | 73/160 [2:33:43<3:02:46, 126.05s/it] 46%|████▋ | 74/160 [2:35:49<3:00:47, 126.14s/it] 47%|████▋ | 75/160 [2:37:55<2:58:43, 126.16s/it] 48%|████▊ | 76/160 [2:40:02<2:56:48, 126.29s/it] 48%|████▊ | 77/160 [2:42:09<2:54:52, 126.41s/it] 49%|████▉ | 78/160 [2:44:16<2:53:10, 126.72s/it] 49%|████▉ | 79/160 [2:46:23<2:51:05, 126.73s/it] 50%|█████ | 80/160 [2:48:29<2:48:47, 126.59s/it] 51%|█████ | 81/160 [2:50:37<2:47:11, 126.98s/it] 51%|█████▏ | 82/160 [2:52:44<2:45:17, 127.15s/it] 52%|█████▏ | 83/160 [2:54:51<2:42:49, 126.88s/it] 52%|█████▎ | 84/160 [2:56:58<2:40:46, 126.93s/it] 53%|█████▎ | 85/160 [2:59:05<2:38:40, 126.94s/it] 54%|█████▍ | 86/160 [3:01:12<2:36:48, 127.15s/it] 54%|█████▍ | 87/160 [3:03:20<2:34:43, 127.17s/it] 55%|█████▌ | 88/160 [3:05:26<2:32:18, 126.92s/it] 56%|█████▌ | 89/160 [3:07:32<2:29:52, 126.66s/it] 56%|█████▋ | 90/160 [3:09:38<2:27:40, 126.58s/it] 57%|█████▋ | 91/160 [3:11:44<2:25:24, 126.44s/it] 57%|█████▊ | 92/160 [3:13:51<2:23:14, 126.39s/it] 58%|█████▊ | 93/160 [3:15:56<2:20:52, 126.15s/it] 59%|█████▉ | 94/160 [3:18:02<2:18:44, 126.13s/it] 59%|█████▉ | 95/160 [3:20:09<2:16:43, 126.20s/it] 60%|██████ | 96/160 [3:22:16<2:14:48, 126.38s/it] 61%|██████ | 97/160 [3:24:22<2:12:41, 126.38s/it] 61%|██████▏ | 98/160 [3:26:28<2:10:34, 126.36s/it] 62%|██████▏ | 99/160 [3:28:35<2:08:28, 126.36s/it] 62%|██████▎ | 100/160 [3:30:41<2:06:20, 126.34s/it]/home/ec2-user/diffusion/utils.py:9: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
plt.figure()
63%|██████▎ | 101/160 [3:32:49<2:04:38, 126.75s/it] 64%|██████▍ | 102/160 [3:34:55<2:02:30, 126.73s/it] 64%|██████▍ | 103/160 [3:37:02<2:00:20, 126.68s/it] 65%|██████▌ | 104/160 [3:39:08<1:58:08, 126.57s/it] 66%|██████▌ | 105/160 [3:41:14<1:55:51, 126.38s/it] 66%|██████▋ | 106/160 [3:43:20<1:53:42, 126.35s/it] 67%|██████▋ | 107/160 [3:45:27<1:51:35, 126.33s/it] 68%|██████▊ | 108/160 [3:47:33<1:49:29, 126.34s/it] 68%|██████▊ | 109/160 [3:49:40<1:47:24, 126.37s/it] 69%|██████▉ | 110/160 [3:51:45<1:45:12, 126.25s/it] 69%|██████▉ | 111/160 [3:53:52<1:43:06, 126.26s/it] 70%|███████ | 112/160 [3:55:59<1:41:11, 126.50s/it] 71%|███████ | 113/160 [3:58:05<1:39:06, 126.53s/it] 71%|███████▏ | 114/160 [4:00:11<1:36:53, 126.38s/it] 72%|███████▏ | 115/160 [4:02:18<1:34:44, 126.33s/it] 72%|███████▎ | 116/160 [4:04:24<1:32:41, 126.40s/it] 73%|███████▎ | 117/160 [4:06:31<1:30:38, 126.47s/it] 74%|███████▍ | 118/160 [4:08:37<1:28:30, 126.44s/it] 74%|███████▍ | 119/160 [4:10:44<1:26:26, 126.50s/it] 75%|███████▌ | 120/160 [4:12:50<1:24:21, 126.53s/it] 76%|███████▌ | 121/160 [4:14:58<1:22:22, 126.73s/it] 76%|███████▋ | 122/160 [4:17:04<1:20:11, 126.62s/it] 77%|███████▋ | 123/160 [4:19:10<1:18:01, 126.52s/it] 78%|███████▊ | 124/160 [4:21:17<1:15:52, 126.46s/it] 78%|███████▊ | 125/160 [4:23:23<1:13:44, 126.41s/it] 79%|███████▉ | 126/160 [4:25:29<1:11:37, 126.41s/it] 79%|███████▉ | 127/160 [4:27:36<1:09:30, 126.39s/it] 80%|████████ | 128/160 [4:29:42<1:07:21, 126.31s/it] 81%|████████ | 129/160 [4:31:48<1:05:15, 126.31s/it] 81%|████████▏ | 130/160 [4:33:54<1:03:09, 126.32s/it] 82%|████████▏ | 131/160 [4:36:00<1:01:00, 126.23s/it] 82%|████████▎ | 132/160 [4:38:06<58:49, 126.06s/it] 83%|████████▎ | 133/160 [4:40:12<56:45, 126.14s/it] 84%|████████▍ | 134/160 [4:42:19<54:45, 126.37s/it] 84%|████████▍ | 135/160 [4:44:26<52:44, 126.58s/it] 85%|████████▌ | 136/160 [4:46:33<50:35, 126.49s/it] 86%|████████▌ | 137/160 [4:48:40<48:32, 126.63s/it] 86%|████████▋ | 138/160 [4:50:46<46:25, 126.62s/it] 87%|████████▋ | 139/160 [4:52:53<44:18, 126.62s/it] 88%|████████▊ | 140/160 [4:54:59<42:10, 126.53s/it] 88%|████████▊ | 141/160 [4:57:06<40:05, 126.63s/it] 89%|████████▉ | 142/160 [4:59:13<37:59, 126.66s/it] 89%|████████▉ | 143/160 [5:01:19<35:52, 126.62s/it] 90%|█████████ | 144/160 [5:03:26<33:46, 126.66s/it] 91%|█████████ | 145/160 [5:05:33<31:39, 126.66s/it] 91%|█████████▏| 146/160 [5:07:39<29:32, 126.59s/it] 92%|█████████▏| 147/160 [5:09:45<27:24, 126.48s/it] 92%|█████████▎| 148/160 [5:11:51<25:16, 126.36s/it] 93%|█████████▎| 149/160 [5:13:58<23:09, 126.35s/it] 94%|█████████▍| 150/160 [5:16:04<21:03, 126.34s/it] 94%|█████████▍| 151/160 [5:18:10<18:56, 126.31s/it] 95%|█████████▌| 152/160 [5:20:17<16:50, 126.27s/it] 96%|█████████▌| 153/160 [5:22:23<14:43, 126.22s/it] 96%|█████████▋| 154/160 [5:24:29<12:37, 126.20s/it] 97%|█████████▋| 155/160 [5:26:35<10:30, 126.17s/it] 98%|█████████▊| 156/160 [5:28:41<08:24, 126.17s/it] 98%|█████████▊| 157/160 [5:30:47<06:18, 126.19s/it] 99%|█████████▉| 158/160 [5:32:54<04:12, 126.26s/it] 99%|█████████▉| 159/160 [5:35:00<02:06, 126.25s/it]100%|██████████| 160/160 [5:37:06<00:00, 126.22s/it]100%|██████████| 160/160 [5:37:06<00:00, 126.42s/it]
wandb: Waiting for W&B process to finish... (success).
wandb:
wandb: Run history:
wandb: train loss █▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▁▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: validation loss █▅▄▄▃▃▃▃▃▂▃▂▂▂▃▂▂▂▂▂▁▁▂▂▁▂▁▂▁▂▂▁▂▂▁▁▁▁▁▁
wandb:
wandb: Run summary:
wandb: train loss 0.05803
wandb: validation loss 0.05928
wandb:
wandb: 🚀 View run wobbly-gorge-1 at: https://wandb.ai/peter-thomas-mchale/train-diffusion-model-on-cifar10/runs/pf6bpay6
wandb: Synced 6 W&B file(s), 16 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230326_140011-pf6bpay6/logs