CouchDB 2.3.1 Connection Refused

[akircher]

Description

Production instance suddenly failed (no recent software updates). Curl/Telnet to 127.0.01:5984 now results in "CouchDB 2.3.1 Connection Refused". Have tried restarting the couchdb service, rebooting the server (x-large on AWS). I don't see any issues in the logs other than a seemingly one-time reduce error.

Steps to Reproduce

telnet 127.0.0.1 5984
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Have also tried with curl and other ports (5986, 6984). I have also tried changing the bind_address from 127.0.0.1 to 0.0.0.0 but that had no apparent effect. I have an identical production server (that replicates with this one) that is still running fine.

Expected Behaviour

CouchDB should accepted incoming connections and Fauxton should be available.

Your Environment

CouchDB 2.3.1
Ubuntu 16 LTS

Additional Context

Log of the initial crash and my subsequent attempts to reboot

[notice] 2020-08-26T16:10:37.646795Z couchdb@127.0.0.1 <0.15411.59> 35ebe29d52 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[notice] 2020-08-26T16:10:37.710732Z couchdb@127.0.0.1 <0.20303.5232> a71b76a545 172.30.2.222:5984 172.30.2.58 undefined GET / 200 ok 0
[notice] 2020-08-26T16:10:38.638786Z couchdb@127.0.0.1 <0.15744.1275> 4b152905af 172.30.2.222:5984 172.30.2.228 admin GET /drug/_changes?feed=continuous&style=all_docs&since=%$
[notice] 2020-08-26T16:10:39.361697Z couchdb@127.0.0.1 <0.8589.5232> 163e07f1d7 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?since=%224646814-g1AAAAJ7eJzLYWB$
[notice] 2020-08-26T16:10:41.356314Z couchdb@127.0.0.1 <0.13133.5232> 4936bf49c9 52.53.218.125:5984 205.185.113.151 undefined GET / 200 ok 0
[notice] 2020-08-26T16:10:41.879808Z couchdb@127.0.0.1 <0.314.0> 52f1b80313 172.30.2.222:5984 172.30.2.228 admin GET /_users/_changes?feed=continuous&style=all_docs&since=%227$
[notice] 2020-08-26T16:10:42.362891Z couchdb@127.0.0.1 <0.9761.5173> bf390effac 172.30.2.222:5984 172.30.2.228 admin GET /user/_changes?feed=continuous&style=all_docs&since=%2$
[notice] 2020-08-26T16:10:42.588798Z couchdb@127.0.0.1 <0.10922.1430> 79df22a428 172.30.2.222:5984 172.30.2.228 admin GET /shipment/_changes?feed=continuous&style=all_docs&sin$
[notice] 2020-08-26T16:10:42.886834Z couchdb@127.0.0.1 <0.6465.118> a920665437 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?feed=continuous&style=all_docs&si$
[notice] 2020-08-26T16:10:47.650844Z couchdb@127.0.0.1 <0.15411.59> 3803086aec 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[notice] 2020-08-26T16:10:48.642918Z couchdb@127.0.0.1 <0.15744.1275> ed7ba8cd72 172.30.2.222:5984 172.30.2.228 admin GET /drug/_changes?feed=continuous&style=all_docs&since=%$
[notice] 2020-08-26T16:10:51.883949Z couchdb@127.0.0.1 <0.314.0> bd75454636 172.30.2.222:5984 172.30.2.228 admin GET /_users/_changes?feed=continuous&style=all_docs&since=%227$
[notice] 2020-08-26T16:10:52.368261Z couchdb@127.0.0.1 <0.9761.5173> e6e73c6c66 172.30.2.222:5984 172.30.2.228 admin GET /user/_changes?feed=continuous&style=all_docs&since=%2$
[notice] 2020-08-26T16:10:52.593764Z couchdb@127.0.0.1 <0.10922.1430> 5b3c99531f 172.30.2.222:5984 172.30.2.228 admin GET /shipment/_changes?feed=continuous&style=all_docs&sin$
[notice] 2020-08-26T16:10:52.890822Z couchdb@127.0.0.1 <0.6465.118> e3293f8847 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?feed=continuous&style=all_docs&si$
[notice] 2020-08-26T16:10:53.980521Z couchdb@127.0.0.1 <0.16447.5232> 644d28a6c0 172.30.2.222:5984 172.30.1.157 undefined GET / 200 ok 1
[info] 2020-08-26T16:10:54.775025Z couchdb@127.0.0.1 <0.2448.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775175Z couchdb@127.0.0.1 <0.2439.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775176Z couchdb@127.0.0.1 <0.2440.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775281Z couchdb@127.0.0.1 <0.2434.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775373Z couchdb@127.0.0.1 <0.2444.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775435Z couchdb@127.0.0.1 <0.2447.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775462Z couchdb@127.0.0.1 <0.2441.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775489Z couchdb@127.0.0.1 <0.2433.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.944593Z couchdb@127.0.0.1 <0.2444.3> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.966501Z couchdb@127.0.0.1 <0.2448.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.969693Z couchdb@127.0.0.1 <0.2447.3> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.975892Z couchdb@127.0.0.1 <0.2441.3> -------- Index update finished for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.989251Z couchdb@127.0.0.1 <0.2434.3> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.994995Z couchdb@127.0.0.1 <0.2433.3> -------- Index update finished for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.996715Z couchdb@127.0.0.1 <0.2440.3> -------- Index update finished for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:55.000703Z couchdb@127.0.0.1 <0.2439.3> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/verified-by$
[notice] 2020-08-26T16:10:56.107851Z couchdb@127.0.0.1 <0.6076.5232> a50e26f55e v2.goodpill.org:8443 172.69.33.52 admin GET /transaction/_design/verified-by-generic/_view/veri$
[info] 2020-08-26T16:10:57.307708Z couchdb@127.0.0.1 <0.3351.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.307863Z couchdb@127.0.0.1 <0.3370.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308013Z couchdb@127.0.0.1 <0.3353.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308007Z couchdb@127.0.0.1 <0.3368.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308106Z couchdb@127.0.0.1 <0.3365.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308156Z couchdb@127.0.0.1 <0.3354.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308233Z couchdb@127.0.0.1 <0.3361.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308249Z couchdb@127.0.0.1 <0.3357.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.456208Z couchdb@127.0.0.1 <0.3370.3> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.518071Z couchdb@127.0.0.1 <0.3361.3> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.543192Z couchdb@127.0.0.1 <0.3354.3> -------- Index update finished for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.552580Z couchdb@127.0.0.1 <0.3365.3> -------- Index update finished for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.583668Z couchdb@127.0.0.1 <0.3351.3> -------- Index update finished for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.627098Z couchdb@127.0.0.1 <0.3353.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.648730Z couchdb@127.0.0.1 <0.3357.3> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/expired-by-$
[notice] 2020-08-26T16:10:57.656013Z couchdb@127.0.0.1 <0.15411.59> a6eb4f8394 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[info] 2020-08-26T16:10:57.687183Z couchdb@127.0.0.1 <0.3368.3> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/expired-by-$
[notice] 2020-08-26T16:10:58.208277Z couchdb@127.0.0.1 <0.8285.5232> 2a7578e2d1 v2.goodpill.org:8443 172.69.35.64 admin GET /transaction/_design/expired-by-generic/_view/expir$
[info] 2020-08-26T16:10:58.322650Z couchdb@127.0.0.1 <0.2719.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.322768Z couchdb@127.0.0.1 <0.2730.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/disposed-by                                                                                           
[notice] 2020-08-26T16:10:37.646795Z couchdb@127.0.0.1 <0.15411.59> 35ebe29d52 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[notice] 2020-08-26T16:10:37.710732Z couchdb@127.0.0.1 <0.20303.5232> a71b76a545 172.30.2.222:5984 172.30.2.58 undefined GET / 200 ok 0
[notice] 2020-08-26T16:10:38.638786Z couchdb@127.0.0.1 <0.15744.1275> 4b152905af 172.30.2.222:5984 172.30.2.228 admin GET /drug/_changes?feed=continuous&style=all_docs&since=%$
[notice] 2020-08-26T16:10:39.361697Z couchdb@127.0.0.1 <0.8589.5232> 163e07f1d7 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?since=%224646814-g1AAAAJ7eJzLYWB$
[notice] 2020-08-26T16:10:41.356314Z couchdb@127.0.0.1 <0.13133.5232> 4936bf49c9 52.53.218.125:5984 205.185.113.151 undefined GET / 200 ok 0
[notice] 2020-08-26T16:10:41.879808Z couchdb@127.0.0.1 <0.314.0> 52f1b80313 172.30.2.222:5984 172.30.2.228 admin GET /_users/_changes?feed=continuous&style=all_docs&since=%227$
[notice] 2020-08-26T16:10:42.362891Z couchdb@127.0.0.1 <0.9761.5173> bf390effac 172.30.2.222:5984 172.30.2.228 admin GET /user/_changes?feed=continuous&style=all_docs&since=%2$
[notice] 2020-08-26T16:10:42.588798Z couchdb@127.0.0.1 <0.10922.1430> 79df22a428 172.30.2.222:5984 172.30.2.228 admin GET /shipment/_changes?feed=continuous&style=all_docs&sin$
[notice] 2020-08-26T16:10:42.886834Z couchdb@127.0.0.1 <0.6465.118> a920665437 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?feed=continuous&style=all_docs&si$
[notice] 2020-08-26T16:10:47.650844Z couchdb@127.0.0.1 <0.15411.59> 3803086aec 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[notice] 2020-08-26T16:10:48.642918Z couchdb@127.0.0.1 <0.15744.1275> ed7ba8cd72 172.30.2.222:5984 172.30.2.228 admin GET /drug/_changes?feed=continuous&style=all_docs&since=%$
[notice] 2020-08-26T16:10:51.883949Z couchdb@127.0.0.1 <0.314.0> bd75454636 172.30.2.222:5984 172.30.2.228 admin GET /_users/_changes?feed=continuous&style=all_docs&since=%227$
[notice] 2020-08-26T16:10:52.368261Z couchdb@127.0.0.1 <0.9761.5173> e6e73c6c66 172.30.2.222:5984 172.30.2.228 admin GET /user/_changes?feed=continuous&style=all_docs&since=%2$
[notice] 2020-08-26T16:10:52.593764Z couchdb@127.0.0.1 <0.10922.1430> 5b3c99531f 172.30.2.222:5984 172.30.2.228 admin GET /shipment/_changes?feed=continuous&style=all_docs&sin$
[notice] 2020-08-26T16:10:52.890822Z couchdb@127.0.0.1 <0.6465.118> e3293f8847 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?feed=continuous&style=all_docs&si$
[notice] 2020-08-26T16:10:53.980521Z couchdb@127.0.0.1 <0.16447.5232> 644d28a6c0 172.30.2.222:5984 172.30.1.157 undefined GET / 200 ok 1
[info] 2020-08-26T16:10:54.775025Z couchdb@127.0.0.1 <0.2448.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775175Z couchdb@127.0.0.1 <0.2439.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775176Z couchdb@127.0.0.1 <0.2440.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775281Z couchdb@127.0.0.1 <0.2434.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775373Z couchdb@127.0.0.1 <0.2444.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775435Z couchdb@127.0.0.1 <0.2447.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775462Z couchdb@127.0.0.1 <0.2441.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.775489Z couchdb@127.0.0.1 <0.2433.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.944593Z couchdb@127.0.0.1 <0.2444.3> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.966501Z couchdb@127.0.0.1 <0.2448.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.969693Z couchdb@127.0.0.1 <0.2447.3> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.975892Z couchdb@127.0.0.1 <0.2441.3> -------- Index update finished for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.989251Z couchdb@127.0.0.1 <0.2434.3> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.994995Z couchdb@127.0.0.1 <0.2433.3> -------- Index update finished for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:54.996715Z couchdb@127.0.0.1 <0.2440.3> -------- Index update finished for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/verified-by$
[info] 2020-08-26T16:10:55.000703Z couchdb@127.0.0.1 <0.2439.3> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/verified-by$
[notice] 2020-08-26T16:10:56.107851Z couchdb@127.0.0.1 <0.6076.5232> a50e26f55e v2.goodpill.org:8443 172.69.33.52 admin GET /transaction/_design/verified-by-generic/_view/veri$
[info] 2020-08-26T16:10:57.307708Z couchdb@127.0.0.1 <0.3351.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.307863Z couchdb@127.0.0.1 <0.3370.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308013Z couchdb@127.0.0.1 <0.3353.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308007Z couchdb@127.0.0.1 <0.3368.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308106Z couchdb@127.0.0.1 <0.3365.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308156Z couchdb@127.0.0.1 <0.3354.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308233Z couchdb@127.0.0.1 <0.3361.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.308249Z couchdb@127.0.0.1 <0.3357.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.456208Z couchdb@127.0.0.1 <0.3370.3> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.518071Z couchdb@127.0.0.1 <0.3361.3> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.543192Z couchdb@127.0.0.1 <0.3354.3> -------- Index update finished for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.552580Z couchdb@127.0.0.1 <0.3365.3> -------- Index update finished for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.583668Z couchdb@127.0.0.1 <0.3351.3> -------- Index update finished for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.627098Z couchdb@127.0.0.1 <0.3353.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/expired-by-$
[info] 2020-08-26T16:10:57.648730Z couchdb@127.0.0.1 <0.3357.3> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/expired-by-$
[notice] 2020-08-26T16:10:57.656013Z couchdb@127.0.0.1 <0.15411.59> a6eb4f8394 172.30.2.222:5984 172.30.2.228 admin GET /account/_changes?feed=continuous&style=all_docs&since=$
[info] 2020-08-26T16:10:57.687183Z couchdb@127.0.0.1 <0.3368.3> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/expired-by-$
[notice] 2020-08-26T16:10:58.208277Z couchdb@127.0.0.1 <0.8285.5232> 2a7578e2d1 v2.goodpill.org:8443 172.69.35.64 admin GET /transaction/_design/expired-by-generic/_view/expir$
[info] 2020-08-26T16:10:58.322650Z couchdb@127.0.0.1 <0.2719.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.322768Z couchdb@127.0.0.1 <0.2730.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/dispo                                                                                           
[info] 2020-08-26T16:10:58.322650Z couchdb@127.0.0.1 <0.2719.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.322768Z couchdb@127.0.0.1 <0.2730.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.322957Z couchdb@127.0.0.1 <0.2727.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.323012Z couchdb@127.0.0.1 <0.2717.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.323083Z couchdb@127.0.0.1 <0.2721.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.323100Z couchdb@127.0.0.1 <0.2725.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.323149Z couchdb@127.0.0.1 <0.2728.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.323180Z couchdb@127.0.0.1 <0.2720.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.386644Z couchdb@127.0.0.1 <0.2720.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.387384Z couchdb@127.0.0.1 <0.2721.3> -------- Index update finished for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.390432Z couchdb@127.0.0.1 <0.2730.3> -------- Index update finished for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.394638Z couchdb@127.0.0.1 <0.2719.3> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.399942Z couchdb@127.0.0.1 <0.2728.3> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.403973Z couchdb@127.0.0.1 <0.2725.3> -------- Index update finished for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.403977Z couchdb@127.0.0.1 <0.2717.3> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/disposed-by$
[info] 2020-08-26T16:10:58.408398Z couchdb@127.0.0.1 <0.2727.3> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/disposed-by$
[notice] 2020-08-26T16:10:58.502339Z couchdb@127.0.0.1 <0.6076.5232> 41a4fbd2ed v2.goodpill.org:8443 172.69.34.203 admin GET /transaction/_design/disposed-by-generic/_view/dis$
[notice] 2020-08-26T16:10:58.648842Z couchdb@127.0.0.1 <0.15744.1275> 6e4c25be1e 172.30.2.222:5984 172.30.2.228 admin GET /drug/_changes?feed=continuous&style=all_docs&since=%$
[info] 2020-08-26T16:10:59.026934Z couchdb@127.0.0.1 <0.24987.0> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.026934Z couchdb@127.0.0.1 <0.24993.0> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.027238Z couchdb@127.0.0.1 <0.25019.0> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.027267Z couchdb@127.0.0.1 <0.24981.0> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.027260Z couchdb@127.0.0.1 <0.24982.0> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.033613Z couchdb@127.0.0.1 <0.24981.0> -------- Index update finished for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.038339Z couchdb@127.0.0.1 <0.24982.0> -------- Index update finished for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.039734Z couchdb@127.0.0.1 <0.25019.0> -------- Index update finished for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.064115Z couchdb@127.0.0.1 <0.24987.0> -------- Index update finished for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/inventory-$
[info] 2020-08-26T16:10:59.072657Z couchdb@127.0.0.1 <0.24993.0> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/inventory-$
[notice] 2020-08-26T16:11:00.920734Z couchdb@127.0.0.1 <0.6076.5232> 639f64fdac v2.goodpill.org:8443 172.69.34.163 admin GET /transaction/_design/inventory-by-generic/_view/in$
[notice] 2020-08-26T16:11:01.889856Z couchdb@127.0.0.1 <0.314.0> 8268344bf0 172.30.2.222:5984 172.30.2.228 admin GET /_users/_changes?feed=continuous&style=all_docs&since=%227$
[notice] 2020-08-26T16:11:02.373732Z couchdb@127.0.0.1 <0.9761.5173> 8f292e1c90 172.30.2.222:5984 172.30.2.228 admin GET /user/_changes?feed=continuous&style=all_docs&since=%2$
[notice] 2020-08-26T16:11:02.597769Z couchdb@127.0.0.1 <0.10922.1430> 002493566d 172.30.2.222:5984 172.30.2.228 admin GET /shipment/_changes?feed=continuous&style=all_docs&sin$
[notice] 2020-08-26T16:11:02.895876Z couchdb@127.0.0.1 <0.6465.118> c963f9bc0d 172.30.2.222:5984 172.30.2.228 admin GET /transaction/_changes?feed=continuous&style=all_docs&si$
[info] 2020-08-26T16:11:03.080876Z couchdb@127.0.0.1 <0.2562.3> -------- Starting index update for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.081079Z couchdb@127.0.0.1 <0.2549.3> -------- Starting index update for db: shards/20000000-3fffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.081445Z couchdb@127.0.0.1 <0.2557.3> -------- Starting index update for db: shards/e0000000-ffffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.081540Z couchdb@127.0.0.1 <0.2554.3> -------- Starting index update for db: shards/40000000-5fffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.081643Z couchdb@127.0.0.1 <0.2598.3> -------- Starting index update for db: shards/a0000000-bfffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.081752Z couchdb@127.0.0.1 <0.2563.3> -------- Starting index update for db: shards/80000000-9fffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.082540Z couchdb@127.0.0.1 <0.2552.3> -------- Starting index update for db: shards/60000000-7fffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.082651Z couchdb@127.0.0.1 <0.2560.3> -------- Starting index update for db: shards/c0000000-dfffffff/transaction.1577570871 idx: _design/refused-by-$
[info] 2020-08-26T16:11:03.227615Z couchdb@127.0.0.1 <0.2562.3> -------- Index update finished for db: shards/00000000-1fffffff/transaction.1577570871 idx: _design/refused-by-$
[error] 2020-08-26T16:11:03.231062Z couchdb@127.0.0.1 <0.20793.5232> d031e849f1 rexi_server: from: couchdb@127.0.0.1(<0.6076.5232>) mfa: fabric_rpc:reduce_view/4 throw:{{badmatch,{error,enospc}},[{couch_btree,'-write_node/3-lc$^0/1-0-',5,[{fil[info] 2020-08-26T16:56:12.589598Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_log start
[info] 2020-08-26T16:56:12.601493Z couchdb@127.0.0.1 <0.9.0> -------- Application folsom started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.718435Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_stats started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.718557Z couchdb@127.0.0.1 <0.9.0> -------- Application khash started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.732216Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_event started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.732338Z couchdb@127.0.0.1 <0.9.0> -------- Application hyper started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.747809Z couchdb@127.0.0.1 <0.9.0> -------- Application ibrowse started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.767282Z couchdb@127.0.0.1 <0.9.0> -------- Application ioq started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.767727Z couchdb@127.0.0.1 <0.9.0> -------- Application mochiweb started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:12.793507Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB 2.3.1 is starting.
[info] 2020-08-26T16:56:12.793507Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB 2.3.1 is starting.

[info] 2020-08-26T16:56:12.794055Z couchdb@127.0.0.1 <0.213.0> -------- Starting couch_sup
[notice] 2020-08-26T16:56:12.836012Z couchdb@127.0.0.1 <0.96.0> -------- config: [features] pluggable-storage-engines set to true for reason nil
[info] 2020-08-26T16:56:13.096101Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB has started. Time to relax.

[info] 2020-08-26T16:56:13.096245Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/
[info] 2020-08-26T16:56:13.096377Z couchdb@127.0.0.1 <0.9.0> -------- Application couch started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.096459Z couchdb@127.0.0.1 <0.9.0> -------- Application ets_lru started on node 'couchdb@127.0.0.1'
[notice] 2020-08-26T16:56:13.121686Z couchdb@127.0.0.1 <0.272.0> -------- rexi_server : started servers
[notice] 2020-08-26T16:56:13.125992Z couchdb@127.0.0.1 <0.276.0> -------- rexi_buffer : started servers
[info] 2020-08-26T16:56:13.126241Z couchdb@127.0.0.1 <0.9.0> -------- Application rexi started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.196991Z couchdb@127.0.0.1 <0.9.0> -------- Application mem3 started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.197026Z couchdb@127.0.0.1 <0.9.0> -------- Application fabric started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.230658Z couchdb@127.0.0.1 <0.9.0> -------- Application chttpd started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.247023Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_index started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.247120Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_mrview started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.247151Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_plugins started on node 'couchdb@127.0.0.1'
[notice] 2020-08-26T16:56:13.361394Z couchdb@127.0.0.1 <0.96.0> -------- config: [features] scheduler set to true for reason nil
[info] 2020-08-26T16:56:13.407297Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_replicator started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.420027Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_peruser started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.437040Z couchdb@127.0.0.1 <0.9.0> -------- Application ddoc_cache started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.456262Z couchdb@127.0.0.1 <0.9.0> -------- Application global_changes started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.456288Z couchdb@127.0.0.1 <0.9.0> -------- Application jiffy started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.465002Z couchdb@127.0.0.1 <0.9.0> -------- Application mango started on node 'couchdb@127.0.0.1'
[info] 2020-08-26T16:56:13.473204Z couchdb@127.0.0.1 <0.9.0> -------- Application setup started on node 'couchdb@127.0.0.1'

[wohali]

Your log shows:

[error] 2020-08-26T16:11:03.231062Z couchdb@127.0.0.1 <0.20793.5232> d031e849f1 rexi_server: from: couchdb@127.0.0.1(<0.6076.5232>) mfa: fabric_rpc:reduce_view/4 throw:{{badmatch,{error,enospc}},[{couch_btree,'-write_node/3-lc$^0/1-0-',5,[{fil[info] 2020-08-26T16:56:12.589598Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_log start

That error,enospc means your disk is out of space.

[akircher]

Thank you so much!!! I increased the AWS Volume Size and now I can connect again.