Assert from rocksdb posix_logger

[neverchanje]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

If possible, provide a recipe for reproducing the error.

Normal rolling-update on an existing cluster. (c4tst-perfomance)

2. What did you expect to see?

No error.

3. What did you see instead?

pegasus_server: /home/wutao1/pegasus/rocksdb/env/posix_logger.h:151: virtual void rocksdb::PosixLogger::Logv(const char*, __va_list_tag*): Assertion `sz == write_size' failed.

(gdb) bt
#0  0x00007f7845b641d7 in raise () from /lib64/libc.so.6
#1  0x00007f7845b658c8 in abort () from /lib64/libc.so.6
#2  0x00007f7845b5d146 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007f7845b5d1f2 in __assert_fail () from /lib64/libc.so.6
#4  0x0000000000924572 in rocksdb::PosixLogger::Logv(char const*, __va_list_tag*) ()
#5  0x0000000000896cb3 in rocksdb::Log(rocksdb::InfoLogLevel, rocksdb::Logger*, char const*, ...) ()
#6  0x00000000008a35ac in rocksdb::ImmutableDBOptions::Dump(rocksdb::Logger*) const ()
#7  0x00000000007e6fd3 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::string const&) ()
#8  0x000000000081c0ef in rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**) ()
#9  0x000000000081d856 in rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**) ()
#10 0x0000000000701e98 in pegasus::server::pegasus_server_impl::start (this=0x57d0000, argc=<optimized out>, argv=<optimized out>)
    at /home/wutao1/pegasus/src/server/pegasus_server_impl.cpp:1515
#11 0x00007f784964e08b in dsn::replication::replication_app_base::open (this=this@entry=0x57d0000)
    at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replication_app_base.cpp:425
#12 0x00007f7849650968 in dsn::replication::replication_app_base::open_internal (this=0x57d0000, r=r@entry=0x2f6f700)
    at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replication_app_base.cpp:346
#13 0x00007f784960e0af in dsn::replication::replica::init_app_and_prepare_list (this=this@entry=0x2f6f700, create_new=create_new@entry=false)
    at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:211
#14 0x00007f784960f45a in dsn::replication::replica::initialize_on_load (this=this@entry=0x2f6f700) at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:121
#15 0x00007f784960f88a in dsn::replication::replica::load (stub=0x27de580, dir=0x2e185b8 "/home/work/ssd6/pegasus/c4tst-perfomance/replica/reps/60.33.pegasus")
    at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:162
#16 0x00007f784962e801 in operator() (__closure=0x2cf8fe0) at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_stub.cpp:397
#17 std::_Function_handler<void(), dsn::replication::replica_stub::initialize(const dsn::replication::replication_options&, bool)::__lambda14>::_M_invoke(const std::_Any_data &) (
    __functor=...) at /home/wutao1/app/include/c++/4.8.2/functional:2071
#18 0x00007f784970b149 in dsn::task::exec_internal (this=this@entry=0x2e1ec60) at /home/wutao1/pegasus/rdsn/src/core/core/task.cpp:180
#19 0x00007f784971eedd in dsn::task_worker::loop (this=0x2c704d0) at /home/wutao1/pegasus/rdsn/src/core/core/task_worker.cpp:211
#20 0x00007f784971f0a9 in dsn::task_worker::run_internal (this=0x2c704d0) at /home/wutao1/pegasus/rdsn/src/core/core/task_worker.cpp:191
#21 0x00007f78464bc600 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>)
    at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#22 0x00007f784712fdc5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f7845c2673d in clone () from /lib64/libc.so.6

4. What version of Pegasus are you using?

pegasus (bdc0299)
rocksdb (dc91f8a9567b65cea9cc7d1cc792ddd764141638)

The cause for this error may be the incompatible upgrade of rocksdb.

[acelyc111]

磁盘满了吗？

[neverchanje]

@acelyc111 不知道，我现在已经清了，清的时候没有看磁盘容量，好像很满了。
之前也有社区用户有这个问题。