Question

我们最终决定将生产服务器ClickHouse版本1.1.54231更新到新的19.6。新版本开始运行，获取所有配置，也可以在服务器上看到数据，但无法在Zookeeper上获取新数据。在日志中出现错误

    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: user_files_path (version 19.6.2.11 (official build))
    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: format_schema_path (version 19.6.2.11 (official build))
    Include not found: clickhouse_remote_servers
    Include not found: clickhouse_compression
    Logging error to /var/log/clickhouse-server/clickhouse-server.log
    Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
    Logging error to console
    2019.05.16 17:06:10.801816 [ 1 ] {} <Error> Application: Listen [0.0.0.0]: 98: Net Exception: Address already in use: 0.0.0.0:8123  If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
    2019.05.16 17:06:10.859589 [ 25 ] {} <Error> analytics..inner.main_page_hits (StorageReplicatedMergeTree): Code: 40, e.displayText() = DB::Exception: Checksums of parts don't match: hash of uncompressed files doesn't match (version 19.6.2.11 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 
1. Using newer version of compression library after server update. 
2. Using another compression method. 
3. Non-deterministic compression algorithm (highly unlikely). 
4. Non-deterministic merge algorithm due to logical error in code. 
5. Data corruption in memory due to bug in code. 
6. Data corruption in memory due to hardware issue. 
7. Manual modification of source data after server startup. 
8. Manual modification of checksums stored in ZooKeeper. We will download merged part from replica to force byte-identical result.
    2019.05.16 17:06:10.874886 [ 23 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8., Stack trace:

    0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x767c3f6]
    1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x39294a2]
    2. /usr/bin/clickhouse-server() [0x6d7fc8c]
    3. /usr/bin/clickhouse-server(DB::DataPartsExchange::Fetcher::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, DB::ConnectionTimeouts const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x950) [0x6d830b0]
    4. /usr/bin/clickhouse-server() [0x6d2d7a4]
    5. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, unsigned long)+0x97f) [0x6d48d5f]
    6. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&)+0x7f2) [0x6d4b012]
    7. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x26b) [0x6d4bd5b]
    8. /usr/bin/clickhouse-server() [0x6d4c31f]
    9. /usr/bin/clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x64) [0x6e6d324]
    10. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x189) [0x6d280d9]
    11. /usr/bin/clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x44e) [0x6d7da4e]
    12. /usr/bin/clickhouse-server() [0x6d7e4c9]
    13. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x1ab) [0x768238b]
    14. /usr/bin/clickhouse-server() [0xb5488bf]
    15. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f553fd076db]
    16. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f553f28688f]
     (version 19.6.2.11 (official build))
    2019.05.16 17:06:10.886495 [ 25 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8.,

有什么想法吗？

Answer 1

您似乎只更新了少数几个副本。

在复制的方案中，使用校验和对后台合并结果进行交叉检查。如果在2个副本上合并的结果不是字节相同的副本，则将该消息发送到日志，然后从其他副本重新下载有问题的部分。更新其他副本后，问题将消失。

将ClickHouse从1.1.54231升级到19.6.2.11

1 个答案: