如何防止单个无响应副本集成员降级整个应用程序？

Question

概述

触发器：副本集成员无响应。

问题：驱动程序将继续将连接请求路由到无响应的副本集成员。等待响应或连接的线程堆积在该成员的“前面”，直到最终耗尽应用程序的线程池为止。整个应用程序停顿数分钟，直到线程超时或返回。

我的问题：这个问题确实让我感到惊讶-在决定将请求路由到哪个成员时，驱动程序是否应该不考虑副本集成员之间的线程分配？其他用户如何处理此行为？

我已经创建了一个示例项目（https://github.com/McKeeAtx/mongo-java-driver-degradation-example），该项目演示了一个无响应的副本集成员如何使客户端应用程序在几分钟内无响应。在测试过程中使用了以下本地设置：

• OpenJDK 1.8.0_121
• macOS High Sierra 10.13.6
• mongod版本v3.4.20
• mongo-java-driver 3.10.2

初始设置

创建三个空目录rs0-0，rs0-1和rs0-2。

启动三个mongod实例：

mongod --replSet rs0 --port 27017 --bind_ip 127.0.0.1 --dbpath rs0-0 --smallfiles --oplogSize 128
mongod --replSet rs0 --port 27018 --bind_ip 127.0.0.1 --dbpath rs0-1 --smallfiles --oplogSize 128
mongod --replSet rs0 --port 27019 --bind_ip 127.0.0.1 --dbpath rs0-2 --smallfiles --oplogSize 128

登录mongo shell并配置副本集

mongo
> rsconf = { _id: "rs0", members: [ { _id: 0, host: "127.0.0.1:27017" }, { _id: 1, host: "127.0.0.1:27018" }, { _id: 2,      host: "127.0.0.1:27019" } ] }
> rs.initiate(rsconf)
{ "ok" : 1 }

创建测试数据库：

rs0:PRIMARY> use test
mongo
switched to db test

运行DataFixture，将10000个文档插入test数据库中的test集合中。

运行示例

启动三个mongod实例：

mongod --replSet rs0 --port 27017 --bind_ip 127.0.0.1 --dbpath rs0-0 --smallfiles --oplogSize 128
mongod --replSet rs0 --port 27018 --bind_ip 127.0.0.1 --dbpath rs0-1 --smallfiles --oplogSize 128
mongod --replSet rs0 --port 27019 --bind_ip 127.0.0.1 --dbpath rs0-2 --smallfiles --oplogSize 128

运行Consumer应用程序。

每次发布​​ConnectionHistogramListener事件时，ConnectionPoolWaitQueueEnteredEvent都会记录一个简单的直方图：

• connected：没有与该成员建立连接的线程
• waiting：等待与成员连接的线程数

示例：

11:47:41 [pool-2-thread-2] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 2, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}

上面的行显示为：

• 连接到127.0.0.1:27019 ：驱动程序为下一个连接请求选择 127.0.0.1:27019
• 127.0.0.1:27019 [已连接：2，正在等待：1] ：2个线程已连接到 127.0.0.1:27019 ，有1个线程正在等待连接
• 127.0.0.1:27018 [已连接：0，正在等待：0] ：0个线程已连接到 127.0.0.1:27018 ，有0个线程正在等待连接
• 127.0.0.1:27017 [已连接：0，正在等待：0] ：0个线程已连接到 127.0.0.1:27017 ，有0个线程正在等待连接

降级单个副本集成员

确定副本集成员的PID：

ps -ef | grep mongo | grep "replSet"
  501 24989 11889   0  7:42AM ttys001    0:30.81 mongod --replSet rs0 --port 27017 --bind_ip 127.0.0.1 --dbpath rs0-0 --smallfiles --oplogSize 128
  501 24990 11905   0  7:42AM ttys002    0:30.02 mongod --replSet rs0 --port 27018 --bind_ip 127.0.0.1 --dbpath rs0-1 --smallfiles --oplogSize 128
  501 24991 11922   0  7:42AM ttys005    0:30.03 mongod --replSet rs0 --port 27019 --bind_ip 127.0.0.1 --dbpath rs0-2 --smallfiles --oplogSize 128

在此示例中，我们选择24991（在端口27019上侦听的成员）。

使用cpulimit减少分配给该进程的CPU百分比：

cpulimit -p 24991 -l 0
Process 24991 found

服务器选择器算法现在继续在三个副本集成员之间轮询连接请求。所有成员都被考虑

• 符合条件
• 不陈旧
• 并在等待时间窗口内。

示例日志：

11:47:34 [pool-2-thread-1] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:34 [pool-2-thread-1] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 0, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:34 [pool-2-thread-1] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 0, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:35 [pool-2-thread-2] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:35 [pool-2-thread-2] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
11:47:35 [pool-2-thread-2] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}

// now the node 127.0.0.1:27019 is 'frozen'
cpulimit -p 24991 -l 0
Process 24991 found

11:47:36 [pool-2-thread-3] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:36 [pool-2-thread-3] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
11:47:36 [pool-2-thread-3] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
11:47:37 [pool-2-thread-4] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:37 [pool-2-thread-4] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 0, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:38 [pool-2-thread-5] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:38 [pool-2-thread-5] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 1, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:39 [pool-2-thread-6] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:39 [pool-2-thread-6] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
11:47:39 [pool-2-thread-6] -    connect to 127.0.0.1:27017: {127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
11:47:40 [pool-2-thread-1] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:40 [pool-2-thread-1] -    connect to 127.0.0.1:27018: {127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:40 [pool-2-thread-1] -    connect to 127.0.0.1:27018: {127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:41 [pool-2-thread-2] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:41 [pool-2-thread-2] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 2, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:42 [pool-2-thread-3] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:42 [pool-2-thread-3] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:43 [pool-2-thread-6] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:43 [pool-2-thread-6] -    connect to 127.0.0.1:27018: {127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:43 [pool-2-thread-6] -    connect to 127.0.0.1:27018: {127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:44 [pool-2-thread-1] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 3286, 127.0.0.1:27017: 2081}
11:47:44 [pool-2-thread-1] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 2, waiting: 3],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
11:47:45 [pool-2-thread-6] - Roundtrip times (micro): {127.0.0.1:27019: 3257, 127.0.0.1:27018: 2885, 127.0.0.1:27017: 1917}
11:47:45 [pool-2-thread-6] -    connect to 127.0.0.1:27019: {127.0.0.1:27019[connected: 2, waiting: 4],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}

节点127.0.0.1:27019未能及时响应请求，但仍由服务器选择算法选择，因为它仍处于根据过时（！）往返数据得出的可接受的等待时间窗口内。在做出每个决定127.0.0.1:27019的支持下，另一个线程堆积起来等待响应或连接。

上面的日志的简化版本显示了线程如何堆积等待127.0.0.1:27019：

127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
127.0.0.1:27019[connected: 0, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
127.0.0.1:27019[connected: 0, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 1, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 1]}
127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 0],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 1],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 2],127.0.0.1:27018[connected: 0, waiting: 1],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 3],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}
127.0.0.1:27019[connected: 2, waiting: 4],127.0.0.1:27018[connected: 0, waiting: 0],127.0.0.1:27017[connected: 0, waiting: 0]}

最终，应用程序的线程池（示例应用程序中具有6个线程的固定池）中的所有线程都将等待127.0.0.1:27019，尽管副本集成员127.0.0.1:27017和{ {1}}仍然可以处理请求。

几分钟后，等待127.0.0.1:27018的线程返回或超时，驱动程序停止将更多请求路由到127.0.0.1:27019。然后，应用程序恢复，并在正常副本集成员127.0.0.1:27019和127.0.0.1:27017之间分配流量。

从用户的角度来看，此行为导致客户端应用程序完全停机，持续数分钟。

Consumer.java

127.0.0.1:27018

Configuration.java

public class Consumer {

    public static void main( String[] args ) throws ExecutionException, InterruptedException, IOException {
        MongoClient client = Configuration.createClient(true);
        MongoDatabase db = client.getDatabase("test");
        final MongoCollection<Document> customers = db.getCollection("test");
        ExecutorService executorService = Executors.newFixedThreadPool(6);

        for (;;) {
            executorService.submit(() -> {
                FindIterable<Document> documents = customers.find();
                documents.into(new LinkedList<>());
            });
            Thread.sleep(1000);
        }
    }
}