我在本地计算机上部署了分布式配置。
1个管理员 1个容器 1个动物园管理员 1个hsql 1 redis
我有以下内容：
stream create --name activityLog --definition "tail --name=/home/bruno/randomName | log"
stream create --name activityLogCounterTap --definition "tap:stream:activityLog > json-to-tuple | filter --expression=payload.reportable.equals(true) | counter --name=activitylogcount"
stream create --name activityLogEventTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=status --name=status"
stream create --name activityLogClientTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=clientType --name=clientType"
stream create --name activityLogIndexCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=index --name=index"
stream create --name activityLogCustomerCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=customer --name=customer"
stream create --name activityLogChannelCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=channel --name=channel"
stream create --name activityLogStrategyOnlineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOnline --name=strategyOnline"
stream create --name activityLogStrategyOfflineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOffline --name=strategyOffline"
我想使用以下部署描述符进行部署：
stream deploy --name activityLog --properties "module.*.count=0"
stream deploy --name activityLogCounterTap --properties "module.*.count=0"
stream deploy --name activityLogEventTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogClientTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogIndexCounterTap --properties "module.*.count=0"
stream deploy --name activityLogCustomerCounterTap --properties "module.*.count=0"
stream deploy --name activityLogChannelCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOnlineCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOfflineCounterTap --properties "module.*.count=0"
消息是json对象。
这个想法是为容器消耗来自apache kafka的消息设置一个任意数字，但是对于这个例子，使用了一个简单的尾部。
我逐个部署，我可以看到使用redis作为传输的性能在部署第4个流后会降级很多。我观察到使用visual vm在入站中存在某种竞争条件。* redis：queue-inbound-channel-adapter1。如果我部署更多流（点击），它将变得无法使用。
（我试图在Visual VM中添加一个线程视图图像，但是我需要10个声誉：s）
似乎在这里锁定：
"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007fe3581a1800 nid=0x29c7 waiting on condition [0x00007fe3d28de000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000788b48d48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:40)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
如果部署的流少于4个，则不会发生等待情况：
"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007ff5101f6800 nid=0x352c runnable [0x00007ff5794a4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:108)
at redis.clients.util.RedisInputStream.fill(RedisInputStream.java:109)
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:45)
at redis.clients.jedis.Protocol.process(Protocol.java:120)
at redis.clients.jedis.Protocol.read(Protocol.java:191)
at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:212)
at redis.clients.jedis.BinaryJedis.brpop(BinaryJedis.java:2068)
at org.springframework.data.redis.connection.jedis.JedisConnection.bRPop(JedisConnection.java:1514)
at org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:154)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
我的第一个问题是为什么没有使用redis作为传输能够处理更多的水龙头。我尝试使用rabbitMq，它可以工作。
我的第二个问题是为什么它首先使用redis因此我部署了内存配置。再次使用内存配置，但这次使用rabbitMQ我可以跟踪传递的消息，我认为不应该发生。
- 编辑 -
关于第一个问题，我在org.apache.commons.pool2.GenericObjectPool中找出了maxTotal参数，默认为8.我进入调试模式并在运行时更改了这个值（无限制= -1），这似乎解决了问题。为了能够将其配置为使用该hack进行部署，我需要创建扩展（从文档中）并覆盖以下bean：
<bean id="messageBus" class="org.springframework.xd.dirt.integration.redis.RedisMessageBus">
<constructor-arg ref="redisConnectionFactory2" />
<constructor-arg ref="codec"/>
<property name="defaultBackOffInitialInterval" value="${xd.messagebus.redis.default.backOffInitialInterval}" />
<property name="defaultBackOffMaxInterval" value="${xd.messagebus.redis.default.backOffMaxInterval}" />
<property name="defaultBackOffMultiplier" value="${xd.messagebus.redis.default.backOffMultiplier}" />
<property name="defaultConcurrency" value="${xd.messagebus.redis.default.concurrency}" />
<property name="defaultMaxAttempts" value="${xd.messagebus.redis.default.maxAttempts}" />
</bean>
<bean id="counterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="fieldValueCounterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisFieldValueCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="gaugeRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisGaugeRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="richGaugeRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisRichGaugeRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="aggregateCounterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisAggregateCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="redisConnectionFactory2" lazy-init="false"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="hostName" value="${spring.redis.host}" />
<property name="port" value="${spring.redis.port}" />
<property name="password" value="" />
<property name="poolConfig" ref="jedisPoolConfig"/>
</bean>
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="-1"/>
</bean>
我首先尝试定义一个redisConnectionFactory bean，它会覆盖默认值，但是它没有那样工作。
我不确定解决方案是否要使server.yml中的maxTotal可配置以允许增加该值，或者如果某种方式存在无法释放poolables的错误。无论如何，我会打开一个希望，希望这可以帮助一个人不去完成我去过的所有工作： - ）
答案 0 :(得分：0)
这里的问题是Spring Boot（Spring XD所基于的）默认情况下将redis连接池大小设置为8（请参阅https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html处的spring.redis.pool.max-active）。这就是问题中提到的GenericObjectPool大小为8的原因（Jedis使用GenericObjectPool）。
同样如第二个堆栈跟踪中所示，Spring XD可以从池中获取连接并阻塞它，直到它收到消息。如果池中只有8个连接，如果在同一个XD容器中部署了多个流/抽头/队列等，这很快就会导致性能很差。
然而，在没有问题中提到的hack的情况下很容易更改默认池大小 - max-active参数已经可以在servers.yml中配置（因为这只是基于spring引导配置），例如
# Redis properties
spring:
redis:
port: 6379
host: 127.0.0.1
pool:
maxActive: 100
maxIdle: 100
# sentinel:
# master: mymaster
# nodes: 127.0.0.1:26379,127.0.0.1:26380,127.0.0.1:26381
目前在Spring XD中没有详细记录，但默认值将记录在下一版本的servers.yml中，请参阅https://jira.spring.io/browse/XD-3733