我在Spring Integration 4.1.0中使用Spring 4.1.2。
我有一个用例,我希望生成一个文件,该文件将包含流向某个频道的每条消息的行。收到的消息都是String
类型。这个文件是一个很好用的文件,这意味着没有必要让这个文件的写入在主流的同一个事务中。因此,可以为用例实现异步线控模式。然而,写入该文件的任何消息都必须与它们最初接收的顺序相同(因此要么1个线程需要处理它们,要么需要聚合器等待多个线程完成,然后按原始顺序写入它们)。
我想了解一下处理这个用例的最高性能方法,所以我尝试了一些测试。为了使它更容易,我的测试没有使用异步wire-tap(但在用例中提到了这一点,因为可能有些建议可能涉及批处理/缓冲解决方案)。
总体流程来自"定义整合流程"此链接的一部分:https://spring.io/guides/gs/integration/
我尝试的主要选项是:
int-file:outbound-channel-adapter
(创建FileWritingMessageHandler
)以及为每条消息附加换行符的转换器(转换器使用SpEL表达式payload + '#{systemProperties['line.separator']}
。)
spring.expression.compiler.mode=OFF
int-file:outbound-channel-adapter
(创建FileWritingMessageHandler
)以及为每条消息附加换行符的转换器(转换器使用SpEL表达式payload.toString() + '#{systemProperties['line.separator']}
。)
spring.expression.compiler.mode=MIXED
payload.toString()
而不是payload
来解决SpEL问题:https://jira.spring.io/browse/SPR-12514 int:logging-channel-adapter
而非int-file:outbound-channel-adapter
(省去必须使用带SpEL表达式的变压器)。RollingRandomAccessFile
和同步记录器使用Log4J2进行测试
spring.expression.compiler.mode=OFF
int:logging-channel-adapter
而非int-file:outbound-channel-adapter
(省去必须使用带SpEL表达式的变压器)。RollingRandomAccessFile
和异步记录器使用Log4J2进行测试。请参阅http://logging.apache.org/log4j/2.0/manual/async.html#Making所有记录器异步
spring.expression.compiler.mode=OFF
int:logging-channel-adapter
而非int-file:outbound-channel-adapter
(省去必须使用带SpEL表达式的变压器)。RollingRandomAccessFile
和异步记录器使用Log4J2进行测试。请参阅http://logging.apache.org/log4j/2.0/manual/async.html#Making所有记录器异步
spring.expression.compiler.mode=MIXED
测试用例1和2流程:
测试用例3到5流程:
输入文件包含XML数据(字符串),其长度在每行1200到1500个字符之间变化(每行是单个消息)。
在我的测试中,我有203,712条消息
以下是时间安排。由于SpEL编译器在一段时间后启动,因此我显示第一项的时间比最后一项更多。
| 1 | 2 | 3 | 4 | 5 |
|SpringInt FileAdapter | SpringInt FileAdapter | Log4j2 RollingRandomAccessFile | Log4j2 RollingRandomAccessFile | Log4j2 RollingRandomAccessFile |
| | | Sync Loggers | Async Loggers | Async with |
|SpEL-compiler=OFF | SpEL-compiler=MIXED | SpEL-compiler=OFF | SpEL-compiler=OFF | SpEL-compiler=MIXED |
|-------------------------|--------------------------|--------------------------------|--------------------------------|------------------------------- |
|Cnt=10000 : 0:00:12.670 | Cnt=10000 : 0:00:17.235 | Cnt=10000 : 0:00:08.222 | Cnt=10000 : 0:00:01.847 | Cnt=10000 : 0:00:01.320 |
|Cnt=20000 : 0:00:24.636 | Cnt=20000 : 0:00:30.208 | Cnt=20000 : 0:00:08.828 | Cnt=20000 : 0:00:02.232 | Cnt=20000 : 0:00:01.839 |
|Cnt=30000 : 0:00:36.179 | Cnt=30000 : 0:00:44.300 | Cnt=30000 : 0:00:09.426 | Cnt=30000 : 0:00:02.512 | Cnt=30000 : 0:00:02.647 |
|... | .... | ... | ... | ... |
|Cnt=180000 : 0:02:58.935 | Cnt=180000 : 0:04:15.528 | Cnt=180000 : 0:00:17.095 | Cnt=180000 : 0:00:08.546 | Cnt=180000 : 0:00:07.936 |
|Cnt=200000 : 0:03:16.473 | Cnt=200000 : 0:04:35.582 | Cnt=200000 : 0:00:18.107 | Cnt=200000 : 0:00:09.548 | Cnt=200000 : 0:00:08.660 |
|Cnt=203712 : 0:03:19.715 | Cnt=203712 : 0:04:39.452 | Cnt=203712 : 0:00:18.284 | Cnt=203712 : 0:00:09.661 | Cnt=203712 : 0:00:08.732 |
拿着一粒盐的时间 - 我没有运行这几十次并取平均值。我也没有提倡log4j2比其他提供的东西更快,比如logback,我只是用它来进行比较。注意:我仅使用文件作为此测试的输入。我指出这一点,因为有人可能建议让Spring Integration将原始文件从fileA复制到fileB。在我们的实际用例中,消息实际上是通过JMS进入的,因此文件到文件的解决方案不是一个真正的选择。
有趣的观点:
FileWritingMessageHandler
比任何log4j2产品慢很多
Log4j2-async占用时间FileWritingMessageHandler
的4.3%(方案1为199.715秒,方案5为8.732秒)。 FileWritingMessageHandler
时间(scenario1为199.715秒,scenario4为9.661秒)。 FileWritingMessageHandler
与spring.expression.compiler.mode=MIXED
(场景#2)的Spring Integration spring.expression.compiler.mode=OFF
实际上比payload + '#{systemProperties['line.separator']}
慢。我认为这是因为在方案#1中我能够使用payload.toString() + '#{systemProperties['line.separator']}
而在方案#2中我必须使用logging-channel-adapter
理想情况下,我不想仅仅使用FileWritingMessageHandler
将消息写入文件 - 似乎我正在混淆该组件。然而,性能上的提升是显着的,所以不幸的是现在我不能排除使用它
所以我的问题是:
FileWritingMessageHandler
以获得更好的文件写入性能外,我还有哪些其他选择?FileWritingMessageHandler
之前批处理或聚合,那么写出批处理组的表现可能会更好。我确信我还可以使用任务执行程序和轮询器(我的用例允许这样做)。如果buffersize
公开FileWritingMessageHandler
属性?StreamWriter
,或者可能提供更具特色的版本,这些版本对我的用例更有效(可能从log4j2记录器中获取一些建议/提示)?payload.toString() + '#{systemProperties['line.separator']}
会更高效吗? <dependencies>
<!-- Testing -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<!-- Spring Integration -->
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-core</artifactId>
<version>${spring.integration.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-file</artifactId>
<version>${spring.integration.version}</version>
</dependency>
<!-- Logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<!-- Binding for JCL (aka Java Common Logging). -->
<!-- Needed since things like the commons libs all use commons-logging which we don't want -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.7.7</version>
<!-- Making scope be runtime so we'll catch any of our own classes that try to use commons-logging when we compile -->
<scope>runtime</scope>
</dependency>
<!-- Binding for Log4J -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<!-- As of 9/12/2014 our company Maven repos does not have 2.0.2 -->
<version>2.0.1</version>
</dependency>
<!-- Log4j API and Core implementation required for binding -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Async loggers for log4j2 require LMAX disruptor, see http://logging.apache.org/log4j/2.x/manual/async.html -->
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
</dependencies>
情况,因为如上所述它实际上比不在SpEL本身中调用toString()慢?以下是用于测试的代码/配置文件。
的pom.xml
package com.xxx;
import java.util.Scanner;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.support.AbstractApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
/**
* Starts the Spring Context and will initialize the Spring Integration routes.
*/
public final class Main {
private static final Logger LOGGER = LoggerFactory.getLogger(Main.class);
private Main() {
}
/**
* Load the Spring Integration Application Context
*
* @param args - command line arguments
*/
public static void main(final String... args) {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("\n=========================================================" + "\n " + "\n Welcome to Spring Integration! " + "\n " + "\n For more information please visit: " + "\n http://www.springsource.org/spring-integration " + "\n " + "\n=========================================================");
}
final AbstractApplicationContext context = new ClassPathXmlApplicationContext("classpath:META-INF/spring/integration/spring-integration-context-usecases.xml");
context.registerShutdownHook();
SpringIntegrationUtils.displayDirectories(context);
final Scanner scanner = new Scanner(System.in);
if (LOGGER.isInfoEnabled()) {
LOGGER.info("\n=========================================================" + "\n " + "\n Please press 'q + Enter' to quit the application. " + "\n " + "\n=========================================================");
}
while (!scanner.hasNext("q")) {
//Do nothing unless user presses 'q' to quit.
}
if (LOGGER.isInfoEnabled()) {
LOGGER.info("Exiting application...bye.");
}
System.exit(0);
}
}
package com.xxx;
import java.io.File;
import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.commons.lang3.time.StopWatch;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.IntegrationMessageHeaderAccessor;
import org.springframework.integration.routingslip.RoutingSlipRouteStrategy;
import org.springframework.integration.splitter.AbstractMessageSplitter;
import org.springframework.integration.support.AbstractIntegrationMessageBuilder;
import org.springframework.integration.transformer.MessageTransformationException;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessageHeaders;
import org.springframework.messaging.MessagingException;
import org.springframework.messaging.core.DestinationResolutionException;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;
/**
* This class is only needed until a bug is fixed in Spring Integration 4.1.0.
* See {@link http://stackoverflow.com/questions/27171978/read-csv-file-concurrently-using-spring-integration}
* Once that is fixed delete this class and use this in the Spring context file.
* <code>
* <splitter input-channel="splitChannel" output-channel="executorChannel" expression="T(org.apache.commons.io.FileUtils).lineIterator(payload)"/>
* </code>
*
*/
public class FileSplitter extends AbstractMessageSplitter {
private static final Logger log = LoggerFactory.getLogger(FileSplitter.class);
int counter = 0;
StopWatch sw = new StopWatch();
public Object splitMessage(Message<?> message) {
if (log.isDebugEnabled()) {
log.debug(message.toString());
}
try {
Object payload = message.getPayload();
Assert.isInstanceOf(File.class, payload, "Expected java.io.File in the message payload");
return org.apache.commons.io.FileUtils.lineIterator((File) payload);
} catch (IOException e) {
String msg = "Unable to transform file: " + e.getMessage();
log.error(msg);
throw new MessageTransformationException(msg, e);
}
}
@Override
protected void produceOutput(Object result, Message<?> requestMessage) {
Iterator<?> iterator = (Iterator<?>) result;
sw.start();
while (iterator.hasNext()) {
++counter;
produceOutputInternal(iterator.next(), requestMessage);
if (counter % 10000 == 0) {
sw.split();
System.out.println("Cnt=" + counter + " : " + sw.toSplitString());
}
}
sw.stop();
System.out.println("completed");
System.out.println("Cnt=" + counter + " : " + sw.toSplitString());
}
private Object getOutputChannelFromRoutingSlip(Object reply, Message<?> requestMessage, List<?> routingSlip, AtomicInteger routingSlipIndex) {
if (routingSlipIndex.get() >= routingSlip.size()) {
return null;
}
Object path = routingSlip.get(routingSlipIndex.get());
Object routingSlipPathValue = null;
if (path instanceof String) {
routingSlipPathValue = getBeanFactory().getBean((String) path);
} else if (path instanceof RoutingSlipRouteStrategy) {
routingSlipPathValue = path;
} else {
throw new IllegalArgumentException("The RoutingSlip 'path' can be of " + "String or RoutingSlipRouteStrategy type, but gotten: " + path);
}
if (routingSlipPathValue instanceof MessageChannel) {
routingSlipIndex.incrementAndGet();
return routingSlipPathValue;
} else {
Object nextPath = ((RoutingSlipRouteStrategy) routingSlipPathValue).getNextPath(requestMessage, reply);
if (nextPath != null && (!(nextPath instanceof String) || StringUtils.hasText((String) nextPath))) {
return nextPath;
} else {
routingSlipIndex.incrementAndGet();
return getOutputChannelFromRoutingSlip(reply, requestMessage, routingSlip, routingSlipIndex);
}
}
}
protected void produceOutputInternal(Object reply, Message<?> requestMessage) {
MessageHeaders requestHeaders = requestMessage.getHeaders();
Object replyChannel = null;
if (getOutputChannel() == null) {
Map<?, ?> routingSlipHeader = requestHeaders.get(IntegrationMessageHeaderAccessor.ROUTING_SLIP, Map.class);
if (routingSlipHeader != null) {
Assert.isTrue(routingSlipHeader.size() == 1, "The RoutingSlip header value must be a SingletonMap");
Object key = routingSlipHeader.keySet().iterator().next();
Object value = routingSlipHeader.values().iterator().next();
Assert.isInstanceOf(List.class, key, "The RoutingSlip key must be List");
Assert.isInstanceOf(Integer.class, value, "The RoutingSlip value must be Integer");
List<?> routingSlip = (List<?>) key;
AtomicInteger routingSlipIndex = new AtomicInteger((Integer) value);
replyChannel = getOutputChannelFromRoutingSlip(reply, requestMessage, routingSlip, routingSlipIndex);
if (replyChannel != null) {
//TODO Migrate to the SF MessageBuilder
AbstractIntegrationMessageBuilder<?> builder = null;
if (reply instanceof Message) {
builder = this.getMessageBuilderFactory().fromMessage((Message<?>) reply);
} else if (reply instanceof AbstractIntegrationMessageBuilder) {
builder = (AbstractIntegrationMessageBuilder<?>) reply;
} else {
builder = this.getMessageBuilderFactory().withPayload(reply);
}
builder.setHeader(IntegrationMessageHeaderAccessor.ROUTING_SLIP, Collections.singletonMap(routingSlip, routingSlipIndex.get()));
reply = builder;
}
}
if (replyChannel == null) {
replyChannel = requestHeaders.getReplyChannel();
}
}
Message<?> replyMessage = createOutputMessage(reply, requestHeaders);
sendOutput(replyMessage, replyChannel);
}
private Message<?> createOutputMessage(Object output, MessageHeaders requestHeaders) {
AbstractIntegrationMessageBuilder<?> builder = null;
if (output instanceof Message<?>) {
if (!this.shouldCopyRequestHeaders()) {
return (Message<?>) output;
}
builder = this.getMessageBuilderFactory().fromMessage((Message<?>) output);
} else if (output instanceof AbstractIntegrationMessageBuilder) {
builder = (AbstractIntegrationMessageBuilder<?>) output;
} else {
builder = this.getMessageBuilderFactory().withPayload(output);
}
if (this.shouldCopyRequestHeaders()) {
builder.copyHeadersIfAbsent(requestHeaders);
}
return builder.build();
}
private void sendOutput(Object output, Object replyChannel) {
MessageChannel outputChannel = getOutputChannel();
if (outputChannel != null) {
replyChannel = outputChannel;
}
if (replyChannel == null) {
throw new DestinationResolutionException("no output-channel or replyChannel header available");
}
if (replyChannel instanceof MessageChannel) {
if (output instanceof Message<?>) {
this.messagingTemplate.send((MessageChannel) replyChannel, (Message<?>) output);
} else {
this.messagingTemplate.convertAndSend((MessageChannel) replyChannel, output);
}
} else if (replyChannel instanceof String) {
if (output instanceof Message<?>) {
this.messagingTemplate.send((String) replyChannel, (Message<?>) output);
} else {
this.messagingTemplate.convertAndSend((String) replyChannel, output);
}
} else {
throw new MessagingException("replyChannel must be a MessageChannel or String");
}
}
}
package com.xxx;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.DirectFieldAccessor;
import org.springframework.context.ApplicationContext;
import org.springframework.expression.Expression;
import org.springframework.integration.file.FileReadingMessageSource;
import org.springframework.integration.file.FileWritingMessageHandler;
/**
* Displays the names of the input and output directories.
*/
public final class SpringIntegrationUtils {
private static final Log logger = LogFactory.getLog(SpringIntegrationUtils.class);
private SpringIntegrationUtils() { }
/**
* Helper Method to dynamically determine and display input and output
* directories as defined in the Spring Integration context.
*
* @param context Spring Application Context
*/
public static void displayDirectories(final ApplicationContext context) {
final File inDir = (File) new DirectFieldAccessor(context.getBean(FileReadingMessageSource.class)).getPropertyValue("directory");
final Map<String, FileWritingMessageHandler> fileWritingMessageHandlers = context.getBeansOfType(FileWritingMessageHandler.class);
final List<String> outputDirectories = new ArrayList<String>();
for (final FileWritingMessageHandler messageHandler : fileWritingMessageHandlers.values()) {
final Expression outDir = (Expression) new DirectFieldAccessor(messageHandler).getPropertyValue("destinationDirectoryExpression");
outputDirectories.add(outDir.getExpressionString());
}
final StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("\n=========================================================");
stringBuilder.append("\n");
stringBuilder.append("\n Input directory is : '" + inDir.getAbsolutePath() + "'");
for (final String outputDirectory : outputDirectories) {
stringBuilder.append("\n Output directory is: '" + outputDirectory + "'");
}
stringBuilder.append("\n\n=========================================================");
logger.info(stringBuilder.toString());
}
}
Java类
log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d{ISO8601} [%t] [%-5p] (%c) - %m%n" />
</Console>
<RollingRandomAccessFile name="fileAppenderMessages" fileName="C:/Users/xxxxx/Desktop/fileadapter-test/usecase3.txt">
<PatternLayout pattern="%m %n" />
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<!-- The Wire-Tap and logging-channel-adapter in the Spring cfg file will use this category name -->
<Logger name="fileLogger" additivity="false">
<AppenderRef ref="fileAppenderMessages" />
</Logger>
<Root level="info">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
配置文件
spring-integration-context-usecases.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/jms http://www.springframework.org/schema/jms/spring-jms.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd">
<int:inbound-channel-adapter id="fileAdapter" ref="fileReadingMessageSource" method="receive" auto-startup="true" channel="files" >
<int:poller fixed-delay="#{T(java.lang.Integer).MAX_VALUE}"/>
</int:inbound-channel-adapter>
<bean id="fileReadingMessageSource" class="org.springframework.integration.file.FileReadingMessageSource">
<property name="directory" value="C:/Users/xxxxx/Desktop/tmg-exchange-gateway-nam/t2"/>
</bean>
<int:channel id="files"/>
<int:splitter input-channel="files" output-channel="stringMessages">
<bean class="com.xxx.FileSplitter" />
</int:splitter>
<int:channel id="stringMessages"/>
<int:transformer expression="payload + '#{systemProperties['line.separator']}'" output-channel="file" auto-startup="true" input-channel="stringMessages"/>
<int-file:outbound-channel-adapter id="file"
mode="APPEND"
charset="UTF-8"
directory="C:/Users/xxxxx/Desktop/fileadapter-test"
auto-create-directory="true"
filename-generator-expression="'usecase2.txt'"/>
</beans>
档案
1. java -Dspring.expression.compiler.mode=OFF com.xxx.Main
Leave context file unchanged.
2. java -Dspring.expression.compiler.mode=MIXED com.xxx.Main
Change context file to have expression="payload.toString() + '#{systemProperties['line.separator']}'"
3. java -Dspring.expression.compiler.mode=OFF com.xxx.Main
Comment out transformer and outbound-channel-adapter.
Change logging-channel-adapter auto-startup="true"
4. java -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dspring.expression.compiler.mode=OFF com.xxx.Main
Comment out transformer and outbound-channel-adapter.
Change logging-channel-adapter auto-startup="true"
5. java -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dspring.expression.compiler.mode=MIXED com.xxx.Main
Comment out transformer and outbound-channel-adapter.
Change logging-channel-adapter auto-startup="true"
可以使用以下设置运行测试:
{{1}}
答案 0 :(得分:2)
感谢您的广泛分析。
说实话,APPEND
模式是出站适配器的相对新增功能,尚未优化。
我怀疑成本仅仅是因为每次写入时使用流都关闭(使用FileCopy.copy()
),这会刷新到磁盘。
我们绝对应该考虑保持BufferedOutputStream
开放的选项。这有点棘手,因为适配器支持为每条消息写入不同的文件。我假设你的用例是你总是写入同一个文件,或一些基于时间戳的文件名。我们可以提供一些优化来保持文件打开,直到有不同文件的请求进入,或者甚至打开几个文件缓冲区。
但是,在某些情况下,如果在一段时间过后没有新消息到达,我们会想要刷新缓冲区。这增加了一些复杂性(但不是很多)。
当然,缺点是当您在内存中缓冲数据时,如果发生电源故障,则存在数据丢失的风险。这是一个经典的权衡 - 性能比。可靠性;现在这个适配器对后者不利。
与往常一样,随时可以打开JIRA问题,我们会一起来看看。