Question

我想将文件从hdfs文件系统复制到本地file系统。我在Java DSL中尝试了一个简单的路由。

from("hdfs2://192.168.1.55:9000/user?owner=honey")
        .to("file:/home/honey/Desktop/DATA");

但它不起作用。这些是日志。

2014-11-13 22:16:22,169 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] util.Shell (Shell.java:326) - setsid exited with exit code 0
2014-11-13 22:16:22,186 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] hdfs2.HdfsConsumer (HdfsConsumer.java:87) - Connected to hdfs file-system 192.168.1.55:9000/hdfs://192.168.1.55:9000/user
2014-11-13 22:16:22,189 INFO  [main] hdfs2.HdfsConsumer (HdfsConsumer.java:84) - Connected to hdfs file-system 192.168.1.55:9000/hdfs://192.168.1.55:9000/user
2014-11-13 22:16:22,189 INFO  [main] impl.DefaultCamelContext (DefaultCamelContext.java:2310) - Route: route1 started and consuming from: Endpoint[hdfs2://192.168.1.55:9000/user?owner=honey]
2014-11-13 22:16:22,190 DEBUG [main] management.DefaultManagementLifecycleStrategy$TimerListenerManagerStartupListener (DefaultManagementLifecycleStrategy.java:893) - Load performance statistics disabled
2014-11-13 22:16:22,190 INFO  [main] impl.DefaultCamelContext (DefaultCamelContext.java:1654) - Total 1 routes, of which 1 is started.
2014-11-13 22:16:22,192 INFO  [main] impl.DefaultCamelContext (DefaultCamelContext.java:1655) - Apache Camel 2.13.0 (CamelContext: camel-1) started in 1.521 seconds
2014-11-13 22:16:22,202 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.Client$Connection (Client.java:371) - The ping interval is 60000 ms.
2014-11-13 22:16:22,204 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.Client$Connection (Client.java:636) - Connecting to /192.168.1.55:9000
2014-11-13 22:16:22,218 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:886) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey: starting, having connections 1
2014-11-13 22:16:22,220 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #0
2014-11-13 22:16:22,227 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #0
2014-11-13 22:16:22,227 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 36ms
2014-11-13 22:16:22,253 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #1
2014-11-13 22:16:22,255 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #1
2014-11-13 22:16:22,255 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 2ms
2014-11-13 22:16:22,256 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #2
2014-11-13 22:16:22,257 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #2
2014-11-13 22:16:22,257 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 1ms
2014-11-13 22:16:22,259 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #3
2014-11-13 22:16:22,261 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #3
so on.....

在下面这些日志中不断重复。但同样的事情，如果我尝试将文件从本地file system复制到hdfs，那么它的工作将完美无瑕。任何人都可以提供任何工作链接或示例吗？这将非常有帮助。

修改现在它对我有用。 readSuffix was read by default，它不会再读取那些已经读过的文件。现在我的问题是为什么有recursive option in consumer side.我想以递归方式使用文件。就像我们可以使用递归选项在File2中扫描一样。

Answer 1

查看Camel的HDFS2组件的文档和源代码，我想这是不可能的。

您可以针对此问题打开CR。或者甚至更好，自己实现这个（首先查看here以了解如何递归扫描HDFS中的目录）并向Apache基金会赠送礼物......

Camel hdfs2到文件协议文件传输

1 个答案: