Question

我有一个水槽，可以将数据写入aws s3存储桶。 Flume配置如下

   aggregator.sinks.s3LogSink.type=hdfs
   aggregator.sinks.s3LogSink.channel=flumeLogAgentFileChannel
   aggregator.sinks.s3LogSink.hdfs.path=s3n://<aws bucket name>
   aggregator.sinks.s3LogSink.hdfs.fileType=DataStream
   aggregator.sinks.s3LogSink.hdfs.writeFormat=Text
   aggregator.sinks.s3LogSink.hdfs.rollCount = 0
   aggregator.sinks.s3LogSink.hdfs.rollSize = 67108864
   aggregator.sinks.s3LogSink.hdfs.batchSize = 1000
   aggregator.sinks.s3LogSink.hdfs.rollInterval = 0

为了支持水槽s3水槽，我还包括以下罐子

 commons-codec-1.4.jar
 commons-httpclient-3.1.jar
 hadoop-common-2.7.1.jar
 commons-configuration-1.6.jar
 hadoop-auth-2.7.1.jar
 aws-java-sdk-1.10.64.jar
 aws-java-sdk-s3-1.10.52.jar
 hadoop-aws.jar
 emr-core-2.4.0.jar

当我开始我的水槽过程时......我得到以下错误

2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
        at java.lang.Class.getConstructor0(Class.java:2895)
        at java.lang.Class.newInstance(Class.java:354)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 23 more

有人可以帮忙吗？

Answer 1

存储桶的正确路径是#pragma mark - Autocomplete Table Data Source Delegate - (void)tableDataSource:(GMSAutocompleteTableDataSource *)tableDataSource didAutocompleteWithPlace:(GMSPlace *)place { NSString *state; NSString *stateAbbreviation; NSArray *addressComponents = place.addressComponents; if (addressComponents) { // Type of addressComponent is <GMSAddressComponent *> (absent from public header UPDATE: Is present in 1.12.) for (GMSAddressComponent *addressComponent in addressComponents) { if ([addressComponent.type isEqualToString:@"administrative_area_level_1"]) { state = addressComponent.name; break; } } } if (state && [[_stateAbbreviations allKeys] containsObject:state]) { stateAbbreviation = _stateAbbreviations[state]; } else { stateAbbreviation = @""; } NSLog(@"City: <%@> State: <%@> Abbr: <%@>", place.name, state, stateAbbreviation); } - (void)tableDataSource:(GMSAutocompleteTableDataSource *)tableDataSource didFailAutocompleteWithError:(NSError *)error { NSLog(@"Error: %@", [error description]); } - (BOOL)tableDataSource:(GMSAutocompleteTableDataSource *)tableDataSource didSelectPrediction:(GMSAutocompletePrediction *)prediction { NSLog(@"Selected Prediction: <%@>", prediction); return YES; } - (void)didUpdateAutocompletePredictionsForTableDataSource: (GMSAutocompleteTableDataSource *)tableDataSource { NSLog(@"-[%@ %@] called.", self.class, NSStringFromSelector(_cmd)); [UIApplication sharedApplication].networkActivityIndicatorVisible = NO; // Reload table data. [_searchResultsTableView reloadData]; } - (void)didRequestAutocompletePredictionsForTableDataSource: (GMSAutocompleteTableDataSource *)tableDataSource { NSLog(@"-[%@ %@] called.", self.class, NSStringFromSelector(_cmd)); [UIApplication sharedApplication].networkActivityIndicatorVisible = YES; // Reload table data. [_searchResultsTableView reloadData]; }

形式

使用ec2实例IAM凭证将Flume接收到s3

1 个答案: