Question

早上好，

我尝试使用批量加载（http://www.datastax.com/dev/blog/bulk-loading）作为指南来实现海量数据转储cassandra示例。

在示例中解决了与脚本（http://www.datastax.com/wp-content/uploads/2011/08/DataImport）的依赖关系，但我发现cassandra库所涵盖的依赖关系不在此处列出的目录中，因为版本I与使用cassandra的dse一起工作2.0。然后尝试覆盖这样的依赖关系得到以下脚本。

#!/bin/sh

# paths to the cassandra source tree, cassandra jar and java

CASSANDRA_HOME="/usr/share/dse/cassandra"
# CASSANDRA_JAR="./apache-cassandra-2.0.10.jar"
JAVA=`which java`

# Java classpath. Must include:
#   - directory of DataImportExample
#   - directory with cassandra/log4j config files
#   - cassandra jar
#   - cassandra depencies jar
CLASSPATH=".:/usr/share/dse/dse.jar:./slf4j-1.7.7/slf4-nop-1.7.7.jar:./slf4j-1.7.7/slf4j-simple-1.7.7.jar:/etc/dse/cassandra"

for jar in $CASSANDRA_HOME/lib/*.jar; do
    CLASSPATH=$CLASSPATH:$jar
done

$JAVA -ea -cp $CLASSPATH -Xmx256M \
        -Dlog4j.configuration=log4j-tools.properties \
        CassandraDataBulk "$@"

评论CASSANDRA_JAR，我使用＆＃34; cassandra-all-2.0.8.39.jar＆＃34;位于文件夹＆＃34; / usr / share / dse / cassandra / lib＆＃34;已经包括在内。
我解决了在1.7.7版本中下载的slf4j依赖项。

由于cassandra版本的不同，我不得不习惯SSTableSimpleUnsortedWriter构建器。

IPartitioner partitioner = new RandomPartitioner();

        SSTableSimpleUnsortedWriter sourcesWriter = new SSTableSimpleUnsortedWriter(
                directory,
                partitioner,
                keyspace,
                table,
                AsciiType.instance, 
                null, 
                64
        );

今天的问题似乎仍然存在依赖关系。在，我得到的跟踪错误。

有依赖但似乎是＆＃34; org.apache.commons.configuration.ConfigurationRuntimeException＆＃34;真正的问题可能是另一个，可能有一个糟糕的配置＆＃34; cassandra.yaml＆＃34;？

谢谢，问候！

[dmdb@vm-dmdb01 ~]$ ./init_env.sh export.csv 
[main] ERROR org.apache.cassandra.cql3.QueryProcessor - Unable to initialize MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
[main] INFO org.apache.cassandra.config.YamlConfigurationLoader - Loading settings from file:/etc/dse/cassandra/cassandra.yaml
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Data files directories: [/data01, /data02]
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Commit log directory: /datatmp/commitlog
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - disk_failure_policy is stop
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - commit_failure_policy is stop
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Global memtable threshold is enabled at 61MB
[main] INFO com.datastax.bdp.snitch.Workload - Setting my workload to Cassandra
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationRuntimeException
    at com.datastax.bdp.config.ConfigUtil.defaultValue(ConfigUtil.java:18)
    at com.datastax.bdp.config.DseConfig.<clinit>(DseConfig.java:51)
    at com.datastax.bdp.snitch.DseDelegateSnitch.<init>(DseDelegateSnitch.java:42)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.Class.newInstance(Class.java:374)
    at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:488)
    at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:508)
    at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:341)
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111)
    at org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.<init>(AbstractSSTableSimpleWriter.java:50)
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:96)
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:80)
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:91)
    at CassandraDataBulk.main(CassandraDataBulk.java:35)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationRuntimeException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 17 more

Answer 1

你错过了一个＆＃34; javaagent＆＃34; java调用中的参数。添加以下内容：

-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar

您的最终通话应如下所示：

$JAVA -ea -cp $CLASSPATH -Xmx256M \
       -Dlog4j.configuration=log4j-tools.properties \
       -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar
       CassandraDataBulk "$@"

注意：根据需要调整jamm.jar的路径

Reference

至于运行时配置错误，请下载apache commons＆＃39; lang＆＃39;库并将其包含在您的类路径中。

Download here

如果您在实施此修复程序后收到新的例外，请下载google-common.jar和guava-16.0.1.jar，并将它们包含在您的类路径中。这些是我自己的批量加载器到目前为止所需的所有JAR。

在cassandra中批量加载时出错

1 个答案: