使用neo4j-import工具导入40M +数据时出错

时间：2015-03-15 07:42:25

标签： shell neo4j

我使用neo4j-import导入40M节点，下面是我的shell：

[luning@pinnacle bin]$ ./neo4j-import --into ../data/weibo.db --nodes:User "/data/weibo/user-header.csv,/data/weibo/users/000000_0.csv,/data/weibo/users/000001_0.csv,/data/weibo/users/000002_0.csv,/data/weibo/users/000003_0.csv,/data/weibo/users/000004_0.csv,/data/weibo/users/000005_0.csv,/data/weibo/users/000006_0.csv,/data/weibo/users/000007_0.csv,/data/weibo/users/000008_0.csv,/data/weibo/users/000009_0.csv,/data/weibo/users/000010_0.csv,/data/weibo/users/000011_0.csv,/data/weibo/users/000012_0.csv,/data/weibo/users/000013_0.csv,/data/weibo/users/000014_0.csv,/data/weibo/users/000015_0.csv,/data/weibo/users/000016_0.csv,/data/weibo/users/000017_0.csv,/data/weibo/users/000018_0.csv,/data/weibo/users/000019_0.csv,/data/weibo/users/000020_0.csv,/data/weibo/users/000021_0.csv,/data/weibo/users/000022_0.csv,/data/weibo/users/000023_1.csv,/data/weibo/users/000024_0.csv,/data/weibo/users/000025_0.csv" --delimiter "TAB"

Nodes
[*>:87.20 MB/s---------------------------|PROPERTIES(2)===============|NOD|v:227.03 MB/s(2)====] 48MImport error: Panic called, so exiting

Neo4j Import Tool
    neo4j-import is used to create a new Neo4j database from data in CSV files. See 
    the chapter "Import Tool" in the Neo4j Manual for details on the CSV file format 
    - a special kind of header is required.
Usage:
--into <store-dir>
    Database directory to import into. Must not contain existing database.
--nodes [:Label1:Label2] "<file1>,<file2>,..."
    Node CSV header and data. Multiple files will be logically seen as one big file 
    from the perspective of the importer. The first line must contain the header. 
    Multiple data sources like these can be specified in one import, where each data 
    source has its own header. Note that file groups must be enclosed in quotation 
    marks.
--relationships [:RELATIONSHIP_TYPE] "<file1>,<file2>,..."
    Relationship CSV header and data. Multiple files will be logically seen as one 
    big file from the perspective of the importer. The first line must contain the 
    header. Multiple data sources like these can be specified in one import, where 
    each data source has its own header. Note that file groups must be enclosed in 
    quotation marks.
--delimiter <delimiter-character>
    Delimiter character, or 'TAB', between values in CSV data. The default option is 
    ,.
--array-delimiter <array-delimiter-character>
    Delimiter character, or 'TAB', between array elements within a value in CSV

我检查了他们的架构。它们都是一致的。它显示

  

导入错误：已发出恐慌，因此退出

有人知道怎么解决吗？

下面是我的stacktrace：

    java.lang.RuntimeException: Panic called, so exiting
    at org.neo4j.unsafe.impl.batchimport.staging.StageExecution.stillExecuting(StageExecution.java:63)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.anyStillExecuting(ExecutionSupervisor.java:79)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.finishAwareSleep(ExecutionSupervisor.java:102)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.supervise(ExecutionSupervisor.java:64)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisors.superviseDynamicExecution(ExecutionSupervisors.java:65)
    at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.executeStages(ParallelBatchImporter.java:226)
    at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.doImport(ParallelBatchImporter.java:151)
    at org.neo4j.tooling.ImportTool.main(ImportTool.java:263)
Caused by: java.lang.RuntimeException: Panic called, so exiting
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.assertHealthy(AbstractStep.java:189)
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.await(AbstractStep.java:180)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep.receive(ExecutorServiceStep.java:82)
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.sendDownstream(AbstractStep.java:226)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep$2.call(ExecutorServiceStep.java:103)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep$2.call(ExecutorServiceStep.java:87)
    at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:217)
Caused by: java.lang.RuntimeException: Panic called, so exiting
    ... 7 more
Caused by: java.lang.RuntimeException: Panic called, so exiting
    ... 7 more
Caused by: org.neo4j.unsafe.impl.batchimport.input.InputException: ERROR in input
  data source: BufferedCharSeeker[buffer:org.neo4j.csv.reader.SectionedCharBuffer@4ac5af5c, seekPos:2764030, line:2882236]
  in field: descriptions:string:4
  for header: [id:string, screenname:string, locations:string, descriptions:string, :IGNORE, profileimageurl:string, gender:string, followerscount:string, friendscount:string, statusescount:string, favouritescount:string, verified:string, verifiedreason:string, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, darenint:string, :IGNORE, :IGNORE, updateddate:string]
  raw field value: 6:19:
  original error: Tried to read in a value larger than effective buffer size 8388608
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:152)
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:42)
    at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
    at org.neo4j.helpers.collection.NestingIterator.fetchNextOrNull(NestingIterator.java:61)
    at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
    at org.neo4j.unsafe.impl.batchimport.staging.IteratorBatcherStep.nextBatchOrNull(IteratorBatcherStep.java:54)
    at org.neo4j.unsafe.impl.batchimport.staging.InputIteratorBatcherStep.nextBatchOrNull(InputIteratorBatcherStep.java:42)
    at org.neo4j.unsafe.impl.batchimport.staging.ProducerStep.process(ProducerStep.java:73)
    at org.neo4j.unsafe.impl.batchimport.staging.ProducerStep$1.run(ProducerStep.java:54)
Caused by: java.lang.IllegalStateException: Tried to read in a value larger than effective buffer size 8388608
    at org.neo4j.csv.reader.BufferedCharSeeker.fillBufferIfWeHaveExhaustedIt(BufferedCharSeeker.java:258)
    at org.neo4j.csv.reader.BufferedCharSeeker.nextChar(BufferedCharSeeker.java:231)
    at org.neo4j.csv.reader.BufferedCharSeeker.seek(BufferedCharSeeker.java:109)
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:81)
    ... 10 more

3 个答案:

答案 0 :(得分：3)

其中一个字段可能有引号并不结束该引用...因此CSV解析器将读取和读取，直到找到下一个引号。很明显，你在那里有一个8M大的领域，这就是我所想的。

答案 1 :(得分：1)

我遇到了同样的错误并删除了特殊字符，例如“*”，“＆amp;”，“/”但保留单引号足以消除错误。

答案 2 :(得分：0)

当我尝试使用此方法将数据导入到我的图表时，我还收到了“导入错误：Executor已关闭”和“导入错误：Panic已调用，因此退出”错误。

当我收到这些错误时，我的数据没有引号字符（“和”）。

什么解决了我的问题是摆脱所有其他特殊字符。

我可能错过了文档中的内容，因为我认为我的节点属性中的所有文本都将作为字符串读入。结果neo4j-import不喜欢像“＆amp;”这样的字符和“/”！

当我编辑我的数据（yay sed！）以仅包含字母数字字符时，导入工具运行良好。

相关问题
最新问题