我的单机可用资源是：
Total machine memory: 2.00 TB
Free machine memory: 1.81 TB
Max heap memory : 910.50 MB
Processors: 192
Configured max memory: 1.63 TB
我的file1.csv文件大小为600GB
我的csv文件中的条目数= 3 000 000 000
标题结构
attempt1
item_col1:ID(label),item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5,item_col6,item_col7,item_col8:IGNORE
Attempt2
item_col1:ID,item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5,item_col6,item_col7,item_col8:IGNORE
Attempt3
item_col1:ID,item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5:LABEL,item_col6,item_col7,item_col8:IGNORE`
Neo4j版本：3.2.1
尝试使用配置组合1
cat ../conf/neo4j.conf | grep "memory"
dbms.memory.heap.initial_size=16000m
dbms.memory.heap.max_size=16000m
dbms.memory.pagecache.size=40g
尝试使用配置组合2
cat ../conf/neo4j.conf | grep "memory"
dbms.memory.heap.initial_size=900m
dbms.memory.heap.max_size=900m
dbms.memory.pagecache.size=4g
尝试使用配置组合3
dbms.memory.heap.initial_size=1000m
dbms.memory.heap.max_size=1000m
dbms.memory.pagecache.size=1g
尝试使用配置组合4
dbms.memory.heap.initial_size=10g
dbms.memory.heap.max_size=10g
dbms.memory.pagecache.size=10g
尝试使用配置组合5（已注释）（无输出）
# dbms.memory.heap.initial_size=10g
# dbms.memory.heap.max_size=10g
# dbms.memory.pagecache.size=10g
尝试了命令
kaushik@machine1:/neo4j/import$ cl
kaushik@machine1:/neo4j/import$ rm -r ../data/databases/
kaushik@machine1:/neo4j/import$ mkdir ../data/databases/
kaushik@machine1:/neo4j/import$ cat ../conf/neo4j.conf | grep active
dbms.active_database=graph.db
kaushik@machine1:/neo4j/import$ ../bin/neo4j-admin import --mode csv -- database social.db --nodes head.csv,file1.csv
Neo4j version: 3.2.1
Importing the contents of these files into /neo4j/data/databases/social.db:
Nodes:
/neo4j/import/head.csv
/neo4j/import/file1.csv
Available resources:
Total machine memory: 2.00 TB
Free machine memory: 1.79 TB
Max heap memory : 910.50 MB
Processors: 192
Configured max memory: 1.61 TB
错误1
Nodes, started 2017-07-14 05:32:51.736+0000
[*NODE:7.63 MB---------------------------------------------------|PROPERTIE|LABEL SCAN--------] 0 ? 0
Done in 40s 439ms
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.neo4j.csv.reader.Extractors$StringArrayExtractor.extract0(Extractors.java:739)
at org.neo4j.csv.reader.Extractors$ArrayExtractor.extract(Extractors.java:680)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/1372918763.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport
错误2
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.neo4j.csv.reader.Extractors$StringArrayExtractor.extract0(Extractors.java:739)
at org.neo4j.csv.reader.Extractors$ArrayExtractor.extract(Extractors.java:680)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/1372918763.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$154/1949503798.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)
错误3
Nodes, started 2017-07-14 05:39:48.602+0000
[NODE:7.63 MB-----------------------------------------------|PROPER|*LABEL SCAN---------------] 0 ? 0
Done in 42s 140ms
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.neo4j.csv.reader.Extractors$StringExtractor.extract0(Extractors.java:328)
at org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract(Extractors.java:287)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/310855317.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$154/679112060.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)
错误4
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.neo4j.csv.reader.Extractors$StringExtractor.extract0(Extractors.java:328)
at org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract(Extractors.java:287)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$118/69048864.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$122/951451297.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)
错误5
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.neo4j.csv.reader.Extractors$StringExtractor.extract0(Extractors.java:328)
at org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract(Extractors.java:287)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$118/950986004.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$122/151277029.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)
一般情况下，如果您可以解释第9章。性能9.1。内存调优的一个例子，对很多初学者都有帮助。 https://neo4j.com/docs/operations-manual/current/performance/
你能给出一个例子来计算dbms.memory.heap.initial_size，dbms.memory.heap.max_size，dbms.memory.pagecache.size以获得500 GB的样本数据集，其中包含3个相同大小的10列的3亿个条目在1TB RAM机器和100个处理器。
答案 0 :(得分：0)
实际上，如果您只是在做节点，那么计算非常简单：
3 * 10^9 * 20 / 1024^3
所以我的堆大小至少为55Gb。 你能试试吗？
此致 汤姆