RAM不足时查询性能

时间:2015-01-31 16:16:16

标签: performance neo4j cypher

的Neo4j企业-2.2.0-M03
27 522 896个节点(394Mb)
111 294 796关系(3609Mb)
33 906 363处房产(1326Mb)
只有8Gb RAM
6Gb用于JVM堆(wrapper.java.initmemory = 6144 wrapper.java.maxmemory = 6144)
用于映射内存的6Gb JVM堆中的5Gb(Windows 8.1)(dbms.pagecache.memory = 5G)

非常简单的查询:

match (:Page {Name:'Dashboard.aspx'})<-[:At]-()
return count(*)

执行47秒。

EagerAggregation
  |
  +Expand(All)
    |
    +NodeUniqueIndexSeek

+---------------------+---------------+---------+---------+-------------+--------------+
|            Operator | EstimatedRows |    Rows |  DbHits | Identifiers |        Other |
+---------------------+---------------+---------+---------+-------------+--------------+
|    EagerAggregation |           436 |       1 |       0 |    count(*) |              |
|         Expand(All) |        189923 | 6132879 | 6132880 |             | ()<-[:At]-() |
| NodeUniqueIndexSeek |             1 |       1 |       1 |             |  :Page(Name) |
+---------------------+---------------+---------+---------+-------------+--------------+

Total database accesses: 6132881

所以,我的问题是:
是否正常表现?
增加性能的唯一方法是增加内存量吗?

UPD:
查询:

match (p:Page {Name:'Dashboard.aspx'}) return p, size((p)<-[:At]-()) as degree

使用以下个人资料信息执行:

neo4j-sh (?)$ profile match (p:Page {Name:'Dashboard.aspx'}) return p, size((p)<-[:At]-()) as degree;
+------------------------------------------+
| p                              | degree  |
+------------------------------------------+
| Node[0]{Name:"Dashboard.aspx"} | 6132879 |
+------------------------------------------+
1 row
71033 ms

Compiler CYPHER 2.2

Planner COST

Projection
  |
  +NodeUniqueIndexSeek

+---------------------+---------------+------+--------+-------------+-------------+
|            Operator | EstimatedRows | Rows | DbHits | Identifiers |       Other |
+---------------------+---------------+------+--------+-------------+-------------+
|          Projection |             1 |    1 |      0 |   degree, p |   p, degree |
| NodeUniqueIndexSeek |             1 |    1 |      1 |           p | :Page(Name) |
+---------------------+---------------+------+--------+-------------+-------------+

Total database accesses: 1

这是启动服务器并执行查询后的message.log

2015-02-01 18:30:30.363+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 264ms.
2015-02-01 18:30:35.492+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 235ms.
2015-02-01 18:30:40.775+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 214ms.
2015-02-01 18:30:46.348+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 247ms.
2015-02-01 18:30:52.799+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 282ms.
2015-02-01 18:31:02.410+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 280ms.
2015-02-01 18:31:10.667+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 233ms.
2015-02-01 18:31:17.137+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 242ms.
2015-02-01 18:31:22.170+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 407ms.
2015-02-01 18:31:27.362+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 369ms.
2015-02-01 18:31:35.883+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 578ms.
2015-02-01 18:31:40.780+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 883ms.
2015-02-01 18:31:42.892+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1209ms.
2015-02-01 18:31:44.574+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1081ms.
2015-02-01 18:31:46.089+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1117ms.
2015-02-01 18:31:47.621+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1106ms.
2015-02-01 18:31:55.700+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 978ms.

图形结构:

neo4j-sh (?)$ match (:Page) return count(*);
+----------+
| count(*) |
+----------+
| 293      |
+----------+
1 row
328 ms
neo4j-sh (?)$ match (:PageView) return count(*);
+----------+
| count(*) |
+----------+
| 27522603 |
+----------+
1 row
17415 ms

每个(:PageView)与(:Page)有关系
有些(:PageView)与next(:PageView)有关系
与此post

中的结构类似的东西

如果我执行一些更复杂的查询,那就是message.log的样子:

2015-02-01 19:13:33.695+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 453) 433.02188MiB -> 410.23123MiB (22.790657MiB) 19.774801% misses, 3.4855824% collisions (736127).
2015-02-01 19:13:33.789+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 453): elementCount:783840 and sizes actual:410.2459MiB, perceived:410.23123MiB (diff:-15404B), registered:410.23123MiB
2015-02-01 19:13:35.742+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1226ms.
2015-02-01 19:13:41.503+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 431ms.
2015-02-01 19:13:49.519+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 55) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.346073% misses, 49.232044% collisions (46993221).
2015-02-01 19:13:49.691+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 55): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-01 19:14:34.228+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 499) 433.02206MiB -> 410.23105MiB (22.79101MiB) 19.79414% misses, 3.1852808% collisions (736127).
2015-02-01 19:14:34.337+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 499): elementCount:783840 and sizes actual:410.24576MiB, perceived:410.23105MiB (diff:-15404B), registered:410.23105MiB
2015-02-01 19:14:51.756+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 65) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.398457% misses, 46.956936% collisions (48775684).
2015-02-01 19:14:51.943+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 65): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-01 19:15:06.604+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 350ms.
2015-02-01 19:15:35.262+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 545) 433.02203MiB -> 410.23123MiB (22.790785MiB) 19.810442% misses, 2.932431% collisions (736127).
2015-02-01 19:15:35.371+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 545): elementCount:783840 and sizes actual:410.24594MiB, perceived:410.23123MiB (diff:-15404B), registered:410.23123MiB
2015-02-01 19:16:19.604+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 69) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.457134% misses, 49.884457% collisions (57486535).
2015-02-01 19:16:19.775+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 69): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-01 19:16:36.336+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 590) 433.02197MiB -> 410.23105MiB (22.790924MiB) 19.824066% misses, 2.7211785% collisions (736127).
2015-02-01 19:16:36.430+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 590): elementCount:783840 and sizes actual:410.24576MiB, perceived:410.23105MiB (diff:-15404B), registered:410.23105MiB
2015-02-01 20:08:27.549+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 7990ms.
2015-02-01 21:08:29.399+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1847ms.
2015-02-01 22:08:31.247+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1815ms.
2015-02-01 23:08:33.206+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1729ms.
2015-02-02 00:08:34.739+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1454ms.
2015-02-02 01:08:36.267+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1474ms.
2015-02-02 02:08:37.801+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1469ms.
2015-02-02 03:08:39.341+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1498ms.
2015-02-02 04:08:40.877+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 1499ms.
2015-02-02 05:07:59.689+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 601) 433.02188MiB -> 410.23114MiB (22.790754MiB) 19.828342% misses, 3.3933303% collisions (940912).
2015-02-02 05:07:59.783+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 601): elementCount:779324 and sizes actual:410.24277MiB, perceived:410.23114MiB (diff:-12198B), registered:410.23114MiB
2015-02-02 05:08:43.570+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 2482ms.
2015-02-02 05:09:01.035+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 628) 433.02216MiB -> 410.23117MiB (22.790974MiB) 19.838266% misses, 5.044218% collisions (1485145).
2015-02-02 05:09:01.144+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 628): elementCount:780385 and sizes actual:410.23215MiB, perceived:410.23117MiB (diff:-1016B), registered:410.23117MiB
2015-02-02 05:09:01.832+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 73) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.506836% misses, 52.426975% collisions (66571698).
2015-02-02 05:09:02.004+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 73): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-02 05:10:01.255+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 661) 433.02188MiB -> 410.2312MiB (22.790682MiB) 19.518581% misses, 4.8153634% collisions (1485145).
2015-02-02 05:10:01.348+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 661): elementCount:780385 and sizes actual:410.23218MiB, perceived:410.2312MiB (diff:-1016B), registered:410.2312MiB
2015-02-02 05:10:05.688+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 83) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.53808% misses, 50.574787% collisions (68605833).
2015-02-02 05:10:05.872+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 83): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-02 05:11:01.378+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 704) 433.02197MiB -> 410.23096MiB (22.791012MiB) 19.545334% misses, 4.541695% collisions (1485145).
2015-02-02 05:11:01.487+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 704): elementCount:780384 and sizes actual:410.23193MiB, perceived:410.23096MiB (diff:-1016B), registered:410.23096MiB
2015-02-02 05:11:40.127+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 87) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.573334% misses, 52.572327% collisions (77262502).
2015-02-02 05:11:40.299+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 87): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-02 05:12:02.504+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 744) 433.02188MiB -> 410.23096MiB (22.79093MiB) 19.567696% misses, 4.3134704% collisions (1485145).
2015-02-02 05:12:02.613+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 744): elementCount:780384 and sizes actual:410.2319MiB, perceived:410.23096MiB (diff:-1016B), registered:410.23096MiB
2015-02-02 05:12:44.793+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 96) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.593792% misses, 50.99522% collisions (78749908).
2015-02-02 05:12:44.996+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 96): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB
2015-02-02 05:13:02.603+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 783) 433.02188MiB -> 410.2308MiB (22.791073MiB) 19.58778% misses, 4.1741714% collisions (1508715).
2015-02-02 05:13:02.713+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 783): elementCount:780498 and sizes actual:410.2316MiB, perceived:410.2308MiB (diff:-812B), registered:410.2308MiB
2015-02-02 05:14:03.232+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 819) 433.02213MiB -> 410.23102MiB (22.791117MiB) 19.604544% misses, 4.001095% collisions (1508715).
2015-02-02 05:14:03.341+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: NodeCache purge (nr 819): elementCount:780497 and sizes actual:410.23178MiB, perceived:410.23102MiB (diff:-812B), registered:410.23102MiB
2015-02-02 05:14:46.253+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 101) 577.36255MiB -> 546.975MiB (30.387543MiB) 49.62446% misses, 52.736645% collisions (88143267).
2015-02-02 05:14:46.425+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: RelationshipCache purge (nr 101): elementCount:11948851 and sizes actual:546.975MiB, perceived:546.975MiB (diff:0B), registered:546.975MiB

1 个答案:

答案 0 :(得分:0)

感谢您提供详细信息。您的graph.db/messages.log也会有所帮助。

如果你只有8G内存,你不能在没有交换的情况下使用6 + 5 = 11G的内存,所以你可能会看到将内存交换到磁盘。

我认为你只是在第一次运行,Neo必须从磁盘加载节点和关系。

Cypher为您的Page执行6.1M db访问。 :At关系意味着什么?

也许你应该通过某种方式对这种关系进行分组。

在Neo4j API中,有一个函数node.getDegree(RelTypes.AT,Direction.IN),它具有恒定的时间复杂度。

我建议你使用它。

这是如何表现的:

匹配(页码:姓名:'Dashboard.aspx'}) 返回p,大小((p)&lt; - [:At] - ())为度

更新

抱歉,我错过了2.1.6中的Neo4j支持REST端点,即使不使用java调用也能提供学位:

请参阅:http://neo4j.com/docs/2.1.6/rest-api-node-degree.html

所以如果你返回节点的node-id:

match (p:Page {Name:'Dashboard.aspx'}) return id(p)
你可以打电话,例如在浏览器中:

:GET /db/data/node/<id>/degree/in/At

或通过http请求

GET http://localhost:7474/db/data/node/<id>/degree/in/At