带有neo4j连接器的spark,neo4j超时了吗?

时间:2017-10-16 14:17:32

标签: scala apache-spark neo4j

编辑:我已经尝试了几项工作来完成工作,似乎没有任何工作:

更新了neo4j-community.vmoptions:

-server
-Xms4096m
-Xmx4096m
-XX:NewSize=1024m

还更新了我的mac以处理更多线程。

C02RH2U9G8WM:~ meuser$ ulimit -u
709
C02RH2U9G8WM:~ meuser$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

最后进行了测试以确保它不是导致问题的有限硬盘空间。一切似乎都结束了,在执行期间我会在另一个窗口中查看我的线程使用情况....

jstack -l 'pid' | grep tid | wc -l

and

ps -elfT | wc -l

没有什么事情似乎失控,所以我真的很困惑为什么我在scala连接到neo4j时运行spark代码时会得到以下错误,这些代码运行得很好,但是当我让它撕掉所有内容时会爆炸。

注意:在我的代码中,我不使用sc.stop()或任何东西,即使我认为资源需要以某种方式清除,同时从spark / scale中运行一个体积大小的循环

我确实检查了日志,结果可能与内存有关?

2017-10-13 01:17:45.073+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: unable to create new native thread unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at org.neo4j.kernel.impl.util.Neo4jJobScheduler.schedule(Neo4jJobScheduler.java:94)
    at org.neo4j.bolt.v1.runtime.concurrent.ThreadedWorkerFactory.newWorker(ThreadedWorkerFactory.java:68)
    at org.neo4j.bolt.v1.runtime.MonitoredWorkerFactory.newWorker(MonitoredWorkerFactory.java:54)
    at org.neo4j.bolt.BoltKernelExtension.lambda$newVersions$1(BoltKernelExtension.java:234)
    at org.neo4j.bolt.transport.ProtocolChooser.handleVersionHandshakeChunk(ProtocolChooser.java:95)
    at org.neo4j.bolt.transport.SocketTransportHandler.chooseProtocolVersion(SocketTransportHandler.java:109)
    at org.neo4j.bolt.transport.SocketTransportHandler.channelRead(SocketTransportHandler.java:58)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
    at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:219)
    at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:631)
    at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:468)
    at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:428)
    at org.neo4j.bolt.transport.TransportSelectionHandler.switchToSocket(TransportSelectionHandler.java:126)
    at org.neo4j.bolt.transport.TransportSelectionHandler.decode(TransportSelectionHandler.java:81)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:745)

我正在一台Mac笔记本电脑上运行这些测试(不是群集或任何其他东西)。一个i7 16GB的内存。

我正在运行一些使用neo4j连接器的spark scala代码。当我在小型数据集上运行它时它工作正常,当我让它处理neo4j数据库中的所有数据时,它会在运行一段时间(几个小时)后超时。通常我不得不重置社区版本重新启动neo4j服务器。在某些地方我可以调整这个时间吗?在Spark / Scala或neo4j configs中?

org.neo4j.driver.v1.exceptions.ClientException: Failed to establish connection with server. Make sure that you have a server with bolt enabled on localhost:7687
    at org.neo4j.driver.internal.connector.socket.SocketClient.negotiateProtocol(SocketClient.java:197)
    at org.neo4j.driver.internal.connector.socket.SocketClient.start(SocketClient.java:76)
    at org.neo4j.driver.internal.connector.socket.SocketConnection.<init>(SocketConnection.java:63)
    at org.neo4j.driver.internal.connector.socket.SocketConnector.connect(SocketConnector.java:52)
    at org.neo4j.driver.internal.pool.InternalConnectionPool.acquire(InternalConnectionPool.java:113)
    at org.neo4j.driver.internal.InternalDriver.session(InternalDriver.java:53)
    at org.neo4j.spark.Executor$.execute(Neo4j.scala:360)
    at org.neo4j.spark.Neo4jRDD.compute(Neo4j.scala:408)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

0 个答案:

没有答案