使用已经实现的日志记录机制的Spark吗?

时间:2018-12-07 07:53:52

标签: java apache-spark logging stack-overflow

我有一个非常庞大的应用程序,它根据从java.util.logging派生的自己的日志记录机制进行所有日志记录。

现在,我想添加一些使用Spark进行统计计算的类,而Spark使用log4j / slf4j进行日志记录。

当我现在尝试调试/运行该应用程序时,由于应用程序和slf4j的日志记录机制不断相互调用,我遇到了StackOverflow错误。

我对java.util.logging中可用的内置JDK日志感到有点熟悉, 我试图在Spark的配置中更改一些设置,并尝试覆盖log4j.properties文件,但它没有解决问题。

是否有任何简便的方法来禁用Spark的日志记录机制,或者以某种方式将其桥接到日志记录库中?

我在SparkConf中尝试过

.set("spark.eventlog.enabled", "false")
.set("spark.logConf", "false")

我在行上收到StackOverFlow错误:

SparkSession spark = sparkBuilder.getOrCreate();

堆栈跟踪是这样的:(实际上,您可以猜测的要长得多)

Thread[AWT-EventQueue-0]
java.lang.RuntimeException: java.lang.StackOverflowError
    at com.companyName.afc.action.InvokeAction.throwCause(InvokeAction.java:209)
    at com.companyName.afc.action.InvokeAction.invoke(InvokeAction.java:166)
    at com.companyName.afc.action.InvokeAction.actionPerformed(InvokeAction.java:138)
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
    at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
    at java.awt.Component.processMouseEvent(Component.java:6533)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
    at java.awt.Component.processEvent(Component.java:6298)
    at java.awt.Container.processEvent(Container.java:2238)
    at java.awt.Component.dispatchEventImpl(Component.java:4889)
    at java.awt.Container.dispatchEventImpl(Container.java:2296)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4897)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4534)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4475)
    at java.awt.Container.dispatchEventImpl(Container.java:2282)
    at java.awt.Window.dispatchEventImpl(Window.java:2746)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
    at java.awt.EventQueue$4.run(EventQueue.java:733)
    at java.awt.EventQueue$4.run(EventQueue.java:731)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.StackOverflowError
    at org.slf4j.impl.JDK14LoggerAdapter.fillCallerData(JDK14LoggerAdapter.java:602)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
    at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
    at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
    at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:588)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:660)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Logger.java:738)

0 个答案:

没有答案