Question

我有这个PySpark脚本：

from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext

sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

sc.addFile("hdfs:///user/cloudera/2904/src/LogFunction.py")
import LogFunction

try:
    df = hive_context.read.json("/user/cloudera/Projet/pareeam.json")
except IOError:
    LogFunction.WarnLog("Nope")
....
....

这是我的LogFunction.py：

import logging
from logging.handlers import RotatingFileHandler
from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

df = hive_context.read.json("/user/cloudera/2904/param.json")
Path = df[df.column.isin("LogRep")].collect()[0][1]
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s :: %(levelname)s :: %(message)s')

debug_handler = RotatingFileHandler(Path+ '/LogDebug.log', 'a', 1000000, 1)
debug_handler.setLevel(logging.DEBUG)
debug_handler.setFormatter(formatter)
logger.addHandler(debug_handler)
info_handler = RotatingFileHandler(Path+ '/LogInfo.log', 'a', 1000000, 1)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(formatter)
logger.addHandler(info_handler)
warning_handler = RotatingFileHandler(Path+ '/LogWarning.log', 'a', 1000000, 1)
warning_handler.setLevel(logging.WARNING)
warning_handler.setFormatter(formatter)
logger.addHandler(warning_handler)

error_handler = RotatingFileHandler(Path+ '/LogError.log', 'a', 1000000, 1)
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(formatter)
logger.addHandler(error_handler)

def WarnLog(a):
    logger.warning(a)

无论如何，我确定我的日志记录正在运行，因为我已经尝试过其他错误（//除以零的异常）。

但是这个例子似乎没有。执行脚本时，我得到

py4j.protocol.Py4JJavaError: An error occurred while calling o31.json.
: java.io.FileNotFoundException: File hdfs://quickstart.cloudera:8020/user/cloudera/Projet/pareeam.json does not exist.

在终端上，但在我的警告文件中什么都没有。有什么帮助为什么它没有捕获错误？谢谢

Answer 1

您正在捕获错误的异常。您的代码捕获IOError，而DataFrameReader.json（内部）抛出py4j.protocol.Py4JJavaError和pyspark.sql.utils.AnalysisException

将代码更改为

from pyspark.sql.utils import AnalysisException

try:
    df = hive_context.read.json("/does/not/exist")
except AnalysisException:
    LogFunction.WarnLog("Nope")

Answer 2

导入AnalysisException之后，应该捕获IOError

from pyspark.sql.utils import AnalysisException


try:
    df = hive_context.read.json("/does/not/exist")
except IOError:
    LogFunction.WarnLog("Nope")

Catch子句不适用于hive_context.read.json函数

2 个答案: