在使用Java发布应用程序时出现问题

时间:2020-09-04 09:14:40

标签: apache-spark

当我运行spark示例时,出现此错误

04/09/04 17:06:35 WARN TaskSchedulerImpl:初始作业未接受任何资源;该任务已被执行。检查您的群集用户界面,以确保工作人员已注册并拥有足够的资源

我在本地运行此应用程序,而我的Spark服务器部署在47.111.185.105

public class WordCountApp {

    private static final Logger logger = LoggerFactory.getLogger(WordCountApp.class);


    public static void main(String[] args) {
        // Should be some file on your system
        String logFile = "src/main/resources/people.json";
        SparkConf conf = new SparkConf().set("spark.shuffle.service.enabled", "false")
                .set("spark.dynamicAllocation.enabled", "false")
                .set("spark.cores.max", "1")
                .set("spark.executor.instances","2")
                .set("spark.executor.memory","500m")
                .set("spark.executor.cores","1")
                .setMaster("spark://47.111.185.105:7077");
                //.set("deploy-mode", "client");
        SparkSession spark = SparkSession.builder()
                .appName("Word Count Application")
                .config(conf)
                .getOrCreate();
        Dataset<String> logData = spark.read().textFile(logFile).cache();
        System.out.println("Spark version = " + spark.version());
        logger.info("Spark version = " + spark.version());
        long numAs = logData.filter((FilterFunction<String>) s -> s.contains("a")).count();
        long numBs = 3;//logData.filter((FilterFunction<String>) s -> s.contains("b")).count();

        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
        logger.info("Lines with a: " + numAs + ", lines with b: " + numBs);
        spark.stop();
    }
}
  • 打包并上传jar文件并使用spark-shell效果很好
  • 检查剩余资源是否足够
  • 尝试重启火花

this is my spark server

0 个答案:

没有答案