Gobblin QuickStart示例异常:ClassNotFoundException:org.apache.gobblin.example.wikipedia.WikipediaSource

时间:2017-10-18 09:14:57

标签: classnotfoundexception

我在quickstart,子部分"运行Gobblin作为守护进程"之后学习gobblin。

我一步一步地作为指南:

  1. 创建配置目录并设置环境变量GOBBLIN_JOB_CONFIG_DIR,并将wikipedia.pull放入其中;
  2. 创建工作目录并设置环境变量GOBBLIN_WORK_DIR。
  3. 启动Gobblin:./ bin / gobblin-standalone.sh start
  4. 
    
        floyddeMac-mini:gobblin-dist floyd$ echo $GOBBLIN_JOB_CONFIG_DIR
        /Users/floyd/plugin/gobblin-dist/config_dir
    
        floyddeMac-mini:gobblin-dist floyd$ echo $GOBBLIN_WORK_DIR
        /Users/floyd/plugin/gobblin-dist/work_dir
    
        floyddeMac-mini:gobblin-dist floyd$ echo $JAVA_HOME
        /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home
    
        floyddeMac-mini:gobblin-dist floyd$ ls $GOBBLIN_JOB_CONFIG_DIR/*
        /Users/floyd/plugin/gobblin-dist/config_dir/wikipedia.pull
    
    
    

    我在日志中遇到了异常:

    
    
    1 2017-10-18 16:29:43 CST INFO  [main] gobblin.runtime.app.ServiceBasedAppLauncher  163 - Starting the Gobblin application and all its associated Services
      2 2017-10-18 16:29:43 CST INFO  [JobScheduler STARTING] gobblin.scheduler.JobScheduler  179 - Starting the job scheduler
      3 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1172 - Using default implementation for ThreadExecutor
      4 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.core.SchedulerSignalerImpl  61 - Initialized Scheduler Signaller of type: class org.quar    tz.core.SchedulerSignalerImpl
      5 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.core.QuartzScheduler  240 - Quartz Scheduler v.2.2.3 created.
      6 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.simpl.RAMJobStore  155 - RAMJobStore initialized.
      7 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.core.QuartzScheduler  305 - Scheduler meta-data: Quartz Scheduler (v2.2.3) 'LocalJobSche    duler' with instanceId 'NON_CLUSTERED'
      8   Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
      9   NOT STARTED.
     10   Currently in standby mode.
     11   Number of jobs executed: 0
     12   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 3 threads.
     13   Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
     14
     15 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1327 - Quartz scheduler 'LocalJobScheduler' initialized from s    pecified file: '/Users/fanjun/plugin/gobblin-dist/conf/quartz.properties'
     16 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1331 - Quartz scheduler version: 2.2.3
     17 2017-10-18 16:29:43 CST INFO  [SchedulerService STARTING] org.quartz.core.QuartzScheduler  575 - Scheduler LocalJobScheduler_$_NON_CLUSTERED started.
     18 2017-10-18 16:29:44 CST WARN  [JobScheduler STARTING] org.apache.hadoop.util.NativeCodeLoader  62 - Unable to load native-hadoop library for your platform...     using builtin-java classes where applicable
     19 2017-10-18 16:29:44 CST INFO  [JobScheduler STARTING] gobblin.scheduler.JobScheduler  413 - Scheduling configured jobs
     20 2017-10-18 16:29:44 CST INFO  [JobScheduler STARTING] gobblin.scheduler.JobScheduler  427 - Loaded 1 job configurations
     21 2017-10-18 16:29:44 CST INFO  [JobScheduler-0] gobblin.util.ClustersNames  74 - no default cluster mapping found
     22 2017-10-18 16:29:44 CST WARN  [JobScheduler-0] gobblin.runtime.AbstractJobLauncher  200 - Creating a job specific SharedResourcesBroker. Objects will only be     shared at the job level.
     23 2017-10-18 16:29:45 CST INFO  [JobScheduler-0] org.reflections.Reflections  229 - Reflections took 466 ms to scan 43 urls, producing 645 keys and 2317 values
     24 2017-10-18 16:29:45 CST INFO  [JobScheduler-0] gobblin.util.reflection.GobblinConstructorUtils  86 - Found accessible constructor for class class gobblin.run    time.FsDatasetStateStore with parameter types [class org.apache.hadoop.fs.LocalFileSystem, class java.lang.String, class java.lang.Integer, class com.google.    common.cache.LocalCache$LocalLoadingCache].
     25 2017-10-18 16:29:45 CST WARN  [JobScheduler-0] gobblin.runtime.JobContext  298 - Property task.data.root.dir is missing.
     26 2017-10-18 16:29:45 CST ERROR [JobScheduler-0] gobblin.scheduler.JobScheduler$NonScheduledJobRunner  519 - Failed to run job PullFromWikipedia
     27 gobblin.runtime.JobException: Failed to run job PullFromWikipedia
     28     at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:352)
     29     at gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:517)
     30     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     31     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     32     at java.lang.Thread.run(Thread.java:745)
     33 Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.ClassNotFoundException: org.apache.gobblin.example.wikipedia.WikipediaSource
     34     at gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:120)
     35     at gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85)
     36     at gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:65)
     37     at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:350)
     38     ... 4 more
     39 Caused by: java.lang.ClassNotFoundException: org.apache.gobblin.example.wikipedia.WikipediaSource
     40     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     41     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     42     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
     43     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    
    
    

    和wikipedia.pull是从github复制的:

    
    
    #
    # Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #    http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    #
    
    job.name=PullFromWikipedia
    job.group=Wikipedia
    job.description=A getting started example for Gobblin
    
    source.class=org.apache.gobblin.example.wikipedia.WikipediaSource
    source.page.titles=LinkedIn,Wikipedia:Sandbox
    source.revisions.cnt=5
    
    wikipedia.api.rooturl=https://en.wikipedia.org/w/api.php
    wikipedia.avro.schema={"namespace": "example.wikipedia.avro","type": "record","name": "WikipediaArticle","fields": [{"name": "revid", "type": ["double", "null"]},{"name": "pageid", "type": ["double", "null"]},{"name": "title", "type": ["string", "null"]},{"name": "user", "type": ["string", "null"]},{"name": "anon", "type": ["string", "null"]},{"name": "userid",  "type": ["double", "null"]},{"name": "timestamp", "type": ["string", "null"]},{"name": "size",  "type": ["double", "null"]},{"name": "contentformat",  "type": ["string", "null"]},{"name": "contentmodel",  "type": ["string", "null"]},{"name": "content", "type": ["string", "null"]}]}
    gobblin.wikipediaSource.maxRevisionsPerPage=10
    
    converter.classes=org.apache.gobblin.example.wikipedia.WikipediaConverter
    
    extract.namespace=org.apache.gobblin.example.wikipedia
    
    writer.destination.type=HDFS
    writer.output.format=AVRO
    writer.partitioner.class=org.apache.gobblin.example.wikipedia.WikipediaPartitioner
    
    data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher
    
    
    

2 个答案:

答案 0 :(得分:2)

您需要从wikipedia.pull中的所有定义中删除前缀org.apache。再次运行它,它将起作用。

实际上,如果你在lib目录中解压缩gobblin-example-0.9.0-642-g13a21ad.jar,导航到example / wikipedia,你可以在那里找到这些类。如果您反编译其中一个,请注意该包不包含org.apache。 package gobblin.example.wikipedia;

答案 1 :(得分:0)

实际上,如果你在lib目录中解压缩gobblin-example-0.9.0-642-g13a21ad.jar,导航到example / wikipedia,你可以在那里找到这些类。如果您反编译其中一个,请注意该包不包含org.apache。 package gobblin.example.wikipedia;