Question

我正在尝试运行我的MapReduce应用程序。

这是主要课程：

public class GPFunctionFitting {
    public static void main(String[] args) {
        // ....
        GPFunctionFitting client;
        try {
            client = new GPFunctionFitting();
            client.runMapReduce(partitionFile, numMapTasks); // line 44
            // ...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

方法：

public ExpressionTree runMapReduce(String partitionFile, int numMapTasks) throws Exception {

        JobConf jobConf = new JobConf("gp-function-fitting-map-reduce" + uuidGen.generateRandomBasedUUID());
        jobConf.setMapperClass(GPMapTask.class);
        jobConf.setReducerClass(GPReduceTask.class);
        jobConf.setCombinerClass(GPCombiner.class);
        jobConf.setNumMapTasks(numMapTasks);
        jobConf.setNumReduceTasks(1);

        TwisterModel driver = new TwisterDriver(jobConf);
        driver.configureMaps(partitionFile); // line 74
        TwisterMonitor monitor = driver.runMapReduce();
        monitor.monitorTillCompletion();
        ExpressionTree solution = ((GPCombiner) driver.getCurrentCombiner()).getResult();

        driver.close();

        return solution;
    }

这是来自eclipse控制台的日志：

JobID:
gp-function-fitting-map-reduce9f25c1da-ab8b-446b-b453-d60f4165b3a9
TransmissionManager: There is a pending/ongoing transfer on
niotcp:///127.0.0.1:3045 for a total of 16384 bytes
TransmissionManager: All transfers have been completed
CommunicationsService: Closing Link =    niotcp://127.0.0.1:3045 null 0 
[main] INFO  cgl.imr.client.TwisterDriver  - MapReduce computation
termintated gracefully.

cgl.imr.base.TwisterException: Could not send all the map executor
requests. First error is Did not receive a response from the worker.14
[Thread-0] DEBUG cgl.imr.client.ShutdownHook  - Shutting down
completed.
    at cgl.imr.client.TwisterDriver.configureMaps(TwisterDriver.java:360)
    at cgl.imr.client.TwisterDriver.configureMaps(TwisterDriver.java:381)
    at map_reduce.GPFunctionFitting.runMapReduce(GPFunctionFitting.java:74)
    at map_reduce.GPFunctionFitting.main(GPFunctionFitting.java:44)

最后，来自终端的错误，我已经运行了twister：

2588033 [Thread-4] ERROR cgl.imr.worker.DaemonWorker  - Could not
instantiate the Mapper. cgl.imr.base.TwisterException: Could not
instantiate the Mapper.     at
cgl.imr.worker.Mapper.<init   (Mapper.java:120)     at
cgl.imr.worker.DaemonWorker.handleMapperRequest(DaemonWorker.java:228)
    at cgl.imr.worker.DaemonWorker.onEvent(DaemonWorker.java:471)   
    at cgl.imr.pubsub.nb.NBPubSubService.onEvent(NBPubSubService.java:154)
    at cgl.narada.service.qos.impl.QosServiceImpl.routeToEventConsumer(QosServiceImpl.java:938)
    at cgl.narada.service.qos.impl.QosServiceImpl.manageEventRoutingToDestinations(QosServiceImpl.java:530)
    at cgl.narada.service.qos.impl.QosServiceImpl.processReceivedEvent(QosServiceImpl.java:424)
    at cgl.narada.service.qos.impl.CommunicationsService.processDataPackets(CommunicationsService.java:236)
    at cgl.narada.service.qos.impl.CommunicationsService.processData(CommunicationsService.java:214)
    at cgl.narada.service.qos.impl.CommunicationsService.handleData(CommunicationsService.java:142)
    at cgl.narada.transport.TransportHandlerImpl.dataReceived(TransportHandlerImpl.java:516)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:323)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.readDataFromSocket(NIOTCPReceiverThread.java:247)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.run(NIOTCPReceiverThread.java:196)
Caused by: java.lang.ClassNotFoundException: map_reduce.GPMapTask   
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)   
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)    
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)    
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)    
at java.lang.ClassLoader.findSystemClass(ClassLoader.java:1004)     
at cgl.imr.util.CustomClassLoader.findClass(CustomClassLoader.java:102)
at cgl.imr.util.CustomClassLoader.loadClass(CustomClassLoader.java:129)
at java.lang.Class.forName0(Native Method)  
at java.lang.Class.forName(Class.java:348)  
at cgl.imr.worker.Mapper.<init   (Mapper.java:116)  ... 24 more

GPMapTask类存在，实现MapTask接口，与主类在同一个包中。分区文件是相应生成的，包含12行，等于请求的映射作业数。

至少有人能让我知道在哪里寻找问题吗？我一直在尝试解决“Class not found exception”，但我开始相信问题在于分区配置。

Answer 1

.class缺少或存在于多个位置。然后编译器会感到困惑，因为类定义不明确。确保所有类都包含在内，只需一次。检查类路径中包含的所有jar的内容。

删除bin文件夹，清理并再次构建项目。使用Twister，正在运行的应用程序的.jar文件必须存在于apps /目录中。无论何时更改（从IDE运行），都要导出最新版本的应用程序。

迭代MapReduce（Twister）运行异常

1 个答案: