Question

我刚写了一个简单的hadoop程序，我试图使用AES算法加密文本文件。我在map方法中逐行读取，加密并写入上下文。很简单。我在map方法中进行加密并使用行偏移量作为键，因此我不需要reducer类。

这是我的代码：

public class Enc {

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String strDataToEncrypt = new String();
            String strCipherText = new String();

            KeyGenerator keyGen = KeyGenerator.getInstance("AES");
            keyGen.init(128);
            SecretKey secretKey = keyGen.generateKey();

            Cipher aesCipher = Cipher.getInstance("AES");
            aesCipher.init(Cipher.ENCRYPT_MODE,secretKey);
            strDataToEncrypt = value.toString();

            byte[] byteDataToEncrypt = strDataToEncrypt.getBytes();
            byte[] byteCipherText = aesCipher.doFinal(byteDataToEncrypt); 
            strCipherText = new BASE64Encoder().encode(byteCipherText);
            System.out.println("cipher text: " +strCipherText);

                    String cipherString =  new String(strCipherText);
                    context.write(key, new Text(cipherString));

                }
    } 

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();

        Job job = new Job(conf, "Enc");
        job.setJarByClass(Enc.class);

        job.setOutputKeyClass(LongWritable.class);
        job.setOutputValueClass(Text.class);

        job.setMapperClass(Map.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.waitForCompletion(true);
    }        
}

我收到以下错误：

The method write(Text, IntWritable) in the type TaskInputOutputContext<LongWritable,Text,Text,IntWritable> is not applicable for the arguments (LongWritable, Text)

我在这里缺少什么？

EDIT_1：

我的最终工作代码在这里：

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.Cipher;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.InvalidKeyException;
import java.security.InvalidAlgorithmParameterException;

import javax.crypto.NoSuchPaddingException;
import javax.crypto.BadPaddingException;
import javax.crypto.IllegalBlockSizeException;

import sun.misc.BASE64Encoder;

public class Enc {

      public static class Map extends Mapper<LongWritable, Text, LongWritable, Text> {
        private Text word = new Text();
        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

            try {
            String strDataToEncrypt = new String();
            String strCipherText = new String();

            KeyGenerator keyGen = KeyGenerator.getInstance("AES");
            keyGen.init(128);
            SecretKey secretKey = keyGen.generateKey();

            Cipher aesCipher = Cipher.getInstance("AES");
            aesCipher.init(Cipher.ENCRYPT_MODE,secretKey);
            strDataToEncrypt = value.toString();

            byte[] byteDataToEncrypt = strDataToEncrypt.getBytes();
            byte[] byteCipherText = aesCipher.doFinal(byteDataToEncrypt); 
            strCipherText = new BASE64Encoder().encode(byteCipherText);
            System.out.println("cipher text: " +strCipherText);

            String cipherString =  new String(strCipherText);
            context.write(key, new Text(cipherString));
            }
            catch (NoSuchAlgorithmException noSuchAlgo)
            {
                System.out.println(" No Such Algorithm exists " + noSuchAlgo);
            }

                catch (NoSuchPaddingException noSuchPad)
                {
                    System.out.println(" No Such Padding exists " + noSuchPad);
                }

                    catch (InvalidKeyException invalidKey)
                    {
                        System.out.println(" Invalid Key " + invalidKey);
                    }

                    catch (BadPaddingException badPadding)
                    {
                        System.out.println(" Bad Padding " + badPadding);
                    }

                    catch (IllegalBlockSizeException illegalBlockSize)
                    {
                        System.out.println(" Illegal Block Size " + illegalBlockSize);
                    }


        }
    } 


    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();

        Job job = new Job(conf, "Enc");
        job.setJarByClass(Enc.class);

        job.setOutputKeyClass(LongWritable.class);
        job.setOutputValueClass(Text.class);

        job.setMapperClass(Map.class);
        //job.setCombinerClass(Reduce.class);
        //job.setReducerClass(Reduce.class);


        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.waitForCompletion(true);
    }        
}

Answer 1

这里的第一个重点是

public class Map extends Mapper<LongWritable, Text, Text, IntWritable>

其中声明您的Map类将LongWritable键和Text值作为输入，并且它将Text键和IntWritable值作为输出。

第二个重点是

context.write(key, new Text(cipherString));

这是您从mapper中提供输出的地方。键的类型为LongWritable，第二个参数为Text。

这里的问题是存在不匹配。您声称映射器在扩展Mapper时输出Text键和IntWritable值，但实际输出的是LongWritable和Text。如果您确实要输出LongWritable和文本，则应将类声明更改为

public class Map extends Mapper<LongWritable, Text, LongWritable, Text>

Answer 2

您的代码context.write(key, new Text(cipherString));在参数中的类型错误。 write方法的参数不仅限于你在作业定义中指定的，你做得对，还包括mapper的通用代码部分：

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable>

，这是输出类型的错误。（IDE不够聪明，不能为你注意这个错误）

将其更改为

public static class Map extends Mapper<LongWritable, Text, LongWritable, Text>

会解决问题。

Hadoop错误：写入方法中的类型不匹配

2 个答案: