Question

在尝试阅读ORC文件时，我面临着令人难以置信的问题（对我而言）。默认情况下，Hive orc文件位于＆＃34; UTF-8＆＃34;编码，或至少应该是。我正在做一个ORC文件的copyToLocal并试图用Java读取ORC文件。

虽然它有一些不需要的字符，但我能够成功读取文件：

当我在配置单元中查询表时，没有不需要的字符：

有人可以帮忙吗？我尝试过各种格式的解码和编码，如（ISO-8859-1到UTF-8），（UTF-8到ISO-8859-1），（ISO-8859-1到UTF-16）等。

编辑：

嗨，我使用下面的java代码来读取ORC文件：

    import org.apache.hadoop.hive.ql.io.orc.Reader;
import org.apache.hadoop.hive.ql.io.orc.RecordReader;
import org.apache.hadoop.hive.serde2.objectinspector.StructField;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;

public class OrcFormat {
    public static void main(String[] argv)
    {
        System.out.println(System.getProperty("file.encoding").toString());
        System.out.println(Charset.defaultCharset().name());

        try {
            Configuration conf = new Configuration();
            Utils.createFile("C:/path/target","opfile.txt","UTF-8");
            Reader reader = OrcFile.createReader(new Path("C:/input/000000_0"),OrcFile.readerOptions(conf));

            StructObjectInspector inspector = (StructObjectInspector)reader.getObjectInspector();

            List<String> keys = reader.getMetadataKeys();
            for(int i=0;i<keys.size();i++){
                System.out.println("Key:"+keys.get(i)+",Value:"+reader.getMetadataValue(keys.get(i)));
            }


            RecordReader records = reader.rows();
            Object row = null;

            List fields = inspector.getAllStructFieldRefs();
            for(int i = 0; i < fields.size(); ++i) {
                System.out.print(((StructField)fields.get(i)).getFieldObjectInspector().getTypeName() + '\t');

            }
            System.out.println();
            int rCnt=0;
            while(records.hasNext())
            {
                row = records.next(row);
                List value_lst = inspector.getStructFieldsDataAsList(row);
                String out = "";

                for(Object field : value_lst) {
                    if(field != null)
                        out+=field;
                    out+="\t";
                }
                rCnt++;

                out = out+"\n";
                byte[] outA = convertEncoding(out,"UTF-8","UTF-8");
                Utils.writeToFile(outA,"C:/path/target","opfile.txt","UTF-8");
                if(rCnt<10){
                    System.out.println(out);
                    System.out.println(new String(outA));
                }else{
                    break;
                }
            }
        }catch (Exception e)
        {
            e.printStackTrace();
        }
    }   

    public static byte[] convertEncoding(String s,String inCharset,String outCharset){
        Charset inC = Charset.forName(inCharset);
        Charset outC = Charset.forName(outCharset);
        ByteBuffer inpBuffer = ByteBuffer.wrap(s.getBytes());
        CharBuffer data = inC.decode(inpBuffer);

        ByteBuffer opBuffer = outC.encode(data);
        byte[] opData = opBuffer.array();
        return opData;
    }
}

从Hive表文件读取时出现字符编码问题

0 个答案: