Question

我正在使用Univocity解析器2.7.3版本。我有一个具有100万条记录的CSV文件，并且将来可能会增加。我仅从文件中读取一些特定的列，以下是我的要求：

请勿在任何时候将CSV内容存储到内存中
如果是纬度或经度列，则忽略/跳过Bean创建 CSV中为空/空白

为了满足这些要求，我尝试实现CsvRoutines，以便不将CSV数据复制到内存中。我在“纬度”和“经度”字段上都使用@Validate批注，并使用错误处理程序不抛出任何异常，以便在验证失败时跳过该记录。

示例CSV：

#version:1.0
#timestamp:2017-05-29T23:22:22.320Z
#brand:test report    
    network_name,location_name,location_category,location_address,location_zipcode,location_phone_number,location_latitude,location_longitude,location_city,location_state_name,location_state_abbreviation,location_country,location_country_code,pricing_type,wep_key
    "1 Free WiFi","Test Restaurant","Cafe / Restaurant","Marktplatz 18","1233","+41 263 34 05","1212.15","7.51","Basel","test","BE","India","DE","premium",""
    "2 Free WiFi","Test Restaurant","Cafe / Restaurant","Zufikerstrasse 1","1111","+41 631 60 00","11.354","8.12","Bremgarten","test","AG","China","CH","premium",""
    "3 Free WiFi","Test Restaurant","Cafe / Restaurant","Chemin de la Fontaine 10","1260","+41 22 361 69","12.34","11.23","Nyon","Vaud","VD","Switzerland","CH","premium",""
    "!.oist*~","HoistGroup Office","Office","Chemin de I Etang","CH-1211","","","","test","test","GE","Switzerland","CH","premium",""
    "test","tess's Takashiro","Cafe / Restaurant","Test 1-10","870-01","097-55-1808","","","Oita","Oita","OITA","Japan","JP","premium","1234B"

TestDTO.java

@Data
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class TestDTO implements Serializable {

    @Parsed(field = "location_name")
    private String name;
    @Parsed(field = "location_address")
    private String addressLine1;
    @Parsed(field = "location_city")
    private String city;
    @Parsed(field = "location_state_abbreviation")
    private String state;
    @Parsed(field = "location_country_code")
    private String country;
    @Parsed(field = "location_zipcode")
    private String postalCode;

    @Parsed(field = "location_latitude")
    @Validate
    private Double latitude;

    @Parsed(field = "location_longitude")
    @Validate
    private Double longitude;

    @Parsed(field = "network_name")
    private String ssid;
}

Main.java

 CsvParserSettings parserSettings = new CsvParserSettings();        
        parserSettings.detectFormatAutomatically();
        parserSettings.setLineSeparatorDetectionEnabled(true);
        parserSettings.setHeaderExtractionEnabled(true);
        parserSettings.setSkipEmptyLines(true);
        parserSettings.selectFields("network_name", "location_name","location_address", "location_zipcode",
                "location_latitude", "location_longitude", "location_city","location_state_abbreviation", "location_country_code");

        parserSettings.setProcessorErrorHandler(new RowProcessorErrorHandler() {
            @Override
            public void handleError(DataProcessingException error, Object[] inputRow, ParsingContext context) {
                //do nothing
            }
        });


        CsvRoutines parser = new CsvRoutines(parserSettings);
        ResultIterator<TestDTO, ParsingContext> iterator = parser.iterate(TestDTO.class, new FileReader("c:\\users\\...\\test.csv")).iterator();


        int i=0;
        while(iterator.hasNext()) {
            TestDTO dto = iterator.next();
            if(dto.getLongitude() == null || dto.getLatitude() == null)
                i++;            
        }

        System.out.println("count=="+i);

问题：

我实际上期望该计数为零，因为我添加了错误处理程序并且没有抛出数据验证异常，但是事实并非如此。我以为@Validate遇到纬度或经度为空的记录（在同一条记录中列也可能为空）时，它将抛出异常，该异常将在错误处理程序处进行处理和忽略/跳过。

基本上，我不希望UniVocity在堆中创建和映射不必要的DTO对象（并导致内存不足），因为传入的CSV文件可能有超过200或300k记录，且纬度/经度为null

我什至尝试在@Validate中添加自定义验证器，但徒劳无功。

有人可以让我知道我在这里想念什么吗？

Answer 1

此处是库的作者。您做对了所有事情。这是一个错误，我刚刚打开了这个问题here，今天就可以解决。

该错误在您选择字段时出现：值的重新排序使验证针对其他问题（在我的测试中，它验证了城市而不是纬度）。

以您为例，只需添加以下代码行即可正常工作：

parserSettings.setColumnReorderingEnabled(false);

这将使行以未选择字段的空值生成，而不是删除空值并在已解析的行中重新排列值。这样可以避免该错误，并使您的程序运行得更快一些。

您还需要在迭代位中测试null：

TestDTO dto = iterator.next();
if(dto != null) { // dto may come null here due to validation
    if (dto.longitude == null || dto.latitude == null)
        i++;
    }
}

希望这会有所帮助，并感谢您使用我们的解析器！

与UniVocity解析器中的CsvRoutines一起使用时，@ Validate不会跳过无效的行

1 个答案: