Spark数据集 - 读取CSV并写入空输出

时间:2017-06-10 01:43:12

标签: java scala apache-spark

我有一个输入文件<div> <md-input-container fxFlex> <textarea #listCode mdInput [(ngModel)]="listInfo.code" placeholder="List Code" uppercase-code maxlength="50" rows="3" required></textarea> <md-hint align="end">{{listCode.value.length}} / 50</md-hint> </md-input-container> {{listInfo.code}} </div>

import { Directive } from '@angular/core';
import { NgControl } from '@angular/forms';

@Directive({
  selector: '[ngModel][uppercase-code]',
  host: {
    '(ngModelChange)': 'ngOnChanges($event)'
  }
})
export class UppercaseCodeDirective {
  constructor(public model: NgControl) {}
  ngOnChanges(event) {
    var newVal = event.replace(/[^A-Za-z0-9_]*/g, '');
    newVal = newVal.toUpperCase();
    this.model.valueAccessor.writeValue(newVal);       
  }
}

我编写了以下源代码,以便获得我所面临的问题的最小,完整和可验证的示例。

用于读取CSV文件的public PreparedStatement setSQL(String sql) throws SQLException { try { pst = connection.prepareStatement(sql); } catch (SQLException e) { e.printStackTrace(); } return pst; } public void testSetSQL() throws SQLException { String sql = "SELECT * FROM student INNER JOIN member ON student.ssn=member.ssn WHERE ssn = ?"; result = dbconn.setSQL(sql); // expsql = ; expectresult = connection.prepareStatement("SELECT * FROM student INNER JOIN member ON student.ssn=member.ssn WHERE ssn = ?"); assertSame(expectresult, result); 类和用于写入输出的test-reading.csv。顺便提一下,它们几乎完全相同,但在实际程序中却完全不同,因为它们代表了输入和输出结构。

剩下的代码启动Spark,读取CSV,将id,sku,price "100002701--425370728",100002701,12159 "100002701--510892030",100002701,11021 "100002701-235195215",100002701,12330 "100002701-110442364",100002701,9901 "100002701-1963746094",100002701,11243 映射到ReadingRecord并写入输出CSV。

问题是:为什么如果我将WritingRecord循环取消注释到ReadingRecord方法,这个Spark程序会停止写入CSV输出?

WritingRecord

1 个答案:

答案 0 :(得分:1)

如果包含已注释的代码,您正在尝试重复使用Iterator b。使用时会修改Iterator

  

特别重要的是要注意,除非另有说明,在调用方法之后永远不应该使用迭代器。

请参阅the Iterator documentation