DoFn中的com.google.cloud.dataflow.sdk.util.IllegalMutationException转换为org.apache.hadoop.hbase.client.Put

时间:2016-09-29 13:24:51

标签: google-cloud-dataflow google-cloud-bigtable

我正在尝试将对象转换为DoFn中的org.apache.hadoop.hbase.client.Put。 我一直收到这个错误:

com.google.cloud.dataflow.sdk.util.IllegalMutationException: DoFn DataBuckettingToBigTableConnectorFn mutated value {"totalColumns":36,"row":"HealthTechnology#MedicalSpecialties#1470010560000#gnip","families":{"SENT":[{"qualifier":"sector","vlen":16,"tag":[],"timestamp":1475153818546},{"qualifier":"industry","vlen":18,"tag":[],"timestamp":1475153818546},{"qualifier":"updateTime","vlen":8,"tag":[],"timestamp":1475153818546},{"qualifier":"timeBucket","vlen":8,"tag":[],"timestamp":1475153818546}]}} after it was output (new value was {"totalColumns":36,"row":"HealthTechnology#MedicalSpecialties#1470010560000#gnip","families":{"SENT":[{"qualifier":"sector","vlen":16,"tag":[],"timestamp":1475153818576},{"qualifier":"industry","vlen":18,"tag":[],"timestamp":1475153818576},{"qualifier":"updateTime","vlen":8,"tag":[],"timestamp":1475153818576},{"qualifier":"timeBucket","vlen":8,"tag":[],"timestamp":1475153818576}]}}). Values must not be mutated in any way after being output.
Caused by: com.google.cloud.dataflow.sdk.util.IllegalMutationException: Value {"totalColumns":36,"row":"HealthTechnology#MedicalSpecialties#1470010560000#gnip","families":{"SENT":[{"qualifier":"sector","vlen":16,"tag":[],"timestamp":1475153818546},{"qualifier":"industry","vlen":18,"tag":[],"timestamp":1475153818546},{"qualifier":"updateTime","vlen":8,"tag":[],"timestamp":1475153818546},{"qualifier":"timeBucket","vlen":8,"tag":[],"timestamp":1475153818546}]}} mutated illegally, new value was {"totalColumns":36,"row":"HealthTechnology#MedicalSpecialties#1470010560000#gnip","families":{"SENT":[{"qualifier":"sector","vlen":16,"tag":[],"timestamp":1475153818576},{"qualifier":"industry","vlen":18,"tag":[],"timestamp":1475153818576},{"qualifier":"updateTime","vlen":8,"tag":[],"timestamp":1475153818576},{"qualifier":"timeBucket","vlen":8,"tag":[],"timestamp":1475153818576}]}}. Encoding was kg4SNkhlYWx0aFRlY2hub2xvZ3kjTWVkaWNhbFNwZWNpYWx0aWVzIzE0NzAwMTA1NjAwMDAjZ25pcBorCikKBFNFTlQSBnNlY3RvchjQ3t-a0LTPAiIQSGVhbHRoVGVjaG5vbG9neRovCi0KBFNFTlQSCGluZHVzdHJ5GNDe35rQtM8CIhJNZWRpY2FsU3BlY2lhbHRpZXMaJwolCgRTRU5UEgp1cGRhdGVUaW1lGNDe35rQtM8CIggAAAFXdgTwiBonCiUKBFNFTlQSCnRpbWVCdWNrZXQY0N7fmtC0zwIiCAAAAVZDdQ4AGh8KHQoEU0VOVBIGc291cmNlGNDe35rQtM8CIgRHbmlwGiMKIQoEU0VOVBIEbWVhbhjQ3t-a0LTPAiIKAAAAD6HuTKnCExorCikKBFNFTlQSDG1lYW5OZWdhdGl2ZRjQ3t-a0LTPAiIKAAAAD6HuTKnCExomCiQKBFNFTlQSDG1lYW5Qb3NpdGl2ZRjQ3t-a0LTPAiIFAAAAAAAaKwopCgRTRU5UEhFzdGFuZGFyZERldmlhdGlvbhjQ3t-a0LTPAiIFAAAAAAAaMwoxCgRTRU5UEhlzdGFuZGFyZERldmlhdGlvbk5lZ2F0aXZlGNDe35rQtM8CIgUAAAAAABozCjEKBFNFTlQSGXN0YW5kYXJkRGV2aWF0aW9uUG9zaXRpdmUY0N7fmtC0zwIiBQAAAAAAGiIKIAoEU0VOVBIIdmFyaWFuY2UY0N7fmtC0zwIiBQAAAAAAGioKKAoEU0VOVBIQdmFyaWFuY2VOZWdhdGl2ZRjQ3t-a0LTPAiIFAAAAAAAaKgooCgRTRU5UEhB2YXJpYW5jZVBvc2l0aXZlGNDe35rQtM8CIgUAAAAAABo1CjMKBFNFTlQSFnNjb3JlRm9sbG93ZXJzV2VpZ2h0ZWQY0N7fmtC0zwIiCgAAAA-h7kypwhMaNQozCgRTRU5UEhVzY29yZU1heFNjb3JlV2VpZ2h0ZWQY0N7fmtC0zwIiCwAAAA_8coFbOYAAGjQKMgoEU0VOVBIac2NvcmVTaWduRm9sbG93ZXJzV2VpZ2h0ZWQY0N7fmtC0zwIiBQAAAAH2GicKJQoEU0VOVBIIbWF4U2NvcmUY0N7fmtC0zwIiCgAAAA9eEbNWPe0aMQovCgRTRU5UEhRudW1iZXJPZk9ic2VydmF0aW9ucxjQ3t-a0LTPAiIIAAAAAAAAAAEaOQo3CgRTRU5UEhxudW1iZXJPZk9ic2VydmF0aW9uc05lZ2F0aXZlGNDe35rQtM8CIggAAAAAAAAAARo5CjcKBFNFTlQSHG51bWJlck9mT2JzZXJ2YXRpb25zUG9zaXRpdmUY0N7fmtC0zwIiCAAAAAAAAAAAGjUKMwoEU0VOVBIYbnVtYmVyT2ZCdWxsT2JzZXJ2YXRpb25zGNDe35rQtM8CIggAAAAAAAAAABo9CjsKBFNFTlQSIG51bWJlck9mQnVsbFBvc2l0aXZlT2JzZXJ2YXRpb25zGNDe35rQtM8CIggAAAAAAAAAABo9CjsKBFNFTlQSIG51bWJlck9mQnVsbE5lZ2F0aXZlT2JzZXJ2YXRpb25zGNDe35rQtM8CIggAAAAAAAAAABo1CjMKBFNFTlQSGG51bWJlck9mQmVhck9ic2VydmF0aW9ucxjQ3t-a0LTPAiIIAAAAAAAAAAAaPQo7CgRTRU5UEiBudW1iZXJPZkJlYXJQb3NpdGl2ZU9ic2VydmF0aW9ucxjQ3t-a0LTPAiIIAAAAAAAAAAAaPQo7CgRTRU5UEiBudW1iZXJPZkJlYXJOZWdhdGl2ZU9ic2VydmF0aW9ucxjQ3t-a0LTPAiIIAAAAAAAAAAAaJAoiCgRTRU5UEgVzY29yZRjQ3t-a0LTPAiIKAAAAD6HuTKnCExosCioKBFNFTlQSDXNjb3JlTmVnYXRpdmUY0N7fmtC0zwIiCgAAAA-h7kypwhMaJwolCgRTRU5UEg1zY29yZVBvc2l0aXZlGNDe35rQtM8CIgUAAAABABosCioKBFNFTlQSDHNjb3JlU3F1YXJlZBjQ3t-a0LTPAiILAAAAEiYBlRXVSeUaNAoyCgRTRU5UEhRzY29yZVNxdWFyZWROZWdhdGl2ZRjQ3t-a0LTPAiILAAAAEiYBlRXVSeUaLgosCgRTRU5UEhRzY29yZVNxdWFyZWRQb3NpdGl2ZRjQ3t-a0LTPAiIFAAAAAQAaJwolCgRTRU5UEgxzdW1Gb2xsb3dlcnMY0N7fmtC0zwIiBgAAAAAGIRoyCjAKBFNFTlQSEnN1bVNjb3JlWEZvbGxvd2VycxjQ3t-a0LTPAiILAAAADsZYjS-kpXIaLAoqCgRTRU5UEhFzdW1TaWduWEZvbGxvd2VycxjQ3t-a0LTPAiIGAAAAAcK2, now kg4SNkhlYWx0aFRlY2hub2xvZ3kjTWVkaWNhbFNwZWNpYWx0aWVzIzE0NzAwMTA1NjAwMDAjZ25pcBorCikKBFNFTlQSBnNlY3RvchiAyeGa0LTPAiIQSGVhbHRoVGVjaG5vbG9neRovCi0KBFNFTlQSCGluZHVzdHJ5GIDJ4ZrQtM8CIhJNZWRpY2FsU3BlY2lhbHRpZXMaJwolCgRTRU5UEgp1cGRhdGVUaW1lGIDJ4ZrQtM8CIggAAAFXdgTwiBonCiUKBFNFTlQSCnRpbWVCdWNrZXQYgMnhmtC0zwIiCAAAAVZDdQ4AGh8KHQoEU0VOVBIGc291cmNlGIDJ4ZrQtM8CIgRHbmlwGiMKIQoEU0VOVBIEbWVhbhiAyeGa0LTPAiIKAAAAD6HuTKnCExorCikKBFNFTlQSDG1lYW5OZWdhdGl2ZRiAyeGa0LTPAiIKAAAAD6HuTKnCExomCiQKBFNFTlQSDG1lYW5Qb3NpdGl2ZRiAyeGa0LTPAiIFAAAAAAAaKwopCgRTRU5UEhFzdGFuZGFyZERldmlhdGlvbhiAyeGa0LTPAiIFAAAAAAAaMwoxCgRTRU5UEhlzdGFuZGFyZERldmlhdGlvbk5lZ2F0aXZlGIDJ4ZrQtM8CIgUAAAAAABozCjEKBFNFTlQSGXN0YW5kYXJkRGV2aWF0aW9uUG9zaXRpdmUYgMnhmtC0zwIiBQAAAAAAGiIKIAoEU0VOVBIIdmFyaWFuY2UYgMnhmtC0zwIiBQAAAAAAGioKKAoEU0VOVBIQdmFyaWFuY2VOZWdhdGl2ZRiAyeGa0LTPAiIFAAAAAAAaKgooCgRTRU5UEhB2YXJpYW5jZVBvc2l0aXZlGIDJ4ZrQtM8CIgUAAAAAABo1CjMKBFNFTlQSFnNjb3JlRm9sbG93ZXJzV2VpZ2h0ZWQYgMnhmtC0zwIiCgAAAA-h7kypwhMaNQozCgRTRU5UEhVzY29yZU1heFNjb3JlV2VpZ2h0ZWQYgMnhmtC0zwIiCwAAAA_8coFbOYAAGjQKMgoEU0VOVBIac2NvcmVTaWduRm9sbG93ZXJzV2VpZ2h0ZWQYgMnhmtC0zwIiBQAAAAH2GicKJQoEU0VOVBIIbWF4U2NvcmUYgMnhmtC0zwIiCgAAAA9eEbNWPe0aMQovCgRTRU5UEhRudW1iZXJPZk9ic2VydmF0aW9ucxiAyeGa0LTPAiIIAAAAAAAAAAEaOQo3CgRTRU5UEhxudW1iZXJPZk9ic2VydmF0aW9uc05lZ2F0aXZlGIDJ4ZrQtM8CIggAAAAAAAAAARo5CjcKBFNFTlQSHG51bWJlck9mT2JzZXJ2YXRpb25zUG9zaXRpdmUYgMnhmtC0zwIiCAAAAAAAAAAAGjUKMwoEU0VOVBIYbnVtYmVyT2ZCdWxsT2JzZXJ2YXRpb25zGIDJ4ZrQtM8CIggAAAAAAAAAABo9CjsKBFNFTlQSIG51bWJlck9mQnVsbFBvc2l0aXZlT2JzZXJ2YXRpb25zGIDJ4ZrQtM8CIggAAAAAAAAAABo9CjsKBFNFTlQSIG51bWJlck9mQnVsbE5lZ2F0aXZlT2JzZXJ2YXRpb25zGIDJ4ZrQtM8CIggAAAAAAAAAABo1CjMKBFNFTlQSGG51bWJlck9mQmVhck9ic2VydmF0aW9ucxiAyeGa0LTPAiIIAAAAAAAAAAAaPQo7CgRTRU5UEiBudW1iZXJPZkJlYXJQb3NpdGl2ZU9ic2VydmF0aW9ucxiAyeGa0LTPAiIIAAAAAAAAAAAaPQo7CgRTRU5UEiBudW1iZXJPZkJlYXJOZWdhdGl2ZU9ic2VydmF0aW9ucxiAyeGa0LTPAiIIAAAAAAAAAAAaJAoiCgRTRU5UEgVzY29yZRiAyeGa0LTPAiIKAAAAD6HuTKnCExosCioKBFNFTlQSDXNjb3JlTmVnYXRpdmUYgMnhmtC0zwIiCgAAAA-h7kypwhMaJwolCgRTRU5UEg1zY29yZVBvc2l0aXZlGIDJ4ZrQtM8CIgUAAAABABosCioKBFNFTlQSDHNjb3JlU3F1YXJlZBiAyeGa0LTPAiILAAAAEiYBlRXVSeUaNAoyCgRTRU5UEhRzY29yZVNxdWFyZWROZWdhdGl2ZRiAyeGa0LTPAiILAAAAEiYBlRXVSeUaLgosCgRTRU5UEhRzY29yZVNxdWFyZWRQb3NpdGl2ZRiAyeGa0LTPAiIFAAAAAQAaJwolCgRTRU5UEgxzdW1Gb2xsb3dlcnMYgMnhmtC0zwIiBgAAAAAGIRoyCjAKBFNFTlQSEnN1bVNjb3JlWEZvbGxvd2VycxiAyeGa0LTPAiILAAAADsZYjS-kpXIaLAoqCgRTRU5UEhFzdW1TaWduWEZvbGxvd2VycxiAyeGa0LTPAiIGAAAAAcK2.
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation(MutationDetectors.java:167) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions(MutationDetectors.java:162) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified(MutationDetectors.java:113) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.verifyOutputUnmodified(ParDo.java:1338) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.output(ParDo.java:1306) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449) ~[google-cloud-dataflow-java-sdk-all-1.7.0.jar:na]

但是,如果我使用的是与org.apache.hadoop.hbase.client.Put(任何自定义类型)不同的类,则不会发生错误

这是当前的DoFn processElementMethod:

@Override
public void processElement(ProcessContext c) throws Exception {
        KV<Key, DataflowBucketedData> element = c.element();
        if(element == null)
            return;
        Key itemKey = element.getKey();
        DataflowBucketedData val = element.getValue();

        String key;
        switch (element.getKey().getKeyType()){
            case Industry:
                key = INDUSTRY_KEY_UTILS.createKey(itemKey.getSectorCode(), itemKey.getIndustryCode(), val.getTimeBucket().getTime(), val.getSource().name().toLowerCase());
                break;
            case Sector:
            key = SECTOR_KEY_UTILS.createKey(itemKey.getSectorCode(), val.getTimeBucket().getTime(), val.getSource().name().toLowerCase());
            break;
        case Symbol:
            key = SYMBOL_KEY_UTILS.createKey(itemKey.getSymbol(), val.getTimeBucket().getTime(), val.getSource().name().toLowerCase());
            break;
        default:
            key = null;
            break;
    }
    if(!Strings.isNullOrEmpty(key)) {
        Put p = createEntity(key, itemKey.getSymbol(), itemKey.getSectorCode(), itemKey.getIndustryCode(), val);
        c.output(p);
    }
}

private Put createEntity(String key, String company, String sectorCode, String industryCode, DataflowBucketedData data){
    Put put = new Put(Bytes.toBytes(key));
    put.setId(key);
    if(!Strings.isNullOrEmpty(company))
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SYMBOL, Bytes.toBytes(company));
    else {
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SECTOR, Bytes.toBytes(sectorCode));
        if(!Strings.isNullOrEmpty(industryCode))
            put.addColumn(Schema.COLUMN_FAMILY, Schema.INDUSTRY, Bytes.toBytes(industryCode));
    }
    if(data.getUpdateTime() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.UPDATE_TIME, Bytes.toBytes(data.getUpdateTime().getTime()));
    if (data.getTimeBucket() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.TIME_BUCKET, Bytes.toBytes(data.getTimeBucket().getTime()));

    if (data.getSource() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SOURCE, Bytes.toBytes(data.getSource().name()));
    if (data.getMean() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.MEAN, Bytes.toBytes(data.getMean()));
    if (data.getMeanNegative() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.MEAN_NEGATIVE, Bytes.toBytes(data.getMeanNegative()));
    if (data.getMeanPositive() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.MEAN_POSITIVE, Bytes.toBytes(data.getMeanPositive()));
    if (data.getStandardDeviation() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.STANDARD_DEVIATION, Bytes.toBytes(data.getStandardDeviation()));
    if (data.getStandardDeviationNegative() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.STANDARD_DEVIATION_NEGATIVE, Bytes.toBytes(data.getStandardDeviationNegative()));
    if (data.getStandardDeviationPositive() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.STANDARD_DEVIATION_POSITIVE, Bytes.toBytes(data.getStandardDeviationPositive()));
    if (data.getVariance() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.VARIANCE, Bytes.toBytes(data.getVariance()));
    if (data.getVarianceNegative() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.VARIANCE_NEGATIVE, Bytes.toBytes(data.getVarianceNegative()));
    if (data.getVariancePositive() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.VARIANCE_POSITIVE, Bytes.toBytes(data.getVariancePositive()));
    if (data.getScoreFollowersWeighted() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_FOLLOWERS_WEIGHTED, Bytes.toBytes(data.getScoreFollowersWeighted()));
    if (data.getScoreMaxScoreWeighted() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_MAX_SCORE_WEIGHTED, Bytes.toBytes(data.getScoreMaxScoreWeighted()));
    if (data.getScoreSignFollowersWeighted() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_SIGN_FOLLOWERS_WEIGHTED, Bytes.toBytes(data.getScoreSignFollowersWeighted()));
    if (data.getMaxScore() != null)
        put.addColumn(Schema.COLUMN_FAMILY, Schema.MAX_SCORE, Bytes.toBytes(data.getMaxScore()));

    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_OBSERVATIONS, Bytes.toBytes(data.getNumberOfObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_OBSERVATIONS_NEGATIVE, Bytes.toBytes(data.getNumberOfObservationsNegative()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_OBSERVATIONS_POSITIVE, Bytes.toBytes(data.getNumberOfObservationsPositive()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BULL_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBullObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BULL_POSITIVE_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBullPositiveObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BULL_NEGATIVE_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBullNegativeObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BEAR_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBearObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BEAR_POSITIVE_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBearPositiveObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.NUMBER_OF_BEAR_NEGATIVE_OBSERVATIONS, Bytes.toBytes(data.getNumberOfBearNegativeObservations()));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE, Bytes.toBytes(BigDecimal.valueOf(data.getScore())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_NEGATIVE, Bytes.toBytes(BigDecimal.valueOf(data.getScoreNegative())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_POSITIVE, Bytes.toBytes(BigDecimal.valueOf(data.getScorePositive())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_SQUARED, Bytes.toBytes(BigDecimal.valueOf(data.getScoreSquared())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_SQUARED_NEGATIVE, Bytes.toBytes(BigDecimal.valueOf(data.getScoreSquaredNegative())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SCORE_SQUARED_POSITIVE, Bytes.toBytes(BigDecimal.valueOf(data.getScoreSquaredPositive())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SUM_FOLLOWERS, Bytes.toBytes(BigDecimal.valueOf(data.getSumFollowers())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SUM_SCORE_X_FOLLOWERS, Bytes.toBytes(BigDecimal.valueOf(data.getSumScoreXFollowers())));
    put.addColumn(Schema.COLUMN_FAMILY, Schema.SUM_SIGN_X_FOLLOWERS, Bytes.toBytes(BigDecimal.valueOf(data.getSumSignXFollowers())));
    return put;
}

正如您所看到的,输出后没有对Put对象进行任何修改。

以下是我使用的库

compile group:'com.google.cloud.dataflow', name:'google-cloud-dataflow-java-sdk-all', version:'1.7.0'
compile group:'com.google.cloud.bigtable', name:'bigtable-hbase-dataflow', version:'0.9.2'

有什么想法吗?

更新: 我在https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/tree/master/java/dataflow-connector-examples抓住了这个例子 它也不起作用,完全相同:

[main] INFO com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner - Executing pipeline using the DirectPipelineRunner.
Disconnected from the target VM, address: '127.0.0.1:50713', transport: 'socket'
Exception in thread "main" com.google.cloud.dataflow.sdk.Pipeline$PipelineExecutionException: com.google.cloud.dataflow.sdk.util.IllegalMutationException: DoFn  mutated value {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801718}]}} after it was output (new value was {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801752}]}}). Values must not be mutated in any way after being output.
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:186)
    at com.google.cloud.bigtable.dataflow.example.HelloWorldWrite.main(HelloWorldWrite.java:113)
Caused by: com.google.cloud.dataflow.sdk.util.IllegalMutationException: DoFn  mutated value {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801718}]}} after it was output (new value was {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801752}]}}). Values must not be mutated in any way after being output.
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.verifyOutputUnmodified(ParDo.java:1344)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.output(ParDo.java:1306)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449)
    at com.google.cloud.bigtable.dataflow.example.HelloWorldWrite$1.processElement(HelloWorldWrite.java:71)
Caused by: com.google.cloud.dataflow.sdk.util.IllegalMutationException: Value {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801718}]}} mutated illegally, new value was {"totalColumns":1,"row":"World","families":{"cf":[{"qualifier":"qualifier","vlen":24,"tag":[],"timestamp":1475158801752}]}}. Encoding was PRIFV29ybGQaNAoyCgJjZhIJcXVhbGlmaWVyGPC19OLitM8CIhh2YWx1ZV8xNC41NDc5NDY0NDY0NzMyNTY, now PRIFV29ybGQaNAoyCgJjZhIJcXVhbGlmaWVyGMC_9uLitM8CIhh2YWx1ZV8xNC41NDc5NDY0NDY0NzMyNTY.
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation(MutationDetectors.java:167)
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions(MutationDetectors.java:162)
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified(MutationDetectors.java:113)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.verifyOutputUnmodified(ParDo.java:1338)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.output(ParDo.java:1306)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449)
    at com.google.cloud.bigtable.dataflow.example.HelloWorldWrite$1.processElement(HelloWorldWrite.java:71)
    at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
    at com.google.cloud.dataflow.sdk.transforms.ParDo.evaluateHelper(ParDo.java:1229)
    at com.google.cloud.dataflow.sdk.transforms.ParDo.evaluateSingleHelper(ParDo.java:1098)
    at com.google.cloud.dataflow.sdk.transforms.ParDo.access$300(ParDo.java:457)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$1.evaluate(ParDo.java:1084)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$1.evaluate(ParDo.java:1079)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:858)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:215)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:102)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:259)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:814)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:526)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:96)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:180)
    at com.google.cloud.bigtable.dataflow.example.HelloWorldWrite.main(HelloWorldWrite.java:113)

1 个答案:

答案 0 :(得分:2)

抱歉你的烦恼。这是云Bigtable连接器中的一个错误。我们为即将发布的0.9.3版本修复了此问题。

您有两种方法可以解决此问题:

  1. 升级到0.9.3-SNAPSHOT。您需要this pom.xml中的存储库部分。在发布官方0.9.3版本之前,我们正在等待确认再修复一次。这应该有希望在下周初发生。
  2. 使用0.9.2并在所有Put.addColumn()次呼叫上设置时间戳参数(System.currentTimeMillis())。