Question

I have a DynamoDB table, let us say here: ReportingTable. Its has following keys to uniquely identify items:

reportingtablePrimaryKey - partition key of table.
merchantId - sort Key of table.
transactionType-timestamp-index - Global Secondary Index of table containing following attributes.
1. transactionType - partition key of our GSI. We are always saving four types of values here. [Cancel, Refund, Shipment, MFNShipment]
2. timestamp- timestamp in epoch when item came into our system and was saved in dynamodb.

Now, the thing I am trying to achieve is, I am to calculate number of items present in DynamoDB table which lie between two timestamps (start and end timestamp).

For that, I came-up with the approach of using our GSI transactionType-timestamp-index and where for the list of values of transactionType and timestamp range, I will pass the key condition which will read all the records and to overcome returned response limiting issue and I will use lastEvaluatedKey in loop to get the other records till end.

Following is the code I am using:

private static int getNumberOfRecordsFromTable(final AmazonDynamoDB dynamoDBclient, final String tableName,
                                                   final String gsiIndex, final List<String> transactionTypes,
                                                   final long startTimeEpoch, final long endTimeEpoch) {
        int numberOfRecords=0;

        Map<String, AttributeValue> lastEvaluatedKey = null;
        Map<String, AttributeValue> valueMap = new HashMap<>();
        valueMap.put(":transaction_type", new AttributeValue().withSS(transactionTypes));
        valueMap.put(":start_time_epoch", new AttributeValue().withN(String.valueOf(startTimeEpoch)));
        valueMap.put(":end_time_epoch", new AttributeValue().withN(String.valueOf(endTimeEpoch)));

        Map<String, String> nameMap = new HashMap<>();
        nameMap.put("#timestamp","timestamp");
        nameMap.put("#transactionType","transactionType");

        final String conditionExpression = "(#transactionType = :transaction_type) " +
                "AND (#timestamp BETWEEN :start_time_epoch AND :end_time_epoch)";

        QueryRequest queryRequest = new QueryRequest()
                .withTableName(tableName)
                .withIndexName(gsiIndex)
                .withKeyConditionExpression(conditionExpression)
                .withExpressionAttributeNames(nameMap)
                .withExpressionAttributeValues(valueMap)
                .withProjectionExpression("#transactionType, #timestamp")
                .withExclusiveStartKey(lastEvaluatedKey)
                .withConsistentRead(false);

        do {
            int numberOfRecordsFetched=0;

            QueryResult queryResult = dynamoDBclient.query(queryRequest);
            lastEvaluatedKey = queryResult.getLastEvaluatedKey();
            numberOfRecordsFetched = queryResult.getScannedCount();
            queryRequest.setExclusiveStartKey(lastEvaluatedKey);

            numberOfRecords = numberOfRecords + numberOfRecordsFetched;
        } while (lastEvaluatedKey != null);

        log.info("Number of {} type messages fetched :: {}", transactionType, numberOfRecords);


    return numberOfRecords;
}

I am getting the following error:

Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more parameter values were invalid: Condition parameter type does not match schema type (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: FJLJTP7NFVKPTSDF2AJRUL0PTJVV4KQNSO5AEMVJF66Q9ASUAAJG)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:3443)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:3419)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeQuery(AmazonDynamoDBClient.java:2318)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:2293)
    at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.getNumberOfRecordsFromTable(GSTDatabaseRecordsCount.java:231)
    at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.countDynamoDBRecords(GSTDatabaseRecordsCount.java:192)
    at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.main(GSTDatabaseRecordsCount.java:123)

Could anyone help me in it?

Answer 1

The reason I was getting error - I was passing list of transactionType which ideally should been passed one by one into the query.

After, adding one more root for-loop to go over every transactionType I was able to fix it.

Please see the code change for reference:

private static int getNumberOfRecordsFromTable(final AmazonDynamoDB dynamoDBclient, final String tableName,
                                                   final String gsiIndex, final List<String> transactionTypes,
                                                   final long startTimeEpoch, final long endTimeEpoch) {
        int numberOfRecords=0;

        for (String transactionType: transactionTypes) {
            Map<String, AttributeValue> lastEvaluatedKey = null;
            Map<String, AttributeValue> valueMap = new HashMap<>();
            valueMap.put(":transaction_type", new AttributeValue().withS(transactionType));
            valueMap.put(":start_time_epoch", new AttributeValue().withN(String.valueOf(startTimeEpoch)));
            valueMap.put(":end_time_epoch", new AttributeValue().withN(String.valueOf(endTimeEpoch)));

            Map<String, String> nameMap = new HashMap<>();
            nameMap.put("#timestamp","timestamp");
            nameMap.put("#transactionType","transactionType");

            final String conditionExpression = "(#transactionType = :transaction_type) " +
                    "AND (#timestamp BETWEEN :start_time_epoch AND :end_time_epoch)";

            QueryRequest queryRequest = new QueryRequest()
                    .withTableName(tableName)
                    .withIndexName(gsiIndex)
                    .withKeyConditionExpression(conditionExpression)
                    .withExpressionAttributeNames(nameMap)
                    .withExpressionAttributeValues(valueMap)
                    .withProjectionExpression("#transactionType, #timestamp")
                    .withExclusiveStartKey(lastEvaluatedKey)
                    .withConsistentRead(false);

            do {
                int numberOfRecordsFetched=0;

                QueryResult queryResult = dynamoDBclient.query(queryRequest);
                lastEvaluatedKey = queryResult.getLastEvaluatedKey();
                numberOfRecordsFetched = queryResult.getScannedCount();
                queryRequest.setExclusiveStartKey(lastEvaluatedKey);

                numberOfRecords = numberOfRecords + numberOfRecordsFetched;
            } while (lastEvaluatedKey != null);

            log.info("Number of {} type messages fetched :: {}", transactionType, numberOfRecords);
        }

        return numberOfRecords;
    }

Issue while querying on DynamoDB table using GSI

1 个答案: