Question

我的桌子大约是220mb，里面有25万条记录。我正在尝试将所有这些数据都放到python中。我意识到这需要是一个分块的批处理过程并循环，但我不知道如何设置批次从上一个停止的地方开始。

有没有办法过滤我的扫描？从我读到的，加载后发生过滤，加载停止在1mb，所以我实际上无法扫描新对象。

任何帮助都将不胜感激。

import boto3
dynamodb = boto3.resource('dynamodb',
    aws_session_token = aws_session_token,
    aws_access_key_id = aws_access_key_id,
    aws_secret_access_key = aws_secret_access_key,
    region_name = region
    )

table = dynamodb.Table('widgetsTableName')

data = table.scan()

Answer 1

我认为关于表扫描的Amazon DynamoDB documentation会回答您的问题。

简而言之，您需要在回复中检查LastEvaluatedKey。以下是使用代码的示例：

import boto3
dynamodb = boto3.resource('dynamodb',
                          aws_session_token=aws_session_token,
                          aws_access_key_id=aws_access_key_id,
                          aws_secret_access_key=aws_secret_access_key,
                          region_name=region
)

table = dynamodb.Table('widgetsTableName')

response = table.scan()
data = response['Items']

while 'LastEvaluatedKey' in response:
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
    data.extend(response['Items'])

Answer 2

boto3提供处理所有分页详细信息的分页器。 Here是扫描分页器的文档页面。基本上，你会像这样使用它：

import boto3

client = boto3.client('dynamodb')
paginator = client.get_paginator('scan')

for page in paginator.paginate():
    # do something

Answer 3

重复Jordon Phillips的回答，这里是你如何通过FilterExpression传递分页：

import boto3

client = boto3.client('dynamodb')
paginator = client.get_paginator('scan')
operation_parameters = {
  'TableName': 'foo',
  'FilterExpression': 'bar > :x AND bar < :y',
  'ExpressionAttributeValues': {
    ':x': {'S': '2017-01-31T01:35'},
    ':y': {'S': '2017-01-31T02:08'},
  }
}

page_iterator = paginator.paginate(**operation_parameters)
for page in page_iterator:
    # do something

Answer 4

删除dynamodb格式类型的代码，如@kungphu所述。

import boto3

from boto3.dynamodb.types import TypeDeserializer
from boto3.dynamodb.transform import TransformationInjector

client = boto3.client('dynamodb')
paginator = client.get_paginator('query')
service_model = client._service_model.operation_model('Query')
trans = TransformationInjector(deserializer = TypeDeserializer())
for page in paginator.paginate():
    trans.inject_attribute_value_output(page, service_model)

Answer 5

原来Boto3捕获“LastEvaluatedKey”作为返回响应的一部分。这可以用作扫描的起点：

data= table.scan(
   ExclusiveStartKey=data['LastEvaluatedKey']
)

我计划在此周围建立一个循环，直到返回的数据只是ExclusiveStartKey

Answer 6

上面建议的2种方法都存在问题：要么编写冗长且重复的代码以在循环中显式处理分页，要么使用具有低级会话的Boto分页器，并且超越了高级Boto对象的优点。

使用Python功能代码提供高级抽象的解决方案允许使用更高级别的Boto方法，同时隐藏AWS分页的复杂性：

import itertools
import typing

def iterate_result_pages(function_returning_response: typing.Callable, *args, **kwargs) -> typing.Generator:
    """A wrapper for functions using AWS paging, that returns a generator which yields a sequence of items for
    every response

    Args:
        function_returning_response: A function (or callable), that returns an AWS response with 'Items' and optionally 'LastEvaluatedKey'
        This could be a bound method of an object.

    Returns:
        A generator which yields the 'Items' field of the result for every response
    """
    response = function_returning_response(*args, **kwargs)
    yield response["Items"]
    while "LastEvaluatedKey" in response:
        kwargs["ExclusiveStartKey"] = response["LastEvaluatedKey"]
        response = function_returning_response(*args, **kwargs)
        yield response["Items"]

    return

def iterate_paged_results(function_returning_response: typing.Callable, *args, **kwargs) -> typing.Iterator:
    """A wrapper for functions using AWS paging, that returns an iterator of all the items in the responses.
    Items are yielded to the caller as soon as they are received.

    Args:
        function_returning_response: A function (or callable), that returns an AWS response with 'Items' and optionally 'LastEvaluatedKey'
        This could be a bound method of an object.

    Returns:
        An iterator which yields one response item at a time
    """
    return itertools.chain.from_iterable(iterate_result_pages(function_returning_response, *args, **kwargs))

# Example, assuming 'table' is a Boto DynamoDB table object:
all_items = list(iterate_paged_results(ProjectionExpression = 'my_field'))

Answer 7

我对文森特的答案有一些疑问，这涉及到将转换应用于LastEvaluatedKey并弄乱了分页。解决方法如下：

import boto3

from boto3.dynamodb.types import TypeDeserializer
from boto3.dynamodb.transform import TransformationInjector

client = boto3.client('dynamodb')
paginator = client.get_paginator('scan')
operation_model = client._service_model.operation_model('Scan')
trans = TransformationInjector(deserializer = TypeDeserializer())
operation_parameters = {
  'TableName': 'tablename',  
}
items = []

for page in paginator.paginate(**operation_parameters):
    has_last_key = 'LastEvaluatedKey' in page
    if has_last_key:
        last_key = page['LastEvaluatedKey'].copy()
    trans.inject_attribute_value_output(page, operation_model)
    if has_last_key:
        page['LastEvaluatedKey'] = last_key
    items.extend(page['Items'])

Answer 8

DynamoDB将scan方法限制为每次扫描1mb数据。

文档： https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.scan

下面是一个示例循环，该循环使用LastEvaluatedKey从DynamoDB表中获取所有数据：

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('your_table_name')

has_items = True
last_key = False
while has_items:

    if last_key:
        data = table.scan(ExclusiveStartKey=last_key)
    else:
        data = table.scan()

    if 'LastEvaluatedKey' in data:
        has_items = True
        last_key = data['LastEvaluatedKey']
    else:
        has_items = False
        last_key = False

    # TODO do something with data['Items'] here.

使用boto3

8 个答案: