Bigtable bulk insert

Question

Can I perform bulk insert in Bigtable using python.

I am using below method to insert which is very slow. I have millions of rows to insert.

def row(self, row_key, filter_=None, append=False):
    """Factory to create a row associated with this table.

Ref Document: https://googlecloudplatform.github.io/google-cloud-python/latest/bigtable/table.html

Thanks.

Answer 1

我认为您正在寻找mutate_rows： https://googlecloudplatform.github.io/google-cloud-python/latest/bigtable/table.html#google.cloud.bigtable.table.Table.mutate_rows

Answer 2

如果您将happybase API与google-bigtable一起使用，则可以使用bigtable happybase批处理方法

from google.cloud import bigtable,happybase
client = bigtable.Client(project=my_project_id, admin=True)
bigtable_instance= client.instance(my_instance_id)
bt_connection = happybase.Connection(instance=bigtable_instance)
bt_table = bt_connection.table(my_bigtable_name)

bt_batch_actions = bt_table.batch()
for key,val in my_info.items():
    bt_batch_actions.put(key.encode('utf-8'):val.encode('utf-8'))
bt_batch_actions.send()

希望这会有所帮助！

Answer 3

Google云python库现在实际上仅支持一个网络请求的批处理。它称为mutations_batcher，可以在here中找到文档。