如何为单个表获取BigQuery存储大小

时间:2015-07-07 11:00:27

标签: google-bigquery

我想计算Google Big Query Storage的表格成本,但我不知道如何分别查看每个表的存储大小。

3 个答案:

答案 0 :(得分:23)

或者从GUI中,您可以使用元数据内部表__TABLES__,例如,这将为您提供GB的大小:

select 
  sum(size_bytes)/pow(10,9) as size
from
  <your_dataset>.__TABLES__
where 
  table_id = '<your_table>'

答案 1 :(得分:7)

有两种方法可以做到这一点,但请注意,以字节为单位的表的大小属性对于主动接收流式插入的表不可用。

一个。使用BQ命令行工具和JQ linux库来解析JSON。

bq --format=json show publicdata:samples.gsod | jq '.numBytes | tonumber'

这个结果:

17290009238

B中。使用REST api进行Tables:get调用

GET https://www.googleapis.com/bigquery/v2/projects/projectId/datasets/datasetId/tables/tableId

这将返回一个完整的JSON,您可以解析并获取numBytes

{
    "kind": "bigquery#table",
    "description": "This dataset contains weather information collected by NOAA, such a…",
    "creationTime": "1335916040125",
    "tableReference": {
        "projectId": "publicdata",
        "tableId": "gsod",
        "datasetId": "samples"
    },
    "numRows": "114420316",
    "numBytes": "17290009238",
    "etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/MTQxMzkzNzk4Nzg0Ng\"",
    "location": "US",
    "lastModifiedTime": "1413937987846",
    "type": "TABLE",
    "id": "publicdata:samples.gsod",
    "selfLink": "https://www.googleapis.com/bigquery/v2/projects/publicdata/datasets…",
    "schema": {
        "fields": [
            {
                "description": "The World Meteorological Organization (WMO) / DATSAV3 station numbe…",
                "type": "INTEGER",
                "name": "station_number",
                "mode": "REQUIRED"
            },
            {
                "description": "The Weather-Bureau-Army-Navy (WBAN) station number where the data w…",
                "type": "INTEGER",
                "name": "wban_number",
                "mode": "NULLABLE"
            },
            {
                "description": "The year the data was collected in",
                "type": "INTEGER",
                "name": "year",
                "mode": "REQUIRED"
            },
            {
                "description": "The month the data was collected in",
                "type": "INTEGER",
                "name": "month",
                "mode": "REQUIRED"
            },
            {
                "description": "The day the data was collected in.",
                "type": "INTEGER",
                "name": "day",
                "mode": "REQUIRED"
            },
            {
                "description": "The mean temperature of the day in degrees Fahrenheit, accurate to …",
                "type": "FLOAT",
                "name": "mean_temp",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_temp.",
                "type": "INTEGER",
                "name": "num_mean_temp_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean dew point of the day in degrees Fahrenheit, accurate to on…",
                "type": "FLOAT",
                "name": "mean_dew_point",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_dew_point.",
                "type": "INTEGER",
                "name": "num_mean_dew_point_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean sea level pressure of the day in millibars, accurate to on…",
                "type": "FLOAT",
                "name": "mean_sealevel_pressure",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_sealevel_pressure…",
                "type": "INTEGER",
                "name": "num_mean_sealevel_pressure_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean station pressure of the day in millibars, accurate to one …",
                "type": "FLOAT",
                "name": "mean_station_pressure",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_station_pressure.…",
                "type": "INTEGER",
                "name": "num_mean_station_pressure_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean visibility of the day in miles, accurate to one tenth of a…",
                "type": "FLOAT",
                "name": "mean_visibility",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_visibility.",
                "type": "INTEGER",
                "name": "num_mean_visibility_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean wind speed of the day in knots, accurate to one tenth of a…",
                "type": "FLOAT",
                "name": "mean_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_wind_speed.",
                "type": "INTEGER",
                "name": "num_mean_wind_speed_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum sustained wind speed reported on the day in knots, accu…",
                "type": "FLOAT",
                "name": "max_sustained_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum wind gust speed reported on the day in knots, accurate …",
                "type": "FLOAT",
                "name": "max_gust_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum temperature of the day in degrees Fahrenheit, accurate …",
                "type": "FLOAT",
                "name": "max_temperature",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates the source of max_temperature.",
                "type": "BOOLEAN",
                "name": "max_temperature_explicit",
                "mode": "NULLABLE"
            },
            {
                "description": "The minimum temperature of the day in degrees Fahrenheit, accurate …",
                "type": "FLOAT",
                "name": "min_temperature",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates the source of min_temperature.",
                "type": "BOOLEAN",
                "name": "min_temperature_explicit",
                "mode": "NULLABLE"
            },
            {
                "description": "The total precipitation of the day in inches, accurate to one hundr…",
                "type": "FLOAT",
                "name": "total_precipitation",
                "mode": "NULLABLE"
            },
            {
                "description": "The snow depth of the day in inches, accurate to one tenth of an in…",
                "type": "FLOAT",
                "name": "snow_depth",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if fog was reported on this day.",
                "type": "BOOLEAN",
                "name": "fog",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if rain was reported on this day.",
                "type": "BOOLEAN",
                "name": "rain",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if snow was reported on this day.",
                "type": "BOOLEAN",
                "name": "snow",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if hail was reported on this day.",
                "type": "BOOLEAN",
                "name": "hail",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if thunder was reported on this day.",
                "type": "BOOLEAN",
                "name": "thunder",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if a tornado was reported on this day.",
                "type": "BOOLEAN",
                "name": "tornado",
                "mode": "NULLABLE"
            }
        ]
    }
}

℃。有一些称为__TABLES____TABLES_SUMMARY__

的元表

您可以运行如下查询:

SELECT size_bytes FROM <dataset>.__TABLES__ WHERE table_id='mytablename'

该查询的__TABLES__部分可能看起来不熟悉。 __TABLES_SUMMARY__是一个元表,包含有关数据集中表的信息。您可以自己使用此元表。例如,查询SELECT * FROM publicdata:samples.__TABLES_SUMMARY__将返回有关publicdata:samples数据集中表格的元数据。您也可以SELECT * FROM publicdata:samples.__TABLES__

可用字段:

__TABLES_SUMMARY__元表的字段(TABLE_QUERY查询中都可用)包括:

  • table_id:表格的名称。
  • creation_time:表格创建后的时间,自1970年1月1日UTC以来的毫秒数。这与表中的creation_time字段相同。
  • type:无论是视图(2)还是常规表(1)。

TABLE_QUERY()中的以下字段,因为它们是__TABLES__的成员,但不是__TABLES_SUMMARY__的成员。他们留在这里是为了历史,并部分记录__TABLES__ metatable:

  • last_modified_time:自1970年1月1日UTC以来的毫秒,表格已更新(元数据或表格内容)。请注意,如果您使用tabledata.insertAll()将记录流式传输到您的表格,则可能会过时几分钟。
  • row_count:表格中的行数。
  • size_bytes:表格的总大小(以字节为单位)。

答案 2 :(得分:5)

您可以使用命令行工具

执行此操作

bq show ds_name.table_name

它将在表格中显示一些信息,包括“总字节数”。在此参考https://cloud.google.com/bigquery/bq-command-line-tool