Question

鉴于某些情况（存储在表中的大数据集），我需要检查Postgres表的字段的唯一性。

出于简化目的，假设我有下表：

id   |    name
--------------
1    |  david
2    |  catrine
3    |  hmida

我想检查字段名称的唯一性;结果是真的到目前为止，我设法使用类似的代码：

select name, count(*)
from test
group by name
having count(*) > 1

请记住，我有一个大数据集，因此我更喜欢由RDBMS处理，而不是通过适配器（例如psycopg2）获取数据。所以我需要尽可能地优化。任何讨厌的想法？

Answer 1

这可能会更快，但不太可靠的解决方案：

t=# create table t (i int);
CREATE TABLE
t=# insert into t select generate_series(1,9,1);
INSERT 0 9
t=# insert into t select generate_series(1,999999,1);
INSERT 0 999999
t=# insert into t select generate_series(1,9999999,1);
INSERT 0 9999999

现在您的查询：

t=# select i,count(*) from t group by i having count(*) > 1 order by 2 desc,1 limit 1;
 i | count
---+-------
 1 |     3
(1 row)

Time: 7538.476 ms

现在从统计数据中查看：

t=# analyze t;
ANALYZE
Time: 1079.465 ms
    t=# with fr as (select most_common_vals::text::text[] from pg_stats where tablename = 't' and attname='i')
    select count(1),i from t join fr on true where i::text = any(most_common_vals) group by i;
     count |   i
    -------+--------
         2 |  94933
         2 | 196651
         2 | 242894
         2 | 313829
         2 | 501027
         2 | 757714
         2 | 778442
         2 | 896602
         2 | 929918
         2 | 979650
         2 | 999259
    (11 rows)

    Time: 3584.582 ms

最后只检查uniq是否只存在一个最常见的值：

t=# select count(1),i from t where i::text = (select (most_common_vals::text::text[])[1] from pg_stats where tablename = 't' and attname='i') group by i;
 count |  i
-------+------
     2 | 1540
(1 row)

Time: 1871.907 ms

<强>更新

在表上收集统计信息后，将修改

pg_stats数据。因此，您有可能在数据分发时没有新的汇总统计数据。以我的样本为例：

t=# delete from t where i = 1540;
DELETE 2
Time: 941.684 ms
t=# select count(1),i from t where i::text = (select (most_common_vals::text::text[])[1] from pg_stats where tablename = 't' and attname='i') group by i;
 count | i
-------+---
(0 rows)

Time: 1876.136 ms
t=# analyze t;
ANALYZE
Time: 77.108 ms
t=# select count(1),i from t where i::text = (select (most_common_vals::text::text[])[1] from pg_stats where tablename = 't' and attname='i') group by i;
 count |   i
-------+-------
     2 | 41377
(1 row)

Time: 1878.260 ms

当然，如果你依赖的只是一个最常见的值，那么失败几率就会降低，但是这种方法又取决于统计数据“新鲜度”。

检查表字段postgres的unicity

1 个答案: