Question

我们最近已将一个大型应用程序和数据库从EngineYard移至Heroku。我们的新数据库的资源少于EngineYard上的资源，并且已经公开了一些可能由于铁耗大而以前才起作用的查询。

一个特别令人讨厌的问题是我们无法从大表中返回计数。该表是应用程序中最大的表，包含超过10亿行。一些客户将拥有成千上万的行，而另一些则可能拥有一亿行。

该表由subscription_id和status索引。

查询很简单：

select count(*)
from my_large_table
where subscription_id = 123
  and status = 'Valid'

不幸的是，此查询似乎对表执行了ExclusiveLock，导致请求排队。对于拥有500万行数据的客户，必须在搅动一个小时后终止查询。该查询实际上使我们的应用程序暂停。

但是，我们需要知道该表中每个客户和状态的计数。它不一定要坚持下去。但这不能像count_estimate过程那样完整，它实际上似乎只是报告虚构的数字。

我确信对此有解决方案。我该怎么做才能得到这个数字？有什么办法可以防止它锁定？

表很大，有很多属性。在我设计citext值时，我过分了，因为我来自MySQL，在那里我习惯了不区分大小写的搜索，并且将它们视为理所当然。我真的只需要在citext的大约4列顶部（attribute1，attribute2）。这些实际上是字段的名称，而不是混淆。该表是基于standard_id的值的不同类型数据的目标。

感谢您的帮助。

/*
 Navicat PostgreSQL Data Transfer

 Source Server         : Heroku myapp-production
 Source Server Version : 100600
 Source Host           : ec2-34-196-135-106.compute-1.amazonaws.com
 Source Database       : d6hrvd8r3u28t0
 Source Schema         : public

 Target Server Version : 100600
 File Encoding         : utf-8

 Date: 02/20/2019 09:37:49 AM
*/

-- ----------------------------
--  Table structure for apps
-- ----------------------------
DROP TABLE IF EXISTS "public"."apps";
CREATE TABLE "public"."apps" (
    "id" int8 NOT NULL DEFAULT nextval('apps_id_seq'::regclass),
    "attribute1" "public"."citext" COLLATE "default",
    "attribute2" "public"."citext" COLLATE "default",
    "attribute3" "public"."citext" COLLATE "default",
    "attribute4" "public"."citext" COLLATE "default",
    "attribute5" "public"."citext" COLLATE "default",
    "attribute6" "public"."citext" COLLATE "default",
    "attribute7" "public"."citext" COLLATE "default",
    "attribute8" "public"."citext" COLLATE "default",
    "attribute9" "public"."citext" COLLATE "default",
    "attribute10" "public"."citext" COLLATE "default",
    "attribute11" "public"."citext" COLLATE "default",
    "attribute12" "public"."citext" COLLATE "default",
    "attribute13" "public"."citext" COLLATE "default",
    "attribute14" "public"."citext" COLLATE "default",
    "attribute15" "public"."citext" COLLATE "default",
    "attribute16" "public"."citext" COLLATE "default",
    "attribute17" "public"."citext" COLLATE "default",
    "attribute18" "public"."citext" COLLATE "default",
    "attribute19" "public"."citext" COLLATE "default",
    "attribute20" "public"."citext" COLLATE "default",
    "attribute21" "public"."citext" COLLATE "default",
    "attribute22" "public"."citext" COLLATE "default",
    "attribute23" "public"."citext" COLLATE "default",
    "attribute24" "public"."citext" COLLATE "default",
    "attribute25" "public"."citext" COLLATE "default",
    "attribute26" "public"."citext" COLLATE "default",
    "attribute27" "public"."citext" COLLATE "default",
    "attribute28" "public"."citext" COLLATE "default",
    "attribute29" "public"."citext" COLLATE "default",
    "attribute30" "public"."citext" COLLATE "default",
    "attribute31" "public"."citext" COLLATE "default",
    "attribute32" "public"."citext" COLLATE "default",
    "attribute33" "public"."citext" COLLATE "default",
    "attribute34" "public"."citext" COLLATE "default",
    "attribute35" "public"."citext" COLLATE "default",
    "attribute36" "public"."citext" COLLATE "default",
    "attribute37" "public"."citext" COLLATE "default",
    "attribute38" "public"."citext" COLLATE "default",
    "attribute39" "public"."citext" COLLATE "default",
    "attribute40" "public"."citext" COLLATE "default",
    "attribute41" "public"."citext" COLLATE "default",
    "attribute42" "public"."citext" COLLATE "default",
    "attribute43" "public"."citext" COLLATE "default",
    "attribute44" "public"."citext" COLLATE "default",
    "attribute45" "public"."citext" COLLATE "default",
    "attribute46" "public"."citext" COLLATE "default",
    "attribute47" "public"."citext" COLLATE "default",
    "attribute48" "public"."citext" COLLATE "default",
    "attribute49" "public"."citext" COLLATE "default",
    "attribute50" "public"."citext" COLLATE "default",
    "created_at" timestamp(6) NOT NULL,
    "updated_at" timestamp(6) NOT NULL,
    "standard_id" int4 NOT NULL,
    "status" "public"."citext" COLLATE "default",
    "listing_id" int4 NOT NULL,
    "repository_id" int4 NOT NULL,
    "subscription_id" int4 NOT NULL,
    "attribute_info" "public"."hstore",
)
WITH (OIDS=FALSE);
ALTER TABLE "public"."apps" OWNER TO "ufn67drbuner1e";

-- ----------------------------
--  Primary key structure for table apps
-- ----------------------------
ALTER TABLE "public"."apps" ADD PRIMARY KEY ("id") NOT DEFERRABLE INITIALLY IMMEDIATE;

-- ----------------------------
--  Indexes structure for table apps
-- ----------------------------
CREATE INDEX  "app_listing_idx" ON "public"."apps" USING btree(listing_id "pg_catalog"."int4_ops" ASC NULLS LAST);
CREATE INDEX  "app_subscription_idx" ON "public"."apps" USING btree(subscription_id "pg_catalog"."int4_ops" ASC NULLS LAST);
CREATE UNIQUE INDEX  "apps_listing_idx" ON "public"."apps" USING btree(listing_id "pg_catalog"."int4_ops" ASC NULLS LAST, "id" "pg_catalog"."int8_ops" ASC NULLS LAST);
CREATE INDEX  "apps_repository_idx" ON "public"."apps" USING btree(repository_id "pg_catalog"."int4_ops" ASC NULLS LAST, subscription_id "pg_catalog"."int4_ops" ASC NULLS LAST);
CREATE INDEX  "listing_and_attr_idx" ON "public"."apps" USING btree(listing_id "pg_catalog"."int4_ops" ASC NULLS LAST, attribute1 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute2 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute3 COLLATE "default" "public"."citext_ops" ASC NULLS LAST);
CREATE INDEX  "listing_and_attr_idx" ON "public"."apps" USING btree(listing_id "pg_catalog"."int4_ops" ASC NULLS LAST, attribute1 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute2 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute3 COLLATE "default" "public"."citext_ops" ASC NULLS LAST);
CREATE INDEX  "listing_and_attr_idx" ON "public"."apps" USING btree(listing_id "pg_catalog"."int4_ops" ASC NULLS LAST, attribute1 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute2 COLLATE "default" "public"."citext_ops" ASC NULLS LAST, attribute3 COLLATE "default" "public"."citext_ops" ASC NULLS LAST);

Answer 1

关于锁定：它不是由您正在运行的SELECT查询引起的。

我唯一的解释是，它正在事务中运行，并且该事务执行了其他操作，导致EXCLUSIVE LOCK被采用。

唯一好的理论是REFRESH MATERIALIZED VIEW CONCURRENTLY比同一笔交易中的金额高。关系扩展锁（在事务处理期间不保留）或ALTER TYPE ... ADD VALUE（仅阻止其他此类语句）之类的其他东西看起来并不像是可疑的。

我不知道Heroku在他们的PostgreSQL中内置了什么功能，但是我敢打赌它不是EXCLUSIVE LOCK上的SELECT。

但是，即使没有这种虚假的锁，对表中的行数进行计数也是缓慢且占用资源的。

如果通常的估算值（pg_stat_get_live_tuples()和pg_class.reltuples不够好，则可以使用触发器：

CREATE TABLE row_counter (
   reloid oid PRIMARY KEY,
   count bigint NOT NULL
);

CREATE FUNCTION count_trig() RETURNS trigger
   LANGUAGE plpgsql AS
$$BEGIN
   IF TG_OP = 'INSERT' THEN
      UPDATE row_counter
      SET count = count + 1
      WHERE reloid = TG_RELID;

      RETURN NEW;
   ELSIF TG_OP = 'DELETE' THEN
      UPDATE row_counter
      SET count = count - 1
      WHERE reloid = TG_RELID;

      RETURN OLD;
   END IF;
END;$$;

CREATE TRIGGER count_trig AFTER INSERT OR DELETE ON my_large_table
   FOR EACH ROW EXECUTE PROCEDURE count_trig();

您只需要在某个时候初始化表即可。

TRUNCATE的语句级触发器留给读者练习。

如何强制Postgres返回合理的行数而不锁定整个表？

1 个答案: