Question

我正在尝试优化用于从具有大型数据集的数据库中获取分页数据的查询。

我的架构如下所示：

CREATE TABLE users (
    user_id TEXT PRIMARY KEY,
    name TEXT,
    custom_fields TEXT
);

CREATE TABLE events (
    event_id TEXT PRIMARY KEY,
    organizer_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE SET NULL ON UPDATE CASCADE,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    start_time INTEGER,
    duration INTEGER
    -- more columns here, omitted for the sake of simplicity
);

CREATE INDEX events_organizer_id_start_time_idx ON events(organizer_id, start_time);
CREATE INDEX events_organizer_id_type_idx ON events(organizer_id, type);
CREATE INDEX events_organizer_id_type_start_time_idx ON events(organizer_id, type, start_time);
CREATE INDEX events_type_start_time_idx ON events(type, start_time);
CREATE INDEX events_start_time_desc_idx ON events(start_time DESC);
CREATE INDEX events_start_time_asc_idx ON events(IFNULL(start_time, 253402300800) ASC);

CREATE TABLE event_participants (
    participant_id TEXT NOT NULL REFERENCES users(user_id) ON DELETE CASCADE ON UPDATE CASCADE,
    event_id TEXT NOT NULL REFERENCES events(event_id) ON DELETE CASCADE ON UPDATE CASCADE,
    role INTEGER NOT NULL DEFAULT 0,
    UNIQUE (participant_id, event_id) ON CONFLICT REPLACE
);

CREATE INDEX event_participants_participant_id_event_id_idx ON event_participants(participant_id, event_id);
CREATE INDEX event_participants_event_id_idx ON event_participants(event_id);

CREATE TABLE event_tag_maps (
    event_id TEXT NOT NULL REFERENCES events(event_id) ON DELETE CASCADE ON UPDATE CASCADE,
    tag_id TEXT NOT NULL,
    PRIMARY KEY (event_id, tag_id) ON CONFLICT IGNORE
);

CREATE INDEX event_tag_maps_event_id_tag_id_idx ON event_tag_maps(event_id, tag_id);

活动表中的 1,500,000 条目， event_participants 中的 2,000,000 。

现在，典型的查询看起来像：

SELECT
    EVTS.event_id,
    EVTS.type,
    EVTS.name,
    EVTS.time,
    EVTS.duration
 FROM events AS EVTS
 WHERE
 EVTS.organizer_id IN(
     'f39c3bb1-3ee3-11e6-a0dc-005056c00008',
     '4555e70f-3f1d-11e6-a0dc-005056c00008',
     '6e7e33ae-3f1c-11e6-a0dc-005056c00008',
     '4850a6a0-3ee4-11e6-a0dc-005056c00008',
     'e06f784c-3eea-11e6-a0dc-005056c00008',
     'bc6a0f73-3f1d-11e6-a0dc-005056c00008',
     '68959fb5-3ef3-11e6-a0dc-005056c00008',
     'c4c96cf2-3f1a-11e6-a0dc-005056c00008',
     '727e49d1-3f1b-11e6-a0dc-005056c00008',
     '930bcfb6-3f09-11e6-a0dc-005056c00008')

    AND EVTS.type IN('Meeting', 'Conversation')
    AND(
      EXISTS (
          SELECT  1 FROM event_tag_maps AS ETM WHERE ETM.event_id = EVTS.event_id AND
             ETM.tag_id IN ('00000000-0000-0000-0000-000000000000', '6ae6870f-1aac-11e6-aeb9-005056c00008', '6ae6870c-1aac-11e6-aeb9-005056c00008', '1f6d3ccb-eaed-4068-a46b-ec2547fec1ff'))
      OR NOT EXISTS (
          SELECT  1 FROM event_tag_maps AS ETM WHERE ETM.event_id = EVTS.event_id)
    )
    AND EXISTS (
        SELECT 1 FROM event_participants AS EPRTS
            WHERE
                EVTS.event_id = EPRTS.event_id
                AND participant_id NOT IN('79869516-3ef2-11e6-a0dc-005056c00008', '79869515-3ef2-11e6-a0dc-005056c00008', '79869516-4e18-11e6-a0dc-005056c00008')
    )
 ORDER BY IFNULL(EVTS.start_time, 253402300800) ASC
 LIMIT 100 OFFSET @Offset;

此外，为了获取查询匹配项的总计数，我将使用上面的查询与count（1）而不是列，没有 ORDER BY 和 LIMIT / OFFSET 条款。

我在这里遇到两个主要问题：

1）随着我增加 @Offset 值，性能急剧下降。差异非常显着 - 从几乎立即到几秒钟。

2）计数查询需要很长时间（秒数）并产生以下执行计划：

0|0|0|SCAN TABLE events AS EVTS
0|0|0|EXECUTE LIST SUBQUERY 1
0|0|0|EXECUTE LIST SUBQUERY 1
0|0|0|EXECUTE CORRELATED SCALAR SUBQUERY 1
1|0|0|SEARCH TABLE event_tag_maps AS ETM USING COVERING INDEX event_tag_maps_event_id_tag_id_idx (event_id=? AND tag_id=?)
1|0|0|EXECUTE LIST SUBQUERY 2
0|0|0|EXECUTE CORRELATED SCALAR SUBQUERY 2
2|0|0|SEARCH TABLE event_tag_maps AS ETM USING COVERING INDEX event_tag_maps_event_id_tag_id_idx (event_id=?)
0|0|0|EXECUTE CORRELATED SCALAR SUBQUERY 3
3|0|0|SEARCH TABLE event_participants AS EPRTS USING INDEX event_participants_event_id_idx (event_id=?)

我不明白为什么要执行完整扫描而不是索引扫描。

使用了其他信息和SQLite设置：

我使用 System.Data.SQLite 提供程序（必须由于自定义函数支持）
页面大小=群集大小（在我的情况下为4096）
缓存大小= 100000
日记帐模式= WAL
临时存储= 2（记忆）
查询未打开任何交易

我是否可以采取任何措施来更改查询/架构或设置，以便尽可能提高性能？

针对分页提取的SQLite性能调优

0 个答案: