SQL查询非常大的问题

时间：2012-09-26 17:59:09

标签： mysql sql refactoring trac

问题：

我正在创建一个Trac报告，显示我们图书馆每章开发周期中每个阶段的门票数量。 门票代表一件作品，通常是个人例程。

例如，即将发布的版本（里程碑）的票数是在 X 章节的同行评审阶段。

共有10个发展阶段和47个章节。

给定的MySQL查询适用于所有10个开发阶段，但只有一个章节并且长度为25行，因此所有章节的整个查询超过1200行。

Trac给出的错误是KeyError: 'numrows'，查询变得很大。

将查询直接输入MySQL时，给出的错误为Out of resources when opening file (Errcode: 24) (23)

问题：

  • 重构 - 这可以做得更好'sql gurus，有一些聪明的技巧/推进技巧吗？

  • 方法 - 我是否需要完全不同的方法？

  • 配置 - 可以将MySQL和/或Trac配置为接受非常大的查询

注意：

表中的数据很小，查询在表观大小限制下执行时间不长。

查询从Trac系统传递到MySQL，这对可以执行的操作设置了一些限制，例如，只能从trac发送一个查询来生成报告。

可以看到Trac报告的示例here.

查询中的%c%*只是我通过脚本生成查询时用于替换实际章节的唯一字符串。 

SELECT '%c%' as Chapter,
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status IN ('new','assigned') ) AS 'New',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='document_interface' ) AS 'Document\
 Interface',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='interface_development' ) AS 'Inter\
face Development',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='interface_check' ) AS 'Interface C\
heck',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='document_routine' ) AS 'Document R\
outine',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='full_development' ) AS 'Full Devel\
opment',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='peer_review_1' ) AS 'Peer Review O\
ne',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%'AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='peer_review_2' ) AS 'Peer Review Tw\
o',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='qa' ) AS 'QA',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%'AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='closed' ) AS 'Closed',
count(id) AS Total,
ticket.id AS _id
FROM engine.ticket
INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine'

5 个答案:

答案 0 :(得分：2)

不会完全改写它......但这是我的建议：

SELECT '%c%' as Chapter,
    SUM(CASE WHEN ticket.status IN ('new','assigned') THEN 1 ELSE 0 END) as `New`,
    ...
    SUM(CASE WHEN ticket.status='closed' THEN 1 ELSE 0 END) as 'Closed',
    count(id) AS Total,
    ticket.id AS _id
FROM engine.ticket
INNER JOIN engine.ticket_custom 
    ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' 
    AND ticket_custom.value LIKE '%c%' 
    AND type='New material' 
    AND milestone='1.1.12' 
    AND component NOT LIKE 'internal_engine'
GROUP BY ticket.id
;

答案 1 :(得分：2)

不是为每个计数创建一个子查询，而是使用case从已为查询获取的数据中计算：

select '%c%' as Chapter,
  sum(case when ticket.status IN ('new','assigned') then 1 else 0 end) as 'New',
  sum(case when ticket.status='document_interface' then 1 else 0 end) as 'DocumentInterface',
  sum(case when ticket.status='interface_development' then 1 else 0 end) as 'Interface Development',
  sum(case when ticket.status='interface_check' then 1 else 0 end) as 'Interface Check',
  sum(case when ticket.status='document_routine' then 1 else 0 end) as 'Document Routine',
  sum(case when ticket.status='full_development' then 1 else 0 end) as 'Full Development',
  sum(case when ticket.status='peer_review_1' then 1 else 0 end) as 'Peer Review One',
  sum(case when ticket.status='peer_review_2' then 1 else 0 end) as 'Peer Review Two',
  sum(case when ticket.status='qa' then 1 else 0 end) as 'QA',
  sum(case when ticket.status='closed' then 1 else 0 end) as 'Closed',
  count(id) as Total,
  ticket.id as _id
from
  engine.ticket
  inner join engine.ticket_custom on ticket.id = ticket_custom.ticket
where
  ticket_custom.name='chapter' and
  ticket_custom.value LIKE '%c%' and
  type='New material' and
  milestone='1.1.12' and
  component NOT LIKE 'internal_engine'

答案 2 :(得分：1)

当通过Trac生成这样的复杂报告时，最好不要使用报告。对于相对简单的查询，报告是可以的，但是当您组织许多不同的阶段和章节时，它们会变得难以处理。

相反，请尝试使用Wiki页面创建“报告”。这将使您可以更好地控制布局和演示，还可以避免完全编写SQL。这是一个虚假报告的示例维基代码，显示按里程碑分组的所有非关闭故障单，然后按状态分类：

= Custom Report =
My custom report, as a wiki page

== Tickets for Milestone A ==
[[TicketQuery(milestone=MilestoneA,status!=closed,group=status,format=table)]]

== Tickets for Milestone B ==
[[TicketQuery(milestone=MilestoneB,status!=closed,group=status,format=table)]]

...

我不知道如何定义您的阶段和章节，因此您需要调整查询参数以使用相应的故障单字段。您还可以使用format和其他选项来调整输出格式。

这只是一个简单的例子。 TicketQuery宏能够生成更复杂的报告。有关可能与您正在寻找的内容更接近（复杂性）的事物的示例，请查看Trac project's release notes。使用单个TicketQuery宏生成整个“详细更改列表”部分（点击“编辑”按钮查看他们是如何做到的）。

另一个例子

要重新创建链接到的图表，您可以执行以下操作：

||= **Id** =||= **Enhancements** =||= **Defects** =||= **Tasks** =||
||[milestone:v1.0 v1.0] || [[TicketQuery(milestone=v1.0,type=enhancement,format=count)]] || [[TicketQuery(milestone=v1.0,type=defect,format=count)]] || [[TicketQuery(milestone=v1.0,type=task,format=count)]] ||
... repeat for each milestone ...

基本上，在宏中使用format=count可以为Guffa的答案中的sum(case ...)语句之一返回数据库返回的内容。

使用宏而不是直接SQL执行此操作的最大好处是它与数据库无关。您不必担心数据库引擎（mysql，sqlite等）之间的差异，Trac数据库布局的变化，处理自定义字段的特殊代码等等。

答案 3 :(得分：0)

Yeek，那很难看。您是否必须在一行中获取所有数据？许多子查询将抨击服务器。你能不进行标准的分组获取并在客户端进行数据透视表填充吗？

e.g。

SELECT count(ticket.id) AS Matches, ticket_custom.name, ticket.status
FROM engine.ticket
INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='qa' AND (ticket.status IN (........))
GROUP BY ticket.id, ticket_custom.name

然后，在伪代码中：

data = array()
while(row = fetch($result)) {
   data[ticket.id]][ticket.status] = row[ticket.status];
}

通过这种方式，您只需运行一个查询，并在获取逻辑中执行一些额外的小工作，以重新创建原始查询正在进行的操作。

答案 4 :(得分：0)

我最终在php编写了我自己的trac报告系统。这样可以更灵活地创建我需要的报告，并且不会涉及荒谬mysql queries。要将脚本与trac项目集成，报告项即 QA 中 QA 阶段的门票数量是超链接到门票本身在trac：http://myhost.co.uk/trac-project/query?id=10&id=15 。此外，NavAddPluggin允许自定义主trac导航栏，以便添加链接到脚本的菜单选项。

相关问题
最新问题