查询的结果集太大了

时间:2011-05-05 15:40:00

标签: mysql sql into-outfile

3 个答案:

答案 0 :(得分:1)

答案很大程度上取决于您对数据的处理方式。通过29个连接查询215列对于非平凡的记录大小永远不会很快。

如果您尝试向用户显示160,000条记录,则应该对结果进行分页,并且一次只能抓取一页。这将使结果集保持足够小,即使是相对低效的查询也会快速返回。在这种情况下,您还需要检查用户选择或操作数据所需的数据量。很有可能你可以将它削减到少数字段和一些聚合(计数,总和等),这将使用户能够明智地决定他们想要使用哪些记录。使用LIMIT和偏移来拉取任意大小的单页。

如果您需要导出数据以进行报告,请确保仅提取报告所需的确切数据。尽可能消除连接,并使用需要子数据聚合的子查询。您需要为常用的连接和条件调整/添加索引。对于您提供的查询,ib.id以及您正在加入的无数外键。您可以不使用布尔列,因为没有足够的不同值来形成有意义的索引。

无论您要完成什么,删除一些连接和列都会固有地加快处理速度。 MySQL填写查询所需要的繁重工作量是您的主要障碍。

答案 1 :(得分:1)

我重新构建了您的查询,希望能够提供显着的性能提升时间。通过使用STRAIGHT_JOIN告诉MySQL按照你所说的顺序(或者我在这里进行了调整)。最内层的第一个查询“PreQuery”根据您的导入包和通用导入的标准别名STARTS,将帐户导入帐户...通过在那里预先应用WHERE子句(并且如您所测试,添加您的限制条款在这里)你是预先加入这些表格,并在他们浪费任何时间试图获取客户,地址等其他信息之前将它们排除在外。在查询中,我调整了连接/左连接以更好地显示基础链接表的关系(主要是为其他人读入)。

正如另一个人所指出的那样,我在PREQUERY中所做的事情可能是一个主要的预查询列表中的“Account.ID”记录的基础,该列表用于通过和页面可用。我很好奇这对你现有的表现,特别是在10000限制范围内。

PREQUERY获取唯一元素(包括下游,银行,月份,年份和类别使用的帐户ID),因此这些表格不必在其余的加入过程中重新加入。

SELECT STRAIGHT_JOIN
      PreQuery.*,
      customer.customer_number,
      customer.name,
      customer.has_bad_address,
      address.line1,
      address.line2,
      address.city,
      state.name,
      address.zip,
      po_box.line1,
      po_box.line2,
      po_box.city,
      po_state.name,
      po_box.zip,
      customer.date_of_birth,
      northway_account.cffna,
      northway_account.cfinsc,
      customer.deceased,
      customer.social_security_number,
      customer.has_internet_banking,
      customer.safe_deposit_box,
      account.has_bill_pay,
      account.has_e_statement,
      branch.number,
      northway_product.code,
      macatawa_product.code,
      account.account_number,
      account.available_line,
      view_macatawa_atm_card.number,
      view_macatawa_debit_card.number,
      uc.code use_class,
      account.open_date,
      account.balance,
      account.affinion,
      northway_account.ytdsc,
      northway_account.ytdodf,
      northway_account.ytdnsf,
      northway_account.rtckcy,
      northway_account.rtckwy,
      northway_account.odwvey,
      northway_account.ytdscw,
      northway_account.feeytd,
      customer.do_not_mail,
      northway_account.aledq1,
      northway_account.aledq2,
      northway_account.aledq3,
      northway_account.aledq4,
      northway_account.acolq1,
      northway_account.acolq2,
      northway_account.acolq3,
      northway_account.acolq4,
      o.officer_number,
      northway_account.avg_bal_1,
      northway_account.avg_bal_2,
      northway_account.avg_bal_3,
      account.maturity_date,
      account.interest_rate,
      northway_account.asslc,
      northway_account.paidlc,
      northway_account.lnuchg,
      northway_account.ytdlc,
      northway_account.extfee,
      northway_account.penamt,
      northway_account.cdytdwaive,
      northway_account.cdterm,
      northway_account.cdtcod,
      account.date_of_last_statement,
      northway_account.statement_cycle,
      northway_account.cfna1,
      northway_account.cfna2,
      northway_account.cfna3,
      northway_account.cfna4,
      northway_account.cfcity,
      northway_account.cfstate,
      northway_account.cfzip,
      northway_account.actype,
      northway_account.sccode,
      macatawa_account.account_type_code,
      macatawa_account.account_type_code_description,
      macatawa_account.advance_code,
      macatawa_account.amount_last_advance,
      macatawa_account.amount_last_payment,
      macatawa_account.available_credit,
      macatawa_account.balance_last_statement,
      macatawa_account.billing_day,
      macatawa_account.birthday_3,
      macatawa_account.birthday_name_2,
      macatawa_account.ceiling_rate,
      macatawa_account.class_code,
      macatawa_account.classified_doubtful,
      macatawa_account.classified_loss,
      macatawa_account.classified_special,
      macatawa_account.classified_substandard,
      macatawa_account.closed_account_flag,
      macatawa_account.closing_balance,
      macatawa_account.compounding_code,
      macatawa_account.cost_center_full,
      macatawa_account.cytd_aggregate_balance,
      macatawa_account.cytd_amount_of_advances,
      macatawa_account.cytd_amount_of_payments,
      macatawa_account.cytd_average_balance,
      macatawa_account.cytd_average_principal_balance,
      macatawa_account.cytd_interest_paid,
      macatawa_account.cytd_number_items_nsf,
      macatawa_account.cytd_number_of_advanes,
      macatawa_account.cytd_number_of_payments,
      macatawa_account.cytd_number_times_od,
      macatawa_account.cytd_other_charges,
      macatawa_account.cytd_other_charges_waived,
      macatawa_account.cytd_reporting_points,
      macatawa_account.cytd_service_charge,
      macatawa_account.cytd_service_charge_waived,
      macatawa_account.date_closed,
      macatawa_account.date_last_activity,
      macatawa_account.date_last_advance,
      macatawa_account.date_last_payment,
      macatawa_account.date_paid_off,
      macatawa_account.ddl_code,
      macatawa_account.deposit_rate_index,
      macatawa_account.employee_officer_director_full_desc,
      macatawa_account.floor_rate,
      macatawa_account.handling_code,
      macatawa_account.how_paid_code,
      macatawa_account.interest_frequency,
      macatawa_account.ira_plan,
      macatawa_account.load_rate_code,
      macatawa_account.loan_rate_code,
      macatawa_account.loan_rating_code,
      macatawa_account.loan_rating_code_1_full_desc,
      macatawa_account.loan_rating_code_2_full_desc,
      macatawa_account.loan_rating_code_3_full_desc,
      macatawa_account.loan_to_value_ratio,
      macatawa_account.maximum_credit,
      macatawa_account.miscellaneous_code_full_desc,
      macatawa_account.months_to_maturity,
      macatawa_account.msa_code,
      macatawa_account.mtd_agg_available_balance,
      macatawa_account.naics_code,
      macatawa_account.name_2,
      macatawa_account.name_3,
      macatawa_account.name_line,
      macatawa_account.name_line_2,
      macatawa_account.name_line_3,
      macatawa_account.name_line_1,
      macatawa_account.net_payoff,
      macatawa_account.opened_by_responsibility_code_full,
      macatawa_account.original_issue_date,
      macatawa_account.original_maturity_date,
      macatawa_account.original_note_amount,
      macatawa_account.original_note_date,
      macatawa_account.original_prepaid_fees,
      macatawa_account.participation_placed_code,
      macatawa_account.participation_priority_code,
      macatawa_account.pay_to_account,
      macatawa_account.payment_code,
      macatawa_account.payoff_principal_balance,
      macatawa_account.percent_participated_code,
      macatawa_account.pmtd_number_deposit_type_1,
      macatawa_account.pmtd_number_deposit_type_2,
      macatawa_account.pmtd_number_deposit_type_3,
      macatawa_account.pmtd_number_type_1,
      macatawa_account.pmtd_number_type_2,
      macatawa_account.pmtd_number_type_6,
      macatawa_account.pmtd_number_type_8,
      macatawa_account.pmtd_number_type_9,
      macatawa_account.principal,
      macatawa_account.purpose_code,
      macatawa_account.purpose_code_full_desc,
      macatawa_account.pytd_number_of_items_nsf,
      macatawa_account.pytd_number_of_times_od,
      macatawa_account.rate_adjuster,
      macatawa_account.rate_over_split,
      macatawa_account.rate_under_split,
      macatawa_account.renewal_code,
      macatawa_account.renewal_date,
      macatawa_account.responsibility_code_full,
      macatawa_account.secured_unsecured_code,
      macatawa_account.short_first_name_1,
      macatawa_account.short_first_name_2,
      macatawa_account.short_first_name_3,
      macatawa_account.short_last_name_1,
      macatawa_account.short_last_name_2,
      macatawa_account.short_last_name_3,
      macatawa_account.statement_cycle,
      macatawa_account.statement_rate,
      macatawa_account.status_code,
      macatawa_account.tax_id_number_name_2,
      macatawa_account.tax_id_number_name_3,
      macatawa_account.teller_alert_1,
      macatawa_account.teller_alert_2,
      macatawa_account.teller_alert_3,
      macatawa_account.term,
      macatawa_account.term_code,
      macatawa_account.times_past_due_01_29,
      macatawa_account.times_past_due_01_to_29_days,
      macatawa_account.times_past_due_30_59,
      macatawa_account.times_past_due_30_to_59_days,
      macatawa_account.times_past_due_60_89,
      macatawa_account.times_past_due_60_to_89_days,
      macatawa_account.times_past_due_over_90,
      macatawa_account.times_past_due_over_90_days,
      macatawa_account.tin_code_name_1,
      macatawa_account.tin_code_name,
      macatawa_account.tin_code_name_2,
      macatawa_account.tin_code_name_3,
      macatawa_account.total_amount_past_due,
      macatawa_account.waiver_od_charge,
      macatawa_account.waiver_od_charge_description,
      macatawa_account.waiver_service_charge_code,
      macatawa_account.waiver_transfer_advance_fee,
      macatawa_account.short_first_name,
      macatawa_account.short_last_name          
FROM
   ( SELECT STRAIGHT_JOIN DISTINCT
         b.name bank,
         ib.YEAR,
         ib.MONTH,
         ip.category,
         Account.ID
         FROM import_bundle ib 
            JOIN generic_import gi ON ib.id = gi.import_bundle_id
               JOIN account_import AI ON gi.id = ai.generic_import_id
                  JOIN Account ON AI.ID = account.account_import_id 
               JOIN import_profile ip ON gi.import_profile_id = ip.id
            JOIN bank b ib.Bank_ID = b.id
      WHERE
            IB.ID = 95
         AND IB.Active = 1
         AND GI.Active = 1
      LIMIT 1000 ) PreQuery
   JOIN Account on PreQuery.ID = Account.ID
      JOIN Customer on Account.Customer_ID = Customer.ID
      JOIN Officer on Account.Officer_ID = Officer.ID
      LEFT JOIN branch ON Account.branch_id = branch.id
      LEFT JOIN cd_type ON account.cd_type_id = cd_type.id
      LEFT JOIN use_class uc ON account.use_class_id = uc.id
      LEFT JOIN account_type at ON account.account_type_id = at.id
      LEFT JOIN northway_account ON account.id = northway_account.account_id
      LEFT JOIN macatawa_account ON account.id = macatawa_account.account_id
      LEFT JOIN view_macatawa_debit_card ON account.id = view_macatawa_debit_card.account_id
      LEFT JOIN view_macatawa_atm_card ON account.id = view_macatawa_atm_card.account_id
      LEFT JOIN original_address OA ON Account.ID = OA.account_id

      JOIN Account_Address AA ON Account.ID = AA.account_id
         JOIN address ON AA.address_id = address.id
            JOIN state ON address.state_id = state.id

      LEFT JOIN Account_po_box APB ON Account.ID = APB.account_id
         LEFT JOIN address po_box ON APB.address_id = po_box.id
            LEFT JOIN state po_state ON po_box.state_id = po_state.id

      LEFT JOIN Account_macatawa_product amp ON account.id = amp.account_id
         LEFT JOIN macatawa_product ON amp.macatawa_product_id = macatawa_product.id
            LEFT JOIN product_type pt ON macatawa_product.product_type_id = pt.id
            LEFT JOIN harte_hanks_service_category hhsc ON macatawa_product.harte_hanks_service_category_id = hhsc.id
            LEFT JOIN core_file_type cft ON macatawa_product.core_file_type_id = cft.id 

      LEFT JOIN Account_northway_product anp ON account.id = anp.account_id
         LEFT JOIN northway_product ON anp.northway_product_id = northway_product.id

答案 2 :(得分:0)