我正在使用数据表来显示来自多个mySQL表的数据(实际上是7个)。目前确实没有太多数据,但我看到“显示7个条目中的1到7个(从642,660,480个条目中过滤掉)。”只显示7个条目需要20秒。一旦我真正开始向数据库添加大量内容,我确信这将无法使用。
我确信有更好的方法来完成我想要做的事情,但这是我能够让它发挥作用的唯一方法。
这是我的服务器端脚本:
$q1 = "'";
$q2 = '"';
$order_id = "CONCAT( ".$q2."<input type='hidden' id='order_id' value='".$q2.", o.id, ".$q2."'><a href='order_details.php?id=".$q2.", o.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", o.id )";
$patient_name = "CONCAT( ".$q2."<input type='hidden' id='patient_name' value='".$q2.", p.first_name, ' ', p.last_name, ".$q2."'><input type='hidden' id='patient_id' value='".$q2.", p.id, ".$q2."'><input type='hidden' id='patient_ssn' value='".$q2.", p.ssn, ".$q2."'><a href='patient_details.php?id=".$q2.", p.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", p.first_name, ' ', p.last_name )";
$doc_name = "CONCAT( ".$q2."<input type='hidden' id='doctor_name' value='".$q2.", d.first_name, ' ', d.last_name, ".$q2."'><input type='hidden' id='doctor_id' value='".$q2.", d.id, ".$q2."'><a href='doctor_details.php?id=".$q2.", d.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", d.first_name, ' ', d.last_name )";
$order_date = "FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y')";
$tests = "GROUP_CONCAT(t.name SEPARATOR ', ')";
$aColumns = array($order_id, $order_date, $doc_name, $patient_name, $tests, 'o.status');
/* Indexed column (used for fast and accurate table cardinality) */
$sIndexColumn = "o.id";
/* DB table to use */
$sTable = "`orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t, `users` u, `events` e";
$sWhere = "WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id AND u.username = o.assigned_username AND e.event_id = o.event_id";
$order_status = isset($_GET['status']) ? $_GET['status'] : 'all';
if($order_status == 'all'){
}else{
$sWhere .= " AND (o.status='Complete' OR o.status='$order_status')";
}
$sGroupBy = "GROUP BY o.id";
/* Database connection information */
$gaSql['user'] = DB_USER;
$gaSql['password'] = DB_PASSWORD;
$gaSql['db'] = DB_NAME;
$gaSql['server'] = DB_SERVER;
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* If you just want to use the basic configuration for DataTables with PHP server-side, there is
* no need to edit below this line
*/
/*
* MySQL connection
*/
$gaSql['link'] = mysql_pconnect( $gaSql['server'], $gaSql['user'], $gaSql['password'] ) or
die( 'Could not open connection to server' );
mysql_select_db( $gaSql['db'], $gaSql['link'] ) or
die( 'Could not select database '. $gaSql['db'] );
/*
* Paging
*/
$sLimit = "";
if ( isset( $_GET['iDisplayStart'] ) && $_GET['iDisplayLength'] != '-1' )
{
$sLimit = "LIMIT ".mysql_real_escape_string( $_GET['iDisplayStart'] ).", ".
mysql_real_escape_string( $_GET['iDisplayLength'] );
}
/*
* Ordering
*/
if ( isset( $_GET['iSortCol_0'] ) )
{
$sOrder = "ORDER BY ";
for ( $i=0 ; $i<intval( $_GET['iSortingCols'] ) ; $i++ )
{
if ( $_GET[ 'bSortable_'.intval($_GET['iSortCol_'.$i]) ] == "true" )
{
$sOrder .= $aColumns[ intval( $_GET['iSortCol_'.$i] ) ]."
".mysql_real_escape_string( $_GET['sSortDir_'.$i] ) .", ";
}
}
$sOrder = substr_replace( $sOrder, "", -2 );
if ( $sOrder == "ORDER BY" )
{
$sOrder = "";
}
}
/*
* Filtering
* NOTE this does not match the built-in DataTables filtering which does it
* word by word on any field. It's possible to do here, but concerned about efficiency
* on very large tables, and MySQL's regex functionality is very limited
*/
if ( $_GET['sSearch'] != "" )
{
$sWhere .= " AND (";
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if($i!=4){ //skip tests column
$sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string( $_GET['sSearch'] )."%' OR ";
}
}
$sWhere = substr_replace( $sWhere, "", -3 );
$sWhere .= ')';
}
/* Individual column filtering */
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if($i!=4){ //skip tests column
if ( $_GET['bSearchable_'.$i] == "true" && $_GET['sSearch_'.$i] != '' )
{
if ( $sWhere == "" )
{
$sWhere = "WHERE ";
}
else
{
$sWhere .= " AND ";
}
$sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string($_GET['sSearch_'.$i])."%' ";
}
}
}
/*
* SQL queries
* Get data to display
*/
$sQuery = "
SELECT SQL_CALC_FOUND_ROWS ".str_replace(" , ", " ", implode(", ", $aColumns))."
FROM $sTable
$sWhere
$sGroupBy
$sOrder
$sLimit
";
//echo $sQuery;
//die();
$rResult = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
/* Data set length after filtering */
$sQuery = "
SELECT FOUND_ROWS()
";
$rResultFilterTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultFilterTotal = mysql_fetch_array($rResultFilterTotal);
$iFilteredTotal = $aResultFilterTotal[0];
/* Total data set length */
$sQuery = "
SELECT COUNT(".$sIndexColumn.")
FROM $sTable
";
$rResultTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultTotal = mysql_fetch_array($rResultTotal);
$iTotal = $aResultTotal[0];
//added to hide filtering
//$iTotal = $iFilteredTotal;
/*
* Output
*/
$output = array(
"sEcho" => intval($_GET['sEcho']),
"iTotalRecords" => $iTotal,
"iTotalDisplayRecords" => $iFilteredTotal,
"aaData" => array()
);
while ( $aRow = mysql_fetch_array( $rResult ) )
{
$row = array();
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if ( $aColumns[$i] != ' ' )
{
/* General output */
$row[] = $aRow[$i];
}
}
$output['aaData'][] = $row;
}
echo json_encode( $output );
以下是服务器端脚本生成的查询:
SELECT SQL_CALC_FOUND_ROWS
CONCAT( "<input type='hidden' id='order_id' value='", o.id, "'><a href='order_details.php?id=", o.id, "'><img src='search.png' border='0'></a> ", o.id ),
FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y'),
CONCAT( "<input type='hidden' id='doctor_name' value='", d.first_name, ' ', d.last_name, "'><input type='hidden' id='doctor_id' value='", d.id, "'><a href='doctor_details.php?id=", d.id, "'><img src='search.png' border='0'></a> ", d.first_name, ' ', d.last_name ),
CONCAT( "<input type='hidden' id='patient_name' value='", p.first_name, ' ', p.last_name, "'><input type='hidden' id='patient_id' value='", p.id, "'><input type='hidden' id='patient_ssn' value='", p.ssn, "'><a href='patient_details.php?id=", p.id, "'><img src='search.png' border='0'></a> ", p.first_name, ' ', p.last_name ), GROUP_CONCAT(t.name SEPARATOR ', '),
o.status
FROM `orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t, `users` u, `events` e
WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id AND u.username = o.assigned_username AND e.event_id = o.event_id AND (o.status='Complete' OR o.status='Draft')
GROUP BY o.id
我正在尝试确定如何在不破坏数据表的搜索和排序功能的情况下优化此操作。我已经为所有表创建了索引并设置了主键,因为我知道如何。有没有办法使用JOIN?
以下是EXPLAIN语句的输出:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE u index NULL PRIMARY 32 NULL 5 Using index; Using temporary; Using filesort
1 SIMPLE o ALL PRIMARY,patient_id,doctor_id,event_id,assigned_use... NULL NULL NULL 6 Using where
1 SIMPLE d eq_ref PRIMARY PRIMARY 4 pasdbadmin.o.doctor_id 1
1 SIMPLE e eq_ref PRIMARY PRIMARY 4 pasdbadmin.o.event_id 1 Using index
1 SIMPLE tst ref order_id,test_id order_id 4 pasdbadmin.o.id 1
1 SIMPLE t eq_ref PRIMARY PRIMARY 4 pasdbadmin.tst.test_id 1
1 SIMPLE p eq_ref PRIMARY PRIMARY 4 pasdbadmin.o.patient_id 1
更新
问题似乎是使用包括查询中的用户和事件表(两者都没有实际使用)的问题。以下是执行速度更快的修订代码:
$q1 = "'";
$q2 = '"';
$order_id = "CONCAT( ".$q2."<input type='hidden' id='order_id' value='".$q2.", o.id, ".$q2."'><a href='order_details.php?id=".$q2.", o.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", o.id )";
$patient_name = "CONCAT( ".$q2."<input type='hidden' id='patient_name' value='".$q2.", p.first_name, ' ', p.last_name, ".$q2."'><input type='hidden' id='patient_id' value='".$q2.", p.id, ".$q2."'><input type='hidden' id='patient_ssn' value='".$q2.", p.ssn, ".$q2."'><a href='patient_details.php?id=".$q2.", p.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", p.first_name, ' ', p.last_name )";
$doc_name = "CONCAT( ".$q2."<input type='hidden' id='doctor_name' value='".$q2.", d.first_name, ' ', d.last_name, ".$q2."'><input type='hidden' id='doctor_id' value='".$q2.", d.id, ".$q2."'><a href='doctor_details.php?id=".$q2.", d.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> ".$q2.", d.first_name, ' ', d.last_name )";
$order_date = "FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y')";
$tests = "GROUP_CONCAT(t.name SEPARATOR ', ')";
$aColumns = array($order_id, $order_date, $doc_name, $patient_name, $tests, 'o.status');
/* Indexed column (used for fast and accurate table cardinality) */
$sIndexColumn = "o.id";
/* DB table to use */
$sTable = "`orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t";
$sWhere = "WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id";
$order_status = isset($_GET['status']) ? $_GET['status'] : 'all';
if($order_status == 'all'){
}else{
$sWhere .= " AND (o.status='Complete' OR o.status='$order_status')";
}
$sJoin = "";
$sGroupBy = "GROUP BY o.id";
/* Database connection information */
$gaSql['user'] = DB_USER;
$gaSql['password'] = DB_PASSWORD;
$gaSql['db'] = DB_NAME;
$gaSql['server'] = DB_SERVER;
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* If you just want to use the basic configuration for DataTables with PHP server-side, there is
* no need to edit below this line
*/
/*
* MySQL connection
*/
$gaSql['link'] = mysql_pconnect( $gaSql['server'], $gaSql['user'], $gaSql['password'] ) or
die( 'Could not open connection to server' );
mysql_select_db( $gaSql['db'], $gaSql['link'] ) or
die( 'Could not select database '. $gaSql['db'] );
/*
* Paging
*/
$sLimit = "";
if ( isset( $_GET['iDisplayStart'] ) && $_GET['iDisplayLength'] != '-1' )
{
$sLimit = "LIMIT ".mysql_real_escape_string( $_GET['iDisplayStart'] ).", ".
mysql_real_escape_string( $_GET['iDisplayLength'] );
}
/*
* Ordering
*/
if ( isset( $_GET['iSortCol_0'] ) )
{
$sOrder = "ORDER BY ";
for ( $i=0 ; $i<intval( $_GET['iSortingCols'] ) ; $i++ )
{
if ( $_GET[ 'bSortable_'.intval($_GET['iSortCol_'.$i]) ] == "true" )
{
$sOrder .= $aColumns[ intval( $_GET['iSortCol_'.$i] ) ]."
".mysql_real_escape_string( $_GET['sSortDir_'.$i] ) .", ";
}
}
$sOrder = substr_replace( $sOrder, "", -2 );
if ( $sOrder == "ORDER BY" )
{
$sOrder = "";
}
}
/*
* Filtering
* NOTE this does not match the built-in DataTables filtering which does it
* word by word on any field. It's possible to do here, but concerned about efficiency
* on very large tables, and MySQL's regex functionality is very limited
*/
if ( $_GET['sSearch'] != "" )
{
$sWhere .= " AND (";
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if($i!=4){ //skip tests column
$sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string( $_GET['sSearch'] )."%' OR ";
}
}
$sWhere = substr_replace( $sWhere, "", -3 );
$sWhere .= ')';
}
/* Individual column filtering */
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if($i!=4){ //skip tests column
if ( $_GET['bSearchable_'.$i] == "true" && $_GET['sSearch_'.$i] != '' )
{
if ( $sWhere == "" )
{
$sWhere = "WHERE ";
}
else
{
$sWhere .= " AND ";
}
$sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string($_GET['sSearch_'.$i])."%' ";
}
}
}
/*
* SQL queries
* Get data to display
*/
$sQuery = "
SELECT SQL_CALC_FOUND_ROWS ".str_replace(" , ", " ", implode(", ", $aColumns))."
FROM $sTable
$sWhere
$sJoin
$sGroupBy
$sOrder
$sLimit
";
$filename = __DIR__.DIRECTORY_SEPARATOR."sql_log.txt";
file_put_contents($filename, $sQuery, FILE_APPEND);
$rResult = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
/* Data set length after filtering */
$sQuery = "
SELECT FOUND_ROWS()
";
$rResultFilterTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultFilterTotal = mysql_fetch_array($rResultFilterTotal);
$iFilteredTotal = $aResultFilterTotal[0];
/* Total data set length */
$sQuery = "
SELECT COUNT(".$sIndexColumn.")
FROM $sTable
";
$rResultTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultTotal = mysql_fetch_array($rResultTotal);
$iTotal = $aResultTotal[0];
//added to hide filtering
//$iTotal = $iFilteredTotal;
/*
* Output
*/
$output = array(
"sEcho" => intval($_GET['sEcho']),
"iTotalRecords" => $iTotal,
"iTotalDisplayRecords" => $iFilteredTotal,
"aaData" => array()
);
while ( $aRow = mysql_fetch_array( $rResult ) )
{
$row = array();
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if ( $aColumns[$i] != ' ' )
{
/* General output */
$row[] = $aRow[$i];
}
}
$output['aaData'][] = $row;
}
echo json_encode( $output );
如果我使用JOINS,更新的查询将是:
SELECT SQL_CALC_FOUND_ROWS
CONCAT( "<input type='hidden' id='order_id' value='", o.id, "'><a href='order_details.php?id=", o.id, "'><img src='search.png' border='0'></a> ", o.id ),
FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y'),
CONCAT( "<input type='hidden' id='doctor_name' value='", d.first_name, ' ', d.last_name, "'><input type='hidden' id='doctor_id' value='", d.id, "'><a href='doctor_details.php?id=", d.id, "'><img src='search.png' border='0'></a> ", d.first_name, ' ', d.last_name ),
CONCAT( "<input type='hidden' id='patient_name' value='", p.first_name, ' ', p.last_name, "'><input type='hidden' id='patient_id' value='", p.id, "'><input type='hidden' id='patient_ssn' value='", p.ssn, "'><a href='patient_details.php?id=", p.id, "'><img src='search.png' border='0'></a> ", p.first_name, ' ', p.last_name ),
GROUP_CONCAT(t.name SEPARATOR ', '),
o.status
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id
WHERE o.status='Complete' OR o.status='Draft'
GROUP BY o.id
问题在于,由于列数组等原因,使用排序和过滤时,DataTables的设计并不适用于JOINS。我很乐意看到一个可以解决这个问题的解决方案
答案 0 :(得分:5)
首先,如果要优化SQL语句,首先要去除那里的HTML垃圾。如果不出意外,它会混淆Statement的实际结构。如果你必须,你可以在优化结束时把它放回去,虽然我会认真投票反对它:你有PHP做标记。为了清楚起见,我已经养成了使用JOIN条款的习惯,我相应地改写了整个Thing。
这个过程给了我的是:
SELECT SQL_CALC_FOUND_ROWS,
o.id, o.created_timestamp, o.status,
d.id, d.first_name, d.last_name,
p.id, p.first_name, p.last_name, p.ssn
GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
JOIN `users` u ON u.username = o.assigned_username
JOIN `events` e ON e.event_id = o.event_id
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id
WHERE o.status='Complete' OR o.status='Draft'
GROUP BY o.id
这里有一些事情要注意:
1)您的主表格为orders
。这也是你正在使用AND的WHERE子句的那个你是GROUPing BY的那个。 id
为主要索引,第二个索引为status
,这不应该太糟糕。
2)你在其他四个表中进行链接是因为我认为是外键。其中两个表从未实际使用过:(很可能)根本不需要加入users
和events
。你应该摆脱那些,顺便说一下,它也删除了文本列(用户名)上不那么出色的连接。确保其余表doctors
和patients
的主键位于各自的id
列中。
3)您在两个表tests_ordered
和tests
上有一个更复杂的联接。所有这一切都是为您提供一个串联的名称字符串,但它确实增加了GROUP BY子句的复杂性。有两种方法可以从这里开始:尝试优化这些连接或从完全选择中删除它们。
3)解决方案A 要优化这些联接,请确保您在tests_ordered.order_id
和tests_ordered.test_id
以及tests.id
上的主索引上有索引。您的声明应如下所示:
SELECT SQL_CALC_FOUND_ROWS,
o.id, o.created_timestamp, o.status,
d.id, d.first_name, d.last_name,
p.id, p.first_name, p.last_name, p.ssn
GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id
WHERE o.status='Complete' OR o.status='Draft'
GROUP BY o.id
3)解决方案B 删除整个测试/ tests_ordered的东西,并将其放在单独的选择中。您的主要选择现在将如下所示:
SELECT SQL_CALC_FOUND_ROWS,
o.id, o.created_timestamp, o.status,
d.id, d.first_name, d.last_name,
p.id, p.first_name, p.last_name, p.ssn
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
WHERE o.status='Complete' OR o.status='Draft'
但是你要么必须每行运行一个额外的SELECT来获取连接的t.name,要么只为当前页面上的所有订单ID执行一个SELECT。后者看起来像这样:
SELECT o.id, GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id
WHERE o.in IN ( <put the 10 ids on your current page here, separated by commas> )
GROUP BY o.id
解决方案A应该在一台不错的机器上运行得非常快。解决方案B应该在任何机器上运行得非常快。索引外键上的直接JOINS很便宜。
4)上述选择中的任何一个都不应该在数量少于6.42亿个订单的数据库中返回6.42亿行,我假设您没有。因为MySQL告诉你它可能需要构建一个完整的笛卡尔积,这也解释了经验丰富的速度。这意味着你的一个直接外键加入变得混乱。最有可能的问题是加入users
- 首先是无用的加入,但无论如何:检查其username
列的唯一性。
答案 1 :(得分:0)
1)一些提示可能是删除所有可以在表示层中使用PHP的连接和不必要的函数,并且只选择列值。
2)仅选择模式中绝对重要的最重要的列,然后可以运行简单的1行选择其他属性。 (即使查询数量增加,它实际上可以加快性能,因为多个连接往往非常慢)
3)如果没有任何帮助,您可能必须导致某些形式的非规范化,您必须在多个表上复制某些关系值以减少连接数。
4)缓存,缓存,cahce - 在数据库级别,在php级别,基本上无处不在......
5)如果您只需要最近使用最近的100 000 000然后开发某种归档功能来减少操作表中的记录数量,那么它在很大程度上依赖于您的数据,例如700 000 000行并且只有当记录存在于存档中时才会搜索大型存档表,这可能很慢但很少会发生......
这些只是一些非常普遍的难题。
答案 2 :(得分:0)
作为@azzit的补充,我可以给你一些我们在数据表服务器端查询中使用的提示: