我们的服务器具有8个核心,正在运行一个Web应用程序(DHIS2),该应用程序使用postgres作为数据库。
有一个大型选择查询,需要几个小时才能执行。 (从终端执行查询) 当运行该查询时,该查询的进程的CPU利用率将跃升至恒定100%。
这会挂起应用程序,并且该应用程序的页面甚至不会加载到浏览器中。这一定是因为其他postgres进程正在等待该查询的进程完成。
但是,当我们的机器中有多个内核时,为什么将单个内核利用率提高到100%会阻止其余进程执行?
在这种情况下,我无法理解多个cpus和内核的概念。 postgres不能识别它们吗?选择查询对另一个查询的依赖性是什么?
有人可能会解释这种现象,并提出一些管理大型查询的方法,尽管可能采用某种Postgres配置?
Postgres Version - 9.6
OS - Ubuntu 16
Database Size - 200GB on disk
DHIS2 Version - 2.30
查询(排除异常值)-
select datasets,
max(ou1.name) Country, ou1.organisationunitid as Country__Id ,
max(ou2.name) state, ou2.organisationunitid as State__Id ,
max(ou3.name) Division, ou3.organisationunitid as Division__Id ,
max(ou4.name) District, max(ou4.code) as District__Code, ou4.organisationunitid as District__Id,
max(ou5.name) Block, max(ou5.code) as Block__Code, ou5.organisationunitid as Block__Id,
max(ou6.name) Facility, max(ou6.code) as Facility__Code,ou6.organisationunitid as Facility__Id,
max(ou.name) as outlierfacility,
max(de.name) as dataelement,
max(coc.name) as category,
concat(max(p.startdate),':',max(p.enddate)) as period,
max(pt.name) as frequency,
_dv.value,
u upperbound,
l lowerbound,
mean,
std
from
(
with stats as (
select dv.sourceid,
dv.dataelementid,
dv.categoryoptioncomboid,
dv.attributeoptioncomboid,
array_agg(distinct dv.periodid) as periods,
array_agg(distinct ds.name) as datasets,
avg(dv.value::float) as mean,
stddev(dv.value::float) as std
from datavalue dv
inner join datasetmembers dsm on dsm.dataelementid = dv.dataelementid
inner join dataelement de on de.dataelementid = dsm.dataelementid
inner join dataset ds on ds.datasetid = dsm.datasetid
inner join period pe on pe.periodid = dv.periodid
inner join periodtype pt on pt.periodtypeid = pe.periodtypeid
inner join categoryoptioncombo coc on dv.categoryoptioncomboid = coc.categoryoptioncomboid
inner join _orgunitstructure ous on ous.organisationunitid = dv.sourceid
where pe.startdate between date('2019-04-29') - interval '6 months' and date('2019-04-29') and pt.name='Monthly'
and de.valueType in ('NUMBER','INTEGER')
and ds.uid in ('123qwe123','123ewq123')
group by dv.sourceid,dv.dataelementid,dv.categoryoptioncomboid,dv.attributeoptioncomboid
)
select dv.*,datasets,mean,std,mean+3*std u,mean-3*std l
from datavalue dv
inner join period pe on pe.periodid = dv.periodid
inner join periodtype pt on pt.periodtypeid = pe.periodtypeid
inner join stats on
stats.dataelementid = dv.dataelementid and
stats.sourceid= dv.sourceid and
stats.categoryoptioncomboid = dv.categoryoptioncomboid and
stats.attributeoptioncomboid = dv.attributeoptioncomboid
where dv.periodid = any(periods)
and (dv.value::float > mean+3*std or dv.value::float < mean-3*std)
) _dv
inner join dataelement de on _dv.dataelementid = de.dataelementid
inner join categoryoptioncombo coc on _dv.categoryoptioncomboid = coc.categoryoptioncomboid
inner join _orgunitstructure ous on _dv.sourceid = ous.organisationunitid
inner join organisationunit ou on ou.organisationunitid = ous.organisationunitid
left join organisationunit ou1 on ou1.organisationunitid = ous.idlevel1
left join organisationunit ou2 on ou2.organisationunitid = ous.idlevel2
left join organisationunit ou3 on ou3.organisationunitid = ous.idlevel3
left join organisationunit ou4 on ou4.organisationunitid = ous.idlevel4
left join organisationunit ou5 on ou5.organisationunitid = ous.idlevel5
left join organisationunit ou6 on ou6.organisationunitid = ous.idlevel6
inner join period p on _dv.periodid = p.periodid
inner join periodtype pt on p.periodtypeid = pt.periodtypeid
group by ou1.organisationunitid,
ou2.organisationunitid,
ou3.organisationunitid,
ou4.organisationunitid,
ou5.organisationunitid,
ou6.organisationunitid,
_dv.dataelementid,_dv.sourceid,_dv.categoryoptioncomboid,_dv.attributeoptioncomboid,_dv.periodid,_dv.value,u,l,mean,std,datasets
order by country,state,division,district,block,facility,dataelement,category