Question

我一直在将所有数据集移动到SPDE库中，因为我在所有方面都获得了出色的性能提升。一直到运行proc转置。在SPDE数据集上执行的时间比在普通v9库中存储的相同数据集长约60倍。数据集按item_id排序。它正被读/写到同一个库中。

有谁知道为什么会这样？我是否遗漏了一些重要的关于SPDE和Proc Transpose不能很好地一起玩的东西？

SPDE Libary

MPRINT(XMLIMPORT_VANTAGE):   proc transpose data = smplus.links_response_mechanism out = smplus.response_mechanism (drop = _NAME_) 
prefix = rm_;
MPRINT(XMLIMPORT_VANTAGE):   by item_id;
MPRINT(XMLIMPORT_VANTAGE):   id lookup_code;
MPRINT(XMLIMPORT_VANTAGE):   var x;
MPRINT(XMLIMPORT_VANTAGE):   run;

NOTE: There were 5866747 observations read from the data set SMPLUS.LINKS_RESPONSE_MECHANISM.
NOTE: The data set SMPLUS.RESPONSE_MECHANISM has 3209353 observations and 14 variables.
NOTE: Compressing data set SMPLUS.RESPONSE_MECHANISM decreased size by 37.98 percent.
NOTE: PROCEDURE TRANSPOSE used (Total process time):
      real time           28:27.63
      cpu time            28:34.64

V9图书馆

MPRINT(XMLIMPORT_VANTAGE):   proc transpose data = mplus.links_response_mechanism out = mplus.response_mechanism (drop = _NAME_) 
prefix = rm_;
MPRINT(XMLIMPORT_VANTAGE):   by item_id;
68                                                         The SAS System                             02:00 Thursday, August 8, 2013

MPRINT(XMLIMPORT_VANTAGE):   id lookup_code;
MPRINT(XMLIMPORT_VANTAGE):   var x;
MPRINT(XMLIMPORT_VANTAGE):   run;

NOTE: There were 5866747 observations read from the data set MPLUS.LINKS_RESPONSE_MECHANISM.
NOTE: The data set MPLUS.RESPONSE_MECHANISM has 3209353 observations and 14 variables.
NOTE: Compressing data set MPLUS.RESPONSE_MECHANISM decreased size by 27.60 percent. 
      Compressed is 32271 pages; un-compressed would require 44572 pages.
NOTE: PROCEDURE TRANSPOSE used (Total process time):
      real time           28.76 seconds
      cpu time            28.79 seconds

Answer 1

在我看来，PROC TRANSPOSE和SPDE存在一些问题。这是一个简单的SSCCE，它有很大的不同;没有你的那么重要，但在某种程度上，这可能是桌面上的一个因素，首先没有特别重要的性能调整。听起来像是打电话给SAS技术支持。

libname spdelib spde 'c:\temp\SPDE Main' 
    datapath=('c:\temp\SPDE Data' 'd:\temp\SPDE Data')
    indexpath=('d:\temp\SPDE Index')
    partsize=512;

libname mainlib 'c:\temp\';


data mainlib.bigdata;
do ID = 1 to 1500000;
  do _varn=1 to 10;
    varname=cats("Var_",_varn);
    vardata=ranuni(7);
    output;
  end;
end;
run;
data  spdelib.bigdata;
do ID = 1 to 1500000;
  do _varn=1 to 10;
    varname=cats("Var_",_varn);
    vardata=ranuni(7);
    output;
  end;
end;
run;
*These data steps take roughly the same amount of time, around 30 seconds each;

proc transpose data=spdelib.bigdata out=spdelib.transdata;
by id;
id varname;
var vardata;
run;
*Run a few times, this takes around 3 to 4 minutes, with 1.5 minutes CPU time;

proc transpose data=mainlib.bigdata out=mainlib.transdata;
by id;
id varname;
var vardata;
run;
*Run a few times, this takes around 30 to 45 seconds, with 20 seconds CPU time;

Answer 2

过去已经存在SPDE和proc比较的已知问题（不是多线程），至少是版本4.1。你用的是什么版本？（可以在“！install / logs”文件夹中看到）。

这绝对是raise with SAS support的内容，为了“加速”事情，我建议使用以下选项提交日志：

proc setinit noalias; run; 
proc options; run; 
%put _ALL_; 
options fullstimer msglevel=i;

此外：

options spdedebug='DA_TRACEIO_OCR CJNL=Trace.txt';

（CJNL选项只是将跟踪消息输出路由到文本文件）

与此同时，您可以利用以下某些SPD特定选项：

http://support.sas.com/kb/11/349.html

Answer 3

当PROC TRANSPOSE与压缩数据集上的BY处理一起使用时，通常会发生此问题。 SAS被迫每次重复读取同一行块，然后对其进行解压缩，直到对所有记录进行完全排序为止。

设置Compress = No选项，它将起作用。请参阅下面的日志，一个程序的Compress = yes，另一个Compress = no，前一个程序是56分钟vs.5秒。

OPTIONS COMPRESS=YES;

50         **tranpose from spde to spde;
51         proc transpose data=spdelib.balancewalkoutput out=spdelib.spdelib_to_spdelib;
52           var metric ;
53           by balancewalk facility_id isretained isexisting isicaapnpl monthofmaturity vintage;
54         run;

NOTE: There were 10000000 observations read from the data set SPDELIB.BALANCEWALKOUTPUT.
NOTE: The data set SPDELIB.SPDELIB_TO_SPDELIB has 160981 observations and 74 variables.
NOTE: Compressing data set SPDELIB.SPDELIB_TO_SPDELIB decreased size by 69.96 percent.
NOTE: PROCEDURE TRANSPOSE used (Total process time):
      real time           56:58.54
      user cpu time       52:03.65
      system cpu time     4:03.00
      memory              19028.75k
      OS Memory           34208.00k
      Timestamp           09/16/2019 06:19:55 PM
      Step Count                        9  Switch Count  22476
      Page Faults                       0
      Page Reclaims                     4056
      Page Swaps                        0
      Voluntary Context Switches        142316
      Involuntary Context Switches      5726
      Block Input Operations            88
      Block Output Operations           569200


OPTIONS COMPRESS=NO;

50         **tranpose from spde to spde;
51         proc transpose data=spdelib.balancewalkoutput out=spdelib.spdelib_to_spdelib;
52           var metric ;
53           by balancewalk facility_id isretained isexisting isicaapnpl monthofmaturity vintage;

18                                                         The SAS System                           16:04 Monday, September 16, 2019

54         run;

NOTE: There were 10000000 observations read from the data set SPDELIB.BALANCEWALKOUTPUT.
NOTE: The data set SPDELIB.SPDELIB_TO_SPDELIB has 160981 observations and 74 variables.
NOTE: PROCEDURE TRANSPOSE used (Total process time):
      real time           26.73 seconds
      user cpu time       14.52 seconds
      system cpu time     11.99 seconds
      memory              13016.71k
      OS Memory           27556.00k
      Timestamp           09/16/2019 04:13:06 PM
      Step Count                        9  Switch Count  24827
      Page Faults                       0
      Page Reclaims                     2662
      Page Swaps                        0
      Voluntary Context Switches        162653
      Involuntary Context Switches      1678
      Block Input Operations            96
      Block Output Operations           1510040

使用SPDE进行proc转置比v9库运行时间长约60倍

3 个答案: