在工作中，我有一个200M行（300多个列，约200GB）的索赔表，该表由成员编号（memno）和服务日期（dos）索引。
我每个月必须从固定的日期范围（例如，2017年1月1日至2018年5月1日）中提取50,000个会员号的索赔，而我在索赔日期中只需要有限的列即可。
MyInputList仅具有1列（memno）。
proc sql;
create table myClaims as
select a.claimno, a.dos, a.memno
from s.claims a inner join myInputList b
on a.memno = b.memno
where a.dos between '01Jan2017'd and '01May2018'd;
quit;
通过PROC SQL运行通常大约需要3-4个小时。数据本身不是由RDMS托管的，我读了很多SAS文章，即PROC SQL做笛卡尔乘积，由于我不需要每条记录全部300列，所以我想知道使用散列表是否会更好。
我的问题：我可以给哈希表“提示”以便它可以利用索引列(memno, dos)吗？
data myClaimsTest (drop=rc);
if 0 then set myInputList;
declare hash vs(hashexp:7, dataset:'myInputList');
vs.definekey('memno');
vs.definedata();
vs.definedone();
do until (eof);
set s.claims (keep=claimno dos) end=eof;
if vs.find()=0 then output;
end;
stop;
run;
新部分（由Richard添加）
运行此代码以获取变量和索引的列表。
dm "clear output"; ods listing; ods noresults; options nocenter; title;
proc contents varum data=all_claims;
run;
dm "output" output; ods results;
在此处复制并粘贴输出的底部。将此示例替换为您的实际列表。
Variables in Creation Order
# Variable Type Len Format
1 claim_id Num 8
2 member_id Num 8
3 claim_date Num 8 YYMMDD10.
Alphabetic List of Indexes and Attributes
# of
Unique Unique
# Index Option Values Variables
1 PICK YES 333338 member_id claim_date
答案 0 :(得分：2)
假设 BIG 是您的200GB索引SAS表，而 SMALL 是您的5万行选择条件行。
BIG索引（键）可能未使用，因为SMALL数据没有足够的信息（变量）来完成与BIG匹配的组合键。
有两种加工方式
您的问题中的哈希码与＃1相对应，而SQL连接为＃2，尽管SQL可能选择采用＃1。
这里是样本数据制作者
%let MEMBER_N = 1e5;
%let CLAIM_RATE = 0.00125;
%let MEMBER_SAMPLE_N = 1e2;
%let STUDY_PROPORTION = 0.001;
data ALL_CLAIMS
( label = "BIG"
index=
(
PICK = (member_id claim_date) / unique
)
);
retain claim_id 0 member_id 0 claim_date 0 member_n 0;
format claim_date yymmdd10.;
do member_id = 1e7 by 1;
claim_n = 1;
do claim_date = '01jan2012'd to '31dec2018'd;
if ranuni(123) > &CLAIM_RATE then continue;
claim_id + 1;
if claim_n = 1 then member_n + 1;
output;
claim_n + 1;
end;
if member_n = &MEMBER_N then leave;
end;
stop;
drop member_n claim_n;
run;
%put note: sample population is %sysevalf(5e4/200e6*100)% of all claims;
%put note: or ~%sysevalf(5e4/200e6*1e6) rows in this example;
data STUDY_MEMBERS(keep=member_id label="SMALL");
* k / n selection method, Proc SURVEYSELECT is better but not always available;
* an early sighting on SAS-L would be https://listserv.uga.edu/cgi-bin/wa?A2=ind9909c&L=SAS-L&P=173979
* Re: Random Selection (Sep 20, 1999);
retain
k %sysevalf(&MEMBER_N*&STUDY_PROPORTION, FLOOR)
n &MEMBER_N
;
set ALL_CLAIMS;
by member_id;
if first.member_id;
if ranuni(123) < k/n then do;
output;
k + (-1);
end;
n + (-1);
if n=0 then stop;
run;
和处理代码
options msglevel=i;
proc sql;
create table ALL_STUDY_SUBSET as
select ALL.claim_id, ALL.claim_date, ALL.member_id
from ALL_CLAIMS ALL inner join STUDY_MEMBERS STUDY
on ALL.member_id = STUDY.member_id
where ALL.claim_date between '01Jan2017'd and '01May2018'd
;
quit;
* extend study data with a date variable that matches the data variable in the ALL index;
data STUDY_MEMBERS_WITH_ITERATED_DATE;
set STUDY_MEMBERS;
do claim_date = '01Jan2017'd to '01May2018'd;
output;
end;
run;
* join on both variables in ALL key;
proc sql;
create table ALL_STUDY_SUBSET2 as
select ALL.claim_id, ALL.claim_date, ALL.member_id
from ALL_CLAIMS ALL inner join STUDY_MEMBERS_WITH_ITERATED_DATE STUDY
on ALL.member_id = STUDY.member_id
and ALL.claim_date = STUDY.claim_date
;
quit;
* full scan with hash based match;
data ALL_STUDY_SUBSET3;
SET ALL_CLAIMS;
if _n_ = 1 then do;
declare hash study (dataset:'STUDY_MEMBERS');
study.defineKey('member_id');
study.defineDone();
end;
if '01jan2017'd <= claim_date <= '01may2018'd;
if study.find() = 0;
run;
* SMALL scan with iterated dates to complete info to allow BIG index (key)
* to be used;
data ALL_STUDY_SUBSET4;
set STUDY_MEMBERS;
do claim_date = '01jan2017'd to '01may2018'd;
set ALL_CLAIMS key=pick;
if _iorc_ = 0 then output;
end;
_error_ = 0;
run;
答案 1 :(得分：1)
我在问题条件略有不同的情况下添加了第二个答案-成员在给定的一天可以有多个声明，并且只有简单的单个变量索引。
每个会员/天对数据进行多次声明
%let MEMBER_N = 1e5;
%let CLAIM_RATE = 0.00125;
%let MULTI_CLAIM_RATE = 0.05; %* iterative rate at which another claim is made on same day a claim is made;
%let STUDY_PROPORTION = 0.001;
data ALL_CLAIMS
( label = "BIG"
index=
(
/* PICK = (member_id claim_date) / unique (not happening) */
member_id
claim_id
)
);
retain claim_id 0 member_id 0 claim_date 0 member_n 0;
format claim_date yymmdd10.;
do member_id = 1e7 by 1;
claim_n = 1;
do claim_date = '01jan2012'd to '31dec2018'd;
if ranuni(123) > &CLAIM_RATE then continue;
if claim_n = 1 then member_n + 1;
do multi_n = 0 by 1 until (ranuni(123) > &MULTI_CLAIM_RATE);
claim_id + 1;
output;
end;
if multi_n > 1 then put 'NOTE: ' member_id= claim_date= multi_n 'claims';
claim_n + 1;
end;
if member_n = &MEMBER_N then leave;
end;
stop;
drop member_n claim_n;
run;
使用claim_date索引来初步选择候选索赔可能没有帮助-您可能在灾难性的一天有成千上万的索赔，您的处理必须在日期范围内迭代日期，通过匹配来设置索赔日期，并对在那些匹配日期发生的每个Claim_id记录进行哈希查找（SMALL：member_id）。您将不得不进行试验，看看这种反直觉的方法对于您的“全部”和“全部”是否可能效果更好。
如果检查SQL日志，您将看到查询优化器选择使用member_id索引（并且内部将迭代找到的行以应用where子句）。未记录的Proc SQL选项_method和_tree可以向您显示其功能-请参阅"The SQL Optimizer Project: _Method and _Tree in SAS®9.1" Lavery（SUGI 30）。
proc sql _method _tree;
create table ALL_STUDY_SUBSET as
select ALL.claim_id, ALL.claim_date, ALL.member_id
from ALL_CLAIMS ALL inner join STUDY_MEMBERS STUDY
on ALL.member_id = STUDY.member_id
where ALL.claim_date between '01Jan2017'd and '01May2018'd
;
quit;
日志摘录
INFO: Index member_id of SQL table WORK.ALL_CLAIMS (alias = ALL) selected for SQL WHERE clause
(join) optimization.
NOTE: SQL execution methods chosen are:
sqxcrta
sqxjndx
sqxsrc( WORK.STUDY_MEMBERS(alias = STUDY) )
sqxsrc( WORK.ALL_CLAIMS(alias = ALL) )
Tree as planned.
/-SYM-V-(ALL.claim_id:1 flag=0001)
/-OBJ----|
| |--SYM-V-(ALL.claim_date:3 flag=0001)
| \-SYM-V-(ALL.member_id:2 flag=0001)
/-JOIN---|
| | /-SYM-V-(STUDY.member_id:1 flag=0001)
| | /-OBJ----|
| | /-SRC----|
| | | \-TABL[WORK].STUDY_MEMBERS opt=''
| |--FROM---|
| | | /-SYM-V-(ALL.claim_id:1 flag=0001)
| | | /-OBJ----|
| | | | |--SYM-V-(ALL.claim_date:3 flag=0001)
| | | | \-SYM-V-(ALL.member_id:2 flag=0001)
| | \-SRC----|
| | |--TABL[WORK].ALL_CLAIMS opt=''
| | | /-NAME--(claim_date:3)
| | \-IN-----|
| | | /-LITN(20820) DATE.
| | | /-RANB---|
| | | | \-LITN(21305) DATE.
| | \-SET----|
| |--empty-
| | /-SYM-V-(STUDY.member_id:1)
| \-CEQ----|
| \-SYM-V-(ALL.member_id:2)
--SSEL---|
和等效的数据步骤
data ALL_STUDY_SUBSET5(label="Presuming a preponderance of members file few claims over their all_claims lifetime");
set STUDY_MEMBERS;
do until (_iorc_);
set ALL_CLAIMS key=member_id;
if _iorc_ = 0 and '01jan2017'd <= claim_date <= '01may2018'd then do;
OUTPUT;
end;
end;
_error_ = 0;
run;
还慢吗？
当情况出现时，尽最大的努力和最佳实践的编程结果还不够快，您将不得不通过包含的系统资源来寻求改进：
(member_id claim_date)添加新的复合索引吗？