如何限制以Pig脚本开头的并发作业数？

时间：2013-12-26 15:18:25

标签： hadoop apache-pig hortonworks-data-platform

我正在尝试使用Hortonworks sandbox为Pig中的POC实现简单的数据处理流程。

这个想法如下：有一些已经处理过的数据。应将新数据集添加到旧数据中，不得重复。

出于测试目的，我使用非常小的数据集（小于10 KB）。 对于虚拟机，我已经分配了4GB的RAM和4个处理器核心中的2个。

这是我的猪脚本：

-- CONFIGURABLE PROPERTIES
%DEFAULT atbInput '/user/hue/ATB_Details/in/1'
%DEFAULT atbOutputBase '/user/hue/ATB_Details/out/1'
%DEFAULT atbPrevOutputBase '/user/hue/ATB_Details/in/empty'

%DEFAULT validData 'valid'
%DEFAULT invalidData 'invalid'
%DEFAULT billDateDimensionName 'tmlBillingDate'
%DEFAULT admissionDateDimensionName 'tmlAdmissionDate'
%DEFAULT dischargeDateDimensionName 'tmlDischargeDate'
%DEFAULT arPostDateDimensionName 'tmlARPostDate'
%DEFAULT patientTypeDimensionName 'dicPatientType'
%DEFAULT patientTypeCodeDimensionName 'dicPatientTypeCode'

REGISTER bdw-all-deps-1.0.jar;

DEFINE toDateDimension com.epam.bigdata.etl.udf.ToDateDimension();
DEFINE toCodeDimension com.epam.bigdata.etl.udf.ToCodeDimension();
DEFINE isValid com.epam.bigdata.etl.udf.atbdetails.IsValidFunc();
DEFINE isGarbage com.epam.bigdata.etl.udf.atbdetails.IsGarbageFunc();
DEFINE toAccounntBalanceCategory com.epam.bigdata.etl.udf.atbdetails.ToBalanceCategoryFunc();
DEFINE isEndOfMonth com.epam.bigdata.etl.udf.IsLastDayOfMonthFunc();
DEFINE toBalanceCategoryId com.epam.bigdata.etl.udf.atbdetails.ToBalanceCategoryIdFunc();

rawData = LOAD '$atbInput';

--CLEANSING
SPLIT rawData INTO garbage IF isGarbage($0),
    cleanLines OTHERWISE;

splitRecords = FOREACH cleanLines GENERATE FLATTEN(STRSPLIT($0, '\\|'));

cleanData = FOREACH splitRecords GENERATE
    $0 AS Id:LONG,
    $1 AS FacilityName:CHARARRAY,
    $2 AS SubFacilityName:CHARARRAY,
    $3 AS PeriodDate:CHARARRAY,
    $4 AS AccountNumber:CHARARRAY,
    $5 AS RAC:CHARARRAY,
    $6 AS ServiceTypeCode:CHARARRAY,
    $7 AS ServiceType:CHARARRAY,
    $8 AS AdmissionDate:CHARARRAY,
    $9 AS DischargeDate:CHARARRAY,
    $10 AS BillDate:CHARARRAY,
    $11 AS PatientTypeCode:CHARARRAY,
    $12 AS PatientType:CHARARRAY,
    $13 AS InOutType:CHARARRAY,
    $14 AS FinancialClassCode:CHARARRAY,
    $15 AS FinancialClass:CHARARRAY,
    $16 AS SystemIPGroupCode:CHARARRAY,
    $17 AS SystemIPGroup:CHARARRAY,
    $18 AS CurrentInsuranceCode:CHARARRAY,
    $19 AS CurrentInsurance:CHARARRAY,
    $20 AS InsuranceCode1:CHARARRAY,
    $21 AS InsuranceBalance1:DOUBLE,
    $22 AS InsuranceCode2:CHARARRAY,
    $23 AS InsuranceBalance2:DOUBLE,
    $24 AS InsuranceCode3:CHARARRAY,
    $25 AS InsuranceBalance3:DOUBLE,
    $26 AS InsuranceCode4:CHARARRAY,
    $27 AS InsuranceBalance4:DOUBLE,
    $28 AS InsuranceCode5:CHARARRAY,
    $29 AS InsuranceBalance5:DOUBLE,
    $30 AS AgingBucket:CHARARRAY,
    $31 AS AccountBalance:DOUBLE,
    $32 AS TotalCharges:DOUBLE,
    $33 AS TotalPayments:DOUBLE,
    $34 AS EstimatedRevenue:DOUBLE,
    $35 AS CreateDateTime:CHARARRAY,
    $36 AS UniqueFileId:LONG,
    $37 AS PatientBalance:LONG,
    $38 AS VendorCode:CHARARRAY;


--VALIDATION
SPLIT cleanData INTO validData IF isValid(*),
    invalidData OTHERWISE;

--Dimension update--

--MACROS
DEFINE mergeDateDimension(validDataSet, dimensionFieldName, previousDimensionFile) RETURNS merged {
    dates = FOREACH $validDataSet GENERATE $dimensionFieldName;
    oldDimensions = LOAD '$previousDimensionFile' USING PigStorage('|') AS (
        id:LONG,
        monthName:CHARARRAY,
        monthId:INT,
        year:INT,
        fiscalYear:INT,
        originalDate:CHARARRAY);
    oldOriginalDates = FOREACH oldDimensions GENERATE originalDate;
    allDates = UNION dates, oldOriginalDates;
    uniqueDates = DISTINCT allDates;
    $merged = FOREACH uniqueDates GENERATE toDateDimension($0);
};


DEFINE mergeCodeDimension(validDataSet, dimensionFieldName, previousDimensionFile, outputIdField) RETURNS merged {
    newCodes = FOREACH $validDataSet GENERATE $dimensionFieldName as newCode;
    oldDim = LOAD '$previousDimensionFile' USING PigStorage('|') AS (
        id:LONG,
        code:CHARARRAY);
    allCodes = COGROUP oldDim BY code, newCodes BY newCode;

    grouped = FOREACH allCodes GENERATE  
        (IsEmpty(oldDim) ? 0L : SUM(oldDim.id)) as id,
        group AS code;
    ranked = RANK grouped BY id DESC, code DESC DENSE;
    $merged = FOREACH ranked GENERATE
        ((id == 0L) ? $0 : id) as $outputIdField,
        code AS $dimensionFieldName;
};

--DATE DIMENSIONS
billDateDim = mergeDateDimension(validData, BillDate, '$atbPrevOutputBase/dimensions/$billDateDimensionName');
STORE billDateDim INTO '$atbOutputBase/dimensions/$billDateDimensionName';

admissionDateDim = mergeDateDimension(validData, AdmissionDate, '$atbPrevOutputBase/dimensions/$admissionDateDimensionName');
STORE admissionDateDim INTO '$atbOutputBase/dimensions/$admissionDateDimensionName';

dischDateDim = mergeDateDimension(validData, DischargeDate, '$atbPrevOutputBase/dimensions/$dischargeDateDimensionName');
STORE dischDateDim INTO '$atbOutputBase/dimensions/$dischargeDateDimensionName';

arPostDateDim =  mergeDateDimension(validData, PeriodDate, '$atbPrevOutputBase/dimensions/$arPostDateDimensionName');
STORE arPostDateDim INTO '$atbOutputBase/dimensions/$arPostDateDimensionName';

--CODE DIMENSION
patientTypeDim = mergeCodeDimension(validData, PatientType, '$atbPrevOutputBase/dimensions/$patientTypeDimensionName', PatientTypeId);
STORE patientTypeDim INTO '$atbOutputBase/dimensions/$patientTypeDimensionName' USING PigStorage('|');

patientTypeCodeDim =  mergeCodeDimension(validData, PatientTypeCode, '$atbPrevOutputBase/dimensions/$patientTypeCodeDimensionName', PatientTypeCodeId);
STORE patientTypeCodeDim INTO '$atbOutputBase/dimensions/$patientTypeCodeDimensionName' USING PigStorage('|');

问题在于，当我运行此脚本时，它永远不会完成（卡住）。 在Job Browser中，我可以看到一个已完成的作业和多个0％进度的作业。

Job browser

如果我注释掉最后三个文件的处理 - 一切正常（即三个并行作业成功）。

我尝试过几种方法来解决这个问题：

  1. -no_multiquery Pig参数 - 允许一次只使用一个作业完全执行脚本。主要缺点是大量生成的作业（26）和非常长的执行时间（所描述的脚本近15分钟，更复杂的版本近40分钟）。
  2. 只使用我开发的部件并通过注释掉其他部件进行测试 - 这不是长期视角的选择。
  3. 更改 mapred-site.xml 中的 mapred.capacity-scheduler.maximum-system-jobs 属性，因此一次应该少于三个作业{{3} }。
  4. 更改 capacity-scheduler.xml 中的 mapred.capacity-scheduler.queue.default.maximum-capacity ，以配置默认队列。但这种方法对我和以前都不起作用。
  5. 为沙箱虚拟机以及映射器和缩减器分配更多内存 - 没有效果。

    6. 所以我的问题是如何限制以Pig脚本开头的并发作业数？ 或者可能还有其他配置修复允许并发执行多个作业？

    [UPDATE]

    如果我使用来自shell控制台的相同输入数据运行相同的脚本 - 一切正常。 所以我认为HUE存在一些问题。

    [UPDATE]

    如果我从控制台启动更复杂的脚本，它也会卡住，但在这种情况下，并行作业的数量是8。

2 个答案:

答案 0 :(得分：3)

上次我们看到这是因为群集有only one map task。

答案 1 :(得分：0)

您可以按照此处所述使用EXEC：

http://pig.apache.org/docs/r0.11.1/perf.html#Implicit-Dependencies

相关问题
最新问题