Question

我正在使用MTurkR将HIT发布到mTurk，我无法延长HIT，直到工人达成共识（或直到总共发布了5个HIT）。

当输入两个不同的响应时，不会扩展HIT。我的代码来自MTurkR documentation pg。我的代码如下：

# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
    #format for sandbox question. Get this from MTURK site
annotation.v="Question1"
assignments.v="2"
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)


# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("Question1"),
  QuestionAgreementThreshold=49, #at least 50% agree
  ExtendIfHITAgreementScoreIsLessThan=50,
  ExtendMinimumTimeInSeconds=3600,
  ExtendMaximumAssignments=5,
  DisregardAssignmentIfRejected=TRUE)

policya=do.call(GenerateHITReviewPolicy,TurkAgreement)


# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
  hits.i=CreateHIT(
    hitlayoutid=layout,
    hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
    annotation=annotation.v[i],
    assignments=assignments.v,
    title=title.v,
    description=description.v,
    reward=reward.v,
    duration=duration.v,
    expiration=expiration.v,
    keywords=keywords.v,
    auto.approval.delay=auto.approval.delay.v,
    qual.req=qualReqs,
    hit.review.policy=policya,
    sandbox=sandbox.v)
    hits=rbind(hits,hits.i)}

代码生成2个HIT（由assignments.v指定），但HIT不会扩展。

我的会话信息如下：

> sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] MTurkR_0.6

loaded via a namespace (and not attached):
[1] digest_0.6.4   RCurl_1.95-4.3 tcltk_3.1.1    XML_3.98-1.1

Answer 1

首先关于您发布的代码的几个要点：

因为您要添加hit.review.policy，所以不需要assignments参数。只有这个声明，两个HIT将立即填充。
在为hit.review.policy定义术语之前，请先查看Amazon's documentation。请特别注意QuestionAgreementThreshold的值。由于大于此值的值被视为约定答案，因此您的原始代码表示大于49％的值是商定的答案。同时检查ExtendIfHITAgreementScoreIsLessThan此值应该是QuestionAgreementThreshold值的+1，以便捕获所有可能的协议值。
最后，确保HIT响应的答案值与QuestionIds对象相对应。当我查看我的HIT的答案时，答案被称为“问题”，所以我在这里替换了这个值。

所有这些都说，这是修改后的代码，现在正确扩展了HIT：

# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
    #format for sandbox question. Get this from MTURK site
annotation.v="question" #NOTE CHANGE HERE
#assignments.v="2"  #NOTE CHANGE HERE (COMMENTED THIS OUT)
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)


# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("question"), #NOTE CHANGE HERE
  QuestionAgreementThreshold=50, #at least 50% agree  #NOTE CHANGE HERE
  ExtendIfHITAgreementScoreIsLessThan=51, #NOTE CHANGE HERE
  ExtendMinimumTimeInSeconds=3600,
  ExtendMaximumAssignments=5,
  DisregardAssignmentIfRejected=TRUE)

policya=do.call(GenerateHITReviewPolicy,TurkAgreement)


# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
  hits.i=CreateHIT(
    hitlayoutid=layout,
    hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
    annotation=annotation.v[i],
    assignments=assignments.v,
    title=title.v,
    description=description.v,
    reward=reward.v,
    duration=duration.v,
    expiration=expiration.v,
    keywords=keywords.v,
    auto.approval.delay=auto.approval.delay.v,
    qual.req=qualReqs,
    hit.review.policy=policya,
    sandbox=sandbox.v)
    hits=rbind(hits,hits.i)}

另请注意MTurkR Package页上的文档。 44（结束）令人困惑。该文档提供了以下示例：

lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 49, # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 50,
...

但论证QuestionAgreementThreshold实际上指定了至少49％的协议（即如果2个土耳其人分开答案，HIT将不会延长）。除非这是意图，否则最好使用以下代码：

lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 50, #NOTE CHANGE HERE # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 51, #NOTE CHANGE HERE
...

扩展HIT直到达成协议MTurkR

1 个答案: