有没有办法让JUnit规则或类似的东西为每次失败的测试提供第二次机会,只需尝试再次运行它。
背景:我有一大堆用JUnit编写的Selenium2-WebDriver测试。由于非常激进的时间(点击后只有短暂的等待时间),一些测试(100个中的1个,总是不同的测试)可能会失败,因为服务器有时响应速度稍慢。但我不能让等待时间太久以至于它肯定足够长,因为那时测试将永远持续下去。) - 所以我认为这个用例即使需要一秒钟,测试也是绿色的是可以接受的尝试。
当然最好有2分(满分3分)(重复3次失败测试,如果其中两项测试是正确的,则将其视为正确),但这将是未来的改进。
答案 0 :(得分:95)
您可以使用TestRule执行此操作。这将为您提供所需的灵活性。 TestRule允许您在测试周围插入逻辑,因此您将实现重试循环:
public class RetryTest {
public class Retry implements TestRule {
private int retryCount;
public Retry(int retryCount) {
this.retryCount = retryCount;
}
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
System.err.println(description.getDisplayName() + ": run " + (i+1) + " failed");
}
}
System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
throw caughtThrowable;
}
};
}
}
@Rule
public Retry retry = new Retry(3);
@Test
public void test1() {
}
@Test
public void test2() {
Object o = null;
o.equals("foo");
}
}
TestRule
的核心是base.evaluate()
,它会调用您的测试方法。因此,围绕此调用,您将进行重试循环。如果在您的测试方法中抛出异常(断言失败实际上是AssertionError
),则测试失败,您将重试。
还有一件事可能有用。您可能只想将此重试逻辑应用于一组测试,在这种情况下,您可以在方法的特定注释的测试上方添加到Retry类中。 Description
包含该方法的注释列表。有关此问题的详细信息,请参阅我对How to run some code before each JUnit @Test method individually, without using @RunWith nor AOP?的回答。
这是CKuck的建议,你可以定义自己的Runner。您需要扩展BlockJUnit4ClassRunner并覆盖runChild()。有关详细信息,请参阅我对How to define JUnit method rule in a suite?的回答。此答案详细说明了如何定义如何为Suite中的每个方法运行代码,您必须为其定义自己的Runner。
答案 1 :(得分:17)
至于我编写自定义跑步者更灵活的解决方案。 上面发布的解决方案(带代码示例)有两个缺点:
这就是为什么我更喜欢编写自定义跑步者的方法。自定义运行程序的代码可能如下:
import org.junit.Ignore;
import org.junit.internal.AssumptionViolatedException;
import org.junit.internal.runners.model.EachTestNotifier;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runner.notification.StoppedByUserException;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;
public class RetryRunner extends BlockJUnit4ClassRunner {
private final int retryCount = 100;
private int failedAttempts = 0;
public RetryRunner(Class<?> klass) throws InitializationError {
super(klass);
}
@Override
public void run(final RunNotifier notifier) {
EachTestNotifier testNotifier = new EachTestNotifier(notifier,
getDescription());
Statement statement = classBlock(notifier);
try {
statement.evaluate();
} catch (AssumptionViolatedException e) {
testNotifier.fireTestIgnored();
} catch (StoppedByUserException e) {
throw e;
} catch (Throwable e) {
retry(testNotifier, statement, e);
}
}
@Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
Description description = describeChild(method);
if (method.getAnnotation(Ignore.class) != null) {
notifier.fireTestIgnored(description);
} else {
runTestUnit(methodBlock(method), description, notifier);
}
}
/**
* Runs a {@link Statement} that represents a leaf (aka atomic) test.
*/
protected final void runTestUnit(Statement statement, Description description,
RunNotifier notifier) {
EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
eachNotifier.fireTestStarted();
try {
statement.evaluate();
} catch (AssumptionViolatedException e) {
eachNotifier.addFailedAssumption(e);
} catch (Throwable e) {
retry(eachNotifier, statement, e);
} finally {
eachNotifier.fireTestFinished();
}
}
public void retry(EachTestNotifier notifier, Statement statement, Throwable currentThrowable) {
Throwable caughtThrowable = currentThrowable;
while (retryCount > failedAttempts) {
try {
statement.evaluate();
return;
} catch (Throwable t) {
failedAttempts++;
caughtThrowable = t;
}
}
notifier.addFailure(caughtThrowable);
}
}
答案 2 :(得分:17)
现在有一个更好的选择。如果您正在使用maven插件,例如:surfire或failsefe,则可以选择添加参数rerunFailingTestsCount
SurFire Api。
这些东西在以下票证中实现:Jira Ticket。
在这种情况下,您不需要编写自定义代码,插件会自动修改测试结果报告
我看到这种方法只有一个缺点:如果某些测试失败,则课前/课后阶段测试不会重新开始。
答案 3 :(得分:6)
您必须自己编写org.junit.runner.Runner
并使用@RunWith(YourRunner.class)
注释您的测试。
答案 4 :(得分:3)
建议的评论是根据this条文章撰写的,并添加了一些内容。
在这里,如果jUnit项目中的某些测试用例得到“失败”或“错误”结果,则该测试用例将再次运行一次。我们总共在这里设置了3个获得成功结果的机会。
因此,我们需要创建规则类,并在您的测试类中添加“ @Rule”通知。
如果不想为每个测试类添加相同的“ @Rule”通知,则可以将其添加到抽象SetProperty类(如果有)中并从其扩展。
规则类别:
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class RetryRule implements TestRule {
private int retryCount;
public RetryRule (int retryCount) {
this.retryCount = retryCount;
}
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
// System.out.println(": run " + (i+1) + " failed");
System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed.");
}
}
System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures.");
throw caughtThrowable;
}
};
}
}
测试类别:
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;
/**
* Created by ONUR BASKIRT on 27.03.2016.
*/
public class RetryRuleTest {
static WebDriver driver;
final private String URL = "http://www.swtestacademy.com";
@BeforeClass
public static void setupTest(){
driver = new FirefoxDriver();
}
//Add this notification to your Test Class
@Rule
public RetryRule retryRule = new RetryRule(3);
@Test
public void getURLExample() {
//Go to www.swtestacademy.com
driver.get(URL);
//Check title is correct
assertThat(driver.getTitle(), is("WRONG TITLE"));
}
}
答案 5 :(得分:0)
此答案基于this answer。
如果需要在每次运行之前重新创建ActivityScenario
(和活动),则可以使用try-with-resources启动它。每次尝试后,ActivityScenario
将自动关闭。
public final class RetryRule<A extends Activity> implements TestRule {
private final int retryCount;
private final Class<A> activityClazz;
private ActivityScenario<A> scenario;
/**
* @param retryCount the number of retries. retryCount = 1 means 1 (normal) try and then
* 1 retry, i.e. 2 tries overall
*/
public RetryRule(int retryCount, @NonNull Class<A> clazz) {
this.retryCount = retryCount;
this.activityClazz = clazz;
}
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i <= retryCount; i++) {
try(ActivityScenario<A> scenario = ActivityScenario.launch(activityClazz)){
RetryRule.this.scenario = scenario;
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
Log.e(LOGTAG,
description.getDisplayName() + ": run " + (i + 1) + " failed: ", t);
}
}
Log.e(LOGTAG,
description.getDisplayName() + ": giving up after " + (retryCount + 1) +
" failures");
throw Objects.requireNonNull(caughtThrowable);
}
};
}
public ActivityScenario<A> getScenario() {
return scenario;
}
}
然后您可以使用getScenario()
方法在测试中访问您的方案。