Question

我有问题。我们使用Matlab测试框架来分析我们的代码库。为了在CI系统TeamCity中跟踪结果，我们使用TAP格式。这里我们有以下问题：

如果测试包括TestClassSetup部分，则TAP结果仅在最后显示，而不在执行过程中显示。这给我们带来了一些问题：

CI系统创建的时间戳可能不正确
如果在测试用例中给出了信息输出，则不会与断言语句一起显示。

我们使用以下（简化的）代码片段来识别并执行TestSuite：

testSuite = matlab.unittest.TestSuite.fromFolder('.');
runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
results = runner.run(testSuite);

对于以下两个类别，该问题是可重现的（内容当然是虚构的，毫无意义……）

classdef SomeTest < matlab.unittest.TestCase
   properties (TestParameter)
       param = {1, 2};
       param2 = {1, 2};
   end        
   methods (TestClassSetup)
       function someSetup(testCase)
           pause(0.1);
       end
   end    
   methods (Test)
       function testMethod(self, param, param2)
          fprintf('I''m here, with the params: %f/%f\n', param, param2);
          pause(0.1);
          self.assertGreaterThan(param, param2); 
       end
   end    
end

classdef SomeOtherTest < matlab.unittest.TestCase
   properties (TestParameter)
       param = {1, 2};
       param2 = {1, 2};
   end    

   methods (Test)
       function testMethod(self, param, param2)
          fprintf('I''m here, with the params: %f/%f\n', param, param2);
          pause(0.1);
          self.assertGreaterThan(param, param2); 
       end
   end

end

如果将所有三个文件都复制到一个文件夹中，然后执行运行程序，则会看到输出（简化了断言）：

1..8
I'm here, with the params: 1.000000/1.000000
not ok 1 - SomeOtherTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 1.000000/2.000000
not ok 2 - SomeOtherTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 2.000000/1.000000
ok 3 - SomeOtherTest/testMethod(param=2,param2=1)
I'm here, with the params: 2.000000/2.000000
not ok 4 - SomeOtherTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 1.000000/1.000000
I'm here, with the params: 1.000000/2.000000
I'm here, with the params: 2.000000/1.000000
I'm here, with the params: 2.000000/2.000000
not ok 5 - SomeTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
# 
not ok 6 - SomeTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
# 
ok 7 - SomeTest/testMethod(param=2,param2=1)
not ok 8 - SomeTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================

我希望在第二种情况下，Assertion语句（以及ok / not ok TAP标志）也与fprintf语句对齐。

有人有主意吗？

Answer 1

TestClassSetup的存在“延迟”了TAP输出的打印的原因是因为TAP输出是一种流格式，并且如果存在任何TestClassSetup代码，则框架实际上尚不知道测试是否会通过。例如，如果您在TestClassTeardown中失败（或通过TestClassSetup中的addTeardown函数调用），则最终结果是所有共享TestClassSetup代码的测试都将失败。

但是，鉴于其流式格式，TAPPLugin希望在知道结果后立即打印出结果。实际上，有一个专门针对这种情况设计的TestRunnerPlugin方法reportFinalizedResult method。

这里的根本问题是，我建议您避免使用disp或fprintf打印到日志。这不太理想，因为插件对使用fprintf打印的任何信息都没有任何了解。另外，除了matlab命令行外，您无法将这些信息重定向到其他任何地方。

但是，如果您改用testCase.log方法，则可以在正确的位置获得诊断信息，并且它会更加灵活。您将能够在不同级别上对其进行记录，因此您可以根据需要打开或关闭它，并控制是否要查看它。它不仅会进入命令行，而且会更好地进入TAP流以及junit xml和pdf / html测试报告等。对于您的情况，如下所示：

runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
results = runner.run(testSuite);

第一次运行时，您看不到任何日志调用，因为它以详细信息“ 3”记录，并且默认值较低（我相信是1级）

1..8
not ok 1 - SomeOtherTest/testMethod(param=value1,param2=value1)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 2 - SomeOtherTest/testMethod(param=value1,param2=value2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 3 - SomeOtherTest/testMethod(param=value2,param2=value1)
not ok 4 - SomeOtherTest/testMethod(param=value2,param2=value2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================
not ok 5 - SomeTest/testMethod(param=value1,param2=value1)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 6 - SomeTest/testMethod(param=value1,param2=value2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 7 - SomeTest/testMethod(param=value2,param2=value1)
not ok 8 - SomeTest/testMethod(param=value2,param2=value2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================

但是，如果将Tap插件（或版本13的Tap插件或Report插件等）配置为在三级登录，则您会看到这些诊断信息，并且它们也位于预期的位置：

runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat('Verbosity', 3));
results = runner.run(testSuite);

您将看到输出。还可以尝试TAPVersion 13，它提供的结构化输出可能会提供更好的结果。

1..8
not ok 1 - SomeOtherTest/testMethod(param=value1,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:18): I'm here, with the params: 1.000000/1.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 2 - SomeOtherTest/testMethod(param=value1,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 3 - SomeOtherTest/testMethod(param=value2,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 2.000000/1.000000
# ================================================================================
not ok 4 - SomeOtherTest/testMethod(param=value2,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 2.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================
not ok 5 - SomeTest/testMethod(param=value1,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/1.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 6 - SomeTest/testMethod(param=value1,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 7 - SomeTest/testMethod(param=value2,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:20): I'm here, with the params: 2.000000/1.000000
# ================================================================================
not ok 8 - SomeTest/testMethod(param=value2,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:20): I'm here, with the params: 2.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================

希望有帮助！

如果存在TestClassSetup，则在执行期间不会显示TAP结果

1 个答案: