TeamCity-使用Kentico Fakes时,NUnit Console Runner未找到所有单元测试

时间:2019-01-15 10:29:31

标签: c# teamcity kentico nunit-3.0 nunit-console

我有3个NUnit测试程序集正在测试为Kentico CMS平台编写的自定义代码。我可以在Visual Studio(使用NUnit适配器)和NUnit控制台运行器中运行这些测试。我现在正在尝试在我的TeamCity构建服务器上运行那些测试。我已经安装了NUnit运行程序,它返回了一些结果,但是我发现并不是所有的测试都在执行。在一种情况下,我的一个装配体报告没有测试夹具,尽管我知道确实存在。

系统详细信息

  • TeamCity v10.0.5(内部版本42677)
  • Nunit 3.8.1
  • Nunit.ConsoleRunner 3.8.0
  • 肯蒂科11
    • Kentico.Libraries-11.0.35
    • Kentico.Libraries.Tests-11.0.35

问题1

我有一个名为gto.ecommerce.core.tests.dll的测试程序集。当它在TeamCity服务器上运行时,我得到以下输出(直接从NUnit控制台运行器获得):

NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\gto.ecommerce.core.tests\bin\Build\gto.ecommerce.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\packages\NUnit.ConsoleRunner.3.8.0\tools
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 6, Passed: 6, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:16:53Z
    End time: 2019-01-15 10:16:54Z
    Duration: 1.219 seconds

Results (nunit3) saved as TestResult.xml

这些与在执行TeamCity构建步骤时返回的结果完全相同。

但是,这里应该进行更多测试。如果将TeamCity构建的二进制文件复制到本地计算机并运行相同的NUnit控制台运行程序命令,则会得到以下结果:

NUnit Console Runner 3.8.0 
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.14393.0
  CLR Version: 4.0.30319.42000

Test Files
    C:\temp\gto-gtoengineering\TeamCity\gto.ecommerce.core.tests\Build\gto.ecommerce.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: Z:\
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 33, Passed: 33, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:19:31Z
    End time: 2019-01-15 10:19:35Z
    Duration: 4.494 seconds

Results (nunit3) saved as TestResult.xml

注意我的机器怎么说,总共有33个测试,这是正确的数字。

问题2

如果我在我拥有的另一个程序集rwy.common.core.tests上运行相同的场景,那么这是TeamCity服务器的结果:

NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dll


Errors, Failures and Warnings

1) Invalid : D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dl
l
Has no TestFixtures

Run Settings
    DisposeRunners: True
    WorkDirectory: D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\packages\NUnit.ConsoleRunner.3.8.0\tools
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:21:16Z
    End time: 2019-01-15 10:21:17Z
    Duration: 1.046 seconds

Results (nunit3) saved as TestResult.xml

但是再次,我将相同的二进制文件复制到本地计算机上,并得到以下结果:

NUnit Console Runner 3.8.0 
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.14393.0
  CLR Version: 4.0.30319.42000

Test Files
    C:\temp\gto-gtoengineering\TeamCity\rwy.common.core.tests\Build\rwy.common.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: Z:\
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 20, Passed: 20, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:22:55Z
    End time: 2019-01-15 10:23:00Z
    Duration: 4.242 seconds

Results (nunit3) saved as TestResult.xml

肯定应该有20个测试,但是在TeamCity服务器上运行时,它说找不到任何测试。

摘要

那么,有谁知道为什么运行我假定为相同版本的NUnit和控制台运行程序的两台不同的机器会产生截然不同的结果?除了Windows,我看不到任何版本号上的明显差异,但是我认为应该使用相同的.NET框架。

我确实使用了大量的TestCaseSourceTestFixtureSource属性来增加参数化测试的测试数量,但是我不确定这是原因-如果它可以在一台机器上运行,则无法看看为什么另一个会有所不同。

1 个答案:

答案 0 :(得分:0)

经过大量跟踪,我发现原因不是NUnit或TeamCity,而是我使用Kentico中的库,特别是我使用Kentico的CMS.Tests库来帮助单元测试我的自定义Kentico代码为explained here

因此,我已经更新了我的问题,使其更加针对Kentico,并将在下面提供解决此问题的解决方案。

确定根本原因

经过一番互联网搜索,我发现NUnit控制台运行程序具有--trace command line option。通过传入--trace=Verbose,我可以将跟踪文件写入到控制台运行程序的工作目录集中,然后比较两台计算机。我的本地开发机器显示正确找到了所有装置,但是TeamCity服务器将生成跟踪文件,其输出类似于以下内容:

InternalTrace: Initializing at level Debug
14:41:48.559 Debug [ 5] DefaultTestAssemblyBuilder: Loading D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dll in AppDomain domain-
14:41:48.570 Debug [ 5] DefaultTestAssemblyBuilder: Examining assembly for test fixtures
14:41:48.579 Debug [ 5] DefaultTestAssemblyBuilder: Found 12 classes to examine
14:41:48.691 Error [ 5] DefaultTestAssemblyBuilder: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.IsCustomAttributeDefined(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Int32 attributeCtorToken, Boolean mustBeInheritable)
   at System.Reflection.CustomAttribute.IsDefined(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit)
   at NUnit.Framework.Internal.Reflect.GetMethodsWithAttribute(Type fixtureType, Type attributeType, Boolean inherit)
   at NUnit.Framework.Internal.TestFixture..ctor(ITypeInfo fixtureType, Object[] arguments)
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo)
   at NUnit.Framework.Api.DefaultTestAssemblyBuilder.GetFixtures(Assembly assembly, IList names)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

我不了解的部分是对Microsoft.VisualStudio.QualityTools.UnitTestFramework的引用,因为它与NUnit无关-这是MSTest库,我的项目中根本没有MSTest单元测试。

那是我意识到这是一个Kentico问题的原因-因为被引用来帮助编写fake Info objects and providers的Kentico CMS.Tests.dll对于NUnit MSTest都适用。

在开发计算机Microsoft.VisualStudio.QualityTools.UnitTestFramework上安装了Visual Studio,如this question中所述,这意味着每次运行测试都没有问题-可以在系统上找到相关的DLL。但是,除非明确安装,否则它在构建服务器上永远不可用。

解决方案

为解决这个问题,我遵循了this advice

  1. Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll复制到我的解决方案中,并将其提交给我的仓库。
  2. 在我所有的测试项目中都包含对此程序集的引用,以确保Copy Local为真。

通过简单地引用此DLL,构建会将其与所有其他依赖项一起复制到bin文件夹中,然后可以在无法全局使用的环境中使用。