How confident I am with my code?

With smart frameworks like XAF it is inevitable that you will create applications one would except they developed by a whole team and not from just one person. A side effect of rich/large applications is that it is impossible to determine if all parts of the application function unless you have tests.

Enter AutoTest from EasyTest

XAF comes together with EasyTest a functional testing framework. EasyTest has an undocumented command the AutoTest. In short  the AutoTest command will go through all your Navigation Items and for all related views will execute the New action if available and open/close the related detailview.

So, lets talk by example. In eXpandFramework there are 38 Demos with a large number of views, each one demonstrating certain features. By using the AutoTest command we can automate the opening of all views therefore we can expect a code coverage larger than 80%.

Following is the RunEasyTests target used in eXpandFramework build script (Xpand.build).

  <Target Name="RunEasyTests" >
    <MSBuild Projects="@(EasyProjects)" Targets="Build" Properties="Configuration=EasyTest" />
    <CallTarget Targets="EasyTestUpdateDBTimeout" />
    <Delete Files="@EasyTestLogs"/>
    <CreateItem Include="@(EasyTestReqs)" AdditionalMetadata="CopiedToDir=%(EasyTests.RelativeDir);" >
      <Output ItemName="EasyTestReqsToDelete" TaskParameter="Include"/>
    </CreateItem>
    <Copy SourceFiles="@(EasytestReqs)" DestinationFolder="%(EasyTests.RelativeDir)"/>
    <Exec ContinueOnError="true"
          WorkingDirectory="%(EasyTests.RelativeDir)"
          Command='%(EasyTests.RelativeDir)\ProcessAsUser.exe $(TestExecutorName) %(EasyTests.Filename).ets'>
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode"/>
    </Exec>
    <Delete Files="%(EasyTestReqsToDelete.CopiedToDir)%(EasyTestReqsToDelete.Filename)%(EasyTestReqsToDelete.Extension)"/>
    <Error Text="Login failed" Condition="'$(ErrorCode)' == '255'" />
    <CallTarget Targets='PrintEasyTestLogs' ></CallTarget>
  </Target>

Next we will discuss in detail each line of the RunEasyTests target.

<MSBuild Projects="@(EasyProjects)" Targets="Build" Properties="Configuration=EasyTest" />

First we need to build all demos under the EasyTest configuration where the required EasyTest components are functional.

<CallTarget Targets="EasyTestUpdateDBTimeout" />

In the above line we call the EasyTestUpdateDBTimeout target that will run DBUpdater for all demos that have long operations in their ModuleUpdaters. We need to explicitly execute the DBUpdater because there is a threshold in time between each EasyTest command so the Login action may fail if moduleupdater takes more than 25sec to complete. 

<Delete Files="@EasyTestLogs"/>

This command will delete all EasyTest logs to make sure that our build script will not pick up any previous failed tests.

<Copy SourceFiles="@(EasytestReqs)" DestinationFolder="%(EasyTests.RelativeDir)"/>

This will copy a set of required files for EasyTest like the TestExecutor, XpandTestAdapters etc in the same path as the test. In the build server assemblies are not in the GAC so by putting them in the same path we minimize assembly binding errors due to version conflicts.

<Exec ContinueOnError="true"           WorkingDirectory="%(EasyTests.RelativeDir)"           Command='%(EasyTests.RelativeDir)\ProcessAsUser.exe $(TestExecutorName) %(EasyTests.Filename).ets'>

This line executes the TestExecutor for all EasyTests, It uses the ProcessAsUser.exe because the build script is actually executed from our build server service using the System Account which has no windows session therefore useful EasyTest features like screenshots are not available. In general we do not want to mess with the system account in order to make our tests run. So the ProcessAsUser reads a username and password from a registry key and by using an RDC connection (Windows Server) creates a new session and executes the tests there under the specified user security privileges. Finally because we are interested to execute all tests without failing the build to fail we set ContinueOnError to true.

<Delete Files="%(EasyTestReqsToDelete.CopiedToDir)%(EasyTestReqsToDelete.Filename)%(EasyTestReqsToDelete.Extension)"/>

This line will delete all EasyTest required files copied in previous steps

<Error Text="Login failed" Condition="'$(ErrorCode)' == '255'" />

If the ProcessAsUser returns 255 then we fail the build with the Login failed message.

<CallTarget Targets='PrintEasyTestLogs' ></CallTarget>

The PrintEasyTestLogs target will go through all EasyTest logs and for any entry with non passed Result it will fail the build with the related error message from the EasyTest xml log.

  <Target Name="PrintEasyTestLogs" >
    <MSBuild.ExtensionPack.Xml.XmlFile
      TaskAction="ReadElements" File="%(EasyTestLogs.FullPath)" Condition="'@(EasyTestLogs->Count())'!='0'"
      XPath="/Tests/Test[@Result='Warning' or @Result='Failed']" ReadChildrenToMetadata="true">
      <Output TaskParameter="Elements" ItemName="Test"/>
    </MSBuild.ExtensionPack.Xml.XmlFile>
    <Error Text="%0D%0AApplicationName: %(Test.ApplicationName)%0D%0AError: %(Test.Error)" Condition="'@(Test->Count())'!='0'"  ContinueOnError="True"></Error>
  </Target>

A big thanks this time to the AutoTest command that provided 80% code coverage for our community framework. I am sure AutoTest will be extended to provide even smarter automated tested in future versions. If you got into that trouble and have interesting ideas to share or contribute for AutoTestV2 please use our community framework forums.