Export results in Xunit XML format

[fedepell]

Connecting also to #7 it would be nice if the framework could output the results in standard Xunit XML format. This way other integration software (ie. Jenkins you mention) can then easily manage and display the results (ie. for Jenkins with the Junit Jenkins plugin).

I believe this pointers can be useful for the format:
https://gist.github.com/erikd/4192748
https://gist.github.com/nil4/7a3cd9c23835ec6b126fe588e836a2e8
https://github.com/windyroad/JUnit-Schema

[sagatowski]

Excellent suggestion! Will move this to a new milestone.

[Aliazzzz]

Dear Jakob,

A file factory could be very useful for generating different types of output, like Xunit XML, JSON or whatever.

I did some research on the this subject and the "Abstract Factory" pattern could be very suitable to implement the filefactory.

However, there are very important details to consider when designing the filefactory as an "Abstract Factory". I found some very interesting information on this topic offline (in some .chm files) and googled for it but I had no luck in finding it quickly. It is written for CODESYS point of view, but I bet it is interoperable with TwinCAT.

In CODESYS there are two possibilities of dynamically creating an FB instance:

A.) By using the operators __NEW and __DELETE and a MemoryPool which has to be parameterized for the respecting application via CODESYS.

However this method has a significant disadvantage:
The MemoryPool is defined once for the application and all the referenced libraries within the application. But it is not possible to determine how much storage space each library is allowed to claim in the MemoryPool.
The coincidental utilization of __NEW and __DELETE with different sized FB instances causes a fragmentation of the MemoryPool. Because of that it is not possible to securely install any system for a non-stop operation throughout the entire year. An occasional reboot of the facility would be necessary to re-establish the MemoryPool.
That is why the MemoryPool and the operators __New and __Delete can only be recommended for applications in very special cases. Especially when dealing with libraries this method should not be used!

B.) By using the library FBFactory there is a second and more reliable possibility to meet the demands of dynamically creating an FB instance.
With the help of the FBFactory Library a static pool (which can be extended into a dynamic pool if needed) for each factory is defined in advance. The management of the memory space is organized in way which makes sure that no fragmentation is possible. Therefore there is no obstacle to a non-stop operation of the system.

FBFactory Library can be used as a base for building factories. In order to get more information about how to use this library for building factories please download the Test Project which you will find in the documentation of the FBFactory Library

So, I'd like to try and implement the filefactory using the 'Abstract Factory' pattern via method B.
If everything works out, it will be usable in TcUnit as well as CfUnit and function as a central filefactory distribution point.

Please let me know what you think of this proposition.

Aliazzz

[sagatowski]

This was exactly the pattern I was thinking of for this issue. Stefan Henneken has written an excellent blog post regarding this design pattern:
https://stefanhenneken.wordpress.com/2014/11/16/iec-61131-6-abstract-factory-english/

Which is exactly what I planned to use for this issue.

Although considering that this library will be included as a library in other peoples code, and there seem to be an allergy (for good reasons) against dynamic memory allocation, I'm all in for the file factory though with pre-allocated memory.

Edit: Please post an update here when you have done an implementation of the Abstract Factory pattern via method B and then I can hopefully use your code!

[Aliazzzz]

I integrated an XML writer into TcUnit.
I think it should suffice to produce files like this;

<?xml version="1.0" encoding="UTF-8" ?> 
   <testsuites id="20140612_170519" name="New_configuration (14/06/12 17:05:19)" tests="225" failures="1262" time="0.001">
      <testsuite id="codereview.cobol.analysisProvider" name="COBOL Code Review" tests="45" failures="17" time="0.001">
         <testcase id="codereview.cobol.rules.ProgramIdRule" name="Use a program name that matches the source file name" time="0.001">
            <failure message="PROGRAM.cbl:2 Use a program name that matches the source file name" type="WARNING">
WARNING: Use a program name that matches the source file name
Category: COBOL Code Review – Naming Conventions
File: /project/PROGRAM.cbl
Line: 2
      </failure>
    </testcase>
  </testsuite>
</testsuites>

[Aliazzzz]

Hi,

Just been searching for the right file markup as I cannot validate it easily;
It seems the following primer is very handy! We just need to verify this.

https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports

<?xml version="1.0" encoding="UTF-8"?>
<testsuites disabled="" errors="" failures="" name="" tests="" time="">
    <testsuite disabled="" errors="" failures="" hostname="" id=""
               name="" package="" skipped="" tests="" time="" timestamp="">
        <properties>
            <property name="" value=""/>
        </properties>
        <testcase assertions="" classname="" name="" status="" time="">
            <skipped/>
            <error message="" type=""/>
            <failure message="" type=""/>
            <system-out/>
            <system-err/>
        </testcase>
        <system-out/>
        <system-err/>
    </testsuite>
</testsuites>

Some of these items can occur multiple times:

• There can only be one testsuites element, since that’s how XML works, but there can be multiple testsuite elements within the testsuites element.
• Each properties element can have multiple property children.
• Each testsuite element can have multiple testcase children.
• Each testcase element can have multiple error, failure, system-out, or system-err children.

[Aliazzzz]

progress so far (small code preview), the 'end' result will be far larger;

(*
	This function block reports the results from the tests into a xUnit compatible XmlFile

*)
FUNCTION_BLOCK FB_XmlFileFormatter IMPLEMENTS I_TestResultFormatter

~~~~

Busy := TRUE;
IF NumberOfFailedTestCases > 0 THEN
    Xml.writeDocumentHeader(Header := '<?xml version=\"1.0\" encoding=\"UTF-8\"?>');
    xml.NewComment('xUnit xml created by TcUnit CI-CD (e.g. Jenkins)');
    xml.NewTag('testsuites');
    FOR SuiteCounter := 1 TO NumberOfTestSuites BY 1 DO
        xml.NewTag('testsuite');
        NumberOfFailedTestsInCurrentSuite :=  GVL_TcUnit.TestSuiteAddresses[SuiteCounter]^.GetNumberOfFailedTests();
        Xml.NewPara('errors', UINT_TO_STRING(NumberOfFailedTestsInCurrentSuite));
        NumberOfTestsInCurrentSuite := GVL_TcUnit.TestSuiteAddresses[SuiteCounter]^.GetNumberOfTests();
        Xml.NewPara('tests', UINT_TO_STRING(NumberOfTestsInCurrentSuite) );
        FOR TestCounter := 1 TO NumberOfTestsInCurrentSuite BY 1 DO
            IF GVL_TcUnit.TestSuiteAddresses[SuiteCounter]^.Tests[TestCounter].IsFailed() THEN
                  ;
            END_IF
        END_FOR
    END_FOR
END_IF
Busy := FALSE;

Current log

<Entry>| ======================================</Entry>
<Entry>| Failed tests : 92</Entry>
<Entry >| Successful tests: 112</Entry>
<Entry >| Tests: 204</Entry>
<Entry >| Test suites: 17</Entry>
<Entry >| ==========TESTS FINISHED RUNNING==========</Entry>
<Entry >| ======================================</Entry>
<Entry >| write: /XmlControl/output.xml succes</Entry>
<Entry ">| ========== WRITING TEST REPORT ==========</Entry>

This yields the following file when running the verifier project
The intermediate result looks very promising :-)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!-- xUnit xml created by CfUnit CI-CD (e.g. Jenkins) -->
<testsuites>
<testsuite errors="22" tests="44"/>
<testsuite errors="2" tests="4"/>
<testsuite errors="1" tests="1"/>
<testsuite errors="0" tests="1"/>
<testsuite errors="38" tests="57"/>
<testsuite errors="1" tests="1"/>
<testsuite errors="21" tests="42"/>
<testsuite errors="1" tests="1"/>
<testsuite errors="1" tests="24"/>
<testsuite errors="1" tests="1"/>
<testsuite errors="3" tests="3"/>
<testsuite errors="1" tests="1"/>
<testsuite errors="0" tests="3"/>
<testsuite errors="0" tests="0"/>
<testsuite errors="0" tests="1"/>
<testsuite errors="0" tests="20"/>
<testsuite errors="0" tests="0"/>
</testsuites>

PS I really dont care for beautifying the xml with tabs etc, as long as the parser accepts the file.

[sagatowski]

Note to self, I'm using this template: https://wiki.yoctoproject.org/wiki/QA/xUnit_XML_Template that I will be using in the TcUnit-Runner + Jenkins plugin.
I've marked and selected the XML that the first version will support in the attached file.