Enable on-target execution of mbed TLS unit tests #930
Conversation
|
In case Jira ticket is missed. Please review @sbutcher-arm @andresag01 @yanesca @RonEld |
|
@sbutcher-arm @andresag01 @yanesca @gilles-peskine-arm On target crypto primitive testing status:
|
|
For detailed log see attached file. Summary is in the bottom |
|
Config used for above tests https://gist.github.com/mazimkhan/b892d2e7d36e7e08faeef367b4498b39 |
dee40a7 to
0581b9f
Compare
|
Travis build now passes. |
tests/Makefile
Outdated
| func.test_suite_x509parse := test_suite_x509parse | ||
| func.test_suite_x509write := test_suite_x509write | ||
| func.test_suite_xtea := test_suite_xtea | ||
| func.test_suite_version := test_suite_version | ||
|
|
There was a problem hiding this comment.
Would now be a good time to rename the test suites to avoid having . in the base names? That makes some scripts more complicated (base names are not “without extension” since they contain a dot) and it might cause problems with static analysis tools that I'd like to run one day.
There was a problem hiding this comment.
I second the idea. However, this PR is already big enough. This can be done separately.
There was a problem hiding this comment.
@mazimkhan - please raise this idea on GitHub so it doesn't get forgotten.
There was a problem hiding this comment.
Filed as #1020 @mazimkhan
| perl scripts/run-test-suites.pl | ||
|
|
||
| test: check | ||
|
|
||
| # Create separate targets for generating embedded tests. | ||
| EMBEDDED_TESTS := $(addprefix embedded_,$(APPS)) |
There was a problem hiding this comment.
Note that function calls and secondary expansion are GNU make features. We publicly claim that we only use portable make constructs, although this is already wrong. But we might not want to make our dependency on GNU make even bigger, if we decide that we want to remove this dependency in the future.
There was a problem hiding this comment.
Yes these are GNU extensions. But we are using GNU make in all our supported dev environments like: Windows mingw, linux, FreeBSD gmake etc. It is very hard to get a decent Makefile without these features.
tests/scripts/mbedtls_test.py
Outdated
| @@ -0,0 +1,273 @@ | |||
| """ | |||
| mbed SDK | |||
| Copyright (c) 2011-2013 ARM Limited | |||
There was a problem hiding this comment.
Wrong year, this is not the mbed SDK, and a short description of the purpose of the file would be useful here.
tests/scripts/mbedtls_test.py
Outdated
| self.__parse(f) | ||
|
|
||
| @staticmethod | ||
| def __escaped_split(str, ch): |
There was a problem hiding this comment.
That method feels overly long for what it does. Since we're keeping all backslashes, you can split on : and then join each string that matches r'(?!\\)(?:\\\\)*\\\Z'with its successor.
There was a problem hiding this comment.
Is this a python regex? I tried testing it and it didn't match:
>>> print re.search(r'(?!\\)(?:\\\\)*\\\Z', 'yahoo\\')
None
There was a problem hiding this comment.
r'(?!\\)?(?:\\\\)*\\\Z' Putting a ? on reverse lookup makes it work. But then it also matches escaped escape.
>>> print re.search(r'(?!\\)?(?:\\\\)*\\\Z', 'yahoo\\\\')
<_sre.SRE_Match object at 0x00000000028A9D98>
There was a problem hiding this comment.
Complexity with the regex and still having to manage a state:'escaped' made me write it in a simple way.
tests/scripts/mbedtls_test.py
Outdated
| """ | ||
| """ | ||
| line = file.readline().strip() | ||
| while line: |
There was a problem hiding this comment.
for line in file:
There was a problem hiding this comment.
There are multiple readline calls in one iteration. This is to parse a test in one go without having to manage states.
There was a problem hiding this comment.
So call file.next() inside the loop.
tests/scripts/mbedtls_test.py
Outdated
| assert len(hex_str) % 2 == 0, "HEX parameter len should be mod of 2: %s" % hex_str | ||
| b = bytearray() | ||
|
|
||
| for i in xrange(len(hex_str) / 2): |
There was a problem hiding this comment.
binascii.unhexlify(hex_str[1:-1])
tests/scripts/mbedtls_test.py
Outdated
| b += hb | ||
| return b | ||
|
|
||
| def run_next_test(self): |
There was a problem hiding this comment.
Please package run_test(name, function, deps, args) as a method that can be called independently of any callback structure.
tests/scripts/mbedtls_test.py
Outdated
| ValueError("Result should return error number. Instead received %s" % value) | ||
| return 0 | ||
|
|
||
| @event_callback('GO') |
There was a problem hiding this comment.
Why use this callback structure? It makes the control flow hard to follow, and makes it hard to e.g. choose to run a selection of tests, abandon the tests midway, etc.
There was a problem hiding this comment.
The callback flow is actually forced by htrun. That is tool that receives k,v pairs from target and dispatches them into this class as events.
tests/scripts/generate_code.py
Outdated
| @@ -0,0 +1,649 @@ | |||
| """ | |||
| mbed SDK | |||
| Copyright (c) 2017-2018 ARM Limited | |||
There was a problem hiding this comment.
Wrong year and not the mbed SDK
tests/scripts/generate_code.py
Outdated
|
|
||
|
|
||
| """ | ||
| Generates code in following structure. |
There was a problem hiding this comment.
That structure doesn't match my reading of the code below
tests/suites/desktop_test.function
Outdated
| int digits = 1; | ||
| int hex = 0; | ||
|
|
||
| for( i = 0; i < strlen( str ); i++ ) |
There was a problem hiding this comment.
No need for this loop, call strtol with a non-null second parameter and check that it's reached the end of the string.
There was a problem hiding this comment.
This file is made out from main_test.function file as part of separating desktop specific and common dispatch code.
We can enhance this code as part of separate PR.
tests/suites/desktop_test.function
Outdated
| " Options:\n" \ | ||
| " -v | --verbose Display full information about each test\n" \ | ||
| " -h | --help Display this information\n\n", \ | ||
| argv[0], \ |
There was a problem hiding this comment.
Wishlist: an option to filter test cases, preferably by regex (like we have ssl-opt.sh -f INCLUDE_REGEX -e EXCLUDE_REGEX).
There was a problem hiding this comment.
Thanks for creating ticket in the issue tracker. It will be implemented as separate PR.
tests/suites/desktop_test.function
Outdated
| mbedtls_exit( 2 ); | ||
| } | ||
| else | ||
| total_errors++; |
There was a problem hiding this comment.
Why is no message printed if a test fails? I find this annoying with the current test suite, it makes it hard to locate failing test cases in the output.
There was a problem hiding this comment.
This is legacy code. It is just a counter to track number of errors. Errors are tracked on variable test_errors. Checked here https://github.com/ARMmbed/mbedtls/pull/930/files/ad84890775d2c35e1d7899b76c12b7894f51fff2#diff-d642c5872a9b41897dee6c4800e899feR594.
Message is printed at the place where test_errors is incremented. Here https://github.com/ARMmbed/mbedtls/pull/930/files/ad84890775d2c35e1d7899b76c12b7894f51fff2#diff-5c77e3830486ba0964d963b0b85814ddR407
tests/suites/embedded_test.function
Outdated
| */ | ||
| #define INCR_ASSERT(p, start, len, step) do \ | ||
| { \ | ||
| assert( p >= start ); \ |
There was a problem hiding this comment.
Please put parentheses around macro arguments: assert( ( p ) >= ( start ) ) etc.
tests/suites/embedded_test.function
Outdated
| * | ||
| * \return count of hex params | ||
| */ | ||
| uint32_t find_hex_count( uint8_t count, uint8_t * data, uint32_t data_len ) |
There was a problem hiding this comment.
Why is the target parsing hex? I'd have expected the host to take care of this.
There was a problem hiding this comment.
It is only parsing received test vectors in hex form. No string to hex conversion on target.
There was a problem hiding this comment.
@mazimkhan There's a hex to string conversion on target. I don't understand why the host doesn't send the data in binary form. We want to minimize the memory usage on the target, and hex takes three times as much until the conversion is finished and the hex data can be freed.
tests/scripts/generate_code.py
Outdated
| :return: | ||
| """ | ||
| exp_code = ''' | ||
| if ( exp_id == {exp_id} ) |
There was a problem hiding this comment.
I'd prefer generating a case block for switch( exp_id ) here. It's more readable and we do have to debug tests sometimes. This requires changing the calling site as well.
tests/scripts/generate_code.py
Outdated
| else: | ||
| noT = '' | ||
| dep_check = ''' | ||
| if ( dep_id == {id} ) |
There was a problem hiding this comment.
Like with exp_id, a switch( dep_id ) would be easier to read.
|
@gilles-peskine-arm Thanks for the useful comments. I have incorporated your comments and replied to some of them. Please review the changes. |
tests/CMakeLists.txt
Outdated
| @@ -24,8 +24,8 @@ function(add_test_suite suite_name) | |||
|
|
|||
| add_custom_command( | |||
| OUTPUT test_suite_${data_name}.c | |||
| COMMAND ${PERL_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl ${CMAKE_CURRENT_SOURCE_DIR}/suites test_suite_${suite_name} test_suite_${data_name} | |||
| DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl mbedtls suites/helpers.function suites/main_test.function suites/test_suite_${suite_name}.function suites/test_suite_${data_name}.data | |||
| COMMAND python scripts/generate_code.py -f suites/test_suite_${suite_name}.function -d suites/test_suite_${data_name}.data -t suites/main_test.function -p suites/desktop_test.function -s suites --help-file suites/helpers.function -o . | |||
There was a problem hiding this comment.
python should surely be replaced with ${PYTHON_EXECUTABLE}? You can't assume you'll hit the intended one on the path.
tests/scripts/mbedtls_test.py
Outdated
| @@ -0,0 +1,296 @@ | |||
| """ | |||
| mbed SDK | |||
There was a problem hiding this comment.
This is not mbed SDK! Please change it to mbed TLS.
tests/scripts/mbedtls_test.py
Outdated
| mbed SDK | ||
| Copyright (c) 2017 ARM Limited | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); |
There was a problem hiding this comment.
The change from Perl to Python is going to break our license scripts. We'll need to fix those first before we merge this.
tests/scripts/mbedtls_test.py
Outdated
| Copyright (c) 2017 ARM Limited | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. |
There was a problem hiding this comment.
We have Apache license here, but no SPDX. Is this the correct header?
tests/suites/desktop_test.function
Outdated
| @@ -0,0 +1,644 @@ | |||
| #line 2 "suites/desktop_test.function" | |||
There was a problem hiding this comment.
Can we change this from desktop_ and embedded_? They don't make sense as names. How about onhost_ and ontarget_
tests/suites/desktop_test.function
Outdated
| @@ -0,0 +1,644 @@ | |||
| #line 2 "suites/desktop_test.function" | |||
There was a problem hiding this comment.
You've prefixed the filename with the local path here and in a couple of other places but not everywhere. It needs to be the same everywhere for consistency. Either add the local path into those places, or remove it here and the other places you've added it.
tests/Makefile
Outdated
| @@ -2,7 +2,7 @@ | |||
| # To compile on SunOS: add "-lsocket -lnsl" to LDFLAGS | |||
| # To compile with PKCS11: add "-lpkcs11-helper" to LDFLAGS | |||
|
|
|||
| CFLAGS ?= -O2 | |||
| CFLAGS ?= -g3 #-O2 | |||
There was a problem hiding this comment.
This line looks like a mistake. We've turned on debugging with -g3 which is fair in test code but we've commented out optimisations.
Why comment them out? Either delete them, or leave them in. Commented out code is bad practice.
tests/CMakeLists.txt
Outdated
| @@ -24,8 +24,8 @@ function(add_test_suite suite_name) | |||
|
|
|||
| add_custom_command( | |||
| OUTPUT test_suite_${data_name}.c | |||
| COMMAND ${PERL_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl ${CMAKE_CURRENT_SOURCE_DIR}/suites test_suite_${suite_name} test_suite_${data_name} | |||
| DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl mbedtls suites/helpers.function suites/main_test.function suites/test_suite_${suite_name}.function suites/test_suite_${data_name}.data | |||
| COMMAND python scripts/generate_code.py -f suites/test_suite_${suite_name}.function -d suites/test_suite_${data_name}.data -t suites/main_test.function -p suites/desktop_test.function -s suites --help-file suites/helpers.function -o . | |||
There was a problem hiding this comment.
We're adding use of python to the library. This needs to be updated in all documentation discussing tools and dependencies.
@mazimkhan - please look for documentation in the library, and ask @RonEld to check for Kb or other documentation on the website.
tests/Makefile
Outdated
| func.test_suite_x509parse := test_suite_x509parse | ||
| func.test_suite_x509write := test_suite_x509write | ||
| func.test_suite_xtea := test_suite_xtea | ||
| func.test_suite_version := test_suite_version | ||
|
|
There was a problem hiding this comment.
@mazimkhan - please raise this idea on GitHub so it doesn't get forgotten.
tests/CMakeLists.txt
Outdated
| @@ -24,8 +24,8 @@ function(add_test_suite suite_name) | |||
|
|
|||
| add_custom_command( | |||
| OUTPUT test_suite_${data_name}.c | |||
| COMMAND ${PERL_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl ${CMAKE_CURRENT_SOURCE_DIR}/suites test_suite_${suite_name} test_suite_${data_name} | |||
| DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_code.pl mbedtls suites/helpers.function suites/main_test.function suites/test_suite_${suite_name}.function suites/test_suite_${data_name}.data | |||
| COMMAND python scripts/generate_code.py -f suites/test_suite_${suite_name}.function -d suites/test_suite_${data_name}.data -t suites/main_test.function -p suites/desktop_test.function -s suites --help-file suites/helpers.function -o . | |||
There was a problem hiding this comment.
By adding use of python, we're making big changes to how we build which could impact any of our supported platforms. Has this been tested on Windows and FreeBSD? As well as Linux?
There was a problem hiding this comment.
Yes these changes are tested on Windows, Linux and FreeBSD.
|
@sbutcher-arm I have incorporated all your code review comments except the documentation ones that I will talk to @RonEld . Can you please elaborate on following:
|
tests/scripts/generate_code.py
Outdated
| pass | ||
|
|
||
|
|
||
| class FileWrapper(file): |
There was a problem hiding this comment.
Isn't FileWrapper(file) basically enumerate(file)?
There was a problem hiding this comment.
ignore my previous comment. yes it can be used like this:
def three_to_five(e):
for i, l in e:
print "three_to_five (%d) %s" % (i, l)
if l == "five\n":
break
with open("123.txt", "r") as f:
e = enumerate(f)
for i, l in e:
print "(%d) %s" % (i, l)
if l == "three\n":
three_to_five(e)
There was a problem hiding this comment.
I will try it in this PR and test.
There was a problem hiding this comment.
FileWrapper adds line number enumeration and still allows access to the file object. Like file.name. enumerate enumerates the file iterable but no access to the iterable for fields like name. Idea is is to pass around a single object with file contents and line number and any other attribute like name.
I will like to keep it. Unless any strong need to change it.
tests/scripts/generate_code.py
Outdated
| return ('!', dep[1:]) if dep[0] == '!' else ('', dep) | ||
|
|
||
|
|
||
| def gen_deps(deps): |
There was a problem hiding this comment.
Just a nitpick but I prefer the one-line format of gen_deps_one_line because the multi-line format makes it harder to see that there aren't complex #else imbrications. Why have both, anyway?
There was a problem hiding this comment.
The one liner is for cases where we have #else. Otherwise this one is used. It nicely annotates #endif with comments to match the corresponding #if. I also like the one liner. But I kept it as it keeps lines short.
sprintf( (char *) buf, "%s\r\n", base ); Above code generates Wformat-overflow warning since both buf and base are of same size. buf should be sizeof( base ) + characters added in the format. In this case format 2 bytes for "\r\n".
2a61854 to
58e9c18
Compare
|
retest |
|
Release job ran into some issues. https://jenkins-internal.mbed.com/job/mbedtls-release/628/flowGraphTable/ @mazimkhan is investigating |
The mbed-os examples build failure is because of AES-XTS ALT implementation not providing required XTS context structure. This issue is dealt with in PR ARMmbed/mbed-os#7243 that is merged and set to release as part of 5.10. 5.10 is not yet released. Hence, it is known issue to be fixed after 5.10 is released in mbed-os examples. |
|
The other failures, from K64F and F429ZI, are stack overflow errors from attempting to benchmark ECDSA (tracked as ARMmbed/mbed-os-example-tls#184) |
| @@ -310,7 +311,7 @@ void entropy_nv_seed( char *read_seed_str ) | |||
| unsigned char entropy[MBEDTLS_ENTROPY_BLOCK_SIZE]; | |||
| unsigned char buf[MBEDTLS_ENTROPY_BLOCK_SIZE]; | |||
| unsigned char empty[MBEDTLS_ENTROPY_BLOCK_SIZE]; | |||
| unsigned char read_seed[MBEDTLS_ENTROPY_BLOCK_SIZE]; | |||
| unsigned char read_seed->x[MBEDTLS_ENTROPY_BLOCK_SIZE]; | |||
There was a problem hiding this comment.
This breaks compilation, it's not compiled with default configuration but once the required defs are enabled this breaks.
|
@itayzafrir thanks for finding the issue in Jenkins CI is stuck due to some Jenkins issue. But it has passed the travis CI. |
This PR enhances the existing unit test framework for execution on target.
Steps to test:
Desktop
Steps to test on desktop are not changed. Hence:
mbed-os
mbedgt -V -n features-mbedtls-tests-mbedtls-test_suite_aes*
Build requires greentea client changes submitted in PR ARMmbed/mbed-os#4364.