TEST: update python script to enable examples smoke test #10521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

0xc0170 merged 5 commits into ARMmbed:master from jamesbeyond:example_test

May 21, 2019

Contributor

jamesbeyond commented May 2, 2019

Description

The goal of this PR is to enable examples smoke test in CI

the new scripts will check examples.json if contains 'test', 'compare_log', 'baud_rate' keys
the new scripts will dump test_spec.json test in examples compiled successfully

The auto generated test_spec.json can be consumed by GreenTea.
This PR is related with ARMmbed/mbed-os-tools#162

examples.json is not required to be changed at moment.
later on when test cases in each example are ready, corresponding example then will need to be updated.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

ciarmcom requested review from OPpuolitaival and a team

May 2, 2019 13:00

Member

ciarmcom commented May 2, 2019

@jamesbeyond, thank you for your changes.
@OPpuolitaival @ARMmbed/mbed-os-tools @ARMmbed/mbed-os-maintainers please review.

ciarmcom added the needs: review label


          TEST: update python script to enable example smoke test

b2611fb

 * it will check examples.json if contains 'test', 'compare_log', 'baud_rate' keys
 * it will dump test_spec.json test in examples compiled successfully

jamesbeyond force-pushed the example_test branch from 226aa33 to b2611fb Compare

May 2, 2019 14:27

bridadan reviewed

View reviewed changes

Contributor

bridadan left a comment

In general this all seems pretty reasonable! I have a few suggestions below,

tools/test/examples/examples_lib.py Outdated

+                  """save the given json data to test_spec.json"""
+                  print ("Dumping json test_specs file {}".format(name))
+                  with open(name, 'w') as outfile:
+                      json.dump(json_spec, outfile , indent=4)

Contributor

bridadan May 2, 2019

I believe we have an existing function that you can use: https://github.com/ARMmbed/mbed-os/blob/master/tools/utils.py#L184

Contributor Author

jamesbeyond May 10, 2019

Thanks for the review, all your comments been addressed

tools/test/examples/examples_lib.py Outdated

+                                      if image:
+                                          image_info = [{"binary_type": "bootable","path": normpath(join(name,image)),"compare_log":log}]
+                                      else:
+                                          print ("Warning: could not found built image for example %s" % name)

Contributor

bridadan May 2, 2019

nit pick on the wording of the warning:

Suggested change

      
                                        print ("Warning: could not found built image for example %s" % name)
          
                                        print ("Warning: could not find built image for example %s" % name)

tools/test/examples/examples_lib.py

+                                  print ("\n#### STDOUT ####\n%s\n#### STDERR ####\n%s\n#### End of STDOUT/STDERR ####\n" % (std_out,std_err))
+                                  if test_example:
+                                      log = example['compare_log'].pop(0)

Contributor

bridadan May 2, 2019

Any reason this needs to be a pop vs the following:

Suggested change

      
                                    log = example['compare_log'].pop(0)
          
                                    log = example['compare_log']

Contributor Author

jamesbeyond May 10, 2019

Here pop() is used because the value of compare_log will be a list of logs files.
The reason for that is because some of the example repos contains more than one example, e.g. the mbed-os-example-tls and mbed-os-example-ble. So the each log file need to match with
the individual sub-example. So we need to pop the file out of list regardless the compilation for each sub-example pass of fail

Contributor

mark-edgeworth May 15, 2019

May I request that this comment is added to the code please.

tools/test/examples/examples_lib.py Outdated

+                                              test_json['builds'][test_group]['tests'][name]={"binaries":image_info}
+                                      else:
+                                          print("Warning: Test for %s will not be generated." % name)
+                                          print("One or more of 'test' 'baud_rate' and 'compare_log' keys are missing from the json file\n")

Contributor

bridadan May 2, 2019

nit pick on wording, missing some commas:

Suggested change

      
                                        print("One or more of 'test' 'baud_rate' and 'compare_log' keys are missing from the json file\n")
          
                                        print("One or more of 'test', 'baud_rate', and 'compare_log' keys are missing from the example config json file\n")

tools/test/examples/examples_lib.py Outdated Show resolved Hide resolved

0xc0170 added needs: work and removed needs: review labels

0xc0170 approved these changes

View reviewed changes

Contributor

0xc0170 left a comment

Besides above suggestions, LGTM

OPpuolitaival suggested changes

View reviewed changes

Contributor

OPpuolitaival left a comment

Need documenting at least

tools/test/examples/examples_lib.py Outdated

                       successes = []
                       compiled = True
                       pass_status = True
+                      if example.has_key('test') and example.has_key('baud_rate') and example.has_key('compare_log'):

Contributor

OPpuolitaival May 3, 2019 •

edited

Loading

This is true for this kind of json object:
{
"name": "Something",
"test": False,
"baund_rate": "invalid",
"compare_log": null
}

Why it need to have "test" key even if the value is not documented or does not even matter?

These values really need to be documented. Actually same problem with whole examples.json. @adbridge do you know any documentation for that json file?

Contributor

adbridge May 3, 2019

No not that i am aware of....

Contributor Author

jamesbeyond May 3, 2019

To me, the test key is just a on/off switch. similar to the compile key, just following the conventions.
It can be used, when we have the compare_log and baud_rate ready, but choose not run the test for some reasons. (e.g. the example requires some special HW, but that's not available on the RaaS, or there are some breaking change happening, we need to turn off tests for some examples temperately )

Contributor

OPpuolitaival May 6, 2019

@jamesbeyond then the condition need to check the value of example['test'] alse. For example:
if example.has_key('test') and example['test'] and example.has_key('baud_rate') and example.has_key('compare_log'):

Contributor Author

jamesbeyond May 10, 2019

Yes, I was checking that condition in later code, after reviewing, I think what you suggested would be more make scene. So this is fixed


          fix issues base on the review

71d7970

Contributor Author

jamesbeyond commented May 10, 2019

all comments been addressed, please review again @bridadan @OPpuolitaival

bridadan approved these changes

View reviewed changes

Contributor

bridadan left a comment

Thanks for addressing my comments!

Contributor

0xc0170 commented May 13, 2019

While waiting for final approvals, CI started

mbed-ci commented May 13, 2019

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 1
Build artifacts

0xc0170 added needs: review and removed needs: work labels

Contributor

0xc0170 commented May 15, 2019

@OPpuolitaival @adbridge Happy with this?

PRs to examples are slowly getting integrated

OPpuolitaival approved these changes

View reviewed changes

adbridge reviewed

View reviewed changes

Contributor

adbridge left a comment

It would be better to use the logging module rather than print but I'm not going to block the PR on that. That could always be a downline improvement.


          add a comment based on review

48e1a56

Contributor Author

jamesbeyond commented May 16, 2019

It would be better to use the logging module rather than print but I'm not going to block the PR on that. That could always be a downline improvement.

Yeah, I just following the original script that using the print. I agree logging module would be a better solution. I think we will consider the improvements in separate PR

Contributor

adbridge commented May 17, 2019

@mark-edgeworth are you happy with the comment?

mark-edgeworth approved these changes

View reviewed changes

Contributor

mark-edgeworth left a comment

Yes, that's what I had in mind 👍

Qinghao Shi added 2 commits

May 18, 2019 23:57


          fix a bug when STDOUT has less than 5 lines

3bb0a8a


          fix a bug where base_path was wrong

35eeef5

Contributor Author

jamesbeyond commented May 20, 2019

I think we got good level of reviews, can we have the CI start? @ARMmbed/mbed-os-maintainers

0xc0170 removed the needs: review label

0xc0170 added the needs: CI label

Contributor

0xc0170 commented May 20, 2019

CI started

Contributor Author

jamesbeyond commented May 20, 2019

seems something wrong with checkout stage of the CI? @ARMmbed/mbed-os-maintainers

Contributor

cmonr commented May 20, 2019

Pipeline restarted.
Recently, GitHub has increased the degree of repo clone throttling.

mbed-ci commented May 20, 2019

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 3
Build artifacts

0xc0170 added ready for merge release-version: 5.13.0-rc1 and removed needs: CI labels

0xc0170 merged commit 7d0cc69 into ARMmbed:master

0xc0170 removed the ready for merge label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-version: 5.13.0-rc1