Skip to content

Add integration tests #11276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Aug 28, 2019
Merged

Add integration tests #11276

merged 3 commits into from
Aug 28, 2019

Conversation

jamesbeyond
Copy link
Contributor

Description

During 2019 SiP workshop, we introduced a set of tests as integration testing on storage and network. We asked partner to run these tests on their platforms fix issues if any. Generally these tests received very positive feedback, and we decide to merge the test into mbed-os master

Pull request type

[ ] Fix
[ ] Refactor
[x] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Reviewers

@MarceloSalazar @maclobdell

@ciarmcom ciarmcom requested review from maclobdell, MarceloSalazar and a team August 21, 2019 11:00
@ciarmcom
Copy link
Member

@jamesbeyond, thank you for your changes.
@maclobdell @MarceloSalazar @ARMmbed/mbed-os-maintainers please review.

MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
TESTS/integration/README.md Outdated

## Execute tests

The integration test requires both the connectivity and and storage device to to configured.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor - remove duplicate 'and' and 'to'

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still pending to remove duplicate 'and'

MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
MarceloSalazar
MarceloSalazar reviewed Aug 21, 2019
View reviewed changes
Copy link

@MarceloSalazar MarceloSalazar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good - I've suggested a few minor changes in the readme doc.

@MarceloSalazar
Copy link

@iriark01 do you want to have a quick look to the readme doc?

@iriark01
Copy link
Contributor

@AnotherButler might have time. If this needs to go in the public docs, it will also need to move repos.

0xc0170
0xc0170 requested changes Aug 21, 2019
View reviewed changes
Copy link
Contributor

@0xc0170 0xc0170 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commits could provide more details. The first one is adding 14 files - what these are? At least an introduction for each new test unit in this commit would be nice to have.

License for alice text is not acceptable, we shall find a replacement for the file used.

TESTS/integration/COMMON/alice.txt Outdated

This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a discussion previously about this one as I recall. To my understanding, the license is not acceptable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file got removed already, it just exist in the git history. do you want me clean up the history and force push a commit as well ? @0xc0170

Copy link
Contributor

@0xc0170 0xc0170 Aug 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I was probably reviewing only the first commit after I checked what it includes. please remove from history . Will review afterwards

@jamesbeyond
Copy link
Contributor Author

Hi @MarceloSalazar @MarceloSalazar @AnotherButler ,
your comments been addressed, please review again.

@MarceloSalazar
Copy link

Reviewed - pending to remove duplicate 'and'.

@AnotherButler
Copy link
Contributor

We need to move the README file content to the mbed-os-5-docs repository. I can do this tomorrow or Friday.

Could someone please explain sample.txt? Why are we copying content from mbed-os-5-docs and pasting it here?

@jamesbeyond
Copy link
Contributor Author

We need to move the README file content to the mbed-os-5-docs repository. I can do this tomorrow or Friday.

Could someone please explain sample.txt? Why are we copying content from mbed-os-5-docs and pasting it here?

Hi @AnotherButler , thanks for reviewing, the sample.txt is the guinea pig, the test case is trying to send this file over the network, then verify the revived copy same as the original one.

BTW, why do we need to move the README to docs repo? the README only related to the tests inside this folder. it might not fit in to the contents of mbed-os-5-docs ? what do you think @MarceloSalazar

@0xc0170
Copy link
Contributor

0xc0170 commented Aug 22, 2019 •
edited
Loading

BTW, why do we need to move the README to docs repo? the README only related to the tests inside this folder. it might not fit in to the contents of mbed-os-5-docs ? what do you think @MarceloSalazar

I was going to ask similar question. My assumption would be to have intergration tests documented in docs and referenced in Readme.md ? However looking at tests, usb have extensive readme for instance similar to these integration tests.
@AnotherButler what shall we do. Clear guidance for tests would be appreciated.

@MarceloSalazar
Copy link

MarceloSalazar commented Aug 22, 2019 •
edited
Loading

I'd love the documentation to be available in os.mbed.com/docs, as it will raise awareness and instruct developers/partners on how to run the tests. At the moment, it's hidden/buried and it's not easy to find it.

bulislaw
bulislaw reviewed Aug 22, 2019
View reviewed changes
Copy link
Member

@bulislaw bulislaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the directory is "COMMON" rather than "common"? Upper case has special meaning in our build system and it may be confusing.

@jamesbeyond
Copy link
Contributor Author

Why the directory is "COMMON" rather than "common"? Upper case has special meaning in our build system and it may be confusing.

Yeah, I think that's the reason we keep it as COMMON, otherwise build system will treat this folder as a test, and trying to build it, but failed in the end

@jamesbeyond
Copy link
Contributor Author

I'd love the documentation to be available in os.mbed.com/docs, as it will raise awareness and instruct developers/partners on how to run the tests. At the moment, it's hidden/buried and it's not easy to find it.

Currently it feels fragmentation in the testing contents in mbeds documentation.
maybe we can have a section in mbed-os-5-docs, focusing on the testing contents, explain to people what it which and how to run them ?

@AnotherButler
Copy link
Contributor

We do have a section in our published documentation about testing: https://os.mbed.com/docs/mbed-os/v5.13/tools/testing.html

It describes our different types of tests and how to build and write them.

If we publish this content there in our published documentation, then it should live in that repository. Having two copies of the same content leads to content out of sync.

@jamesbeyond
Copy link
Contributor Author

HI @AnotherButler, I the one you pointed out is the test tooling, but not the actual tests itself.
For example, these tests I am trying to merge is about integration test on Storage and Connection.
and as far as I know, there are testing about

They are not in the mbed-os-5-docs

@MarceloSalazar
Copy link

The integration tests are critical and involve many components - IMO deserves its own section in the docs in the porting section.
I'd suggest moving the content under the porting process here: https://os.mbed.com/docs/mbed-os/v5.13/porting/index.html

@jamesbeyond
Copy link
Contributor Author

Hi @MarceloSalazar, do you suggest we move the docs from this repo to mbed-os-5-docs repo? or do we leave the docs here and duplicate the README in the docs repo ?

@MarceloSalazar
Copy link

We shouldn't have any duplication.
mbed-os-5-docs is the best place for this document, IMO.

@jamesbeyond jamesbeyond mentioned this pull request Aug 23, 2019
Merged
@jamesbeyond
Copy link
Contributor Author

I remove the README file, and moved it to docs repo. This PR should be ready for review and CI @MarceloSalazar @0xc0170

@0xc0170
Copy link
Contributor

0xc0170 commented Aug 26, 2019

CI started

@0xc0170 0xc0170 changed the title Merge integration tests into mbed-os master Add integration tests Aug 26, 2019
@mbed-ci
Copy link

mbed-ci commented Aug 26, 2019

Test run: FAILED

Summary: 1 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_greentea-test

@MarceloSalazar MarceloSalazar self-requested a review August 27, 2019 06:38
@0xc0170
Copy link
Contributor

0xc0170 commented Aug 27, 2019

Restarted tests

@0xc0170
Copy link
Contributor

0xc0170 commented Aug 27, 2019

CI restarted

@mbed-ci
Copy link

mbed-ci commented Aug 27, 2019

Test run: SUCCESS

Summary: 4 of 4 test jobs passed
Build number : 2
Build artifacts

@0xc0170 0xc0170 merged commit d2c0591 into master Aug 28, 2019
@0xc0170 0xc0170 deleted the sip-workshop branch October 4, 2021 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bulislaw bulislaw bulislaw left review comments

@MarceloSalazar MarceloSalazar MarceloSalazar approved these changes

@0xc0170 0xc0170 0xc0170 approved these changes

@maclobdell maclobdell Awaiting requested review from maclobdell

Assignees
No one assigned
Labels
release-version: 5.14.0-rc1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@jamesbeyond @ciarmcom @MarceloSalazar @iriark01 @AnotherButler @0xc0170 @mbed-ci @bulislaw