tools: check that part size is not exceeding region size #9021

naveenkaje · 2018-12-07T22:21:12Z

Description

Check that the Intel hex file segment fits within the region. Add a test to the tools test suite to verify the same.

Pull request type

[X] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Reviewers

@theotherjimmy

naveenkaje · 2018-12-07T22:22:05Z

Addressing #9012 (comment)

ciarmcom · 2018-12-08T00:00:25Z

@naveenkaje, thank you for your changes.
@ARMmbed/mbed-os-tools @ARMmbed/mbed-os-maintainers please review.

theotherjimmy

part.maxaddr may be past the end of ROM, as introduced by #8097

theotherjimmy · 2018-12-10T15:37:28Z

OOps, got to give you a path forward. Support that use case please.

cmonr · 2018-12-21T03:37:54Z

@naveenkaje Has any progress been made on this PR?

naveenkaje · 2019-01-03T16:27:43Z

Will make changes and upload.

0xc0170 · 2019-01-16T10:12:59Z

Ready for another review (it was updated)

theotherjimmy · 2019-01-16T16:19:54Z

@naveenkaje Would it be possible to add some tests for merge_region_list?

naveenkaje · 2019-01-24T21:08:02Z

@theotherjimmy Here are logs from a test in which target.restrict_size is set to overflow and with this change, such a condition is caught.

Test Log:

Configuration in mbed_app.json
"DISCO_L476VG": {
    "target.restrict_size": "0x15690"
}

Build log

Using ROM regions application, post_application in this build.
  Region application: size 0x15000, offset 0x8000000
  Region post_application: size 0xeb000, offset 0x8015000
Compile [  0.1%]: except.S
Compile [  0.2%]: mbed_tz_context.c
Compile [  0.4%]: MCR20Drv.c
Compile [  0.5%]: rf_configuration.c
Compile [  0.6%]: fslittle_test.c
...
Compile [ 99.8%]: stm_spi_api.c
Compile [ 99.9%]: us_ticker.c
Compile [100.0%]: trng_api.c
Link: mbed-os-example-blinky_application
Elf2Bin: mbed-os-example-blinky_application
Merging Regions
  Filling region application with ./BUILD/DISCO_L476VG/GCC_ARM-DEBUG/mbed-os-example-blinky_application.bin
[ERROR] Contents of region application does not fit
[mbed] ERROR: "/usr/bin/python" returned error.
       Code: 1
       Path: "/home/navkaj01/code/mbed-os-example-blinky"
       Command: "/usr/bin/python -u /home/navkaj01/code/mbed-os-example-blinky/mbed-os/tools/make.py -t GCC_ARM -m DISCO_L476VG --profile debug --source . --build ./BUILD/DISCO_L476VG/GCC_ARM-DEBUG -c"
       Tip: You could retry the last command with "-v" flag for verbose output
---

theotherjimmy · 2019-01-25T14:58:17Z

@naveenkaje I meant some unit tests. That way we can have a quick sanity check and faster feedback in case anyone wants to refactor this code.

naveenkaje · 2019-01-29T15:38:21Z

Test results

$ python -m pytest -v tools/test/build_api/build_api_test.py
========================================================= test session starts ==========================================================
platform linux2 -- Python 2.7.12, pytest-3.3.0, py-1.7.0, pluggy-0.6.0 -- /usr/bin/python                                               
cachedir: .cache                                                                                                                        
<snip>                                                                                    
plugins: hypothesis-3.88.3                                                                                                              
collected 8 items                                                                                                                       
                                                                                                                                        
tools/test/build_api/build_api_test.py::BuildApiTests::test_always_complete_build PASSED                                         [ 12%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_app_config PASSED                                      [ 25%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_no_app_config PASSED                                   [ 37%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_app_config PASSED                                      [ 50%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_no_app_config PASSED                                   [ 62%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_merge_region_no_fit PASSED                                           [ 75%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_app_config PASSED                                  [ 87%] 
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_no_app_config PASSED                               [100%] 
                                                                                                                                        
======================================================= 8 passed in 1.25 seconds =======================================================

theotherjimmy

The new test cannot fail: it catches the only exception of the function under test and does not assert.

Please add an assert.

theotherjimmy · 2019-01-29T15:44:14Z

tools/test/build_api/build_api_test.py

+    @patch('tools.build_api.Resources')
+    @patch('tools.build_api.mkdir')
+    @patch('os.path.exists')
+    @patch('tools.build_api.prepare_toolchain')


I don't think you need this patch

theotherjimmy · 2019-01-29T15:47:01Z

tools/test/build_api/build_api_test.py

+        """
+        Test that merge region fails as expected when part size overflows region size.
+        """
+        with patch("tools.build_api.intelhex_offset") as _intelhex_offset:


Could you move this patch into the @patch directorator section above the funnction?

theotherjimmy · 2019-01-29T15:47:33Z

tools/test/build_api/build_api_test.py

+            region_list = [region_application, region_post_application]
+            region_list = list(region_list)
+            toolchain = prepare_toolchain(self.src_paths, self.build_path, self.target,
+                                          self.toolchain_name, notify=notify)


You don't need a toolchain object at all, so you can delete this line and the line above

theotherjimmy · 2019-01-29T15:47:57Z

tools/test/build_api/build_api_test.py

+            mock_prepare_toolchain.config = MagicMock(
+                                has_regions=True, name=None, lib_config_data=None)
+            region_list = [region_application, region_post_application]
+            region_list = list(region_list)


This line is not needed, as region_list is already a list.

theotherjimmy · 2019-01-29T15:50:36Z

tools/test/build_api/build_api_test.py

@@ -238,5 +240,41 @@ def test_build_library_no_app_config(self, mock_prepare_toolchain, mock_exists,
        self.assertEqual(args[1]['app_config'], None,
                         "prepare_toolchain was called with an incorrect app_config")

+    @patch('tools.build_api.Resources')
+    @patch('tools.build_api.mkdir')
+    @patch('os.path.exists')


os.path.exsits is not used, so you don't need this patch either.

theotherjimmy · 2019-01-29T15:52:31Z

Come to think of it, you probably don't need to patch tools.build_api.Resources or tools.build_api.mkdir, either.

naveenkaje · 2019-01-29T17:52:13Z

tools/test/build_api/build_api_test.py

+
+        notify = MockNotifier()
+        region_list = [region_application, region_post_application]
+        toolchain = prepare_toolchain(self.src_paths, self.build_path, self.target,


toolchain needed to pass to merge_region_list()

merge_region_list(region_list, res, notify, toolchain.config)

Nope. It needs a config object, or stand in. That's not a toolchain object.

Thanks @theotherjimmy got it. Updated the patch.

theotherjimmy

Improving, but we still don't assert any behavior of merge_region_list. Please add an assert to the return value or in either the try or the except path, enforcing that only one of them is possible.

theotherjimmy · 2019-01-29T19:55:26Z

tools/test/build_api/build_api_test.py

+        mock_intelhex_offset.return_value = IntelHex({0:2, max_addr:0})
+
+        self.assertEqual(0, mock_intelhex_offset.return_value.minaddr())
+        self.assertEqual(max_addr, mock_intelhex_offset.return_value.maxaddr())


These asserts don't test any tools code.

since we use the IntelHex() library to generate this object, we can forego these asserts?

naveenkaje · 2019-01-29T23:51:33Z

Without fix 060d3d3

$ python -m pytest -v tools/test/build_api/build_api_test.py                                          │
======================================================== test session starts ========================================================│
platform linux2 -- Python 2.7.12, pytest-3.3.0, py-1.7.0, pluggy-0.6.0 -- /usr/bin/python                                            │
cachedir: .cache                                                                                                                     │
<snip>
plugins: hypothesis-3.88.3                                                                                                           │
collected 8 items                                                                                                                    │
                                                                                                                                     │
tools/test/build_api/build_api_test.py::BuildApiTests::test_always_complete_build PASSED                                      [ 12%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_app_config PASSED                                   [ 25%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_no_app_config PASSED                                [ 37%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_app_config PASSED                                   [ 50%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_no_app_config PASSED                                [ 62%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_merge_region_no_fit FAILED                                        [ 75%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_app_config PASSED                               [ 87%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_no_app_config PASSED                            [100%] │
                                                                                                                                     │
============================================================= FAILURES ==============================================================│
______________________________________________ BuildApiTests.test_merge_region_no_fit ______________________________________________

With fix 060d3d3

======================================================== test session starts ========================================================│
platform linux2 -- Python 2.7.12, pytest-3.3.0, py-1.7.0, pluggy-0.6.0 -- /usr/bin/python                                            │
cachedir: .cache                                                                                                                     │
<snip>
plugins: hypothesis-3.88.3                                                                                                           │
collected 8 items                                                                                                                    │
                                                                                                                                     │
tools/test/build_api/build_api_test.py::BuildApiTests::test_always_complete_build PASSED                                      [ 12%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_app_config PASSED                                   [ 25%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_library_no_app_config PASSED                                [ 37%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_app_config PASSED                                   [ 50%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_build_project_no_app_config PASSED                                [ 62%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_merge_region_no_fit PASSED                                        [ 75%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_app_config PASSED                               [ 87%] │
tools/test/build_api/build_api_test.py::BuildApiTests::test_prepare_toolchain_no_app_config PASSED                            [100%] │
                                                                                                                                     │
===================================================== 8 passed in 1.30 seconds ======================================================│

If config is specified, check that part size is not exceeding the region. Normally we now assume that part.maxaddr() can be beyond end of rom.

…size

0xc0170 · 2019-02-04T16:20:45Z

CI started

mbed-ci · 2019-02-04T18:15:36Z

Test run: SUCCESS

Summary: 12 of 12 test jobs passed
Build number : 1
Build artifacts

NirSonnenschein · 2019-02-05T08:20:02Z

@0xc0170 CI passed, does this need another review?

NirSonnenschein · 2019-02-06T12:41:49Z

@naveenkaje , please restore the template to the PR description and fill in.

naveenkaje mentioned this pull request Dec 7, 2018

Build No Longer Fails if 'target.restrict_size' Exceeded #9012

Closed

ciarmcom requested review from a team December 8, 2018 00:00

ciarmcom added the needs: review label Dec 8, 2018

theotherjimmy suggested changes Dec 10, 2018

View reviewed changes

cmonr added needs: work release-version: 5.11.1 and removed needs: review labels Dec 10, 2018

cmonr removed the release-version: 5.11.1 label Dec 27, 2018

naveenkaje force-pushed the fix_part_size_check branch from 2ac5a4a to 2e74b8b Compare January 7, 2019 21:39

0xc0170 added needs: review and removed needs: work labels Jan 16, 2019

theotherjimmy approved these changes Jan 16, 2019

View reviewed changes

cmonr added needs: work and removed needs: review labels Jan 16, 2019

naveenkaje force-pushed the fix_part_size_check branch from 2e74b8b to 56b53e2 Compare January 29, 2019 15:23

theotherjimmy suggested changes Jan 29, 2019

View reviewed changes

naveenkaje force-pushed the fix_part_size_check branch from 56b53e2 to 6c68f14 Compare January 29, 2019 17:51

naveenkaje commented Jan 29, 2019

View reviewed changes

naveenkaje force-pushed the fix_part_size_check branch from 6c68f14 to a190ec8 Compare January 29, 2019 19:51

theotherjimmy suggested changes Jan 29, 2019

View reviewed changes

naveenkaje force-pushed the fix_part_size_check branch from a190ec8 to bf70ab3 Compare January 29, 2019 23:48

theotherjimmy approved these changes Jan 30, 2019

View reviewed changes

naveenkaje force-pushed the fix_part_size_check branch 2 times, most recently from 1306bb1 to 3f39f99 Compare January 31, 2019 15:40

tools: check part size is not exceeding region size

4393e4b

If config is specified, check that part size is not exceeding the region. Normally we now assume that part.maxaddr() can be beyond end of rom.

naveenkaje force-pushed the fix_part_size_check branch from 3f39f99 to 7e169d5 Compare January 31, 2019 23:03

tools: build_api_test: add test to detect when part overflows region …

43da2f2

…size

naveenkaje force-pushed the fix_part_size_check branch from 7e169d5 to 43da2f2 Compare January 31, 2019 23:10

0xc0170 added needs: CI and removed needs: work labels Feb 4, 2019

0xc0170 added the release-version: 5.11.4 label Feb 4, 2019

0xc0170 added ready for merge and removed needs: CI labels Feb 5, 2019

0xc0170 approved these changes Feb 5, 2019

View reviewed changes

adbridge merged commit 645c1ce into ARMmbed:master Feb 7, 2019

adbridge removed the ready for merge label Feb 7, 2019

tools: check that part size is not exceeding region size #9021

tools: check that part size is not exceeding region size #9021

Uh oh!

Conversation

naveenkaje commented Dec 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Reviewers

Uh oh!

naveenkaje commented Dec 7, 2018

Uh oh!

ciarmcom commented Dec 8, 2018

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

theotherjimmy commented Dec 10, 2018

Uh oh!

cmonr commented Dec 21, 2018

Uh oh!

naveenkaje commented Jan 3, 2019

Uh oh!

0xc0170 commented Jan 16, 2019

Uh oh!

theotherjimmy commented Jan 16, 2019

Uh oh!

naveenkaje commented Jan 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Jan 25, 2019

Uh oh!

naveenkaje commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theotherjimmy commented Jan 29, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naveenkaje commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xc0170 commented Feb 4, 2019

Uh oh!

mbed-ci commented Feb 4, 2019

Uh oh!

NirSonnenschein commented Feb 5, 2019

Uh oh!

NirSonnenschein commented Feb 6, 2019

Uh oh!

Uh oh!

naveenkaje commented Dec 7, 2018 •

edited

Loading

naveenkaje commented Jan 24, 2019 •

edited

Loading

naveenkaje commented Jan 29, 2019 •

edited

Loading

naveenkaje commented Jan 29, 2019 •

edited

Loading