Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing #1700

coalman321 · 2024-06-07T04:05:08Z

Propagates the custom_data and ros_distro fields from the underlying storage into the metadata.yaml file during re-indexing.

Previously this data was not extracted from the storage and the subsequent fields in metadata.yaml were left as default.
The implementation in this PR uses the custom_data and ros_distro metadata from the newest storage file contained within the log as custom data can be modified in the storage API between successive split files.

This fixes #1699

Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]>

Signed-off-by: coalman321 <[email protected]>

rosbag2_cpp/src/rosbag2_cpp/reindexer.cpp

fujitatomoya

lgtm with green CI

MichaelOrlov

@coalman321 Thanks for the PR. Overall looks good to me.
However, I have a proposal to check if temp_metadata.custom_data is not empty before assigning it to the metadata_.custom_data and the same for the ros_distro.

coalman321 · 2024-06-08T23:06:48Z

@coalman321 Thanks for the PR. Overall looks good to me. However, I have a proposal to check if temp_metadata.custom_data is not empty before assigning it to the metadata_.custom_data and the same for the ros_distro.

So I changed my thoughts on implementation a few times during this PR. My guiding thought was that the youngest split file (most recent) should be the most up to date version of custom_data. I was concerned about the case where the user had edited the info between the split files and having any sort of filtering could cause the re-indexed data to not reflect this updated state. The metadata in the final split file written should already be an aggregation of this information by nature of the storage implementation and thus be the most "accurate" representation of the information the user intends to store. In my thought this extends to being empty as well.

If you think this thought process doesn't match up with the designed intent, then I would be happy to modify it.

MichaelOrlov · 2024-06-10T20:12:05Z

@coalman321 Your assumption that the latest metadata should have the most up to date version of the custom_data is correct.
However, this is true for generic normal workflow. Although it might not be true if recording finishes abnormally, the metadata record could get corrupted or be incomplete.
IMO, it wouldn't hurt to add a sanity check that temp_metadata.custom_data is not empty before copying from it for a worst-case scenario.

coalman321 · 2024-06-10T20:49:48Z

I had not considered the abnormal case where the recording is not closed properly. Ill adjust the logic to work only if the fields are not empty.

Signed-off-by: coalman321 <[email protected]>

rosbag2_cpp/src/rosbag2_cpp/reindexer.cpp

Signed-off-by: coalman321 <[email protected]>

MichaelOrlov

@coalman321 Thank you for your contribution.
LGTM with green CI.

coalman321 · 2024-06-11T18:36:55Z

@coalman321 Thank you for your contribution. LGTM with green CI.

Happy to help. Also looks like ament_cppcheck failed, but Github cant show status for it. It might just need to be re-run.

MichaelOrlov

@coalman321 May I ask you to modify https://github.com/ros2/rosbag2/blob/rolling/rosbag2_tests/test/rosbag2_tests/test_reindexer.cpp tests to check custom_data and ROS_DISTRO in the reindexer test?

Signed-off-by: coalman321 <[email protected]>

coalman321 · 2024-06-11T22:35:16Z

@coalman321 May I ask you to modify https://github.com/ros2/rosbag2/blob/rolling/rosbag2_tests/test/rosbag2_tests/test_reindexer.cpp tests to check custom_data and ROS_DISTRO in the reindexer test?

Added verification for both in the reindexer test as requested.

MichaelOrlov

@coalman321 Thanks for updating the tests.
LGTM with green CI.

MichaelOrlov · 2024-06-12T00:02:32Z

Pulls: #1700
Gist: https://gist.githubusercontent.com/MichaelOrlov/23b7ed7f99ad70b70f6d33a8bc52b264/raw/ffe2b74684675f2ba3d508aeae04d238473dce94/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_cpp rosbag2_tests
TEST args: --packages-above rosbag2_cpp rosbag2_tests
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/14060

Linux
Linux-aarch64
Linux-rhel
Windows

MichaelOrlov · 2024-06-12T01:53:58Z

@coalman321 Unfortunately the CI failed on Windows with the following errors:

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(194,55): error C2678: binary '<': no operator found which takes a left-hand operand of type 'std::chrono::system_clock::time_point' (or there is no acceptable conversion) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include\system_error(191,28): message : could be 'bool std::operator <(const std::error_code &,const std::error_code &) noexcept' [found using argument-dependent lookup] (compiling source file C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include\system_error(277,28): message : or       'bool std::operator <(const std::error_condition &,const std::error_condition &) noexcept' [found using argument-dependent lookup] (compiling source file C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(194,55): message : while trying to match the argument list '(std::chrono::system_clock::time_point, std::chrono::time_point<std::chrono::steady_clock,std::chrono::nanoseconds>)' [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(195,53): error C2679: binary '=': no operator found which takes a right-hand operand of type 'std::chrono::time_point<std::chrono::steady_clock,std::chrono::nanoseconds>' (or there is no acceptable conversion) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

Signed-off-by: coalman321 <[email protected]>

coalman321 · 2024-06-12T02:15:06Z

@coalman321 Unfortunately the CI failed on Windows with the following errors:

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(194,55): error C2678: binary '<': no operator found which takes a left-hand operand of type 'std::chrono::system_clock::time_point' (or there is no acceptable conversion) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include\system_error(191,28): message : could be 'bool std::operator <(const std::error_code &,const std::error_code &) noexcept' [found using argument-dependent lookup] (compiling source file C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include\system_error(277,28): message : or       'bool std::operator <(const std::error_condition &,const std::error_condition &) noexcept' [found using argument-dependent lookup] (compiling source file C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(194,55): message : while trying to match the argument list '(std::chrono::system_clock::time_point, std::chrono::time_point<std::chrono::steady_clock,std::chrono::nanoseconds>)' [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

C:\ci\ws\src\coalman321\rosbag2\rosbag2_cpp\src\rosbag2_cpp\reindexer.cpp(195,53): error C2679: binary '=': no operator found which takes a right-hand operand of type 'std::chrono::time_point<std::chrono::steady_clock,std::chrono::nanoseconds>' (or there is no acceptable conversion) [C:\ci\ws\build\rosbag2_cpp\rosbag2_cpp.vcxproj]

Looks like it was an issue with the clock type that was being used. GCC must alias the system clock and the high-res clock in chrono. The change uses the exact same time point spec from the BagMetadata structure now.

MichaelOrlov · 2024-06-12T05:13:27Z

Re-run Windows CI job

Windows

MichaelOrlov · 2024-06-12T21:37:47Z

https://github.com/Mergifyio backport jazzy iron

mergify · 2024-06-12T21:37:56Z

backport jazzy iron

✅ Backports have been created

#1710 Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) has been created for branch jazzy
#1711 Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) has been created for branch iron

… during re-indexing (#1700) * propagate custom data and ros_distro during reindexing Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]> * fix ament uncrustify issue Signed-off-by: coalman321 <[email protected]> * only update metadata if size is nonzero Signed-off-by: coalman321 <[email protected]> * fix uncrustify issue and switch to empty Signed-off-by: coalman321 <[email protected]> * update reindexer test to verify custom data and ros_distro changes Signed-off-by: coalman321 <[email protected]> * Convert to use exact clock type for windows CI Signed-off-by: coalman321 <[email protected]> --------- Signed-off-by: coalman321 <[email protected]> (cherry picked from commit 804432c)

MichaelOrlov · 2024-06-12T21:43:34Z

https://github.com/Mergifyio backport humble

mergify · 2024-06-12T21:43:39Z

backport humble

✅ Backports have been created

#1712 Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) has been created for branch humble but encountered conflicts

… during re-indexing (#1700) * propagate custom data and ros_distro during reindexing Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]> * fix ament uncrustify issue Signed-off-by: coalman321 <[email protected]> * only update metadata if size is nonzero Signed-off-by: coalman321 <[email protected]> * fix uncrustify issue and switch to empty Signed-off-by: coalman321 <[email protected]> * update reindexer test to verify custom data and ros_distro changes Signed-off-by: coalman321 <[email protected]> * Convert to use exact clock type for windows CI Signed-off-by: coalman321 <[email protected]> --------- Signed-off-by: coalman321 <[email protected]> (cherry picked from commit 804432c) # Conflicts: # rosbag2_tests/test/rosbag2_tests/test_reindexer.cpp

… during re-indexing (#1700) (#1710) * propagate custom data and ros_distro during reindexing Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]> * fix ament uncrustify issue Signed-off-by: coalman321 <[email protected]> * only update metadata if size is nonzero Signed-off-by: coalman321 <[email protected]> * fix uncrustify issue and switch to empty Signed-off-by: coalman321 <[email protected]> * update reindexer test to verify custom data and ros_distro changes Signed-off-by: coalman321 <[email protected]> * Convert to use exact clock type for windows CI Signed-off-by: coalman321 <[email protected]> --------- Signed-off-by: coalman321 <[email protected]> (cherry picked from commit 804432c) Co-authored-by: Cole Tucker <[email protected]>

…ml file during re-indexing (backport #1700) (#1711) * Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (#1700) * propagate custom data and ros_distro during reindexing Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]> * fix ament uncrustify issue Signed-off-by: coalman321 <[email protected]> * only update metadata if size is nonzero Signed-off-by: coalman321 <[email protected]> * fix uncrustify issue and switch to empty Signed-off-by: coalman321 <[email protected]> * update reindexer test to verify custom data and ros_distro changes Signed-off-by: coalman321 <[email protected]> * Convert to use exact clock type for windows CI Signed-off-by: coalman321 <[email protected]> --------- Signed-off-by: coalman321 <[email protected]> (cherry picked from commit 804432c) * Remove "metadata.ros_distro" update since we don't have in on Iron Signed-off-by: Michael Orlov <[email protected]> * Skip check for `custom_metadata` in test for mcap storage plugin. - Rationale: Mcap storage doesn't store serialized metadata in bag directly on Iron and Humble. Backporting of the relevant #1423 is not possible without updating mcap vendor package to the v1.1.0 because we have dependencies from the mcap_reader_->metadataIndexes(); API which became available only in the foxglove/mcap#902 Signed-off-by: Michael Orlov <[email protected]> --------- Signed-off-by: Michael Orlov <[email protected]> Co-authored-by: Cole Tucker <[email protected]> Co-authored-by: Michael Orlov <[email protected]>

coalman321 requested a review from a team as a code owner June 7, 2024 04:05

coalman321 requested review from gbiggs and hidmic and removed request for a team June 7, 2024 04:05

propagate custom data and ros_distro during reindexing

eb54d97

Signed-off-by: Cole Tucker Signed-off-by: coalman321 <[email protected]>

coalman321 force-pushed the rolling branch from 34a1eeb to eb54d97 Compare June 7, 2024 04:09

fix ament uncrustify issue

ff626cc

Signed-off-by: coalman321 <[email protected]>

fujitatomoya reviewed Jun 7, 2024

View reviewed changes

rosbag2_cpp/src/rosbag2_cpp/reindexer.cpp Outdated Show resolved Hide resolved

fujitatomoya approved these changes Jun 7, 2024

View reviewed changes

MichaelOrlov changed the title ~~[Jazzy] Propagate custom data and ros_distro during reindexing~~ Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing Jun 7, 2024

MichaelOrlov approved these changes Jun 8, 2024

View reviewed changes

only update metadata if size is nonzero

eba82f4

Signed-off-by: coalman321 <[email protected]>

MichaelOrlov requested changes Jun 10, 2024

View reviewed changes

rosbag2_cpp/src/rosbag2_cpp/reindexer.cpp Outdated Show resolved Hide resolved

fix uncrustify issue and switch to empty

3539dd8

Signed-off-by: coalman321 <[email protected]>

coalman321 requested a review from MichaelOrlov June 11, 2024 17:17

MichaelOrlov approved these changes Jun 11, 2024

View reviewed changes

MichaelOrlov reviewed Jun 11, 2024

View reviewed changes

update reindexer test to verify custom data and ros_distro changes

aadf842

Signed-off-by: coalman321 <[email protected]>

MichaelOrlov approved these changes Jun 11, 2024

View reviewed changes

Convert to use exact clock type for windows CI

cb496a8

Signed-off-by: coalman321 <[email protected]>

MichaelOrlov merged commit 804432c into ros2:rolling Jun 12, 2024

This was referenced Jun 12, 2024

[jazzy] Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) #1710

Merged

[iron] Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) #1711

Merged

mergify bot mentioned this pull request Jun 12, 2024

[humble] Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing (backport #1700) #1712

Closed

Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing #1700

Propagate "custom_data" and "ros_distro" in to the metadata.yaml file during re-indexing #1700

Uh oh!

Conversation

coalman321 commented Jun 7, 2024

Uh oh!

Uh oh!

fujitatomoya left a comment

Choose a reason for hiding this comment

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

coalman321 commented Jun 8, 2024

Uh oh!

MichaelOrlov commented Jun 10, 2024

Uh oh!

coalman321 commented Jun 10, 2024

Uh oh!

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

coalman321 commented Jun 11, 2024

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

coalman321 commented Jun 11, 2024

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

MichaelOrlov commented Jun 12, 2024

Uh oh!

MichaelOrlov commented Jun 12, 2024

Uh oh!

coalman321 commented Jun 12, 2024

Uh oh!

MichaelOrlov commented Jun 12, 2024

Uh oh!

MichaelOrlov commented Jun 12, 2024

Uh oh!

mergify bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

MichaelOrlov commented Jun 12, 2024

Uh oh!

mergify bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

mergify bot commented Jun 12, 2024 •

edited

Loading

mergify bot commented Jun 12, 2024 •

edited

Loading