rosbag2_storage: add type description hash to topic metadata #1272

james-rms · 2023-04-04T02:59:47Z

Adds a type description hash to topic metadata.

james-rms · 2023-04-04T05:17:00Z

Gist: https://gist.githubusercontent.com/james-rms/0f3cca8fa197c87ba9e3bdb15cf55c3d/raw/c11aae907ae19932cdffcefbe11c930ff3d890bc/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_storage_mcap
TEST args: --packages-above rosbag2_storage_mcap
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11766

Linux
Linux-aarch64
Windows

james-rms · 2023-04-04T23:08:28Z

Gist: https://gist.githubusercontent.com/james-rms/ff8241cdc1abf5fd3359098ddff54941/raw/c11aae907ae19932cdffcefbe11c930ff3d890bc/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_compression rosbag2_cpp rosbag2_examples rosbag2_py rosbag2_storage rosbag2_storage_sqlite3 rosbag2_transport
TEST args: --packages-above rosbag2_compression rosbag2_cpp rosbag2_examples rosbag2_py rosbag2_storage rosbag2_storage_sqlite3 rosbag2_transport
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11780

Linux
Linux-aarch64
Windows

james-rms · 2023-04-04T23:25:12Z

Gist: https://gist.githubusercontent.com/james-rms/fab2a65dd9c9a9677e8d9fc0481d9042/raw/c11aae907ae19932cdffcefbe11c930ff3d890bc/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_compression rosbag2_cpp rosbag2_examples_cpp rosbag2_py rosbag2_storage rosbag2_storage_sqlite3 rosbag2_transport
TEST args: --packages-above rosbag2_compression rosbag2_cpp rosbag2_examples_cpp rosbag2_py rosbag2_storage rosbag2_storage_sqlite3 rosbag2_transport
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11782

Linux
Linux-aarch64
Windows

rosbag2_storage_sqlite3/src/rosbag2_storage_sqlite3/sqlite_storage.cpp

rosbag2_transport/src/rosbag2_transport/recorder.cpp

MichaelOrlov

@james-rms Thanks for PR.
Overall changes make sense to me.
I am curious about when implementation of storing type_hash in sqlite3_storage and mcap_storage will be implemented?
I saw some place holders in sqlite3_storage but nothing in mcap_storage plugin.

Since we are changing metadata version need to increment it in one more place

rosbag2/rosbag2_py/src/rosbag2_py/_storage.cpp

Line 235 in fb8b047

pybind11::arg("version") = 6,

Also I found some missing variable declarations (see details in code comments) - seems code not compiles after your recent changes.

Please note that we will have feature freeze at April 17 and it will be nice to merge this PR before then.

rosbag2_transport/src/rosbag2_transport/recorder.cpp

rosbag2_transport/CMakeLists.txt

rosbag2_transport/src/rosbag2_transport/recorder.cpp

rosbag2_transport/include/rosbag2_transport/recorder.hpp

rosbag2_storage_sqlite3/src/rosbag2_storage_sqlite3/sqlite_storage.cpp

rosbag2_transport/src/rosbag2_transport/recorder.cpp

Signed-off-by: James Smith <[email protected]>

rosbag2_transport/src/rosbag2_transport/recorder.cpp

Signed-off-by: Emerson Knapp <[email protected]>

MichaelOrlov

@james-rms Overall already looks good, although I think it would be great to add test coverage.
I've made some exercise in #1288. Feel free to merge or cherry pick it.

Signed-off-by: James Smith <[email protected]>

Signed-off-by: Michael Orlov <[email protected]>

james-rms · 2023-04-13T01:46:57Z

@MichaelOrlov cherry-picked, also added more detailed coverage into recorder.

MichaelOrlov · 2023-04-13T03:07:59Z

@MichaelOrlov cherry-picked, also added more detailed coverage into recorder.

@james-rms May I ask you to rename newly added function parameters endpoints to a something more meaningful like
topic_endpoints_info?

Signed-off-by: James Smith <[email protected]>

rosbag2_storage_mcap/test/rosbag2_storage_mcap/test_mcap_storage.cpp

Signed-off-by: James Smith <[email protected]>

james-rms · 2023-04-13T05:13:41Z

Gist: https://gist.githubusercontent.com/james-rms/46185d994e5da2f8702c59149550830d/raw/a510a74b328f208ffbc7608cad10504606c7ab89/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_compression rosbag2_cpp rosbag2_transport rosbag2_py rosbag2_storage_mcap rosbag2_storage_sqlite3
TEST args: --packages-above rosbag2_compression rosbag2_cpp rosbag2_transport rosbag2_py rosbag2_storage_mcap rosbag2_storage_sqlite3
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11885

Linux
Linux-aarch64
Windows

MichaelOrlov

Approving

MichaelOrlov · 2023-04-13T20:54:53Z

re-run Windows build

Windows

MichaelOrlov · 2023-04-13T21:07:18Z

@james-rms I am curious how CI build passing green on Linux if
RPR job fails with error message

22:12:44 --- stderr: rosbag2_transport
22:12:44 In file included from /tmp/ws/src/rosbag2/rosbag2_transport/src/rosbag2_transport/recorder.cpp:15:
22:12:44 /tmp/ws/src/rosbag2/rosbag2_transport/include/rosbag2_transport/recorder.hpp:190:64: error: ‘rosidl_type_hash_t’ does not name a type
22:12:44   190 | ROSBAG2_TRANSPORT_PUBLIC std::string type_hash_to_string(const rosidl_type_hash_t & type_hash);
22:12:44       |                                                                ^~~~~~~~~~~~~~~~~~
22:12:44 /tmp/ws/src/rosbag2/rosbag2_transport/src/rosbag2_transport/recorder.cpp:389:39: error: ‘rosidl_type_hash_t’ does not name a type
22:12:44   389 | std::string type_hash_to_string(const rosidl_type_hash_t & type_hash)
22:12:44       |                                       ^~~~~~~~~~~~~~~~~~

Which is looks reasonable to fail since we don't have #include "rosidl_runtime_c/type_hash.h" where rosidl_type_hash_t supposed to be defined.

james-rms · 2023-04-13T21:09:47Z

@MichaelOrlov RPR is failing because that's a recent addition to rosidl_runtime_c that hasn't been released yet, it's just in the rolling branch of rosidl. rosidl_runtime_c is a transitive dependency, there's no way we can have access to the definition of TopicEndpointInfo without also having the definition of rosidl_type_hash_t available.

clalancette · 2023-04-14T00:39:21Z

@MichaelOrlov RPR is failing because that's a recent addition to rosidl_runtime_c that hasn't been released yet, it's just in the rolling branch of rosidl. rosidl_runtime_c is a transitive dependency, there's no way we can have access to the definition of TopicEndpointInfo without also having the definition of rosidl_type_hash_t available.

It's all been released now, it is still rebuilding. In theory the Rpr jobs should all work again in about 12 hours.

emersonknapp · 2023-04-14T01:25:48Z

Note: CI was all green, updated the comment - somebody had removed something from the build queue, changing the pending ID, so we found the actual build and it was green. Action CI build_and_test also passed

MichaelOrlov · 2023-04-14T02:50:20Z

@clalancette Thanks for heads up and letting us know. 👍

ros-discourse · 2023-04-24T20:58:01Z

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-4-20-2023/31087/1

james-rms requested a review from MichaelOrlov April 4, 2023 02:59

james-rms requested a review from a team as a code owner April 4, 2023 02:59

james-rms requested review from hidmic and emersonknapp and removed request for a team April 4, 2023 02:59

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch 12 times, most recently from a9479fc to b55434f Compare April 4, 2023 05:15

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch 3 times, most recently from 74894dc to b3caa30 Compare April 4, 2023 21:06

emersonknapp reviewed Apr 5, 2023

View reviewed changes

rosbag2_storage_sqlite3/src/rosbag2_storage_sqlite3/sqlite_storage.cpp Outdated Show resolved Hide resolved

emersonknapp reviewed Apr 5, 2023

View reviewed changes

rosbag2_transport/src/rosbag2_transport/recorder.cpp Outdated Show resolved Hide resolved

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch 2 times, most recently from b3fe1ab to fb8b047 Compare April 7, 2023 02:56

MichaelOrlov requested changes Apr 11, 2023

View reviewed changes

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch from a2d6e26 to fc8fbfd Compare April 11, 2023 11:18

rosbag2_storage: add type description hash to topic metadata

1a02f38

Signed-off-by: James Smith <[email protected]>

increase versions

c9c5f5a

Signed-off-by: James Smith <[email protected]>

emersonknapp reviewed Apr 12, 2023

View reviewed changes

rosbag2_transport/src/rosbag2_transport/recorder.cpp Outdated Show resolved Hide resolved

Fix syntax errors

48e9d04

Signed-off-by: Emerson Knapp <[email protected]>

MichaelOrlov mentioned this pull request Apr 12, 2023

Add test coverage for get_all_topics_and_types(..) and get_metadata(..) #1288

Closed

MichaelOrlov reviewed Apr 12, 2023

View reviewed changes

test against multiple endpoint infos

833918b

Signed-off-by: James Smith <[email protected]>

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch from 547859d to 833918b Compare April 13, 2023 01:42

Add test coverage for get_all_topics_and_types(..) and get_metadata(..)

71ba37f

Signed-off-by: Michael Orlov <[email protected]>

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch 2 times, most recently from 42667ae to 71ba37f Compare April 13, 2023 02:14

james-rms added 2 commits April 13, 2023 13:49

add topic information equality operator

c745425

Signed-off-by: James Smith <[email protected]>

sqlite3: fix test

4e93bfc

Signed-off-by: James Smith <[email protected]>

MichaelOrlov reviewed Apr 13, 2023

View reviewed changes

rosbag2_storage_mcap/test/rosbag2_storage_mcap/test_mcap_storage.cpp Outdated Show resolved Hide resolved

mcap: fix read_metadata test

236a419

Signed-off-by: James Smith <[email protected]>

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch from 2472472 to 5a72fdf Compare April 13, 2023 05:06

rename arg

434bbb3

Signed-off-by: James Smith <[email protected]>

james-rms force-pushed the jrms/topic-metadata-type-description-hash branch from 5a72fdf to 434bbb3 Compare April 13, 2023 05:10

MichaelOrlov approved these changes Apr 13, 2023

View reviewed changes

james-rms merged commit 7b5cd7d into rolling Apr 14, 2023

delete-merged-branch bot deleted the jrms/topic-metadata-type-description-hash branch April 14, 2023 01:24

rosbag2_storage: add type description hash to topic metadata #1272

rosbag2_storage: add type description hash to topic metadata #1272

Uh oh!

Conversation

james-rms commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

james-rms commented Apr 4, 2023

Uh oh!

james-rms commented Apr 4, 2023

Uh oh!

james-rms commented Apr 4, 2023

Uh oh!

Uh oh!

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

james-rms commented Apr 13, 2023

Uh oh!

MichaelOrlov commented Apr 13, 2023

Uh oh!

Uh oh!

james-rms commented Apr 13, 2023 • edited by emersonknapp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelOrlov left a comment

Choose a reason for hiding this comment

Uh oh!

MichaelOrlov commented Apr 13, 2023

Uh oh!

MichaelOrlov commented Apr 13, 2023

Uh oh!

james-rms commented Apr 13, 2023

Uh oh!

clalancette commented Apr 14, 2023

Uh oh!

emersonknapp commented Apr 14, 2023

Uh oh!

MichaelOrlov commented Apr 14, 2023

Uh oh!

ros-discourse commented Apr 24, 2023

Uh oh!

Uh oh!

james-rms commented Apr 4, 2023 •

edited

Loading

james-rms commented Apr 13, 2023 •

edited by emersonknapp

Loading