Skip to content

Add custom_metadata support for RecordBatch IPC messages #135

@rustyconover

Description

@rustyconover

Describe the enhancement requested

The Arrow IPC spec supports custom_metadata (key-value pairs) on RecordBatch messages via the Message.custom_metadata field. The Swift FlatBuffers generated code (org_apache_arrow_flatbuf_Message) already has full read/write support for this field, but the Swift RecordBatch model, reader, and writer all ignore it.

Proposed changes

  • Add a customMetadata: [String: String] property to RecordBatch (defaulting to [:] for backward compatibility)
  • Add addMetadata builder methods on RecordBatch.Builder
  • Serialize metadata into the FlatBuffers Message wrapper when writing (covers file, streaming, and Flight paths via toMessage)
  • Deserialize metadata from the FlatBuffers Message wrapper when reading (covers readFile, readStreaming, and fromMessage)

Design notes

  • [String: String] matches the pragmatic API used by pyarrow and most Arrow consumers. Duplicate keys from other implementations are deduplicated (last wins). Dictionary order is not preserved across round-trips.
  • Flight support comes for free since toMessage(batch:) delegates to writeRecordBatch.
  • ArrowTable.from(recordBatches:) does not propagate per-batch metadata, which is expected since ArrowTable is a table-level abstraction.

Files affected

File Change
Sources/Arrow/ArrowTable.swift Add customMetadata property and builder methods
Sources/Arrow/ArrowWriter.swift Serialize metadata into FlatBuffers Message
Sources/Arrow/ArrowReader.swift Deserialize metadata from FlatBuffers Message
Tests/ArrowTests/IPCTests.swift Round-trip, builder API, cross-language, and multi-batch tests

I have an implementation ready and will open a PR referencing this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions