Core: Minor metadata table code harmonization for readable_metrics #7613
Conversation
| if (partitionType.fields().size() < 1) { | ||
| // avoid returning an empty struct, which is not always supported. instead, drop the partition | ||
| // field (id 102) | ||
| // avoid returning an empty struct, which is not always supported. |
There was a problem hiding this comment.
This is not related, but fixing longstanding uneven line breaks introduced by the spotless refactor
| * file projection | ||
| * | ||
| * @return file projection with required columns to read readable metrics | ||
| * file projection. |
There was a problem hiding this comment.
I think the return does not convey additional information, so removed it in favor of the method comment for brevity
| StructProjection structProjection = structProjection(projection); | ||
| return CloseableIterable.transform(entryAsStruct, structProjection::wrap); | ||
|
|
||
| return CloseableIterable.transform( |
There was a problem hiding this comment.
I was trying to make both branches be more alike:
- calculate final struct projection
- calculate 'file' projection if needed for reading manifest
- Use these to transform the result
5de22b6 to
141c9ab
Compare
| private CloseableIterable<? extends ManifestEntry<? extends ContentFile<?>>> entries( | ||
| Schema newFileProjection) { | ||
| return ManifestFiles.open(manifest, io, specsById).project(newFileProjection).entries(); | ||
| Schema fileStructProjection) { |
There was a problem hiding this comment.
Make this method a bit more functional (input/output is more clear)
|
@dramaticlly fyi |
dramaticlly
left a comment
There was a problem hiding this comment.
thank you @szehon-ho for the refactoring. I think now logic in BaseFiles is much easier to read
| private StructLike withReadableMetrics( | ||
| ContentFile<?> file, Types.NestedField readableMetricsField) { | ||
| int metricsPosition = projection.columns().indexOf(readableMetricsField); | ||
| int columnCount = projection.columns().size(); |
There was a problem hiding this comment.
maybe we can reuse the projectionColumnCount as variable name above ?
There was a problem hiding this comment.
Yea, projectionColumnCount is I guess the description of what it is, but the way I think about the method withReadableMetrics is actually more towards the goal, of making a struct.
So I think it's better to have the variables in the end: struct, structSize, metricsStruct, metricsPosition
(projectionColumnCount being structSize). Changed the references to match. What do you think?
There was a problem hiding this comment.
thanks for the explanation, I think structSize sounds great!
| * ensuring that the underlying metrics used to create that column are part of the final | ||
| * projection. | ||
| * | ||
| * @param projectionSchema projection to transform |
There was a problem hiding this comment.
nit: projection to transform read a bit weird as first time I read it I am not clear on why do we need to transform, maybe requested projection or intended projection which correspond to returning of actual projection ?
There was a problem hiding this comment.
I like that, changed to 'requestedProjection'
f071cf5 to
45db9e8
Compare
45db9e8 to
be003e9
Compare
|
Merged, as this is just a continuation/code fixup of #7539. |
Some minor cleanup discussed: #7539 (comment) ,
Try to make the readable_metrics code of BaseEntriesTable and BaseFilesTable align, by extracting common class and making code more similar.