Core: Add REST API for committing changes against multiple tables

[nastra]

The main purpose of this PR is to extract the REST-specific changes out of #6948

[nastra]

CommitTableRequest does already exist in this file unfortunately, hence naming it CommitTransactionTableRequest

[rdblue]

I think we could refactor to work around this. Could we just use a list of CommitTableRequest?

[nastra]

the existing CommitTableRequest doesn't have an identifier field unfortunately

[rdblue]

Does this make sense with multiple tables? The route exists. Maybe this is a bad request?

[nastra]

The only case where I thought it might make sense to throw a 404 is when the server gets changes for a table that doesn't exist (anymore). Alternatively we could handle this case via a general CommitFailedException, but CommitFailedException is retryable and I don't think we'd want to retry in this particular case

[rdblue]

I'm undecided. On one hand, that would already work and we wouldn't need to find a response code for "concurrent delete" that avoids the retry. On the other, the route doesn't represent that table.

I think I'm inclined to go with what you have here right now. It makes sense and we can always deprecate its use later.

[rdblue]

As I noted above, I think it would be better to use requirements and updates instead of mixing the updates with a TableMetadata.

I think there is a drawback to that approach, though. Any catalog that commits using metadata location rather than requirements and updates would be harder to update for multi-table transactions because those require the base metadata and the new metadata.

Overall, I think I would still prefer the requirements and updates. There are a couple of reasons:

1. Implementations of TableOperations almost always refresh immediately before attempting a commit anyway. Loading table metadata should not be a performance problem.
2. Some operations don't refresh table metadata before attempting to commit because they do not retry. For those cases, we have to address that they are not normally retried and use generic retry logic. That's what requirements/updates already do.

For a concrete example of point 2 above, consider UpdateSchema that does not retry. If the table changes concurrently, then the schema update fails. That's because the operation doesn't validate that it can still apply the schema changes. However, the REST protocol does provide a way to validate a schema update can still be applied, by sending an assert-last-assigned-field-id requirement.

If we were to try to perform a schema update using a multi-table commit method, that method doesn't know what is update is happening and will always retry. To make that retry safe, we should use the requirements.

[nastra]

I completely agree with you on just using requirements and updates in TableCommit due to the points you mentioned. The only difficulty is that requirements are currently at the rest-layer and we'd have to refactor them out to make them generally available/usable. I didn't want to overcomplicate this PR with such a refactoring and decided to keep the base metadata in the class for now, which we use to build the requirements from

[rdblue]

Looks like this is the primary blocker, so I recommend we get started on the refactor and get some of the other pieces of this PR (like REST updates and request objects) in.

[rdblue]

I think that this method should check whether the underlying catalog implements commitTransaction. If it does, then it should delegate to the catalog. If not, it should throw an exception.

The CatalogHandlers act as a reference implementation for the REST protocol, so I think it would be problematic if we included a method that took an atomic multi-table commit and implemented it as a sequence of individual commits.

[rdblue]

It seems confusing to have a CommitTableRequest and an UpdateTableRequest that are basically the same thing but with or without the identifier. Then there is also the naming problem in the spec. Those issues are a bit of a red flag that we need to address duplication.

I think there are two options. First, we could reuse the UpdateTableRequest directly and pull the identifier to a higher level, like this:

    CommitTransactionRequest:
      type: array
      items:
        $ref: '#/components/schemas/TransactionCommit'

    TransactionCommit:
      type: object
      required:
        - identifier
        - commit
      properties:
        identifier:
          $ref: '#/components/schemas/TableIdentifier'
        request:
          $ref: '#/components/schemas/CommitTableRequest'

    CommitTableRequest:
      type: object
      required:
        - requirements
        - updates
      properties:
        requirements:
          type: array
          items:
            $ref: '#/components/schemas/TableRequirement'
        updates:
          type: array
          items:
            $ref: '#/components/schemas/TableUpdate'

That's okay, but still fairly awkward. The next option is to reuse the existing schema directly and just add an optional identifier field:

    CommitTransactionRequest:
      type: array
      items:
        description: Each table commit request must provide an `identifier`
        $ref: '#/components/schemas/CommitTableRequest'

    CommitTableRequest:
      type: object
      required:
        - requirements
        - updates
      properties:
        identifier:
          description: Table identifier to update; must be present for CommitTransactionRequest
          $ref: '#/components/schemas/TableIdentifier'
        requirements:
          type: array
          items:
            $ref: '#/components/schemas/TableRequirement'
        updates:
          type: array
          items:
            $ref: '#/components/schemas/TableUpdate'

I prefer the second option. It's not a problem to add an optional field and ensure it is set when the object is used in a transaction. We'd also want to check that the identifier is either not set or is set to the same table in a normal table update.

[rdblue]

Looks like we may need to refactor how we produce requirements. This would not work for non-REST catalogs.

[nastra]

yes I agree, we'd have to pull them out of the REST layer and make them work for non-REST catalogs

[rdblue]

I can see why you'd want to use a transaction to get the bulk of the work done before committing, but I don't think this actually works because you're committing the transactions sequentially below.

I think to test we need to pass the commitTransaction call through and implement it either for an InMemoryCatalog or JDBC.

[rdblue]

Great progress, @nastra! I think we can get this in fairly quickly. Thanks for getting this ready!

[rdblue]

@nastra, can you move the REST spec updates and request objects (with tests) to a different PR? I think those are about ready to go and this is currently blocked by producing requirements and refactoring that code. That way we can get some of this in rather than waiting.

[nastra]

@rdblue I've opened #7741 that contains the REST-specific changes and #7750 that contains the UpdateRequirement refactorings.

[nastra]

this is required to be public so that it can be used in RESTSessionCatalog.
It might be also worth adding a Builder to UpdateTableRequest so that we don't have to make it public here.
In this case we have two options:

• write a manual builder
• generate a builder from the constructor definition (I've opened Core: Generate Builder for UpdateTableRequest #7838 to indicate how this would look like)

[rdblue]

Maybe we should have a create method like TableCommit? Or one that accepts a TableCommit and returns the correct request? That would be simpler than a builder.

[nastra]

sure that would also work. I'll make the respective changes in the next PR

[nastra]

added in #7867

[rdblue]

Javadoc?

[nastra]

added in #7867

[rdblue]

Nothing major that would block this, but I made a couple of comments. Thanks, @nastra! Good to have transactions moving forward.