Runtime APIs and abstractions #991
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
And some minor stuff: - Removes buf - Tidies up SQL library download - Moves server to designated folder
|
Looks great! One note looking at |
* Adding initial implementation of metrics views API * Addressing PR comments
…ctors (#1013) * Initial structure * Adding abstractions for validation and a test for local file * Addressing a few PR comments * Addressing some other PR comments
djbarnwal
pushed a commit
that referenced
this pull request
Aug 3, 2023
* Setup Postgres for metadata and db testing * Align runtime and cloud server config * Adds registry, repo, catalog, and olap abstraction And some minor stuff: - Removes buf - Tidies up SQL library download - Moves server to designated folder * Propose runtime APIs * Autogenerate svelte-query client for runtime APIs * Adding missing API fields * Add SQL protobufs to runtime * fix: forgot to commit makefile * Fix sql deps install in CI * Add READMEs describing the repo/runtime * Fix arm sql release script after moving pom.xml * Run prettier on contributing.md * Minor improvements to api * Remove Runtime struct * Spec initial driver APIs * Runtime server version * Implement handlers for instances and repos * Fix bugs * Added stubs for registry store functions in sqlite driver * Adding initial implementation of metrics views API (#1001) * Adding initial implementation of metrics views API * Addressing PR comments * Ad-hoc create source parser * Implement registry for sqlite * Implemented repos * Fix failing tests * [Runtime] Connectors APIs and support local file, s3 and future connectors (#1013) * Initial structure * Adding abstractions for validation and a test for local file * Addressing a few PR comments * Addressing some other PR comments * Integrate connectors * Implement catalog and migrations handlers * Add embedded catalog support to duckdb * Add connectors metadata endpoint * Move duckdb connector overrides to own file * Add connection caching to runtime server * Bugfixes * Add example usage with curl * Regenerate runtime client * Set embed_catalog = true in DuckDBClient * Add boot queries * Incorporate small fix for multi-statement queries in duckdb * Can't use args in multi-statement query Co-authored-by: Aditya Hegde <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR proposes endpoints and abstractions for the runtime. I'm implementing the endpoints over the next couple of days, but would love some feedback on the API signatures now. These are the main files for feedback at the moment:
runtime/api/runtime.protoproposes APIs for the runtime. The APIs are expressed as a gRPC service with a mapping to REST endpoints. There are APIs for file artifacts ("repos"), runtime deployments ("instances"), catalog data, queries, explore APIs (metrics views APIs), and connector metadata (profiling endpoints are still TBD)web-common/src/runtime-clientcontains a generatedsvelte-queryclient for all these APIsNOTE: The comments in
runtime/api/runtime.protoexplain most of the concepts/terminology, such as "repos".Workflows
Below I've tried to list how to deal with metrics views and sources using the proposed APIs. I realise these might not fit perfectly into the existing Node frontend, so let's work to make the ends meet. In addition to any simplifications you can think of, we can also add new/temporary APIs to the runtime to bridge the gap as necessary.
NOTE that for convenience, we can have the runtime automatically create an instance and repo for the current directory and map the keyword
localto those – so in the APIs below, you just need to insertlocalin place ofinstance_idorrepo_id.Workflow for metrics views:
CREATE METRICS VIEWand write the artifact to disk usingPOST /v1/repos/{repo_id}/objects/-/metrics/name.sqlPOST /v1/instances/{instance_id}/migrate/singleWorkflow for connectors:
/v1/connectors/metato get list of available connectors and their properties – to generate UICREATE SOURCEstatement and write the artifact to disk usingPOST /v1/repos/{repo_id}/objects/-/sources/name.sqlPOST /v1/instances/{instance_id}/migrate/singlePOST /v1/instances/{instance_id}/catalog/{name}/refreshTODOs
runtime/README.md(Note: Go CI/CD are failing due to a SQL bug – should get fixed soon.)