Use Buf to generate protos and standardize layout (rilldata#1276)

* Move proto to root and generate using buf

* Update developer docs

* Reconfig and regen Orval

* Add buf linting

* Fix test failures

* Fix lint errors not related to analytical APIs

* Add details to proto ci

* Merge main

* Fix yaml errors

* Fix yaml error 2

* Restore openapi error types

* Restore error types

* Fix orval generation rules

* Fix prettier

.github/workflows/proto-check.yml

@@ -0,0 +1,14 @@
+name: Check .proto files
+on:
+  pull_request:
+    paths:
+      - "proto/**"
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: bufbuild/buf-setup-action@v1
+      - uses: bufbuild/buf-lint-action@v1
+        with:
+          input: "proto"

CONTRIBUTING.md

@@ -10,7 +10,7 @@ This is a full list of development dependencies:
 - [Node.js 18](https://nodejs.org/en/) (we recommend installing it with [nvm](https://github.com/nvm-sh/nvm))
 - [Go 1.19](https://go.dev) (on macOS, install with `brew install go`)
 - [GraalVM](https://www.graalvm.org) and [Maven](https://maven.apache.org) for Java (we recommend installing both through [sdkman](https://sdkman.io))
-- [Protocol Buffers](https://grpc.io/docs/protoc-installation/) (`protoc`) (on macOS, install with `brew install protobuf`)
+- [Buf](https://buf.build) (Protocol Buffers) (on macOS, install with `brew install bufbuild/buf/buf`)
 
 ## Technologies
 
@@ -40,6 +40,7 @@ Here's a guide to the top-level structure of the repository:
 - `.github` and `.travis.yml` contain CI/CD workflows. We allow both, but the goal is to move fully to Github Actions.
 - `cli` is an implementation of a new CLI in Go (in progress, not launched yet).
 - `docs` contains the user-facing documentation that we deploy to [docs.rilldata.com](https://docs.rilldata.com).
+- `proto` contains protocol buffer definitions for all Rill components, which notably includes our API interfaces.
 - `runtime` is our data plane, responsible for querying and orchestrating data infra. It currently supports DuckDB and Druid.
 - `server-cloud` contains the backend control plane for a multi-user version of Rill (in progress, not launched yet).
 - `sql` contains our SQL parser and transpiler. It's based on Apache Calcite and is used heavily by the `runtime`.

Makefile

@@ -4,4 +4,9 @@ cli:
 	npm run build
 	mkdir -p cli/pkg/web/embed/dist
 	cp -r web-local/build/ cli/pkg/web/embed/dist
-	go build -o rill cli/main.go 
+	go build -o rill cli/main.go 
+
+.PHONY: proto.generate
+proto.generate:
+	rm -rf proto/gen
+	cd proto && buf generate

cli/cmd/start/start.go

@@ -10,7 +10,7 @@ import (
 	"github.com/mattn/go-colorable"
 	"github.com/rilldata/rill/cli/pkg/browser"
 	"github.com/rilldata/rill/cli/pkg/web"
-	"github.com/rilldata/rill/runtime/api"
+	runtimev1 "github.com/rilldata/rill/proto/gen/rill/runtime/v1"
 	_ "github.com/rilldata/rill/runtime/connectors/gcs"
 	_ "github.com/rilldata/rill/runtime/connectors/https"
 	_ "github.com/rilldata/rill/runtime/connectors/s3"
@@ -88,7 +88,7 @@ func StartCmd() *cobra.Command {
 			}
 
 			// Create instance and repo configured for local use
-			_, err = server.CreateInstance(context.Background(), &api.CreateInstanceRequest{
+			_, err = server.CreateInstance(context.Background(), &runtimev1.CreateInstanceRequest{
 				InstanceId:   localInstanceID,
 				Driver:       olapDriver,
 				Dsn:          olapDSN,
@@ -98,7 +98,7 @@ func StartCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
-			_, err = server.CreateRepo(context.Background(), &api.CreateRepoRequest{
+			_, err = server.CreateRepo(context.Background(), &runtimev1.CreateRepoRequest{
 				RepoId: localRepoID,
 				Driver: "file",
 				Dsn:    repoDSN,
@@ -115,7 +115,7 @@ func StartCmd() *cobra.Command {
 
 			// Trigger a migration
 			logger.Infof("Hydrating project at '%s'", repoAbs)
-			res, err := server.Migrate(context.Background(), &api.MigrateRequest{
+			res, err := server.Migrate(context.Background(), &runtimev1.MigrateRequest{
 				InstanceId: localInstanceID,
 				RepoId:     localRepoID,
 			})

proto/README.md

@@ -0,0 +1,33 @@
+# `proto/`
+
+This directory contains the protocol buffer definitions for all Rill components. Instead of placing `.proto` files in their respective sub-projects, we follow the convention of having a single `proto` folder to make cross-component imports and codegen easier.
+
+We use [Buf](https://buf.build) to lint and generate protocol buffers. The layout and style of our `.proto` files follow their [style guide](https://docs.buf.build/best-practices/style-guide).
+
+## Defining APIs
+
+We define APIs as gRPC services. All APIs should be defined centrally in this directory, but implemented in their respective sub-package of the monorepo. 
+
+For all APIs, we setup [gRPC-Gateway](https://grpc-ecosystem.github.io/grpc-gateway/) to map the gRPC definitions to a RESTful API. The mapping rules are done inline in the `proto` file, using `google.api.http` annotations ([docs here](https://github.com/googleapis/googleapis/blob/master/google/api/http.proto#L44)). 
+
+Using protocol buffers to define dual RPC and REST interfaces is a technique widely used at Google. We suggest taking a look at their excellent [API design guide](https://cloud.google.com/apis/design/resources), which describes this pattern (notice it's multiple pages).
+
+## Generating
+
+After changing a `.proto` file, you should re-generate the bindings:
+
+1. Install Buf if you haven't already ([docs](https://docs.buf.build/installation))
+2. From the repo root, run:
+```bash
+make proto.generate
+```
+
+### Typescript runtime client
+
+We separately have a generated TypeScript client for the runtime in `web-common/src/runtime-client`. If relevant, you can re-generate it by running:
+
+```bash
+npm run generate:runtime-client -w web-common
+```
+
+(This is not automated as the frontend may currently be pinned to an older version of the runtime.)

proto/buf.gen.yaml

@@ -0,0 +1,30 @@
+version: v1
+managed:
+  enabled: true
+  go_package_prefix:
+    default: github.com/rilldata/rill
+    except:
+      - buf.build/googleapis/googleapis
+plugins:
+  - remote: buf.build/protocolbuffers/plugins/go:v1.28.1-1
+    out: gen
+    opt:
+      - paths=source_relative
+  - remote: buf.build/grpc/plugins/go:v1.2.0-1
+    out: gen
+    opt:
+      - paths=source_relative
+  - remote: buf.build/grpc-ecosystem/plugins/grpc-gateway:v2.13.0-1
+    out: gen
+    opt:
+      - paths=source_relative
+      - logtostderr=true
+      - generate_unbound_methods=true
+  - remote: buf.build/grpc-ecosystem/plugins/openapiv2:v2.13.0-1
+    out: gen
+    opt:
+      - logtostderr=true
+      - allow_merge=true,merge_file_name=rill/runtime/v1/runtime
+      - output_format=yaml
+      # - openapi_naming_strategy=simple # Removes "V1" prefix from types
+      # - simple_operation_ids=true # Removes "RuntimeService_" prefix from operations

proto/buf.lock

@@ -0,0 +1,11 @@
+# Generated by buf. DO NOT EDIT.
+version: v1
+deps:
+  - remote: buf.build
+    owner: googleapis
+    repository: googleapis
+    commit: 5abafbf55b5c4c07ad5fb06d2599560f
+  - remote: buf.build
+    owner: grpc-ecosystem
+    repository: grpc-gateway
+    commit: b98ae2f8ce63452e97e6bae65c5add1b

proto/buf.yaml

@@ -0,0 +1,11 @@
+version: v1
+name: buf.build/rilldata/rill
+lint:
+  use:
+    - DEFAULT
+breaking:
+  use:
+    - FILE
+deps:
+  - buf.build/googleapis/googleapis
+  - buf.build/grpc-ecosystem/grpc-gateway