-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
glob support for local, s3 and gcs #1578
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this PR! Really happy to see that you were able to jump in and identify all the relevant places to make changes.
In addition to the various nits below, I suggest we change some abstractions:
-
Adding the
PrepareBlob
method to theConnector
interface is not necessary. Instead, how about just renamingConsumeAsFile
toConsumeAsFiles
, which returns a list of one or more filenames to ingest instead? (In the future, we could update this to return a complex Iterator object that allows consuming downloaded files before the full download has completed – this is a feature we've been asked to look into soon.)- This also enables moving the concurrent download logic into the
blob
connector instead of in the DuckDB driver, which makes the abstractions neater
- This also enables moving the concurrent download logic into the
-
I think overloading the
blob
connector to also work for local files is making things more complex than they need to be. Local files feel pretty different, since they don't need to be copied to a temp directory and deleted. I would consider just proxying the pattern directly to duckdb as we did before, even for globs.
Hey Benjamin Thanks for the review comments.
|
@k-anshul, about (1.), that reminds me, we actually have some notes on how to improve this abstraction by using an iterator object (like the "orchestrator" you describe): #1179. Maybe this will address your concern? If you like this idea, then since we'd like to merge this into a nightly soon, I'd suggest just returning a list of paths for an MVP (even if it means using more disk space). Then, we can add the iterator afterwards to improve performance. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also consider refactoring the blob
connector into an actual connector implementing the Connector
interface (i.e. just ConsumeAsFile
). Then the s3
and gcs
connectors could just map their config keys to a generic blob
config object, and then proxy to the blob
connector.
Update: This maybe isn't such a good idea since it would expose the blob connector directly to the end user, which might not be ideal. But the idea of moving as much logic into blob
as possible to make the s3
and gcs
implementations as simple as possible I think still holds.
Maybe refactor FetchFileNames
into a NewBlobHandler
constructor, and add a ConsumeAsFiles
implementation on BlobHandler
?
MaxSize int64 `yaml:"glob.max_size,omitempty" mapstructure:"glob.max_size,omitempty"` | ||
MaxDownload int `yaml:"glob.max_download,omitempty" mapstructure:"glob.max_download,omitempty"` | ||
MaxIterations int64 `yaml:"glob.max_iterations,omitempty" mapstructure:"glob.max_iterations,omitempty"` | ||
PageSize int `yaml:"glob.page_size,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's update these to match the names in the blob
connector
runtime/drivers/duckdb/connectors.go
Outdated
func getMultiSourceReader(paths []string) (string, error) { | ||
ext := fileutil.FullExt(paths[0]) | ||
dir := filepath.Dir(paths[0]) | ||
if ext == "" { | ||
return "", fmt.Errorf("invalid file") | ||
} else if strings.Contains(ext, ".csv") || strings.Contains(ext, ".tsv") || strings.Contains(ext, ".txt") { | ||
return fmt.Sprintf("read_csv_auto('%s/*%s')", dir, ext), nil | ||
} else if strings.Contains(ext, ".parquet") { | ||
return fmt.Sprintf("read_parquet('%s/*%s')", dir, ext), nil | ||
} else { | ||
return "", fmt.Errorf("file type not supported : %s", ext) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Inferring a glob from
paths[0]
is too fragile. Theread_parquet
file supports passing a list of files (see docs), maybe theread_csv_auto
does too? - This function also replaces
getSourceReader(path string)
as it is just a more generic version of it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- The reason I am reading all files from a directory is because passing the entire list of files in a sql query can potentially reach the query limit if any(haven't been able to find any in the docs although some sql systems have limits on the number of characters in a query).
That's the reason I am downloading the files in a separate temp directory per ingestion and passing the directory itself to duck db.
I agree to your point that inferring dir path from filename can be fragile. We can think of passing the directory name itself from the connector if required.
Please let me know your thoughts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- In case we continue to load all data form directory we will need getSourceReader for local ingestion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested and there doesn't appear to be a query limit. So let's do it the "right" way and pass each file separately. Here's the test script I used:
import duckdb
import os.path
n = 100000
paths = []
for i in range (0, n):
name = f"data/data_{i}.csv"
f = open(name, "w")
f.write(f"a,b\n{i},{i}\n")
f.close()
paths.append(os.path.join(os.getcwd(), name))
sql = "create table foo as select * from "
sql += "read_csv_auto(['"
sql += "','".join(paths)
sql += "'])"
con = duckdb.connect(database=':memory:')
con.execute(sql)
con.execute("select * from foo limit 20")
print(con.fetchall())
runtime/drivers/duckdb/connectors.go
Outdated
func (c *connection) ingestLocalGlob(ctx context.Context, source *connectors.Source, glob string) error { | ||
query := fmt.Sprintf("CREATE OR REPLACE TABLE %s AS (SELECT * FROM '%s');", source.Name, glob) | ||
c.logger.Info("running query %v", zap.String("query", query)) | ||
return c.Exec(ctx, &drivers.Statement{Query: query, Priority: 1}) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function ingestFile
is only used for local files. Rename it to ingestLocalFile
and merge this logic into it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ingestFile has lot of path resolution logic. Also the query pattern for glob ingestion and file ingestion is lot different. I would prefer to keep both functions different. We can rename ingestFile to ingestLocalFile.
Please let me know your thoughts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For local files, I think we also need the logic in ingestFile
for globs. Namely, that paths should be resolved relative to env.RepoDSN
and that we should support custom delimiters for CSVs. So that's why I think it would be best to make it one function.
This seems similar to our initial PrepareBlob and then using blobHandler to downloadFiles implementation. |
It seem the duckdb isn't able to identify file types from glob patterns |
It's okay to just proxy to DuckDB for now, but would be nice to add a better local walker once we've got the main functionality in this PR in place. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In addition to the below, I still feel the blob
package is more convoluted than necessary. I would suggest moving things into one file and structuring it something like this:
package blob
type Options struct {
Path string // maybe contains a glob pattern
MaxTotalSize int64
// other glob options
}
func (o *Options) validate() error {
// ...
}
type Connector struct {
bucket *blob.Bucket
opts *Options
}
func New(bucket, opts) (*Connector, err) {
// 1. validate opts
// 2. create and return connector
}
func (c *Connector) ConsumeAsFiles(ctx) ([]string, error) {
// 1. filenames, err := c.fetchFileNames(ctx)
// 2. tmps, err = c.downloadAll(ctx, filenames)
// 3. return tmps
}
func (c *Connector) fetchFileNames(ctx) // ...
func (c *Connector) downloadAll(ctx) // ...
func (c *Connector) downloadSingle(ctx) // ...
func (c *Connector) validateLimits(size, listed, matched) // ... can use c.opts to read limits
It's won't be a normal connector since it won't be exposed through the Connector interface, instead it's a util connector since it'll be used and configured by the S3 and GCS connectors (who are responsible for connecting to the bucket and passing it in).
What do you think?
if conf.Format == ".csv" && conf.CSVDelimiter != "" { | ||
from = fmt.Sprintf("read_csv_auto('%s', delim='%s')", path, conf.CSVDelimiter) | ||
from = fmt.Sprintf("read_csv_auto(['%s'], delim='%s')", path, conf.CSVDelimiter) | ||
} else { | ||
from, err = getSourceReader(path) | ||
from, err = getSourceReader(localPaths) | ||
if err != nil { | ||
return err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
conf.Format
is just based on the path, not fileutil.FullExt(paths[0])
as in getSourceReader
. This can lead to the CSV delimiter not propagating when the glob pattern doesn't end in .csv
as you pointed out. Maybe simplify and pass CSVDelimiter
into getSourceReader
instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, it could actually just call ingestFiles
after expanding the glob with doublestar
, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually that's not possible. If glob pattern doesn't end with .csv that we would always get complete path from doublestar.Filepath and would load it using file names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but the ParseConfig
func doesn't do that when it sets conf.Format
. Hence, it's best not to rely on it.
// for files downloaded locally from remote sources | ||
func (c *connection) ingestFiles(ctx context.Context, source *connectors.Source, filenames []string) error { | ||
from, err := getSourceReader(filenames) | ||
if err != nil { | ||
return err | ||
} | ||
query := fmt.Sprintf("CREATE OR REPLACE TABLE %s AS (SELECT * FROM %s);", source.Name, from) | ||
return c.Exec(ctx, &drivers.Statement{Query: query, Priority: 1}) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, we should also pass CSVDelimiter
to getSourceReader
here (for remote sources)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the existing functionality also the CSVDelimiter seems to be getting used only for locally uploaded files. Is this a new functionality that we want to give to users ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's probably a mistake. So let's fix it here by passing it into getSourceReader
and setting it if it's a CSV
// increasing this limit can increase speed ingestion | ||
// but may increase bottleneck at duckdb or network/db IO | ||
// set without any benchamarks | ||
const concurrentBlobDownloadLimit = 8 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also make this a config parameter, defaulting to 8. Let's call it GlobMaxConcurrentDownloads
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a very implementation specific config. I think this shouldn't be exposed to the user.
Also although tough to argue without any benchmarks but we should keep the default high since the go routines are expected to be mostly busy in IO only.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like this to be configurable since I'm not sure if some users might run into gateway problems (e.g. if there are limits on open connections to same host on single OS, or someone uses an S3-compatible data lake that has rate limits, etc.). In support, it might be handy to have people try reduce this to 1.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure about the internal impl of the go cdk but it may already be having a connection pool on openBucket call. Will try to check in the code.
This sounds good. However I would definitely want to give downloadAll and downloadSingle option to clients. |
The only clients are the S3 and GCS connectors – and they configure by either passing a glob or non-glob path. They know that only a single download happens if they don't pass a glob. It's better to have minimal public interfaces and keep internals private. If a future use case needs more granular access to the internals, then we can refactor private functions to public functions, but not before :) |
As I mentioned earlier downloading all files at one shot may not be a very good impl. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving and merging to get this into a nightly – it mostly works as expected now. There are some review comments and minor problems (listed in the issue) that remain to be fixed, but let's do that in a new PR.
* Use RefreshedOn in runtime query cache (#1559) * Remove timestamp required from CTA buttons * Dashboard without time dimension * Remove logs * Add lint fixes * Address review comments * Use single container component * Add grid to measures * Increase max size for measures * Metrics View: Value exclusion should keep nulls (#1535) * exclude with nulls * exclude with nulls * exclude with nulls for timeseries * Add explicit is null check instead of coalesce with UUID * fix failing test * Adress review + Resolve filter code duplication Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Remove redundant import * Fix imports * Remove time params * Connection pool for DuckDB (#1405) * connection pooling for duckdb * close channel * set pool size * fix test * fix stopping of pw * formatting * start mulitple jobs are getting unpaused * conn pool dequeue * fix test * better concurrency test of priority worker * comment * Merging with context affinity * Pass pool size through DSN * Formatting * Added a priority semaphore * Update duckdb driver to use priority semaphore * Maybe fix failing test case * Document priorityqueue.Semaphore * Bind time series to one connection * Use sqlx.Conn + fix hanging errors * Temporarily committing spending benchmark in examples * fix failing test * formatting * Commit js benchmark * Removing benchmarks * duckdb driver fix, tests with pool size > 1, separate meta connection * use conn from pool in migrate * use single conn in migrate * use built in connection pool * fix conn release race condition * fix linter errors * fmt * gofmt fix * gofmt fix * fix tests * upgrade duckdb driver * Meta and OLAP sems; ensuring safe release * Use WithConnection for temporary tables * formatting * Review Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Improve handling of time ranges (#1560) * Improve handling of time ranges (maybe) * Run prettier * Update tests * Update web-local/src/lib/temp/time-control-types.ts Co-authored-by: Hamilton Ulmer <[email protected]> * Make `rill init` Fail When Passed Arguments (Probably By Accident) (#1566) Make `rill init --example <example>` fail instead of quietly using the "default" example project when used instead of `--example=example`. The latter form is required due to a bug in cobra. See #1398 (comment) * Left align measures * Change copy, add tooltips * Remove CTA from metrics no timeseries icon * dyanamic number of columns upto 3 based on measures * Fix nightly builds * Remove old runtime release CI * more from r to f for integer formatting in summary numbers (#1572) * Show error on refresh failure (#1492) * Show error on refresh failure * Improving error display in sources * Fix lint * Fixing navigation blocked during profiling (#1561) * Release notes 0.18 (#1565) * release notes for 0.18 * Update 0.18.md Co-authored-by: Marissa Gorlick <[email protected]> * Handle long labels * Increase margin top to 20 * For measures with timeseries align center * Align measures on the left side * Align measures with timecharts * Align measures from the top * Increase gap between measures * use observer in measures container * changes the edit tooltip (#1575) * When a model query is blank, hide the table & inspector (#1568) * adds in a placeholder when a model is blank * ensures preview table is hidden when model is empty * makes a more dramatic yet more subtle style for empty models * Fix S3 Region not recognised (#1583) * changing aws.region to region * fixing ci * Add S3/GCS region to docs (#1581) * Add region to docs As proposed in #1577 * Update project-files.md * Update project-files.md * Check rows.Err() after all relevant queries (#1540) * check rows.Err() after all relevant DB queries * removed extra lines * fixed test cases * requested changes * adding err check for drivers.Result * revert deleted code * Use measure height and prevent overflow * Dispatch trigger for nightlies (#1586) * Testing trigger dynamics * Enable manual nightly trigger * Typo * Improve comments * Glob support for local_file, S3 and GCS (#1578) * initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Adding gci for import grouping (#1587) * adding gci for import groupin * cleanup requested changes * Run DuckDB tests in CI (#1584) * fix olap test * run duckdb olap test always * Reflow measures on window resize * Fix spinner alignment * reflow on resize * Use max content for measures Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Aditya Hegde <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]>
* queries-caching-impl * queries-caching-impl * caching for time grain query * caching: column numeric histogram * caching: column numeric histogram * caching: table cardinality fix * caching: rug histogram * caching: time range * caching: time range * caching: column cardinality * caching: rollup interval * caching: column with all nulls * caching: code style * caching: rug * caching: code style * caching: ts fix * caching: ts fix * caching: ts fix * caching: code style * caching: timeseries * caching: table columns * caching: metricsview totals * caching: metricsview toplist * caching: metricsview toplist * caching: metricsview timeseries * Making metrics dashboard less strict * adds rudimentary dashboard support for no time series * Removing model path filter (#1488) * Adding support for reading public buckets (#1491) * Refresh when local file changes (#1489) * Refresh when local file changes * Fixing test failure * spaces in timestamp column name (#1490) * spaces in timestamp column name * spaces in timestamp column name * spaces in timestamp column name Co-authored-by: egor-ryashin <[email protected]> * npm install * Temporarily disable docker releases * Setup golangci-lint in rill-developer (#1417) * golangci-lint initial commit * added new line * adding .golangci.yml config * latest version and comment GH action on pull_request * go version update * go version update + prettier * for current branch * typo * run with config file * adding timeout * commenting disable all * default run * setup go changes * go setup changes * lint version latest * Adding golangci.yml file * with config arg * adding goreportcard-action * removed some linters from enable list * fix suggested by golangci linters * updaing GH action args with config and verbose * errors fix by linters * trigger ci check * trigger ci check * trigger ci check * trigger ci check * fixed errcheck issues * on push main branch and pull_request * adding golangci-lint-setup for test * golangci-lint error fix * more fixes * fixed statuscode const etc * fixes after merging with main * PR requested changes * Fixed PR changes and other errors * prettier on go-test.yml * Final changes * fixed PR requested changes * fixed some more changes after merge with main branch * reverted errcheck and gofumpt and fixed the changes * gofmt * updated CONTRIBUTING.md for golint setup * updated GH actions for golint * test action * fixed some more errors from linters * updated GH action and fixed typo * udpated CONTRIBUTING.md for golint instructions * changes in GH actions for golint * Release notes 0.17 (#1497) * release notes for 0.17 * fixing static image in docusaurus * Maybe fixes builds Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Update 0.17.md * Fix suppress tooltip not being reactive (#1493) * [Dashboard] Fix jarring rearrange when include/exclude is toggled from filter pills (#1494) * Fixing misused older types * Fixing the jarring rearrange on toggling include/exclude * Update help links to new docs structure (#1499) Co-authored-by: Marissa Gorlick <[email protected]> * Move design system to `web-common` (#1486) * Move files from `web-local` to `web-common` * Move files out of design system and into app * Move utils into `web-common` * Add typescript & eslint config files to `web-common` * Update imports (design system) * Delete blank file * Update imports (application) * Move duckdb data types to `web-common` * Fix import of chip component * Fix two missing imports * Edit path in the menu component's README * Add `svelte` as a `web-common` dependency * Remove old paths from typscript and vite configs * Fix imports in `/dev` routes * Add back imports to fix lint * Add `web-common` to tailwind config * Move to one top-level `eslint` file * Move `data-types` into `web-common/components` * Move `data-graphic` into `web-common/components` * Bugfix: show nav border * Add more data file extensions to `.prettierignore` * Fix prettier warning * Publish nightly docker image (#1500) * golint improvements (#1502) * skip proto files and only-new-issues is set true * testing with adding few errors * revert errors and commented dupl * adding dupl * dupl with thresold setting * removed dupl linter * removing only-new-issues * Improved dimension validations * Addressing PR comments * Enable dashboards with no time dimension (#1544) * Use RefreshedOn in runtime query cache (#1559) * Remove timestamp required from CTA buttons * Dashboard without time dimension * Remove logs * Add lint fixes * Address review comments * Use single container component * Add grid to measures * Increase max size for measures * Metrics View: Value exclusion should keep nulls (#1535) * exclude with nulls * exclude with nulls * exclude with nulls for timeseries * Add explicit is null check instead of coalesce with UUID * fix failing test * Adress review + Resolve filter code duplication Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Remove redundant import * Fix imports * Remove time params * Connection pool for DuckDB (#1405) * connection pooling for duckdb * close channel * set pool size * fix test * fix stopping of pw * formatting * start mulitple jobs are getting unpaused * conn pool dequeue * fix test * better concurrency test of priority worker * comment * Merging with context affinity * Pass pool size through DSN * Formatting * Added a priority semaphore * Update duckdb driver to use priority semaphore * Maybe fix failing test case * Document priorityqueue.Semaphore * Bind time series to one connection * Use sqlx.Conn + fix hanging errors * Temporarily committing spending benchmark in examples * fix failing test * formatting * Commit js benchmark * Removing benchmarks * duckdb driver fix, tests with pool size > 1, separate meta connection * use conn from pool in migrate * use single conn in migrate * use built in connection pool * fix conn release race condition * fix linter errors * fmt * gofmt fix * gofmt fix * fix tests * upgrade duckdb driver * Meta and OLAP sems; ensuring safe release * Use WithConnection for temporary tables * formatting * Review Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Improve handling of time ranges (#1560) * Improve handling of time ranges (maybe) * Run prettier * Update tests * Update web-local/src/lib/temp/time-control-types.ts Co-authored-by: Hamilton Ulmer <[email protected]> * Make `rill init` Fail When Passed Arguments (Probably By Accident) (#1566) Make `rill init --example <example>` fail instead of quietly using the "default" example project when used instead of `--example=example`. The latter form is required due to a bug in cobra. See #1398 (comment) * Left align measures * Change copy, add tooltips * Remove CTA from metrics no timeseries icon * dyanamic number of columns upto 3 based on measures * Fix nightly builds * Remove old runtime release CI * more from r to f for integer formatting in summary numbers (#1572) * Show error on refresh failure (#1492) * Show error on refresh failure * Improving error display in sources * Fix lint * Fixing navigation blocked during profiling (#1561) * Release notes 0.18 (#1565) * release notes for 0.18 * Update 0.18.md Co-authored-by: Marissa Gorlick <[email protected]> * Handle long labels * Increase margin top to 20 * For measures with timeseries align center * Align measures on the left side * Align measures with timecharts * Align measures from the top * Increase gap between measures * use observer in measures container * changes the edit tooltip (#1575) * When a model query is blank, hide the table & inspector (#1568) * adds in a placeholder when a model is blank * ensures preview table is hidden when model is empty * makes a more dramatic yet more subtle style for empty models * Fix S3 Region not recognised (#1583) * changing aws.region to region * fixing ci * Add S3/GCS region to docs (#1581) * Add region to docs As proposed in #1577 * Update project-files.md * Update project-files.md * Check rows.Err() after all relevant queries (#1540) * check rows.Err() after all relevant DB queries * removed extra lines * fixed test cases * requested changes * adding err check for drivers.Result * revert deleted code * Use measure height and prevent overflow * Dispatch trigger for nightlies (#1586) * Testing trigger dynamics * Enable manual nightly trigger * Typo * Improve comments * Glob support for local_file, S3 and GCS (#1578) * initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Adding gci for import grouping (#1587) * adding gci for import groupin * cleanup requested changes * Run DuckDB tests in CI (#1584) * fix olap test * run duckdb olap test always * Reflow measures on window resize * Fix spinner alignment * reflow on resize * Use max content for measures Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Aditya Hegde <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]> * Check for zero heights * address PR comments * Increase column gap and fix node bug Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Eric P Green <[email protected]> Co-authored-by: Himadri Singh <[email protected]> Co-authored-by: Dhiraj Barnwal <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]>
* initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]>
* queries-caching-impl * queries-caching-impl * caching for time grain query * caching: column numeric histogram * caching: column numeric histogram * caching: table cardinality fix * caching: rug histogram * caching: time range * caching: time range * caching: column cardinality * caching: rollup interval * caching: column with all nulls * caching: code style * caching: rug * caching: code style * caching: ts fix * caching: ts fix * caching: ts fix * caching: code style * caching: timeseries * caching: table columns * caching: metricsview totals * caching: metricsview toplist * caching: metricsview toplist * caching: metricsview timeseries * Making metrics dashboard less strict * adds rudimentary dashboard support for no time series * Removing model path filter (#1488) * Adding support for reading public buckets (#1491) * Refresh when local file changes (#1489) * Refresh when local file changes * Fixing test failure * spaces in timestamp column name (#1490) * spaces in timestamp column name * spaces in timestamp column name * spaces in timestamp column name Co-authored-by: egor-ryashin <[email protected]> * npm install * Temporarily disable docker releases * Setup golangci-lint in rill-developer (#1417) * golangci-lint initial commit * added new line * adding .golangci.yml config * latest version and comment GH action on pull_request * go version update * go version update + prettier * for current branch * typo * run with config file * adding timeout * commenting disable all * default run * setup go changes * go setup changes * lint version latest * Adding golangci.yml file * with config arg * adding goreportcard-action * removed some linters from enable list * fix suggested by golangci linters * updaing GH action args with config and verbose * errors fix by linters * trigger ci check * trigger ci check * trigger ci check * trigger ci check * fixed errcheck issues * on push main branch and pull_request * adding golangci-lint-setup for test * golangci-lint error fix * more fixes * fixed statuscode const etc * fixes after merging with main * PR requested changes * Fixed PR changes and other errors * prettier on go-test.yml * Final changes * fixed PR requested changes * fixed some more changes after merge with main branch * reverted errcheck and gofumpt and fixed the changes * gofmt * updated CONTRIBUTING.md for golint setup * updated GH actions for golint * test action * fixed some more errors from linters * updated GH action and fixed typo * udpated CONTRIBUTING.md for golint instructions * changes in GH actions for golint * Release notes 0.17 (#1497) * release notes for 0.17 * fixing static image in docusaurus * Maybe fixes builds Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Update 0.17.md * Fix suppress tooltip not being reactive (#1493) * [Dashboard] Fix jarring rearrange when include/exclude is toggled from filter pills (#1494) * Fixing misused older types * Fixing the jarring rearrange on toggling include/exclude * Update help links to new docs structure (#1499) Co-authored-by: Marissa Gorlick <[email protected]> * Move design system to `web-common` (#1486) * Move files from `web-local` to `web-common` * Move files out of design system and into app * Move utils into `web-common` * Add typescript & eslint config files to `web-common` * Update imports (design system) * Delete blank file * Update imports (application) * Move duckdb data types to `web-common` * Fix import of chip component * Fix two missing imports * Edit path in the menu component's README * Add `svelte` as a `web-common` dependency * Remove old paths from typscript and vite configs * Fix imports in `/dev` routes * Add back imports to fix lint * Add `web-common` to tailwind config * Move to one top-level `eslint` file * Move `data-types` into `web-common/components` * Move `data-graphic` into `web-common/components` * Bugfix: show nav border * Add more data file extensions to `.prettierignore` * Fix prettier warning * Publish nightly docker image (#1500) * golint improvements (#1502) * skip proto files and only-new-issues is set true * testing with adding few errors * revert errors and commented dupl * adding dupl * dupl with thresold setting * removed dupl linter * removing only-new-issues * Improved dimension validations * Addressing PR comments * Enable dashboards with no time dimension (#1544) * Use RefreshedOn in runtime query cache (#1559) * Remove timestamp required from CTA buttons * Dashboard without time dimension * Remove logs * Add lint fixes * Address review comments * Use single container component * Add grid to measures * Increase max size for measures * Metrics View: Value exclusion should keep nulls (#1535) * exclude with nulls * exclude with nulls * exclude with nulls for timeseries * Add explicit is null check instead of coalesce with UUID * fix failing test * Adress review + Resolve filter code duplication Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Remove redundant import * Fix imports * Remove time params * Connection pool for DuckDB (#1405) * connection pooling for duckdb * close channel * set pool size * fix test * fix stopping of pw * formatting * start mulitple jobs are getting unpaused * conn pool dequeue * fix test * better concurrency test of priority worker * comment * Merging with context affinity * Pass pool size through DSN * Formatting * Added a priority semaphore * Update duckdb driver to use priority semaphore * Maybe fix failing test case * Document priorityqueue.Semaphore * Bind time series to one connection * Use sqlx.Conn + fix hanging errors * Temporarily committing spending benchmark in examples * fix failing test * formatting * Commit js benchmark * Removing benchmarks * duckdb driver fix, tests with pool size > 1, separate meta connection * use conn from pool in migrate * use single conn in migrate * use built in connection pool * fix conn release race condition * fix linter errors * fmt * gofmt fix * gofmt fix * fix tests * upgrade duckdb driver * Meta and OLAP sems; ensuring safe release * Use WithConnection for temporary tables * formatting * Review Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Improve handling of time ranges (#1560) * Improve handling of time ranges (maybe) * Run prettier * Update tests * Update web-local/src/lib/temp/time-control-types.ts Co-authored-by: Hamilton Ulmer <[email protected]> * Make `rill init` Fail When Passed Arguments (Probably By Accident) (#1566) Make `rill init --example <example>` fail instead of quietly using the "default" example project when used instead of `--example=example`. The latter form is required due to a bug in cobra. See #1398 (comment) * Left align measures * Change copy, add tooltips * Remove CTA from metrics no timeseries icon * dyanamic number of columns upto 3 based on measures * Fix nightly builds * Remove old runtime release CI * more from r to f for integer formatting in summary numbers (#1572) * Show error on refresh failure (#1492) * Show error on refresh failure * Improving error display in sources * Fix lint * Fixing navigation blocked during profiling (#1561) * Release notes 0.18 (#1565) * release notes for 0.18 * Update 0.18.md Co-authored-by: Marissa Gorlick <[email protected]> * Handle long labels * Increase margin top to 20 * For measures with timeseries align center * Align measures on the left side * Align measures with timecharts * Align measures from the top * Increase gap between measures * use observer in measures container * changes the edit tooltip (#1575) * When a model query is blank, hide the table & inspector (#1568) * adds in a placeholder when a model is blank * ensures preview table is hidden when model is empty * makes a more dramatic yet more subtle style for empty models * Fix S3 Region not recognised (#1583) * changing aws.region to region * fixing ci * Add S3/GCS region to docs (#1581) * Add region to docs As proposed in #1577 * Update project-files.md * Update project-files.md * Check rows.Err() after all relevant queries (#1540) * check rows.Err() after all relevant DB queries * removed extra lines * fixed test cases * requested changes * adding err check for drivers.Result * revert deleted code * Use measure height and prevent overflow * Dispatch trigger for nightlies (#1586) * Testing trigger dynamics * Enable manual nightly trigger * Typo * Improve comments * Glob support for local_file, S3 and GCS (#1578) * initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Adding gci for import grouping (#1587) * adding gci for import groupin * cleanup requested changes * Run DuckDB tests in CI (#1584) * fix olap test * run duckdb olap test always * Reflow measures on window resize * Fix spinner alignment * reflow on resize * Use max content for measures Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Aditya Hegde <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]> * Check for zero heights * address PR comments * Increase column gap and fix node bug Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Eric P Green <[email protected]> Co-authored-by: Himadri Singh <[email protected]> Co-authored-by: Dhiraj Barnwal <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]>
* initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]>
* queries-caching-impl * queries-caching-impl * caching for time grain query * caching: column numeric histogram * caching: column numeric histogram * caching: table cardinality fix * caching: rug histogram * caching: time range * caching: time range * caching: column cardinality * caching: rollup interval * caching: column with all nulls * caching: code style * caching: rug * caching: code style * caching: ts fix * caching: ts fix * caching: ts fix * caching: code style * caching: timeseries * caching: table columns * caching: metricsview totals * caching: metricsview toplist * caching: metricsview toplist * caching: metricsview timeseries * Making metrics dashboard less strict * adds rudimentary dashboard support for no time series * Removing model path filter (#1488) * Adding support for reading public buckets (#1491) * Refresh when local file changes (#1489) * Refresh when local file changes * Fixing test failure * spaces in timestamp column name (#1490) * spaces in timestamp column name * spaces in timestamp column name * spaces in timestamp column name Co-authored-by: egor-ryashin <[email protected]> * npm install * Temporarily disable docker releases * Setup golangci-lint in rill-developer (#1417) * golangci-lint initial commit * added new line * adding .golangci.yml config * latest version and comment GH action on pull_request * go version update * go version update + prettier * for current branch * typo * run with config file * adding timeout * commenting disable all * default run * setup go changes * go setup changes * lint version latest * Adding golangci.yml file * with config arg * adding goreportcard-action * removed some linters from enable list * fix suggested by golangci linters * updaing GH action args with config and verbose * errors fix by linters * trigger ci check * trigger ci check * trigger ci check * trigger ci check * fixed errcheck issues * on push main branch and pull_request * adding golangci-lint-setup for test * golangci-lint error fix * more fixes * fixed statuscode const etc * fixes after merging with main * PR requested changes * Fixed PR changes and other errors * prettier on go-test.yml * Final changes * fixed PR requested changes * fixed some more changes after merge with main branch * reverted errcheck and gofumpt and fixed the changes * gofmt * updated CONTRIBUTING.md for golint setup * updated GH actions for golint * test action * fixed some more errors from linters * updated GH action and fixed typo * udpated CONTRIBUTING.md for golint instructions * changes in GH actions for golint * Release notes 0.17 (#1497) * release notes for 0.17 * fixing static image in docusaurus * Maybe fixes builds Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Update 0.17.md * Fix suppress tooltip not being reactive (#1493) * [Dashboard] Fix jarring rearrange when include/exclude is toggled from filter pills (#1494) * Fixing misused older types * Fixing the jarring rearrange on toggling include/exclude * Update help links to new docs structure (#1499) Co-authored-by: Marissa Gorlick <[email protected]> * Move design system to `web-common` (#1486) * Move files from `web-local` to `web-common` * Move files out of design system and into app * Move utils into `web-common` * Add typescript & eslint config files to `web-common` * Update imports (design system) * Delete blank file * Update imports (application) * Move duckdb data types to `web-common` * Fix import of chip component * Fix two missing imports * Edit path in the menu component's README * Add `svelte` as a `web-common` dependency * Remove old paths from typscript and vite configs * Fix imports in `/dev` routes * Add back imports to fix lint * Add `web-common` to tailwind config * Move to one top-level `eslint` file * Move `data-types` into `web-common/components` * Move `data-graphic` into `web-common/components` * Bugfix: show nav border * Add more data file extensions to `.prettierignore` * Fix prettier warning * Publish nightly docker image (#1500) * golint improvements (#1502) * skip proto files and only-new-issues is set true * testing with adding few errors * revert errors and commented dupl * adding dupl * dupl with thresold setting * removed dupl linter * removing only-new-issues * Improved dimension validations * Addressing PR comments * Enable dashboards with no time dimension (#1544) * Use RefreshedOn in runtime query cache (#1559) * Remove timestamp required from CTA buttons * Dashboard without time dimension * Remove logs * Add lint fixes * Address review comments * Use single container component * Add grid to measures * Increase max size for measures * Metrics View: Value exclusion should keep nulls (#1535) * exclude with nulls * exclude with nulls * exclude with nulls for timeseries * Add explicit is null check instead of coalesce with UUID * fix failing test * Adress review + Resolve filter code duplication Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Remove redundant import * Fix imports * Remove time params * Connection pool for DuckDB (#1405) * connection pooling for duckdb * close channel * set pool size * fix test * fix stopping of pw * formatting * start mulitple jobs are getting unpaused * conn pool dequeue * fix test * better concurrency test of priority worker * comment * Merging with context affinity * Pass pool size through DSN * Formatting * Added a priority semaphore * Update duckdb driver to use priority semaphore * Maybe fix failing test case * Document priorityqueue.Semaphore * Bind time series to one connection * Use sqlx.Conn + fix hanging errors * Temporarily committing spending benchmark in examples * fix failing test * formatting * Commit js benchmark * Removing benchmarks * duckdb driver fix, tests with pool size > 1, separate meta connection * use conn from pool in migrate * use single conn in migrate * use built in connection pool * fix conn release race condition * fix linter errors * fmt * gofmt fix * gofmt fix * fix tests * upgrade duckdb driver * Meta and OLAP sems; ensuring safe release * Use WithConnection for temporary tables * formatting * Review Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Improve handling of time ranges (#1560) * Improve handling of time ranges (maybe) * Run prettier * Update tests * Update web-local/src/lib/temp/time-control-types.ts Co-authored-by: Hamilton Ulmer <[email protected]> * Make `rill init` Fail When Passed Arguments (Probably By Accident) (#1566) Make `rill init --example <example>` fail instead of quietly using the "default" example project when used instead of `--example=example`. The latter form is required due to a bug in cobra. See #1398 (comment) * Left align measures * Change copy, add tooltips * Remove CTA from metrics no timeseries icon * dyanamic number of columns upto 3 based on measures * Fix nightly builds * Remove old runtime release CI * more from r to f for integer formatting in summary numbers (#1572) * Show error on refresh failure (#1492) * Show error on refresh failure * Improving error display in sources * Fix lint * Fixing navigation blocked during profiling (#1561) * Release notes 0.18 (#1565) * release notes for 0.18 * Update 0.18.md Co-authored-by: Marissa Gorlick <[email protected]> * Handle long labels * Increase margin top to 20 * For measures with timeseries align center * Align measures on the left side * Align measures with timecharts * Align measures from the top * Increase gap between measures * use observer in measures container * changes the edit tooltip (#1575) * When a model query is blank, hide the table & inspector (#1568) * adds in a placeholder when a model is blank * ensures preview table is hidden when model is empty * makes a more dramatic yet more subtle style for empty models * Fix S3 Region not recognised (#1583) * changing aws.region to region * fixing ci * Add S3/GCS region to docs (#1581) * Add region to docs As proposed in #1577 * Update project-files.md * Update project-files.md * Check rows.Err() after all relevant queries (#1540) * check rows.Err() after all relevant DB queries * removed extra lines * fixed test cases * requested changes * adding err check for drivers.Result * revert deleted code * Use measure height and prevent overflow * Dispatch trigger for nightlies (#1586) * Testing trigger dynamics * Enable manual nightly trigger * Typo * Improve comments * Glob support for local_file, S3 and GCS (#1578) * initial commit * download plus ingestion support * adding config for limit * linting issues * linting issues * const name declaration * ut fix * config fix * config fix * batch mode fix * read parquet changes * ingestion by folder * review comments * unused code * relative local paths * local invalid path handling * review nit picks * review nit picks * Update error messages Co-authored-by: anshul khandelwal <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> * Adding gci for import grouping (#1587) * adding gci for import groupin * cleanup requested changes * Run DuckDB tests in CI (#1584) * fix olap test * run duckdb olap test always * Reflow measures on window resize * Fix spinner alignment * reflow on resize * Use max content for measures Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Aditya Hegde <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]> * Check for zero heights * address PR comments * Increase column gap and fix node bug Co-authored-by: egor-ryashin <[email protected]> Co-authored-by: Hamilton Ulmer <[email protected]> Co-authored-by: Egor Riashin <[email protected]> Co-authored-by: Benjamin Egelund-Müller <[email protected]> Co-authored-by: rakeshsharma14317 <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: Eric P Green <[email protected]> Co-authored-by: Himadri Singh <[email protected]> Co-authored-by: Dhiraj Barnwal <[email protected]> Co-authored-by: Nishant Bangarwa <[email protected]> Co-authored-by: Parag Jain <[email protected]> Co-authored-by: Christian G. Warden <[email protected]> Co-authored-by: Marissa Gorlick <[email protected]> Co-authored-by: marcelteraflow <[email protected]> Co-authored-by: Anshul <[email protected]> Co-authored-by: anshul khandelwal <[email protected]>
No description provided.