glob support for local, s3 and gcs

[k-anshul]

No description provided.

[begelundmuller]

Thank you for this PR! Really happy to see that you were able to jump in and identify all the relevant places to make changes.

In addition to the various nits below, I suggest we change some abstractions:

1. Adding the PrepareBlob method to the Connector interface is not necessary. Instead, how about just renaming ConsumeAsFile to ConsumeAsFiles, which returns a list of one or more filenames to ingest instead? (In the future, we could update this to return a complex Iterator object that allows consuming downloaded files before the full download has completed – this is a feature we've been asked to look into soon.)

   • This also enables moving the concurrent download logic into the blob connector instead of in the DuckDB driver, which makes the abstractions neater

2. I think overloading the blob connector to also work for local files is making things more complex than they need to be. Local files feel pretty different, since they don't need to be copied to a temp directory and deleted. I would consider just proxying the pattern directly to duckdb as we did before, even for globs.

[k-anshul]

Hey Benjamin

Thanks for the review comments.

1. The reason I am downloading the objects from duckdb driver is to download files in batch mode and ingest in duck db. If we download objects directly in the connector than we will have to copy entire files first to the disk and then load it into db(increasing the max disk usage). Also with a blob handler its easy for us to migrate to a design where we may not even download the entire content to local filesystem and load in db but load partial results.
   I agree to your point that downloading files via driver doesn't seem very good. May be we need some orchestrator that consumes these files from connector and passes it to duck db which is currently being done in driver itself.
   Please let me know your thoughts

2. The reason I wanted to do that was to have a common handling of all connectors. We can pass the glob directly as well.

[begelundmuller]

@k-anshul, about (1.), that reminds me, we actually have some notes on how to improve this abstraction by using an iterator object (like the "orchestrator" you describe): #1179. Maybe this will address your concern?

If you like this idea, then since we'd like to merge this into a nightly soon, I'd suggest just returning a list of paths for an MVP (even if it means using more disk space). Then, we can add the iterator afterwards to improve performance.

[begelundmuller]

Also consider refactoring the blob connector into an actual connector implementing the Connector interface (i.e. just ConsumeAsFile). Then the s3 and gcs connectors could just map their config keys to a generic blob config object, and then proxy to the blob connector.

Update: This maybe isn't such a good idea since it would expose the blob connector directly to the end user, which might not be ideal. But the idea of moving as much logic into blob as possible to make the s3 and gcs implementations as simple as possible I think still holds.

Maybe refactor FetchFileNames into a NewBlobHandler constructor, and add a ConsumeAsFiles implementation on BlobHandler?

[begelundmuller]

Let's update these to match the names in the blob connector

[begelundmuller]

1. Inferring a glob from paths[0] is too fragile. The read_parquet file supports passing a list of files (see docs), maybe the read_csv_auto does too?
2. This function also replaces getSourceReader(path string) as it is just a more generic version of it

[k-anshul]

1. The reason I am reading all files from a directory is because passing the entire list of files in a sql query can potentially reach the query limit if any(haven't been able to find any in the docs although some sql systems have limits on the number of characters in a query).
   That's the reason I am downloading the files in a separate temp directory per ingestion and passing the directory itself to duck db.
   I agree to your point that inferring dir path from filename can be fragile. We can think of passing the directory name itself from the connector if required.
   Please let me know your thoughts.

[k-anshul]

2. In case we continue to load all data form directory we will need getSourceReader for local ingestion

[begelundmuller]

I tested and there doesn't appear to be a query limit. So let's do it the "right" way and pass each file separately. Here's the test script I used:

import duckdb
import os.path

n = 100000
paths = []
for i in range (0, n):
    name = f"data/data_{i}.csv"
    f = open(name, "w")
    f.write(f"a,b\n{i},{i}\n")
    f.close()
    paths.append(os.path.join(os.getcwd(), name))


sql = "create table foo as select * from "
sql += "read_csv_auto(['"
sql += "','".join(paths)
sql += "'])"

con = duckdb.connect(database=':memory:')
con.execute(sql)
con.execute("select * from foo limit 20")
print(con.fetchall())

[begelundmuller]

The function ingestFile is only used for local files. Rename it to ingestLocalFile and merge this logic into it

[k-anshul]

ingestFile has lot of path resolution logic. Also the query pattern for glob ingestion and file ingestion is lot different. I would prefer to keep both functions different. We can rename ingestFile to ingestLocalFile.
Please let me know your thoughts.

[begelundmuller]

For local files, I think we also need the logic in ingestFile for globs. Namely, that paths should be resolved relative to env.RepoDSN and that we should support custom delimiters for CSVs. So that's why I think it would be best to make it one function.

[k-anshul]

1. ng the PrepareBlob method to the Connector interface is

This seems similar to our initial PrepareBlob and then using blobHandler to downloadFiles implementation.
I would suggest lets keep the implementation same for this initial PR. In later PRs we can use the iterator kind of implementation that you had mentioned earlier.

[k-anshul]

It seem the duckdb isn't able to identify file types from glob patterns
For example with a gcs pattern like this : gs://<bucket>/folder/* we are able to indentify the file types and we are using the relevant read_parquet vs read_csv functions but proxying these kind of patterns directly to duckdb for local files aren't working.
Please let me know if we need to handle this for local files as well. In that case we can use doublestar filewalk

[begelundmuller]

It seem the duckdb isn't able to identify file types from glob patterns
For example with a gcs pattern like this : gs://<bucket>/folder/* we are able to indentify the file types and we are using the relevant read_parquet vs read_csv functions but proxying these kind of patterns directly to duckdb for local files aren't working.
Please let me know if we need to handle this for local files as well. In that case we can use doublestar filewalk

It's okay to just proxy to DuckDB for now, but would be nice to add a better local walker once we've got the main functionality in this PR in place.

[begelundmuller]

In addition to the below, I still feel the blob package is more convoluted than necessary. I would suggest moving things into one file and structuring it something like this:

package blob

type Options struct {
  Path string // maybe contains a glob pattern
  MaxTotalSize int64
  // other glob options
}

func (o *Options) validate() error {
  // ...
}

type Connector struct {
  bucket *blob.Bucket
  opts *Options
}

func New(bucket, opts) (*Connector, err) {
  // 1. validate opts
  // 2. create and return connector
}

func (c *Connector) ConsumeAsFiles(ctx) ([]string, error) {
  // 1. filenames, err := c.fetchFileNames(ctx)
  // 2. tmps, err = c.downloadAll(ctx, filenames)
  // 3. return tmps
}

func (c *Connector) fetchFileNames(ctx) // ...

func (c *Connector) downloadAll(ctx) // ...

func (c *Connector) downloadSingle(ctx) // ...

func (c *Connector) validateLimits(size, listed, matched) // ... can use c.opts to read limits

It's won't be a normal connector since it won't be exposed through the Connector interface, instead it's a util connector since it'll be used and configured by the S3 and GCS connectors (who are responsible for connecting to the bucket and passing it in).

What do you think?

[begelundmuller]

conf.Format is just based on the path, not fileutil.FullExt(paths[0]) as in getSourceReader. This can lead to the CSV delimiter not propagating when the glob pattern doesn't end in .csv as you pointed out. Maybe simplify and pass CSVDelimiter into getSourceReader instead?

[begelundmuller]

In that case, it could actually just call ingestFiles after expanding the glob with doublestar, right?

[k-anshul]

Actually that's not possible. If glob pattern doesn't end with .csv that we would always get complete path from doublestar.Filepath and would load it using file names.

[begelundmuller]

Yes, but the ParseConfig func doesn't do that when it sets conf.Format. Hence, it's best not to rely on it.

[begelundmuller]

Actually, we should also pass CSVDelimiter to getSourceReader here (for remote sources)

[k-anshul]

In the existing functionality also the CSVDelimiter seems to be getting used only for locally uploaded files. Is this a new functionality that we want to give to users ?

[begelundmuller]

Yeah, it's probably a mistake. So let's fix it here by passing it into getSourceReader and setting it if it's a CSV

[begelundmuller]

Let's also make this a config parameter, defaulting to 8. Let's call it GlobMaxConcurrentDownloads.

[k-anshul]

This seems like a very implementation specific config. I think this shouldn't be exposed to the user.
Also although tough to argue without any benchmarks but we should keep the default high since the go routines are expected to be mostly busy in IO only.

[begelundmuller]

I would like this to be configurable since I'm not sure if some users might run into gateway problems (e.g. if there are limits on open connections to same host on single OS, or someone uses an S3-compatible data lake that has rate limits, etc.). In support, it might be handy to have people try reduce this to 1.

[k-anshul]

Not sure about the internal impl of the go cdk but it may already be having a connection pool on openBucket call. Will try to check in the code.

[k-anshul]

runtime/connectors/blob/blobdownloader.go

This sounds good. However I would definitely want to give downloadAll and downloadSingle option to clients.

[begelundmuller]

runtime/connectors/blob/blobdownloader.go

This sounds good. However I would definitely want to give downloadAll and downloadSingle option to clients.

The only clients are the S3 and GCS connectors – and they configure by either passing a glob or non-glob path. They know that only a single download happens if they don't pass a glob.

It's better to have minimal public interfaces and keep internals private. If a future use case needs more granular access to the internals, then we can refactor private functions to public functions, but not before :)

[k-anshul]

runtime/connectors/blob/blobdownloader.go

This sounds good. However I would definitely want to give downloadAll and downloadSingle option to clients.

The only clients are the S3 and GCS connectors – and they configure by either passing a glob or non-glob path. They know that only a single download happens if they don't pass a glob.

It's better to have minimal public interfaces and keep internals private. If a future use case needs more granular access to the internals, then we can refactor private functions to public functions, but not before :)

As I mentioned earlier downloading all files at one shot may not be a very good impl.
There could be use cases where we want to download only a subset of files basis olap driver.
Also I would only expose download functionality and let clients run it in parallel if they want to speed up.

[begelundmuller]

Approving and merging to get this into a nightly – it mostly works as expected now. There are some review comments and minor problems (listed in the issue) that remain to be fixed, but let's do that in a new PR.