connection pooling for duckdb

[pjain1]

See details and todos in issue: #1316

[pjain1]

Modified priority worker test to test for concurrency and priority correctness. Still there are pending things to take care of -

1. Since we are creating DB handles, the way embedded duckdb is behaving is that for each DB handle (created using sql.Open) is a separate instance so schemas that are created through one handle are not available on other. If we go with this approach then we will have to limit pool size to 1 for in-memory db.
2. Introduce transactions/callback for handling temp tables used within an api call.
3. More testing to make sure correctness in case of multiple writes because of Multiple running database instances on same database file duckdb/duckdb#77 (comment)

[begelundmuller]

(Moved to top)

[begelundmuller]

Just a note that I experienced a bug when using this PR with pool size > 1 when doing an initial reconcile against an empty DB. I wonder if some information is not propagated across connections right away. Let's investigate further before merging.

[begelundmuller]

• Also note the linter errors in CI

[begelundmuller]

This should be db.SetMaxOpenConns(cfg.PoolSize) so we don't create more connections than expected (this can be useful when debugging). Instead, make the semaphore size:

semSize := cfg.PoolSize - 1
if semSize < 1 {
  semSize = 1
}

In the case where cfg.PoolSize is 1, it means the meta calls may block a while, but that's fine (since for prod use cases, we will use pool size > 1).

[pjain1]

using the built in pool so semSize not needed anymore, for now removed metaConn altogether. Will probably handle that separately

[pjain1]

when using built in pool, we will have to create and keep metaConn explicitly as we don't have control over the pool

[pjain1]

should we just panic here ?

[pjain1]

why ignore the error from release ? Should we log it in case there is any connection leak ?

[begelundmuller]

I guess we should. Looking in database/sql, it looks like the only error it can return is if you try to close the conn twice, so not handling it shouldn't lead to any leaks currently

[pjain1]

I guess Go code does not throw any runtime exceptions/errors like Java so it can be ignored and in case of exceptional situations like low memory etc. it should already panic and print in the logs? Only caveat is if in the future, the API changes and starts throwing new error but not a priority to handle now, can be added in the backlog.

[begelundmuller]

Yeah, panics will always propagate. Generally, it's good to handle all errors and only ignore specific named errors, since as you mention, more errors may be returned in future. But for defer statements, I often see people pick simpler code. I added a backlog to add good handling for all defer errors.

[pjain1]

should we catch and log error from rows.Close() instead of ignoring

[begelundmuller]

In a perfect world, yeah. We have this many places, we can add a backlog item for it

[begelundmuller]

I think this is now in a mergable state. I've also done an experience pass, and it seems good. Don't want to merge it until after code freeze, though. Will also do a second review with fresh eyes tomorrow.

[begelundmuller]

Second review looks good as well