Create table

[pbsds]

Is PRQL strictly a query language, or should it also support defining and modifying the database schema/tables?

[max-sixty]

I think we should focus on the analytical query language, and leave DDL-like to SQL (or at least to the future).

A key assumption behind PRQL is that people spend a great deal of time writing purely analytical queries, and SQL is insufficiently focused on that need. If that's assumption is correct, we can benefit from more focus.

On the margin, I could imagine "now write this as a table please" being helpful. But that could exist outside of PRQL (e.g. the way dbt does it), or we could add something later if that becomes helpful.

Does that make sense?

[max-sixty]

Closed with https://github.com/max-sixty/prql/pull/19

[culebron]

@max-sixty I see your point about focusing. Do you see the language being used mostly in Rust, or elsewhere? What's the use case that you're adressing with selection queries?

The problem is that if you limit it to selection queries, the resulting library will require other tools, and it puts it in the same position like SqlAlchemy in Python -- that one didn't become popular among analysts, because they already have Pandas library, and can just fetch the entire table and do whatever they need in Pandas. (IDK what's the ecosystem in Julia.)

For an analyst (I've been one) it would be compelling if the end results can be saved somewhere. Exporting to, say, CSV, would work, but many will just stick with Python/Pandas or Julia.

From a data engineer perspective (me now), the projects are larger, and must run without manual interaction, and can't be loaded in RAM, hence the need for SQL servers, and saving/updating data is a big issue.

[qharlie]

I personally need it outside of Rust, I had planned on implementing one in Python, and let PRQL be the 'spec' to which all the libraries implement. That's a lot of duplicated effort I realize but I don't know else to accomplish it, I personally wouldn't install a Rust dependency for my Python project.

[max-sixty]

My intention is that PRQL is language & environment agnostic — while the transpiler is written in rust (though I would love to see other impls — @qorrect that sounds great!) — it'll build a binary which can be used anywhere that uses SQL and can run a command.

For an analyst (I've been one) it would be compelling if the end results can be saved somewhere. Exporting to, say, CSV, would work, but many will just stick with Python/Pandas or Julia.

From a data engineer perspective (me now), the projects are larger, and must run without manual interaction, and can't be loaded in RAM, hence the need for SQL servers, and saving/updating data is a big issue.

For sure — because PRQL just transpiles to SQL, all the existing tools that you use to manage SQL can still be used — anything to save the results of SQL queries, or write to a new table in a DB. I added https://github.com/max-sixty/prql/pull/31 to try and emphasize the point, please let me know if I'm being unclear though.

[aljazerzen]

I see PRQL as data manipulation language, which can be transpiled into SQL, but could also have pandas, polars, dplyr or data.table backends.

Currently, each of these libraries uses their own APIs for data manipulation, some of them more reasonable than others. Different APIs for essentially the same operations are not developer friendly.

My hopes are that we could construct a language with all tooling that would be easy to integrate with these libraries so developers could write PRQL while actually using any of the backends mentioned.

While this could have been done with SQL, it is not well defined and is quite old in its ideas. That's why PRQL is in a great position to become de-facto data manipulation language, which you can use when working on a database, in Python machine learning or R data analysis. In the best case scenario, PRQL would be simple enough to be used as a replacement of spreadsheets.

If this is the goal for PRQL, DDL is not really in scope, since it is only applicable to databases.

[vanillajonathan]

I don't like the NOT NULL, because it is two words instead of just one, and it contains "not" which is a negation, I think REQUIRED would be better than NOT NULL. But I don't like that SQL assumes that columns are nullable unless you explicitly declare them as NOT NULL, I would like to inverse this relationship and have columns be required unless explicitly declared to be nullable, so instead of having REQUIRED I would prefer the columns which can be null declared so as OPTIONAL or NULLABLE.

However, it is verbose so it could just a be a question mark on the data type as is common in many programming languages. I don't like the SQL data types either because they're ambiguous.

This is how I would like to create a table:

create table Users
{
    id: int32,
    first_name: string,
    last_name: string,
    bio: string?,
}

[maelp]

Is there a way to use s-string to still do a "CREATE OR REPLACE TABLE" while the content of the table would be generated by a PRQL query?

eg something like

let items = (
  somePrqlQuery
)

s"CREATE OR REPLACE TABLE MyTable AS SELECT * FROM items"

[aljazerzen]

I don't think this can work, because s-strings are supposed to be expressions, not top-level statements.

I suggest doing this with an external tool.

[maelp]

you mean doing it in a different language like SQL ? or rather compiling a PRQL query to SQL, then "hacking the compiled string" to create a database from it?

[max-sixty]

For example, in dbt, all statements are select statements. But then running dbt run will surround the select statements with CREATE OR REPLACE TABLE and write the result to a table.

Separating those are important, particularly for analytical queries — we want to be able to run SELECT lots, without worrying about destructive actions — for example, in the playground, we run the query on every keystroke.

So I totally agree that at some point someone needs to create the table! I think that can be by wrapping the output of the PRQL query, rather than integrated into the PRQL lang...

[maelp]

Interesting, perhaps the lib / language could have an escape-hatch for this, to make sure that the query generated can be included in a CREATE TABLE statement without issue?

I'd happily do it that way if I have a way to ensure that the PRQL output is something I can always enclose in something like CREATE OR REPLACE TABLE MyTable AS ...someOutput

[max-sixty]

Interesting, perhaps the lib / language could have an escape-hatch for this, to make sure that the query generated can be included in a CREATE TABLE statement without issue?

I think it always can! Because it's always a query result. Unless I'm missing something?