dbt-prql

[max-sixty]

I've been thinking of ways that people could really start using PRQL practically. Somehow, something needs to transform their PRQL into SQL between the user and the DB.

Some ideas:

• An specific interactive environment, a bit like PyPrql have built / are building
• An editor plugin, similar to what Malloy have built
• A plugin to a batch tool, such as dbt (Thoughts on dbt #13)

They are all compelling! It might be that the dbt plugin is the approach that can get the most amount of traction the fastest. While it might not take advantage of some of the things that PRQL allows in the long term — auto-complete, type-inference — it also has much lower requirements — just compile the query when otherwise compiling the SQL.

dbt would be the natural choice given how widely it's used.

A problem with dbt packages is that IIUC they don't allow for executing arbitrary code — they can only execute macros, which have a very defined scope.

So a couple of alternatives:

1. dbt package + import hack — We write something that patches the dbt jinja context to allow a prql macro which compiles the contents to SQL — e.g. {{% prql ... %}}

• The advantage of this is that it's exactly like running existing dbt models, but people can write PRQL
• The disadvantage is that it requires extreme python hackery — check out @mtkennerly's approach to patching poetry here. Ours would be a bit easier than this, since we don't need to account for multiple installation approaches, but it wouldn't be simple.
• We'd be using dbt's internal APIs, which could break on dbt upgrades.
• I'm not sure whether the dbt folks would approve? (@kwigley @drewbanin ?) I'm sure they'd approve of us building on top of dbt, but not so sure about patching their library's jinja environment...

2. Wrapper tool — We write a tool which pre-compiles files for dbt; like prql dbt models/ -- dbt run -m foo

• The advantage is that the implementation would be extremely simple
• The disadvantage is that it adds a layer of indirection to dbt — It means prql might need to understand dbt's file structure, and run a similar process to that which dbt is already running to walk the file tree — in order to compile .prql files to .sql, so that dbt sees the .sql files. Processes which run dbt now need to run a different command.
• Up a level, it offloads complexity onto to user, rather than encapsulates it in the tool — something we should try and avoid.
• But possibly it's the practical approach is the short-term.

Either of these requires a way of allowing jinja in PRQL — probably we could just treat {{ }} as comments without giving up anything in the language. (Hopefully people don't need as much jinja with PRQL, given we have abstractions like functions, but we don't need to be strict about it)

This could either be implemented by calling out to PyPrql, or through embedding the rust binary in a python package — this is something dbt do already with https://github.com/dbt-labs/dbt-extractor.

[chamini2]

Hi! I spent some time seeing if this could be done with fal-ai/fal, the first blocker I found was the Jinja macros

Either of these requires a way of allowing jinja in PRQL — probably we could just treat {{ }} as comments without giving up anything in the language. (Hopefully people don't need as much jinja with PRQL, given we have abstractions like functions, but we don't need to be strict about it)

I fixed it for pyprql but have realized that the Python implementation is maybe a little bit different than the one specified here.

The fal script I was intending to use looks like this for pyprql

# prql_to_sql.py
from typing import List
from pathlib import Path

from pyprql import to_sql


def process_pyprql_dirs(source_dirs: List[str]):
    for source_dir in source_dirs:
        print("processing dir", source_dir)
        source_path = Path(source_dir)
        prql_paths = source_path.rglob("*.prql")
        for prql_path in prql_paths:
            sql_path = prql_path.with_suffix(".sql")
            process_prql_file(prql_path, sql_path)


def process_prql_file(prql_path: Path, sql_path: Path):
    print("reading file", prql_path)
    with open(prql_path, "r") as handle:
        pyprql_source = handle.read()

    sql_source = to_sql(pyprql_source, verbose=True, dbt=True)

    print("writing file", sql_path)
    with open(sql_path, "w") as handle:
        handle.write(sql_source)


process_pyprql_dirs(["./models"])

To use the rust version of prql, we can do this instead:

from typing import List
from pathlib import Path

import subprocess


def process_pyprql_dirs(source_dirs: List[str]):
    for source_dir in source_dirs:
        source_path = Path(source_dir)
        print("processing dir", source_path.relative_to("."))
        prql_paths = source_path.rglob("*.prql")
        for prql_path in prql_paths:
            sql_path = prql_path.with_suffix(".sql")
            process_prql_file(prql_path, sql_path)


def process_prql_file(prql_path: Path, sql_path: Path):

    command_list = ["prql", "compile", prql_path]

    # Assure all command parts are str
    command_list = list(map(str, command_list))

    print(f"\nExecuting command: {' '.join(command_list)}\n")

    sql_buffer = open(sql_path, "w")
    process = subprocess.Popen(command_list, stdout=sql_buffer)

    process.wait()

    if process.returncode != 0:
        raise RuntimeError(f"Error processing {prql_path}")


model_paths = ["./models"]
process_pyprql_dirs(model_paths)

and in the schema yml we would just have to add top-level:

version: 2

fal:
  scripts:
    before:
      - prql_to_sql.py

---

My conclusion is that this is an almost usable solution. We would need to add Jinja macros to the rust grammar.

Let me know if this is something you are willing to do to see if I get a fal solution out there.

[max-sixty]

Hi @chamini2 , thanks for the code, and the PR in PyPrql. I'm familiar with fai and we'd be keen to be compatible with that — it arguably dominates the "wrapper tool" option for many people.

It would be very reasonable to add a pest rule to ignore {{ }} blocks; I don't think that should be too difficult, though would require some familiarity with pest and being able to write a small test in rust. IF you'd be up for that, that would be great.

FYI we've been discussing potentially using the rust compiler in PyPrql now that the compiler is becoming more fully-featured. If we / @qorrect goes down that path, we could let fai work with a pip-installed dependency, with full access to the prql compiler.

[max-sixty]

Copying from discord:

dbt is probably the way that PRQL will be initially successful if it is successful — it's extremely popular as a tool that brings running data pipelines into the 2010s. So the sort of people who care about expressing their data transformations in a better way are also the people using dbt.

And I imagine that PRQL's early users will more experienced people. Beginners will be a great audience but are unlikely to be early adopters.

So if we can make it really easy for someone to write PRQL rather than SQL in a dbt file, that would be such a good way to encourage people to try it. And people can do it for a single query in a project of SQL queries, so it's easy to start.

The problem is that it will be gnarly python — there is no official way to run arbitrary executables in a dbt query — so we would need to hook into python's import system, and patch the dbt jinja templates to let us do that.

[max-sixty]

I made some limited progress on this. Here's a jinja2 Extension which converts a block from PRQL to SQL:

from jinja2 import nodes, Environment
from jinja2.ext import Extension
from jinja2.nodes import Const

import pyprql

class PrqlExtension(Extension):
    tags = {'prql'}

    def parse(self, parser):
        line_number = next(parser.stream).lineno
        prql = parser.parse_statements(['name:endprql'], drop_needle=True)
        return nodes.CallBlock(self.call_method('_to_sql', [Const('')]), [], [], prql).set_lineno(line_number)

    def _to_sql(self, args, caller):
        prql = caller()
        sql = pyprql.to_sql(prql)
        output = f"""
-- SQL created from PRQL. Original PRQL:
{chr(10).join(f'-- {line}' for line in prql.splitlines())}

{sql}
"""
        return output

Then this

template = Environment(extensions=[PrqlExtension]).from_string("""
{% prql %}
from employees
filter (age | in 5..10)
{% endprql %}
""")
print(template.render())

Generates:

-- SQL created from PRQL. Original PRQL:
-- 
-- from employees
-- filter (age | in 20..30)

SELECT   employees.* FROM   employees WHERE   age BETWEEN 20   AND 30

(not sure what's going on with the linebreaks in the output)

Things remaining:

• Is this form the best design? i.e. {% prql %} block. I'm not that familiar with Jinja. Would {% set lang=prql %} be another option?
• Allow jinja in PRQL (not sure if the current mode allows)
• Monkeypatch dbt
• Hack imports so it can monkeypatch dbt
• Decide how we want to install — should this be a separate package dbt-prql with a dependency on pyprql? Same repo as pyprql? Or here?

[drewbanin]

hey @max-sixty - thanks for tagging me in here! And apologies for the delay on following up with you 😓

I have not had the chance to dig as deep into prql as I'd have liked yet! Can you give me a sense of how you'd imagine prql interfacing with some of the dbt DAG/config constructs (ref, source, and config)? Would those just be jinja interpolated into the prql query, or is there some prql-native construct that we could leverage here?

We're thinking about Python support for dbt models (see here) & it sort of presents an opportunity for us to be a little bit more agnostic to language in dbt's compiler. My personal instinct is that this could potentially be a sort of "plugin" system built into dbt. If we did that, then we'd be able to build parsers/compilers for code written in:

• sql (+ jinja)
• python
• prql (+ jinja??)
• etc

We do not currently have any concrete plans to do this, but I just wanted to call out that this possibility exists! Curious what y'all think about something like this, and also really interested in learning more if the answer is prql+jinja, or just prql with configs/sources/refs/macro calls baked into the language itself!

[max-sixty]

Can you give me a sense of how you'd imagine prql interfacing with some of the dbt DAG/config constructs (ref, source, and config)? Would those just be jinja interpolated into the prql query, or is there some prql-native construct that we could leverage here?

Our primary interest is to integrate with dbt, deferring to it for things it's already very good at — so I didn't have any plans to innovate on the existing constructs. So this would be valid:

{% prql %}
from {{ ref('employees') }}
filter (age | in 5..10)
{% endprql %}

Because PRQL could ignore anything within {{}} blocks, either PRQL or dbt's jinja blocks could be parsed first.

Ofc we're open-minded if you have thoughts, and I could imagine a deeper integration in the future. We're big fans of dbt and see it as one of the primary ways that PRQL could grow from a popular idea into a way of making people's work better.

---

We're thinking about Python support for dbt models (see here) & it sort of presents an opportunity for us to be a little bit more agnostic to language in dbt's compiler. My personal instinct is that this could potentially be a sort of "plugin" system built into dbt. I

I've been following along on those discussions! I agree that proposal would be great.

Though tbc, the PRQL extension would require a tiny increase in the dbt interface relative to that proposal. IIUC, that proposal involves building a python interface to SQL, somewhat similar to the pipeline concept that PRQL uses — e.g. upstream_model.where(col("city")=="Philly") would build SQL text (I'm not sure whether it would identify whether it requires a WHERE / HAVING / a new CTE?).

The PRQL extension would just require an ability to add jinja Extensions onto the existing dbt jinja parser. I was planning to see if PyPrql could monkeypatch that on, but it would be much preferred if there were an actual interface for this, similar to how there's an interface for dbt packages.

[drewbanin]

ok - got it! There's actually more on Python models for dbt here: dbt-labs/dbt-core#5261. We're not planning on building our own ORM-style interface or anything... just piggybacking on the capabilities of the different data warehouses out there.

The reason I bring up Python models is because it ends up requiring two changes:

• a change in dbt Core to know about Python files (identified with .py extenstions) & parse out configs/refs/sources from the Python file AST
• additions to dbt adapters (Eg. Snowflake, BigQuery, Databricks) which use new APIs to materialize these Python models in the warehouse

So, I think that support for something like prql ends up looking kind of like the first bullet point above -- a change in Core that implements different parsing logic across .sql, .py, or (potentially) .prql files :). I guess it ends up being a sort of matrix of compatibility across languages & warehouses, but I am still pretty fuzzy here and we're very much thinking through it in real time!

Just to set expectations: I don't believe that we'll be building this sort of plugin system by the end of the year, but @jtcohen6 and I did have a quick catch-up about prql the other day and we're definitely incorporating it into out mental model!

[max-sixty]

Ah great, that's very clear. Thanks for the link. I was overestimating the size of the .py issue above; I see now that it's delegating that to the databases that can run python files.

So, I think that support for something like prql ends up looking kind of like the first bullet point above -- a change in Core that implements different parsing logic across .sql, .py, or (potentially) .prql files :).

Yes, very much so!

I guess it ends up being a sort of matrix of compatibility across languages & warehouses, but I am still pretty fuzzy here and we're very much thinking through it in real time!

Yes...

At least for PRQL, there's a way of encoding the warehouse / dialect — https://lang.prql.builders/query-header/dialect.html — so possibly one small additional feature of the extension would be for it to add dialect:bigquery given it's already in the jinja context.

Just to set expectations: I don't believe that we'll be building this sort of plugin system by the end of the year, but @jtcohen6 and I did have a quick catch-up about prql the other day and we're definitely incorporating it into out mental model!

Great, let's keep in touch. I may try and implement the hacky monkeypatch, unless you have an objection. And if it's OK, I may drop you a message offline asking if you have any feedback on PRQL, somewhat separate from this thread.

[drewbanin]

ok - amazing! Really excited to follow along with what you're building here & would be happy to chat offline! I'm not quite as close to the nitty-gritty implementation details of dbt Core as I once was, but feel free to drop me a line if I can share any pointers for the hacks :)

[jtcohen6]

+1 - very cool convo! I'm just lurking for now, though I took another look through the docs earlier today, very impressive progress in the past few months!

[max-sixty]

This now works! https://github.com/prql/dbt-prql

As per the Readme:

• The implementation is some dark art
• It doesn't pass through {{...}} until we release a version of PyPrql which includes Allow jinja expressions through, for dbt-prql #571

Note that if you're seeing this in the future, we may move the code to another repo, so check out this repo or PyPrql.