Implement aggregation and grouping pushdown by gruuya · Pull Request #1 · splitgraph/Multicorn

gruuya · 2021-12-13T13:20:55Z

Multicorn support for Python FDW instances pushdown of an arbitrary combination of bare aggregations and/or groupings.

Accompanying PR demonstrating a particular implementation (in Elasticsearch) is Multicorn aggregation/grouping pushdown support postgres-elasticsearch-fdw#1.
For now it does not support pushdown of HAVING clauses or WHERE clauses in case of aggregations. This case results in full record fetch and then subsequent filtering/aggregation on the PG side.
Does not support pushdown of ORDER BY clauses, but in this case it does push down the aggregation, and performs only the ordering of returned aggregations on the PG side (so it's an improvement, albeit there's still some work to be done on doing sorting on the remote server).
Also not supported are aggregations with DISTINCT or COUNT(*) for the time being (defaults to full record fetch and subsequent processing on PG side).
Implementation was guided by postgres_fdw and other FDW implementations.

CU-1x57q56

The current implementation provides a mechanism for pushing down aggregation and/or grouping queries into the foreign data source. The Python side of the implementation will now receive two new kwargs, `aggs` and `group_clauses`, in which case it should return the corresponding aggreagation result. Still left to implement is consulting the Python side whether remote aggregation is possible at all, and if so which agregation functions are valid. Also missing are some more advanced aggregation cases (aggregating multiple functions, or handling `HAVING` clause for example). This is to be implemented separately.

Add a method to FDW Python instance that provides info on whether the pushdown is supported at all, and if so gives data for more granular decisions (for now only list of aggregation functions). Consult this method in `multicornGetForeignUpperPaths`.

Currently the parsing is incomplete for simple WHERE clauses due to the lack of T_OpExpr and T_Const cases in multicorn_foreign_expr_walker. Therefore, all WHERE clauses will be treated as local conditions, and not pushed down.

For the first iteration disable pushdown of `COUNT(*)`, like for `DISTINCT` clauses. These can be added later on, and tested on their eqivalents in ES, `doc_count` and `cardinality`.

mildbyte

I took a first pass at this and left some comments; will try understanding it deeper in the morning. Pretty impressive!

mildbyte · 2021-12-14T19:12:13Z

python/multicorn/__init__.py

+        The FDW has to inspect every sort, and respond which one are handled.
+        The sorts are cumulatives.


Copypaste error here?

Yup, thanks.

mildbyte · 2021-12-14T19:12:33Z

python/multicorn/__init__.py

+
+        Return:
+            None if pushdown not supported, otherwise a dictionary containing
+            more granular details for the planning phase, in the form:


Needs docs on the expected dict output

Adding docs for it in the next commit.

mildbyte · 2021-12-14T19:14:12Z

python/multicorn/__init__.py

+                column to be used in the aggregation operation. Result should be
+                returned under the provided aggregation key.
+            group_clauses (list): A list of columns used in GROUP BY statements.
+                The result should be returned for each column name provided.


What does this mean -- does every row we return need to have an entry for everything in columns + aggs?

What I meant to say is that whenever there is a group_clauses kwarg, then for each column specified there the returned response should have a corresponding value for each row using that column name as the key.

I re-worded the docstring as above, hopefully this clarifies it.

mildbyte · 2021-12-14T19:27:01Z

src/python.c

+        p_object = PyMapping_GetItemString(p_upperrel_pushdown, "agg_functions");
+        if (p_object != NULL && p_object != Py_None)
+		{
+            state->agg_functions = PyMapping_Keys(p_object);


I don't think you ever DECREF state->agg_functions, so this will slowly leak. I'd extract the contents into a separate List here and get rid of the PyObject here so that you also don't have to mess with the Python API in foreign_expr_walker.

Good catch. I initially tried the route you mentioned but was stuck extracting Python Unicode objects into a PG List, so I went with this instead. Let me get back at this.

Ok, I've now added storing of supported agg functions to a List.

mildbyte · 2021-12-14T19:30:25Z

src/python.c

+            foreach(lc_groupc, state->group_clauses)
+            {
+                PyObject *column = PyUnicode_FromString(strVal(lfirst(lc_groupc)));
+                PyList_Append(group_clauses, column);


I think (but not entirely sure, since https://docs.python.org/3/c-api/list.html#c.PyList_Append doesn't mention it -- some evidence in https://stackoverflow.com/questions/3512414/does-this-pylist-appendlist-py-buildvalue-leak) that PyList_Append increments the refcounter, so you need to DECREF the column here.

Makes sense, added DECREF.

mildbyte · 2021-12-14T19:47:38Z

src/multicorn.c

 }
 #endif

+/*


That's a lot of code! Can you mark the parts taken from other FDWs (here and in deparse) and parts that you added yourself so that I know where to concentrate the review? Currently it kind of makes sense to me but knowing where it came from would make it clearer.

Sure, I can do that. I can add some comments like // MY CODE START and // MY CODE END if that helps. Just keep in mind that the parts taken from other FDWs are also trimmed down, i.e. I've thrown away the irrelevant stuff so it's not 1-1.

Done - I've enclosed all deviations from common FDW code (as used in postgres_fdw and other implementations) with the above comments in multicorn.c and deparse.c (other files should be more easier to parse I think).

Again worth mentioning that common FDW code that I've "appropriated" was pruned.

mildbyte · 2021-12-14T19:56:40Z

src/deparse.c

+
+            initStringInfo(agg_key);
+            appendStringInfoString(agg_key, strVal(function));
+            appendStringInfoString(agg_key, ".");


Just to check my understanding, does the Python FDW get a dict of {"functionname.colname": {"function": "functionname", "column": "colname"}} and is then expected to return a surrogate functionname.colname column in its response? e.g. https://github.com/splitgraph/postgres-elasticsearch-fdw/pull/1/files#diff-45ed0634a3ed30705f0b30dce58a096decc81bdf04af2df3906bc56d692c3de4R88-R92

Yes, that is correct.

mildbyte · 2021-12-14T19:57:39Z

src/python.c

+        pushdown_upperrel = true;
+    }
+
+	Py_DECREF(p_upperrel_pushdown);


Should this DECREF be inside of the if (p_upperrel_pushdown != NULL && p_upperrel_pushdown != Py_None) like other decrefs?

We'd need to decrement a Py_None reference, which would leak if the Py_DECREF was in the if statement I believe.

That said, in the case of p_upperrel_pushdown being a null pointer it seems the proper approach is to use Py_XDECREF, like in the case of pythonDictToTuple function.

Also seems like I should do something similar for p_object inside the outer if statement.

Adding those changes now.

gruuya added 4 commits December 9, 2021 16:24

Remove redundant code and fix compiler warnings

96744b4

gruuya requested a review from mildbyte December 13, 2021 13:20

gruuya self-assigned this Dec 13, 2021

gruuya mentioned this pull request Dec 13, 2021

Multicorn aggregation/grouping pushdown support splitgraph/postgres-elasticsearch-fdw#1

Merged

gruuya added 4 commits December 13, 2021 14:17

Remove obsolete struct

b37a39e

Enable queries for aggregation/grouping

8541184

Fix aggregation queries with DISTINCT clauses

a7bbfd5

Fix COUNT(*) statements from crashing

ab9d3ab

For the first iteration disable pushdown of `COUNT(*)`, like for `DISTINCT` clauses. These can be added later on, and tested on their eqivalents in ES, `doc_count` and `cardinality`.

mildbyte reviewed Dec 14, 2021

View reviewed changes

gruuya added 6 commits December 15, 2021 09:52

Fix a couple of PyObject DECREF's and clarify docstrings

7b05afb

Store supported aggregation functions as a PG List object

051165e

Add comments stressing deviations from commonly used FDW code

85e6cbc

Fix cases where aggregations of integers do not return an integer

8efb2e8

Add comment with explanation for type conversion

df5e576

Fix typo in comment

67ffe22

gruuya mentioned this pull request Dec 17, 2021

Enable aggregation and grouping pushdown in the engine splitgraph/sgr#581

Merged

gruuya added 3 commits December 22, 2021 09:49

Bring back comment on qual re-checking

757e00c

Fix memory leak when converting row value to PyLong

a651e11

Refine the comments in deparse.c and multicorn.c where needed

55feb66

gruuya merged commit 3391858 into master Dec 27, 2021

gruuya deleted the implement-agg-grouping-pushdown-cu-1x57q56 branch December 27, 2021 15:29

gruuya mentioned this pull request Jan 21, 2022

Push grouping to the fdw level. Segfault-Inc/Multicorn#215

Open

		The FDW has to inspect every sort, and respond which one are handled.
		The sorts are cumulatives.

               }
               #endif
+              /*

Conversation

gruuya commented Dec 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mildbyte left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gruuya commented Dec 13, 2021 •

edited

Loading