New Features on _.each

[dwt]

Ah, all clear regarding the missing comment. But yes, I would like to add as many overloads as are possible with _.each - so keep coming back if you find something that can be implemented.

With some delay... :)

_.each

• __rmul__ to support 2*_.each. Similar for other __r...__ functions, see e.g. _(dir(int)).filter(lambda it: it.startswith("__r")).to(list).
• Invoking methods on _.each doesn't work as expected, e.g. _(dir(int)).filter(_.each.startswith("__r")).to(list) doesn't filter anything out.
• The ,$,| operators could be overloaded as eager boolean operators (like numpy does). Though precedence rules are an issue, it would allow writing _(range(10)).filter( (_.each > 3) & (_.each < 7) ).to(list).

Prefices

Regarding the prefixes: I really don't know what the best way here would be. I kind of like that the default case is not lazy and I find the I prefix intuitive (though python2-ish). At the same time I haven't found a good prefix for a non lazy variant.

That being said - I tried at least to be internally consistent, but am of course open to suggestions.

"Iterator by default" can be annoying for debugging, since lazy evaluation often doesn't produce useful stack traces.
"Iterator by default" can be annoying for debugging, because it delays the failure. In

import fluentpy as _

def fail(value):
    assert False, "fail"

items = _(range(10)).imap(fail)

for item in items:
    print(item)

the line where items is defined doesn't appear in the backtrace while with map it would. So, for my personal use-cases your current choice almost always will be preferable actually.

On the other hand, making eager evaluation the default means that code written with fluentpy will be brittle. The original purpose of a function might work well with _(...).filter, but will suddenly fail if the passed iterable is large or infinite.

Documentation (1)

It could be useful to point out, that the wrappers can be used as decorators to side-step the limited features of lambdas. E.g.

items = _(range(5))
@items.call
def items(its):
    for it in its:
        yield it
        yield it
print(items.to(list))

This would be yet another advantage of the suffix notation for filtering effects...

Documentation (2)

Ad-hoc, I thought it might be useful to provide custom extensions, e.g. let's say implementing a .groupby that groups non-consecutive elements. Until I realized, that this is covered by .call. It might be worth pointing this out explicitly.

Subprocess utility

A wrapper around subprocess.Popen would be useful, that allows to do something like

lines = _(["hello","world"]).pipe(["sed", "s/l/x/g"]).to(list)

It can be done with .call and a function wrapping around subprocess.Popen; In order to prevent deadlocks, it requires multiple threads though; I had some success building a function that allowed _(...).call(pipe(...)).... by having a thread feed the input iterator to the Popen.stdin and returning an iterator over Popen.stdout.

Originally posted by @kbauer in #5 (comment)

[dwt]

@kbauer There's a lot to unwrap here, so lets go step by step.

1. Adding support for __r${operator}__ methods seems like a great idea, and is something I'd love to do. (Patches welcome!)

3. Invoking methods on _.each: That is actually problematic, as right now _.each.foo('bar')._ creates a operator.methodcaller(name='foo', args=['bar']). Maybe the underlying problem here is that all operations on _.each need to be terminated by an unwrap ._ to actually arm the object and use it. I have found however that in my own usage this is easily forgotten, and there is really no good way I have found to output good debug information when this happens. If you happen to have an Idea on how to best move forward and make this more user friendly, I'd love to know, as I am currently kind of stuck here.

4. Implementing operators like &, |, … A similar problem here, _.each is kind of two faced, almost all operations leave an eager wrapper behind that will swallow up and build operations as you apply them to it - except binary operators, which will immediately instantiate the actual operation. My thinking here was that _.each.foo()['bar'].baz._ is easy to write and symetrical, while (_.each < 3)._ would suck. So to get your example working we would need to first change _.each < 3 to not evaluate to an operator immediately, but stay eager (otherwise there is no class on which to implement the __op__ and __rop__ methods on). This would require at least one major release and also at least a deprecation period. Then this can be implemented, but requires that it is written like this: _(range(10)).filter( ( (_.each > 3) & (_.each < 7) )._ ).to(list). I am a bit unsure about this. I really like the uniformity, in that all operators on _.each return another greedy wrapper and each _.each wrapper has to consistently manually be terminated / turned into an operator before it can be used. But it seems to me that this makes the problem of hard to debug errors if used wrong every harder. :-( One thing I considered, is to add type checks to some crucial methods (mostly the collection operations) to catch this error, but since that is only a very partial solution I haven't gone ahead with this yet.

5. i-prefixes and the question wether eager or lazy evaluation should be the default for iterators. In my own usage I have found that a) debugging code written with this library can be surprisingly hard and b) that I found it relatively easy to audit / check code written with it to convert it to laziness after the code works. So I am leaning towards staying with the default on eager evaluation to ease debugging and accelerating testing. I am curious, what kind of datasets you are working with? I'd love to know more about how this library is used.

6. Documentation 1). I actually had no idea this works. Thats pretty amazing. Definitely goes into the documentation.

7. Documentation 2) I'm not entirely sure what you mean here and suspect that I might learn something from you again here. Could you please provide an example?

8. Wrapping subprocess.popen() That sounds like a great idea, to make going to and from shell executables simpler. Would you mind opening a WIP pull request with the code you have so far so we can collaborate on that? I already have special code to make python -m fluentpy "code that uses fluent with less prefixes" and thus makes it simpler to inject python into one off shell pipes. Also going the other way around sounds awesome.

[dwt]

Thinking more about subprocess.popen() it may be that that is out of scope of this library and that it would be better to use a library like sh together with fluentpy to achieve the effect we are looking towards.

[dwt]

@kbauer: I have pushed a first version of support for reverse operators on _.each I'd like your feedback for that.

[kbauer]

ad 1) The __r*__ operators work as expected as far as I can tell. 👍

ad 3 "methods on _.each"): You could have .map auto-convert their argument to an callable function, i.e. if the argument is a wrapper object, add ._ automatically.

ad 4 "boolean operators") For boolean operators, upon reading more I don't see a good solution. Numpy was a bad comparison on my side, as it is a bit of a special case there: There is no expectation of short circuiting with arrays, and the distinction of logical and and binary & is moot. It is probably a case best left to lambda it: it and it.

ad 5 "prefix convention") I mostly meant it as a remark about the changed conventions from Python2 to Python3. There is an expectation now, that map should work with infinite sequences. And it easily could bite me, when I write a script that filters through log files from a simulation, which may in edge-cases become rather large. With the data I am currently working with, it isn't likely to occur, but I've had such cases in the past.

If the affected script is both written and used by myself, it is an easy fix. And correspondingly I was originally annoyed with the extra effort caused by Python 3's lazy iterators, exactly because of the debugging issues. But when the same software is reused by a third party, a crash and/or low performance due to eager evaluation suddenly becomes an issue.

I assume that such concerns where what moved Python 3 towards "lazy by default".

ad 6 "documentation 1") In hindsight, I am increasingly sceptical about that usage, mostly because the function name doesn't describe what the function does or returns, but where the result will be stored. Since there's no precedence for using decorators like this, I'm not sure anymore if they should be used like that. Plus, it essentially becomes an equivalent of the "same name twice per line" style

items = range(5)
items = map(lambda it: 2*it, items)
items = filter(lambda it: it%2==0, items)

that fluentpy help avoid, only less obviously.

ad 7 "extensions") I was originally concerned about the utility of libraries like fluentpy due to concerns about what happens if I need a sequence algorithm that isn't implemented in the library, see e.g. that "duplicate items" function in (6/"Documentation (2)"). I was dreading the necessity of subclassing _ to be able to write

_(range(10)).duplicateItems()

and almost lost interest over that misunderstanding. It could help users along to explicitly mention that the same can be achieved with

_(range(10)).call(duplicateItems)

ad 8 "pipe") Will try, but it took me days just to answer to your comments :/ For now I am just attaching my experient (pipe.py.txt).

Regarding sh: It looks quite interesting, and most of my scripting is Unix anyway (or WSL if on Windows). It is however somewhat inconvenient to use with fluentpy.

_(range(10)).map(lambda it: str(it)+"\n").call(lambda it: sh.sed("s/^/>> /", _in=it)).to(list)
                                                      ^^                         ^^

In scripting contexts, something like

_(strings).sh.sed("s/^/>> /").to(list)

would be preferable. As it is, that would make sh a thing wrapper to the sh module, that sets _in. And maybe for consistency some ish, that also sets _iter=True...

The downside would be, that it would introduce a feature that doesn't work on Windows. My popen based experiment is platform independent by contrast. Will look into it more.

[dwt]

ad 1) The __r*__ operators work as expected as far as I can tell. 👍

Nice!

ad 3 "methods on _.each"): You could have .map auto-convert their argument to an callable function, i.e. if the argument is a wrapper object, add ._ automatically.

I thought about that, but that leaves the unfortunate fact that each is used in far more contexts than just .map(). And that would make the rules when you have to self unwrap and when it becomes automatic even stranger. Just think about code like this:

up = _.each.toupper()
_('fooo').map(up).print()
_('fooo').call(up._).print()

Where the second call _('fooo').call(up._) needs that manual ._ at the end? I still am really uncontent with the current state of affair, but I worry that such a change would make the library even more confusing to use.

ad 4 "boolean operators") For boolean operators, upon reading more I don't see a good solution. Numpy was a bad comparison on my side, as it is a bit of a special case there: There is no expectation of short circuiting with arrays, and the distinction of logical and and binary & is moot. It is probably a case best left to lambda it: it and it.

I agree.

ad 5 "prefix convention") I mostly meant it as a remark about the changed conventions from Python2 to Python3. There is an expectation now, that map should work with infinite sequences. And it easily could bite me, when I write a script that filters through log files from a simulation, which may in edge-cases become rather large. With the data I am currently working with, it isn't likely to occur, but I've had such cases in the past.

I have to say that I kind of agree. I just don't see a good way forward. About the most clear thing I see would be to provide an alternative import that had the defaults switched, .i.e. something like import fluentpy.lazy as _, and then start deprecating the I-prefixed versions of the methods? That would still require to find a good prefix for the non lazy versions?

Thinking about this more, maybe having two differen import paths is actually the best solution? That way you can do all your development with import fluentpy as _, which also doesn't make any small scripts confusing, and then, once you are done debugging you switch to import fluentpy.lazy as _? I don't quite know yet, as that would also make iterative development on a library that uses fluent quite hard.

ad 6 "documentation 1") In hindsight, I am increasingly sceptical about that usage, mostly because the function name doesn't describe what the function does or returns, but where the result will be stored. Since there's no precedence for using decorators like this, I'm not sure anymore if they should be used like that. Plus, it essentially becomes an equivalent of the "same name twice per line" style

items = range(5)
items = map(lambda it: 2*it, items)
items = filter(lambda it: it%2==0, items)

that fluentpy help avoid, only less obviously.

That is true. In the past I have used this style to solve that problem:

seen = set()
def havent_seen(number):
    if number in seen:
        return False
    seen.add(number)
    return True


(
    _(day1_input)
    .icycle()
    .iaccumulate()
    .idropwhile(havent_seen)
    .get(0)
    ._
)

Which kind of is better, but also takes some code completely out of context as it now has to live above the fluent chain, completely out of context?

Ideally you could write something like this:

seen = set()


(
    _(day1_input)
    .icycle()
    .iaccumulate()
    .idropwhile(
        def havent_seen(number):
            if number in seen:
                return False
            seen.add(number)
            return True
    )
    .get(0)
    ._
)

But sadly in python def isn't an expression and probably never will be. I have also in the past thought about strange hacks with using context managers to inject statements into fluent chains, but nothing has come of it yet. :-(

ad 7 "extensions") I was originally concerned about the utility of libraries like fluentpy due to concerns about what happens if I need a sequence algorithm that isn't implemented in the library, see e.g. that "duplicate items" function in (6/"Documentation (2)"). I was dreading the necessity of subclassing _ to be able to write

_(range(10)).duplicateItems()

and almost lost interest over that misunderstanding. It could help users along to explicitly mention that the same can be achieved with

_(range(10)).call(duplicateItems)

Indeed, that is something that I should make more clear in the documentation. -> I'll ping you later with a first draft.

ad 8 "pipe") Will try, but it took me days just to answer to your comments :/ For now I am just attaching my experient (pipe.py.txt).

Regarding sh: It looks quite interesting, and most of my scripting is Unix anyway (or WSL if on Windows). It is however somewhat inconvenient to use with fluentpy.

_(range(10)).map(lambda it: str(it)+"\n").call(lambda it: sh.sed("s/^/>> /", _in=it)).to(list)
                                                      ^^                         ^^

In scripting contexts, something like

_(strings).sh.sed("s/^/>> /").to(list)

would be preferable. As it is, that would make sh a thing wrapper to the sh module, that sets _in. And maybe for consistency some ish, that also sets _iter=True...

The downside would be, that it would introduce a feature that doesn't work on Windows. My popen based experiment is platform independent by contrast. Will look into it more.

Yes, windows support would be really nice. Still I would like to give an sh adapter a shot, just to see how it works. Perhaps as an example of how to extend fluent?

Maybe an interested user has an idea how this could work? Might be worth it opening a bug just for that.

I'm guessing that in the meantime, it would be quite good to have an interface to add your own wrappers (extending existing ones is already quite easy, as you can just assign additional properties to existing wrapper classes). Though I think that both of these use cases are not documented yet -> todo.

[dwt]

Let's break out work on popen () integration into it's own bug.

[dwt]

Regarding better documentation, see 1e11288

[dwt]

@kbauer I've released 2.1 with support for r* methods and some bug fixes. Hope you like it!

[kbauer]

@dwt Nice :)

There is a little typo: each.__ror_ should be each.__ror__, as a consequence 3|each does not work.

[dwt]

@kbauer Good catch! Update released

[dwt]

I'l close this issue for now, if I missed something and this needs reopening, please feel free to tell me.