This repository was archived by the owner on Mar 20, 2023. It is now read-only.
This repository was archived by the owner on Mar 20, 2023. It is now read-only.
Bulk Helpers #5
Closed
Description
Is there a way to call an async version of the bulk helpers? I couldn't find away to do this when looking around.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
honzakral commentedon Oct 10, 2016
Unfortunately there is currently no way to do this. To make this work we'd have to reimplement the
_process_bulk_chunkhelper and all the others that use it. It shouldn't be too expensive time-wise.stickperson commentedon Dec 9, 2016
@honzakral I took a look at implementing
helpers.bulkthe other day._process_bulk_chunkandstreaming_bulkare generators which might be tricky to implement. Asynchronous iterators were introduced in python 3.5, but seeing as this library uses python 3.4 that might not be possible.Thoughts?
Archelyst commentedon Jan 18, 2017
Having async helpers would be really helpful. It's really not that hard. However, the final piece for this only came with Python 3.6: async generators. Those allow for almost identical code with the synchronous version. And I'd argue that people who use async are probably willing to use Python 3.6.
stickperson commentedon Jan 18, 2017
Yup. Here's a link to the PEP: https://www.python.org/dev/peps/pep-0525/
Still waiting to hear what @honzakral thinks.
honzakral commentedon Jan 21, 2017
I would love to have async bulk helpers and it makes a lot of sense. If at all possible I would prefer the solution to be at least
3.5compatible, not relying on3.6.If anybody wants to take a stab at it I would be happy to help with reviews and feedback (also if any changes in
elasticsearch-pywould make this easier we can do so). If no one is interested I will definitely try it but I am not sure of the time frame..eranhirs commentedon Feb 24, 2017
I would like to add that an async version of the
Scanhelper is also necessary, if someone is already taking a stab at the bulk helpers.eranhirs commentedon Dec 7, 2017
Regarding this issue, what do you think about this answer?
honzakral commentedon Dec 7, 2017
@eranhirs nothing about elasticsearch's
bulkAPI is asynchronous unfortunately.mazzma12 commentedon Aug 10, 2018
Any update on this one? It has been a year and python3.6 adoption has become larger
elastic#5 async scan proto
amitripshtos commentedon Oct 28, 2018
For a quick solution, I created my own bulk helper:
https://gist.github.com/amitripshtos/efd280e88376623b491c8682f417d597
However, if you guys think we can use the python3.6 pep with async generators, I can create a proper PR with that helper.
I think it's a critical helper we need in this package , and I'm more than happy to help :)
elastic#5 Adds .travis.yml
elastic#5 Adds .travis.yml
elastic#5 Adds .travis.yml
elastic#5 adds basic scan test
elastic#5 adds scroll cleanup test
elastic#5 adds tests for cleanup and exception re-raising
elastic#5 adds examples
elastic#5 adds syntax support for tests in 3.5
elastic#5 adds syntax support for tests in 3.5
bisoldi commentedon Mar 21, 2019
I see some commits against a fork for this issue, has there been any progress in getting this into the codebase?
mjzarrin commentedon May 16, 2019
Hi,
Any update?
In my tests, using elasticsearch-py-async for indexing bulk data like Pandas Dataframe, doesn't make sense as it brought more pressure on elasticsereach server which caused some Timeout Error.
Although, all part of my codes are optimized by concurrently, I have to use blocking es bulk helper.
for me asynchronous bulk helper is the most valuable feature. I would be so thankful if you take more attention on developing it.
sethmlarson commentedon Jul 20, 2020
There are asynchronous bulk helpers available in
elasticsearch[async]7.8.0+, see #81 for more info.