Working with data pipelines
To work with asynchronous data pipelines, we are going to use one of the many open-source libraries that handles steps and orchestration behind the scenes. Our main goal is not to provide custom implementations of the filter-pipe architecture but rather to use it to demonstrate how asynchronous programming can be applied to some steps even though the whole business case might require a combination of both synchronous and asynchronous solutions.
A suitable use case for asynchronous data pipelines
In our case, we want to find the word that has the largest number of related words in any language; to do so we must process a Parquet file that contains a structured, comprehensive, multilingual public etymology dataset, etymology being the study of the origin and historical development of words. For example, the word algorithm is derived from words in four languages (Middle English, Anglo-Norman, Medieval Latin and Arabic). The following graph shows you...
