Join our community | Newsletter | Docs | Twitter | Blog | YouTube | Contact us

This repository contains sample pipelines developed using Ploomber.

Note: We recommend you to go through the first tutorial to learn the basics of Ploomber.

Use Colab:

Use Binder (free, hosted JupyterLab):

Or run locally:

pip install ploomber

# list examples
ploomber examples

# download example with name
ploomber examples --name {name}

# example
ploomber examples --name templates/mlflow

Each example contains a README.md file that describes it; a README.ipynb is also available with the same contents but in Jupyter notebook format and with command outputs. In addition, files for pip (requirements.txt) and conda (environment.yml) are provided for local execution.

Starting points for common use cases. Use them to ramp up a project quickly.

1. templates/etl Download a data file, upload it to a database, process it, and plot with Python and R.

2. templates/exploratory-analysis Sample pipeline that explores penguins data.

3. templates/google-cloud Use Google Cloud and Ploomber to develop a scalable and production-ready pipeline.

4. templates/ml-advanced ML pipeline using the Python API. Shows how to create a Python package, test it with pytest, and train models in parallel.

5. templates/ml-basic Download data, clean it, generate features and train a model.

6. templates/ml-intermediate Training and serving ML pipelines with integration testing to evaluate training data quality.

7. templates/ml-online Load data, generate features, train a model, and deploy model with flask.

8. templates/mlflow Train a grid of models and log them to MLflow.

9. templates/python-api Loads, clean, and plot data using the Python API.

10. templates/pytorch Using GPUs to train models in Ploomber Cloud.

11. templates/shell Create a pipeline with shell scripts as tasks.

12. templates/spec-api-directory Create a pipeline from a directory with scripts (without a pipeline.yaml file).

13. templates/spec-api-r Load, clean and plot data with R.

14. templates/spec-api-sql Use SQL scripts to manipulate data in a database, dump a table, and plot it with Python.

Short and to-the-point examples showing how to use a specific feature.

2. cookbook/dynamic-params Pipeline parameters whose values are computed at runtime.

3. cookbook/file-client Upload task's products upon execution (local, S3, GCloud storage)

4. cookbook/grid An example showing how to create a grid of tasks to train models with different parameters.

5. cookbook/hooks Task hooks

6. cookbook/incremental A pipeline that processes new records from a database and uploads them.

7. cookbook/nested-cv Nested cross-validation for model selection and hyperparameter tuning.

8. cookbook/python-load Load pipeline.yaml file in a Python session to customize initialization.

9. cookbook/report-generation Generating HTML/PDF reports.

10. cookbook/serialization Shows how to use the serializer and unserializer decorators.

11. cookbook/sql-dump A minimal example showing how to dump a table from a SQL database.

12. cookbook/variable-number-of-products Shows how to create tasks whose number of products depends on runtime conditions.

In-depth tutorials for learning. These are part of the documentation.

5. guides/cron This guide shows how to schedule Ploomber pipelines using cron.

6. guides/debugging Tutorial showing techniques for debugging pipelines.

7. guides/first-pipeline Introductory tutorial to learn the basics of Ploomber.

8. guides/intro-to-ploomber Introductory tutorial to learn the basics of Ploomber.

9. guides/logging Tutorial showing how to add logging to a pipeline.

10. guides/monitoring To show the capabilities we'll run our pipeline monitoring through a pre-built Ploomber template: ml-basic.

11. guides/parametrized Tutorial showing how to parametrize pipelines and change parameters from the command-line.

12. guides/refactor Using Soorgeon to convert a notebook into a Ploomber pipeline.

13. guides/serialization Tutorial explaining how the serializer and unserializer fields in a pipeline.yaml file work.

14. guides/sql-templating Introductory tutorial teaching how to develop modular SQL pipelines.

15. guides/testing Tutorial showing how to use a task's on_finish hook to test data quality.

16. guides/versioning A tutorial showing how to version pipeline products.

The simplest way to get started with Ploomber is via the Spec API, which allows you to describe pipelines using a pipeline.yaml file, most examples on this repository use the Spec API. However, if you want more flexibility, you may write pipelines with Python.

The templates/python-api/ directory contains a project written using the Python API. And the python-api-examples/ includes some tutorials and more examples.

In Ploomber 0.21, we introduced a simplified API to write pipelines in a single Jupyter notebook (or .py) file. This is a great option for small projects.

You can find the examples in the micro-pipelines/ directory.