AI Systems

Engineering
Prof. Roberto Pietrantuono

Lecture 2
Outline
• The Functional Architecture
• From What to How: The Implementation Framework
• Development Process
• DevOps, MLOps
• A first simple full production pipeline
How to build an AI
system?
 The system engineering high-level V-Model
Stakeholder’s requirements Solution

System System
development Integration, verification, realization
and validation planning

Upper-level Upper-level
system element system element Architecture
Architecture
development realization Integration
decompositio
and
n and
Verification
definition

Lowerrlevel Lowerrlevel
system element system element
development realization
A Functional
AI System Architecture

«What» the AI system consists of…

The "WHAT" Users (missions)

Sensors Sources

Structured data Unstructured data

Data conditioning
Information
(Un)supervised L.
Reinforcement L. Machine learning
Others… Knowledge
Human-Machine Teaming
Human
Human-Machine cooperation Insight
Machine

CPU GPU TPU Modern computing Custom Quantum

Robust and Responsible AI

Explainable Metrics and bias V&V (standards Security, Policy, ethics,
AI assessment & benchmarks) Privacy Safety and training
Adapted from:
Artificial Intelligence: A Systems Approach from Architecture Principles to Deployment, 2024, David R. Martinez, Bruke M. Kifle.
Data conditioning – Managing Big Data: The 4 “Vs”
Activity Signature Data representation
Detection (Veracity) Data Input (Variety)
Cued Uncued Single Many

Increasing Difficulty

Increasing Difficulty
High
Signal-to-noise

Dimensionality
Low
Low

High
Problem scale Latency requirement
Entities (Volume) Collection time (Velocity)
Few. Many Few Entit. Many

Increasing Difficulty
Increasing Difficulty
Local

Long
Response Time
Relationships

Global

Short
 Data conditioning
Key Questions
Questions Comments
What insight does the customer need? Work from the expected result back to the needed data

What data input is required to achieve Make sure that we have curated data
the desired insight?
Are the incoming data and required Forwardly deployed applications required batched processing in the
processing batched or streaming? cloud, streaming, or both
Can we solve the AI problem using ML Deep learning is very powerful when we have data that is (statistically)
techniques other than Deep Learning? representative of the population, scenarios, or both

Do we need edge computing? We might need to operate under low-capacity and intermittent data links

How do we protect our AI system from Ai systems are very fragile and easy to fool
adversarial AI (e.g., data poisoning)?
Data conditioning
Data Storage Solutions
▪ Since the advent of big data, the has been a surge in DB types over traditional
relational DB for transactional processing
▪ Distributed storage systems for structured big data
▪ Hadoop distributed file system (HDFS) (+ MapReduce for distributed data processing model +
YARN for resource allocation/scheduling on clusters) became a standard
▪ Can work with stream-processing-oriented systems (e.g., Spark, Kafka, Ray)
▪ Rapid ingestion of streaming data
▪ Often coupled with distributed processing systems (e.g., Spark, Kafka, Ray)
▪ NoSQL solutions, more suitable for unstructured data
▪ Consistency vs Availability vs Partition tolerance needs to be considered too (CAP
Theorem)
▪ Computational resources (e.g., storage on the edge in IoT)
 SQL and NoSQL comparison
SQL NoSQL
Relational database with fixed columns and rows Nonrelational databases typically found with unstructured
typically found with structured data data

Follows a specific schema; for example, last name Relational data (structured data) can be stored, but no in a
(row) followed by first name, home address, phone, … row-column pair. Instead, data is nested in a single data
structure
Much easier to comply with the ACID properties Harder to meet ACID properties (often use BASE (Basically
Available, Soft State, Eventually Consistent) properties).

Difficult to scale to large volume and velocity Easier to scale, optimized for developer productivity, runs
well on distributed clusters

Originated in the early 1970s with the need of Originated in the mid-2000s with he need for rapid ingestion
transactional processing (e.g., internet searches)
NoSQL Database types
Key-value stores are the simplest NoSQL (e.g.:
Key-value databases Customer #ID and Attributes). Example: Amazon
DynamoDB

Like before, but stores data in RAM. Example: Redis,

Key-value cache Memcached, Apache Ignite

Key is a document ID. Value is the hierarchical tree

Document databases data structures (e.g., JSON, XML, YAML)containing the
doc. Example: MongoDB

Each column family can be compared to a container of

Wide-column databases. rows in an RDBMS table. Rows do not have to have the
same columns (e.g.: Google Bigtable, Cassandra)

Data is stored in nodes (e.g., people) and edges

Graph databases (relationships social in social networks). Examples:
Neo4J
Example. Key-value store
<Key=CustomerID>
<Value=Object>
Customer
Billing Address

Orders
Order
ShippomgAdress

OrderPayment

OrderItem
Product
Example: Wide-col db
 Data conditioning
Data Wrangling/Munging
Stage Issues/Activities
Data discovery Analyze and understand your data. Are the required data available? Are
they representative? Is there any bias?

Data structuring Organize raw data into structured data

Cleaning, data quality Corrections, formatting, duplication, missing or incomplete data, outlier
detection, bias-variance trade-off (over/underfitting), imputation,
rescaling, pruning, feature selection, re-sampling

Preparation, Enriching Curse-of-dimensionality, Dimensionaltiy reduction, Data augmentation

(Determine if additional data is neededValidating), Data labeling, Data
drift
Validating Checking consistency, accuracy, privacy and security, bias

Publishing Prepare the data set for use downstream, which could include use for
users or software.
Data conditioning

▪ Data Management Strategy

▪ To what extent can your data be made Findable,
Accessible, Interoperable and Re-usable? (FAIR
principles)
▪ What data can be shared/disclosed and
how? What tools/repos for sharing?
▪ Naming conventions
▪ How/where data are stored (security (e.g.,
encryption), privacy, reliability (e.g., fault-
tolerant replicas, distributed systems)
▪ What (if any) data can be re-used (licensing
options)
▪ How they need to be consumed (streaming/batch
processing – previous slides)
Machine Learning – By strategy
Learning

Semi-
Supervised Unsupervised Self-supervised Reinforcement
supervised

Off-policy vs
Classification Regression Inductive Clustering Association
On-policy

(Non-)Linear Offline vs
Decision Tree Transductive K-Means Rule mining
Regression Online

Value-, Policy-,
SVC SVR Hierarchical Itemset mining
Model-based

Neural
K-NN Probabilistic
Networks

Regression
Neural Network
Trees
Machine Learning – Type of models
• Discriminative models
• capture the conditional probability p(Y | X)
• discriminate between different kinds of data instances
• Has to learn a boundary

• Generative models
• capture the joint probability p(X, Y), or just p(X) if there are no labels.
• can generate new data instances
• Has to capture correlations in the data
• Ex.: GAN, (Variational ) Autoencoders
Machine Learning – Limitations/Challenges
• Opacity, explainability (DNN)
• Bias (Fairness)
• Computational resources (e.g., training), human resources (e.g.,
labelling)
• Over-Under fitting
• Vulnerabilities
• Bias-Variance Trade-Off, Hallucination

• “Engineering” challenges (storage, processing, distribution,

requirements,…)
An end-to-end AI system architecture
implementation framework

From input to AI product or service: The «How»

 Operational
excellence

Quality Cost
Assurance optimization
Stakeholder Business goals
assessment (stakeholders req)

Monitoring Data conditioning

Deployment Machine learning

Robust and Resp. AI Modern computing

Human-Machine
Teaming
Risk Reliability and
Management Security

Source: Artificial Intelligence: A Systems Approach from Architecture

Performance
Principles to Deployment, 2024, David R. Martinez, Bruke M. Kifle. Metrics
Adapted from AWS framework
 From what to how
Technology Description Risk Analysis
System analysis trade-offs Threshold vs target req, incorporating Can the AI capability be scaled? Are
stakeholders business goals and AI requirements realistic? Risk in
capabilities provided achieving target vs threshold reqs
Functional architecture System architecture building block End-to-end measure of performance
Sensors and sources Identify the data needed Are the required data available?
Data condit. ML, computing, HMT Data preparation; ML algorithms Technology management and
selection; computing (enterprise/edge) technology levels of maturity
infrastructure; address HMT
cooperation level
Robust AI Vulnerability to adversarial AI What’s the likelihood vs consequence
of unintentional/intentional adversaries
Responsible AI Use tools to incorporate FASTEPS To what degree are the harms mitigated
and FASTEPS principles implemented?
 From what to how
Process (1/2) Description Risk Analysis
Stakeholder needs Business goals drive AI requirements Are the needs and goals realistic?
Integration, V&V I-V&V at each stage of V-Model Is it exercised at all process levels?
Development, deployment, Monitoring
Operational Excellence Ability to deliver business value based Regards all the items in “Technology”,
on the AI value proposition, and to must be assessed for each building
continually improve processes and block
procedures through monitoring
Cost Optimization Ability to avoid unneeded costs or Ability to minimize cost while meeting
suboptimal resources, throughout the business needs and goals
whole system life cycle
Reliability & Security Ability to meet requirements in a What is the likelihood of a failure and
secured way; (Reliability: ability to its associated consequences?
tolerate and recover from failures, and
to dynamically get resources to meet
the demand. Security: abiltiy to protect
information, systems, and assets). It
applies to all building blocks
 From what to how
Process (2/2) Description Risk Analysis
Operational metrics Performance metrics for each building Including accuracy, precision, recall, F-
block, e.g., based on benchmarks (gold scores, and others relative to model
standards). performance and at system-level.
Include efficient use of computing
resources and scaling
Risk Management Include technical risk (e.g., Must address the broad range of risks
performance risk), management risk in the context of likelihood vs
(e.g., cost and schedule), consequences (i.e., impact)
organizational risk (e.g., societal risk)
Quality Assurance Broad term to assert that Should be addressed from design to
specifications are met (including, for development, deployment and
instance, size, weight, power, shock, monitoring. Standard compliance.
vibration, …). ISO 9000 standard family
Cross-cutting system engineering Tools and approaches to assess Employing these tools should also
performance prior to deployment (e.g., include the use of operational metrics.
modelling, simulation, emulation, etc.) These should be checked for quality
 From what to how
People Description Risk Analysis
Leadership and management Leaders define and drive the AI Leaders must assess the strategic
strategic direction and road map. risks. Managers address the execution
Managers focus on the strategy risks, e.g., is the necessary talent
execution. available to execute a task?
Systems thinker and systems engineers Every AI team must include a person In concert with the AI leadership and
with responsibility for and authority management, the system engineer
over the overall AI system meeting a set must assess the overall AI system risks
of system requirement
Sta ff performance Management must set clear staff Tools are available, discussed later
performance goals, and assess
progress several times
Multicultural environment AI project requires a multidisciplinary Assess the organization’s mentorship
team including engineers, data and coaching approaches
scientist, ML experts, computing
technologists, and human-machine
social engineers
Effective communication Effective communication at all levels This includes ethics issues
within the team and to stakeholders
Still on “how”: An ML perspective
Discover Clarity on the business question.
Clear arguments for ML over another approach.
Definition of the KPIs and metrics you want to optimize.
A sketch of the route to value.
Play Detailed understanding of the data. Working proof of concept.
Agreement on the model/algorithm/logic that will solve the problem.
Evidence that a solution is doable within realistic resource scenarios.
Evidence that good ROI can be achieved.
Develop A working solution that can be hosted on appropriate and available infrastructure
Thorough test results and performance metrics (for algorithms and software).
An agreed retraining and model deployment strategy.
Unit tests, integration tests, and regression tests.
Solution packaging and pipelines
Deploy A working and tested deployment process.
Provisioned infrastructure with appropriate security and performance characteristics.
Mode retraining and management processes. An end-to-end working solution!

CRISP-DM: Cross Industry Standard Process for Data Mining

Business Understanding (Discover), Data Understanding (Discover), Data prep (Play), Modelling (Play and Develop),
Evaluation (Develop and Deploy), Deployment
 Operational
MORE ON HOW:
excellence

Quality Cost
Assurance optimization
Stakeholder Business goals
assessment (stakeholders req)

Monitoring Data conditioning

Deploy

Develop
Deployment Machine learning

Robust and Resp. AI Modern computing

Human-Machine
Teaming
Risk Reliability and
Management Security

Performance
Metrics
Still on “how”: An ML perspective
 Operational
MORE ON HOW:
excellence

Quality Cost
Assurance optimization
Stakeholder Business goals
assessment (stakeholders req)

Monitoring Data conditioning

Develop
Deployment Machine learning

Robust and Resp. AI Modern computing

Human-Machine
Teaming
Risk Reliability and
Management Security

Performance
Metrics
The final AI Product/Service heterogeneity (1/2)

Aerospace (drones,
airplane design, Agriculture Automation (e.g., Chemicals
satellites, ATC (precision farming) self-driving cars) (cleaning, hygiene)
prediction)

Climate change (water Consumer goods and Cybersecurity Digital enterprise

sustainability and services (marketing, (malware, transformation,
purification, greenhouse advertising,
effects reduction, floods, manufacturing, e-
adversarial AI, Industry 4.0
erosion, energy commerce, lifelong ransomware, deep (robotics process
distribution, forest fires)
learning) fakes) automation)

Electronics and Engineering and

Enterprise construction (site Education (virtual
computing (data planning, drones teacher)
centers, cloud) for monitoring)
The final AI Product/Service heterogeneity (2/2)

Healthcare (medical
Finance (banking, Fitness (wearable diagnosis, medical
Education (virtual
fraud detection, devices, clothing, equipment,
teacher)
trading) nutrition) anxiety/depression,
surgical robots)

Security and safety

Oil and gas Haptic sensing (biometrics, remote Software
(exploration, natural (prosthetics, identification Engineering (AI for
resources delivery exoskeleton) systems, law testing, deisgn, …)
enforcement)

Virtual Worlds and

Utilities (water, gas,
Telecommunication X-reality
electricity, power Virtual personal
s (5G, 6G, (augmented, mixed,
generation via solar assistants
connectivity) virtual reality;
or wind)
metaverse)
The AI Systems Spectrum
Chat Bots, Deep Fakes, Virtual Personal AIoT, e.g.,
Robotics Autonomous
Malware detectors, sw Assistants, AI- infrastructure
Systems Driving Systems
engineering… Powered Avatars monitoring

AI(Software)-centric, System-centric perspective

Low Risk High-Risk Unacceptable Risk

Societal Risk (e.g., AI Act)

Search-and-discovery AI as a Teammate Full Autonomy

Level of autonomy
The AI Systems Spectrum
Chat Bots, Deep Fakes, Virtual Personal AIoT, e.g.,
Robotics Autonomous
Malware detectors, sw Assistants, AI- infrastructure
Systems Driving Systems
engineering… Powered Avatars monitoring

AI(Software)-centric, System-centric perspective

• AI(Software)-centric (our next classes)

• DevOps (and MLOps, LLMOps) has a greater focus
• Software engineering and related processes/architectures have greater focus (e.g.,
Microservice architectures, cloud/containerization)
• Examples we’ll look at: prediction/classification services, LLM-based chatbots
• System-centric (later in the course)
• System as a whole has a greater focus
• Model-based System Engineering, Simulation/Digital Twin, IoT, SysML
• Examples we’ll look at: Autonomous Driving System, UAV
Examples
 AI(Software)-centric perspective
LLM-based agents
User Interface Executor
Result Planning Memory
Response Synthesis
Task
Preferences
Goal …
Prompt LLM Knowledge
Prompt Task

Context Models Tools

Data
LLMs
LLMs
Domain LLMs
Experts
 System-centric perspective:
The ADS/ADAS Example
Sensing Perception Route Planner
▪ Architecture/SysML Lidar Object Detection Plans routes via
▪ MBSE Camera Obstacle Detection path search on
global map
GPS Road edge detect.
▪ Digital twins Accelerometers,
Gyros
World Model Monopoly Board
Static and moving Keep tracks of
obstacles what to do via FSM
Road map
Actuators Position/speed Motion Planner
Steering Plans short-term
Braking moves by
Speed searching for a
short path
 AI-centric
Systems
Following the process…
Technical requriements
▪ Anaconda
▪ Conda, possibly poetry
▪ IDE
▪ E.g., PyCharm, Spyder
▪ Atlassian Jira
▪ Git, GitHub
▪ AWS
Discover, Play, Develop, Deploy
A simple example of a full pipeline - Instructions
▪ Install Anaconda
▪ Clone the repo
▪ Create and activate the Environment
git clone https://github.com/rpietrantuono/AISE_Ch2.git
conda env create –f env.yml
conda activate env
conda env list

>>>env_test /Users/robertopietrantuono/anaconda3/envs/test
>>>env * /Users/robertopietrantuono/anaconda3/envs/env
>>>others ...

Deactivate any active environment first.

Alternatively, use Anaconda Navigator

Discover, Play, Develop, Deploy
Stakeholder’s requirements
Discover, Play, Develop, Deploy
Requirements and the
Stakeholder’s requirements “discover” phase, as well
as the whole project agile
life cycle (SCRUM in this
case), can be managed with
the help of management
tools like Jira Atlassian

Can be integrated with other tools, like GitHub

Discover, Play, Develop, Deploy

The code is in a Jupyter notebook, which is run by a user

interactively in a web browser.
The code sporadically calls methods to simply check or
explore elements of the data (for example, df.head() and
df.dtypes).
Discover, Play, Develop, Deploy

There is ad hoc code for plotting (and it’s Absolutely fine in a this exploratory phase, but
not very intuitive!).
we need to take code like this and make it into
There is a variable called tmp, which is
not very descriptive. something suitable for your production ML
pipelines... Let’s go to the «Develop» stage
Discover, Play, Develop, Deploy
Select a development process
Methodology Pros Cons
Agile Flexibility is expected. If not well managed, can
easily have scope drift.
Faster dev to deploy
cycles. Sprints or Kanban may
not work well for some
projects.
Waterfall Clearer path to Lack of flexibility.
deployment.
Higher admin overheads.
Clear staging and
ownership of tasks.
 Discover, Play, Develop, Deploy
Manage your artefacts – Code Version Control Strategies
Engineer A

git checkout -b pipeline1spark

# Configure an ML pipeline, which consists of three stages: tokenizer,

hashingTF, and lr.
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(),
outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

git checkout -b pipeline1spark

 Discover, Play, Develop, Deploy
Manage your artefacts – Code Version Control Strategies
Engineer B create another
branch with A’s code in it
git pull origin pipeline1spark
git checkout pipeline1spark Engineer B adds
git checkout -b pipeline some code and push

lr = LogisticRegression(maxIter=model_config["maxIter"],
regParam=model_config["regParam"])

git push origin pipeline

main There are three branches now:

pipeline1spark
pipeline
Engineer C doesn’t know where the
updated code is (assume main)
 Discover, Play, Develop, Deploy
Manage your artefacts – Code Version Control Strategies
# Branch pipeline1spark - Commit 1 (Engineer A)
lr = LogisticRegression(maxIter=10, regParam=0.001) If these commits both
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) get pushed to the main
branch at the same
# Branch pipeline - Commit 2 (Engineer B) time, then we will get
lr = LogisticRegression(maxIter=model_config["maxIter"], what is called a merge
regParam=model_config["regParam"]) conflict,
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

<<<<<<< HEAD
lr = LogisticRegression(maxIter=10, regParam=0.001)
E.g., If engineer A
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
======= pushes…
lr = LogisticRegression(maxIter=model_config["maxIter"],
regParam=model_config["regParam"])
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
>>>>>>> pipeline
Discover, Play, Develop, Deploy
Manage your artefacts – Code Version Control Strategies
• A Git Workflow
• Define types of branches, such as:
• Main contains your official releases and should only contain the stable version of
code
• Dev acts as the main point for branching from and merging to for most work in the
repository; it contains the ongoing development of the code base and acts as a
staging area before main
• Feature branches should not be merged straight into the main branch; everything
should branch off from dev and then be merged back into dev.
• Release branches are created from dev to kick off a build or release process before
being merged into main and dev and then deleted.
• Hotfix branches are for removing bugs in deployed or production software. You can
branch this from main before merging into main and dev when done.
Tag 0.1 Tag 0.2 Tag 1.0

Several bugs fixed

for production:
hotfix 0.2

Start of release
branch for 1.0
Bug fixes
Incorporate bug continuously
fixes in develop merged back

Major feature
for next release

From here on,

Features for «next release»
future release means the
release after 1.0
from https://lucamezzalira.com/2014/03/10/git-flow-vs-github- flow/.
 Discover, Play, Develop, Deploy
Manage your artefacts – Code Version Control Strategies

• Pull Requests
• To make known your intention to
merge into another branch and
allow another team member to
review your code before this
executes
• Enables code review
• You do this whenever you want to
merge your changes and update
them into dev or main branches.
 Discover, Play, Develop, Deploy
Manage your artefacts – Model Version Control
pip install mlflow

If you already activated the provided environment, this is not needed

import pandas as pd
from prophet import Prophet
from prophet.diagnostics import cross_validation Main code excerpts:
from prophet.diagnostics import performance_metrics Relevant imports
import mlflow
import mlflow.pyfunc
 Discover, Play, Develop, Deploy
Manage your artefacts – Model Version Control
class ProphetWrapper(mlflow.pyfunc.PythonModel):
def __init__(self, model): Wrapper class
self.model = model inheriting from
super().__init__() mlflow model
object
def load_context(self, context):
from prophet import Prophet
return

def predict(self, context, model_input):

future = self.model.make_future_dataframe(
periods=model_input["periods"][0])
return self.model.predict(future)
 Discover, Play, Develop, Deploy
Manage your artefacts – Model Version Control
with mlflow.start_run():
# create Prophet model train_predict
model = Prophet( Wrapper class to
yearly_seasonality=seasonality_params['yearly’], improve reuse;
weekly_seasonality=seasonality_params['weekly’], MLflow-related
daily_seasonality=seasonality_params['daily’] code reported here
)
# train and predict
model.fit(df_train)
# Evaluate Metrics
df_cv = cross_validation(model, initial="366 days", period="180 days",
horizon="365 days")
df_p = performance_metrics(df_cv)
# Log parameter, metrics, and model to MLflow
mlflow.log_metric("rmse", df_p.loc[0, "rmse"])
mlflow.pyfunc.log_model("model", python_ model=ProphetWrapper(model))
print( "Logged model with URI: runs:/{run_id}/model".format(
run_id=mlflow.active_run().info.run_id ) )
 Discover, Play, Develop, Deploy
Manage your artefacts – Model Version Control
mlflow ui

To get the IP port of MLFlow server

First open the

import mlflow browser at the
… provided IP-Port,
then set tracking in
mlflow.set_tracking_uri("http://localhost:5000") your code

with mlflow.start_run():
…
Discover, Play, Develop, Deploy

▪ How to get your solution built out into the real world ?
▪ On-premise deployment, on owned infrastructures
▪ Legacy software
▪ Strong constraints on data location and processing
▪ Pros: security, privacy
▪ Cons: Need (physical) resources and specialists, e.g., for
configuring networking, load balancing, infrastructure
maintenance…
Discover, Play, Develop, Deploy

• How to get your solution built out into the real world ?
• Infrastructure-as-a-Service (IaaS)
• On-demand access to physical and virtual servers, cloud-hosted
storage and networks, and the back-end IT infrastructure on a pay-as-
you-go basis

• Extreme scalability, no maintenance

• Google Cloud, AWS Simple Storage Service (S3), Elastic Compute

Cloud (EC2), Microsoft Azure
Discover, Play, Develop, Deploy

How to get your solution built out into the real world ?
• Platform-as-a-Service (PaaS)
• On-demand access to a ready-to-use, complete cloud hosting platform for
developing, running, maintaining and managing applications.

• Don’t need to worry about underlying infrastructure.

• Examples , Google App Engine, AWS Elastic Beanstalk, AWS Lambda

functions (serverless functions), Databricks on Spark cluster
infrastructure
Discover, Play, Develop, Deploy

How to integrate and manage your deployment and continuous update cycles?
Discover, Play, Develop, Deploy
Life cycle Activity Details Example of
stage Tools
Testing Unit tests: tests aimed at testing the functionality pytest or
smallest pieces of code. unittest
Integration tests: ensure that interfaces within the Selenium
code and to other solutions work.
Acceptance tests: business focused tests. Behave

UI tests: ensuring any frontends behave as

Dev expected.
Linting Raise minor stylistic errors and bugs. flake8 or bandit

Formatting Enforce well- formatted code automatically. black or sort

Building The final stage of bringing the solution together. Docker, twine,
or pip
Discover, Play, Develop, Deploy
Life cycle Activity Details Example of Tools
stage
Training Train the model Any ML package

Predicting Run the predictions or inference step Any ML package

Building Creating the pipelines and application sklearn pipelines,

logic in which the model is embedded. Spark ML pipelines,
ML ZenML.
Staging Tag and release the appropriate version of MLflow or Comet.ml.
your models and pipelines.
Monitoring Track the solution performance and raise Seldon, Neptune. ai,
alerts when necessary Evidently.ai, or
Arthur.ai.
Discover, Play, Develop, Deploy
Life cycle Activity Details Example of Tools
stage
Releasing Taking the software you have built and Twine, pip, GitHub, or
storing it somewhere central for reuse BitBucket.
Deploying Pushing the software you have built to the Docker, GitHub
appropriate target location and Actions, Jenkins,
Ops environment. TravisCI, or CircleCI.
Monitoring Tracking the performance and utilization of DataDog, Dynatrace, or
the underlying infrastructure and general Prometheus.
software performance, alerting where
necessary.
 Discover, Play, Develop, Deploy
name: Python package Define the name of the
on: [push] GitHub Actions workflow
and what Git event will
jobs: trigger
build:
runs-on: ubuntu-latest
strategy:
matrix: list the jobs you want to
python-version: [‘3.9’, ‘3.10’] execute and their config.

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }} define the step. «uses»
uses: actions/setup-python@v4 exploits pre-defined
with: standard actions
python-version: ${{ matrix.python-version }}
 Discover, Play, Develop, Deploy
- name: Install dependencies installs the relevant
run: dependencie
| python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

- name: Lint with flake8

run: Run linting
| # stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics #
exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line- length=127
--statistics run tests, specifying a
working-directory
- name: Test with pytest keyword to only run
run: pytest in that
pytest directory.
working-directory: testing
 Discover, Play, Develop, Deploy
Continuous model performance testing
@pytest.fixture
def test_dataset() -> Union[np.array, np.array]:
# Load the dataset
X, y = load_wine(return_X_y=True) Load a dataset
# create an array of True for 2 and False otherwise
y = y == 2
# Train and test split
X_train, X_test, y_train, y_test = train_test_split(X, y,
random_state=42) Load a model from
return X_test, y_test HuggingFace (could be
from Mlflow or any other
@pytest.fixture repo)
def model() -> sklearn.ensemble._forest.RandomForestClassifier:
REPO_ID = "electricweegie/mlewp-sklearn-wine"
FILENAME = "rfc.joblib"
model = joblib.load(hf_hub_download(REPO_ID, FILENAME)) return model
 Discover, Play, Develop, Deploy
Continuous model performance testing
def test_model_inference_types(model, test_dataset):
assert isinstance(model.predict(test_dataset[0]), np.ndarray)
assert isinstance(test_dataset[0], np.ndarray)
assert isinstance(test_dataset[1], np.ndarray)

def test_model_performance(model, test_dataset):

metrics = classification_report(y_true=test_dataset[1],
y_pred=model.predict(test_dataset[0]), output_dict=True)
assert metrics['False']['f1-score'] > 0.95
assert metrics['False']['precision'] > 0.9
assert metrics['True']['f1-score'] > 0.8
assert metrics['True']['precision'] > 0.8
To sum up…