PHASE 2 DOCUMENT:DATA WRANGLING AND ANALYSIS IN

HEALTH MONITORING AND DIAGNOSIS

INTRODUCTION

In Phase 2 of our project, we focus on enhancing health monitoring and diagnosis systems. This
phase involves implementing advanced data analytics to improve accuracy in detecting health issues.
We integrate AI-based predictive models to pre-emptively identify potential risks. Additionally, this
stage encompasses user-friendly interfaces to ensure seamless adoption by healthcare
professionals. Our goal is to significantly improve patient outcomes through innovative technology.

OBJECTIVES

• Early Detection of Health Conditions: Monitor vital signs and other health indicators to
identify potential health issues before they become severe.
• Continuous Health Assessment: Enable ongoing monitoring of a person's health to track
changes over time, allowing for real-time assessment and timely interventions.
• Improved Diagnostic Accuracy: Use data-driven insights and advanced diagnostic tools to
increase the precision and reliability of medical diagnoses.
• Personalized Health Recommendations: Tailor healthcare advice and interventions based on
individual health data to create more effective treatment plans.
• Patient Empowerment and Engagement: Encourage patients to take an active role in their
health by providing them with accessible health data and insights.

DATASET DESCRIPTION

A health monitoring and diagnosis dataset encompasses patient demographic information, including
age, gender, and unique identifiers. It contains medical history, lab test results, and vital signs such
as heart rate, blood pressure, and temperature. Often, the dataset incorporates data from wearable
devices and IoT sensors. To ensure privacy, all data is secured and anonymized in compliance with
regulatory standards.

DATA WRANGLING TECHNIQUES

1.DATA DESCRIPTION

head(): Displays the first five rows of the dataset to give an initial view of the data, useful for seeing
the structure and content of the dataset.

tail(): Shows the last five rows, helping to identify if there are any anomalies or missing data at the
end.

info(): Provides a summary of the dataset, including the number of rows, data types, and the count
of non-null entries for each column.
describe(): Offers descriptive statistics for numerical columns, such as mean, median, min, max, and
standard deviation.

CODE

import pandas as pd

data = {

'Patient_ID': [101, 102, 103],

'Age': [25, 30, 45],

'Weight': [70, 65, 80],

'Height': [175, 170, 180],

'Blood_Pressure': ['120/80', '110/75', '130/85'],

'Heart_Rate': [70, 72, 80]}

df = pd.DataFrame(data)

print("DataFrame Head:")

print(df.head())

print("\nDataFrame Tail:")

print(df.tail())

print("\nDataFrame Info:")

print(df.info())

print("\nDataFrame Describe:")

print(df.describe())

OUTPUT
2.NULL DATA HANDLING

Identification of Null Data: Using isnull().sum(), you can identify the columns with missing data and
count the number of null values in each column.

Null Data Imputation:

• For numerical columns like 'Age', you can impute missing values with the mean, median, or
other statistical measures.
• For categorical columns like 'Gender', you can impute with the mode (most frequent value)
or a fixed value.

Null Data Removal: If necessary, you can remove rows with any missing data using dropna(). This
approach is suitable when the amount of null data is relatively low or when the impact of missing
values on analysis could be significant.

CODE

import pandas as pd

import numpy as np

data = {

'Patient_ID': [101, 102, 103],

'Age': [25, np.nan, 45],

'Weight': [70, 65, np.nan],

'Height': [175, 170, np.nan],

'Blood_Pressure': ['120/80', '110/75', None],

'Heart_Rate': [70, 72, np.nan]

df = pd.DataFrame(data)

print("\nCount of Null Data in Each Column:")

print(df.isnull().sum())

df= df.dropna()

OUTPUT
3.DATA VALIDATION

Data Integrity Check: Using .unique() to ensure 'Patient_ID' values are unique helps maintain the
integrity of patient records, avoiding issues caused by duplicate entries.

Data Consistency Verification: Checking unique values in 'Age', 'Blood_Pressure', and 'Heart_Rate'
can help verify consistency across the dataset. It can also help identify potential outliers or incorrect
data entries.

CODE

import pandas as pd

import numpy as np

data = {

'Patient_ID': [101, 102, 103],

'Age': [25, 30, 45],

'Weight': [70, 65, 80],

'Height': [175, 170, 180],

'Blood_Pressure': ['120/80', '110/75', '130/85'],

'Heart_Rate': [70, 72, 80]

df = pd.DataFrame(data)

print("Unique Patient IDs:")

print(df['Patient_ID'].unique())

print("\nUnique Age Values:")

print(df['Age'].unique())

print("\nUnique Blood Pressure Values:")

print(df['Blood_Pressure'].unique())

print("\nUnique Heart Rate Values:")

print(df['Heart_Rate'].unique())

OUTPUT
4.DATA RESHAPING

Reshaping Rows and Columns:Transposing a DataFrame allows you to swap rows and columns,
providing a different perspective on the data.

Transporting Data:Transposing a DataFrame can facilitate data transportation by allowing you to

align the data in a way that suits exportation or integration with other systems.

CODE

import pandas as pd

import numpy as np

data = {

'Patient_ID': [101, 102, 103],

'Age': [25, 30, 45],

'Weight': [70, 65, 80],

'Height': [175, 170, 180],

'Blood_Pressure': ['120/80', '110/75', '130/85'],

'Heart_Rate': [70, 72, 80]

df = pd.DataFrame(data)

print("Original DataFrame:")

print(df)

df_transposed = df.T

print("\nTransposed DataFrame:")

print(df_transposed)

OUTPUT
5.DATA MERGING

Combining datasets :involves merging multiple DataFrames into one, allowing you to bring together
various sources of health-related information for a comprehensive view.

Joining data: involves merging DataFrames based on a shared key, with different join types
determining which records are included.

CODE

import pandas as pd

import numpy as np

data1 = {

'Patient_ID': [101, 102, 103],

'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 45],

'Gender': ['F', 'M', 'M']

df1 = pd.DataFrame(data1)

data2 = {

'Patient_ID': [101, 102, 104],

'Weight': [70, 65, 80],

'Height': [175, 170, 180],

'Blood_Pressure': ['120/80', '110/75', '130/85']

df2 = pd.DataFrame(data2)

Merged_data = pd.merge(df1, df2, on='Patient_ID')

print("Merged DataFrame:")

print(Merged_data)

OUTPUT
6.DATA AGGREGATION

Grouping Data: The groupby('Patient_ID') method groups the data by 'Patient_ID'. This allows you to
perform operations on a per-patient basis.

Aggregating Data: The agg() method calculates the mean of 'Weight', 'Height', and 'Heart_Rate' for
each group. This is helpful to get average values for each patient.

CODE

import pandas as pd

import numpy as np

data = {

'Patient_ID': [101, 102, 103],

'Weight': [70, 65, 80],

'Height': [175, 170, 180],

'Blood_Pressure': ['120/80', '110/75', '130/85'],

'Heart_Rate': [70, 72, 80]

df = pd.DataFrame(data)

grouped = df.groupby('Patient_ID')

agg_mean = grouped.agg({

'Weight': 'mean',

'Height': 'mean',

'Heart_Rate': 'mean'

})

print(agg_mean)

OUTPUT
7.EXPLORATORY DATA ANALYSIS(EDA)

Univariate Analysis: In health monitoring and diagnosis, histograms are often used to analyze key
metrics like age, weight, or heart rate.

Bivariate Analysis:In health monitoring, this can help identify correlations or patterns, such as the
relationship between weight and height or age and blood pressure.

Multivariate Analysis:In health monitoring and diagnosis, this can reveal complex relationships and
aid in comprehensive data analysis by visualizing how several health metrics relate to each other.

CODE

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

data = {

'Patient_ID': [101, 102, 103, 104, 105, 106, 107, 108],

'Age': [25,50, 45, 30, 35, 40, 60, 55],

'Heart_Rate': [70, 72, 80, 78, 74, 69, 85, 76],

df = pd.DataFrame(data)

# ---- Histogram for Univariate Analysis ----

sns.histplot(df['Age'], bins=20)

plt.title("Histogram of Patient Age")

plt.show()

#---Scatter plot for Bivariate Analysis---

sns.scatterplot(x='Age', y='Heart_Rate', data=df)

plt.title("Scatter Plot of Age vs Heart Rate")

plt.show()

#---Pair plot for Multivariate Analysis---

sns.pairplot(df)

plt.suptitle("Pair Plot for Health Monitoring Data", y=1.02) # y=1.02 for proper title placement

plt.show()
OUTPUT
8.FEATURE ENGINEERING

Creating User Profiles: Creates simple user profiles containing key attributes and health-related
information.

Temporal Analysis: The temporal analysis tracks changes in health metrics over time, allowing you
to examine trends and variations.

Content Embeddings: Content embeddings convert categorical data into numerical representations,
making it easier to use in machine learning or advanced analysis.

CODE

#---Code for Creating User’s Profile---

import pandas as pd

import numpy as np
data = {

'Patient_ID': [101, 102, 103, 104],

'Name': ['Alice', 'Bob', 'Charlie', 'David'],

'Age': [25, 30, 45, 50],

'Weight': [70, 65, 80, 85],

'Height': [175, 170, 180, 178],

'Heart_Rate': [70, 72, 80, 78],

'Blood_Pressure': ['120/80', '110/75', '130/85', '140/90']

df = pd.DataFrame(data)

user_profiles = df[['Patient_ID', 'Name', 'Age', 'Weight', 'Height', 'Heart_Rate', 'Blood_Pressure']]

print("User Profiles:")

print(user_profiles)

#---Code for Temporal Analysis---

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

data = {

'Patient_ID': [101, 102, 103, 104],

'Checkup_Time': ['2024-01-01 08:00:00', '2024-01-02 14:30:00', '2024-01-03 09:15:00', '2024-01-

04 17:45:00'],

'Heart_Rate': [70, 72, 78, 80]

df = pd.DataFrame(data)

df['Checkup_Time'] = pd.to_datetime(df['Checkup_Time'])

df['Hour'] = df['Checkup_Time'].dt.hour

print("Data with Hour Extracted from Checkup Time:")

print(df)

#Code for Content Embeddings

import pandas as pd

import gensim
from gensim.models import Word2Vec

data = {

'Patient_ID': [101, 102, 103, 104],

'Symptoms': [

"fever cough fatigue",

"sore throat cough congestion",

"nausea headache fatigue",

"headache dizziness"

],

'Diagnosis': ["flu", "common cold", "migraine", "dehydration"]

df = pd.DataFrame(data)

symptom_sentences = [symptom.split() for symptom in df['Symptoms']]

word2vec_model = Word2Vec(symptom_sentences, vector_size=100, window=5, min_count=1,

workers=4)

cough_embedding = word2vec_model.wv['cough']

print("Word2Vec Embedding for 'cough':")

print(cough_embedding)

similar_to_cough = word2vec_model.wv.most_similar('cough', topn=5)

print("\nTop 5 words similar to 'cough':")

for word, similarity in similar_to_cough:

print(f"{word}: {similarity:.2f}")

unique_words = list(word2vec_model.wv.key_to_index.keys())

from sklearn.decomposition import PCA

pca = PCA(n_components=2)

word_embeddings_2d = pca.fit_transform([word2vec_model.wv[word] for word in unique_words])

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))

plt.scatter(word_embeddings_2d[:, 0], word_embeddings_2d[:, 1], alpha=0.7)

for i, word in enumerate(unique_words):

plt.annotate(word, (word_embeddings_2d[i, 0], word_embeddings_2d[i, 1]))

plt.title("2D Projection of Word2Vec Embeddings")

plt.xlabel("PCA 1")

plt.ylabel("PCA 2")

plt.show()

OUTPUT

USER’S PROFILE

TEMPORAL ANALYSIS

CONTENT EMBEDDINGS
ASSUMED SCENARIO

• SCENARIO: The project aims to improve early diagnosis and personalized care by integrating
patient data with electronic health records for seamless communication with healthcare
providers.
• OBJECTIVE: To design a health monitoring and diagnosis system that continuously collects
and analyzes physiological data, enabling early detection of health risks, personalized care,
and improved patient outcomes.
• TARGET AUDIENCE : The individuals seeking proactive health management, especially those
with chronic conditions or at risk for them, as well as healthcare providers looking for
advanced tools to support patient care. It also caters to researchers and developers focused
on creating innovative health monitoring technologies.

CONCLUSION

Phase 2 of the health monitoring and diagnosis project marks a significant advancement in
personalized healthcare and remote health management. By incorporating sophisticated wearable
technology, AI-driven analytics, and seamless telemedicine integration, this phase allows for more
comprehensive monitoring and early detection of health issues. The enhanced data-driven approach
fosters proactive healthcare, with the potential to reduce costs and improve patient outcomes. As
the project progresses, it underscores the importance of data security and patient privacy, ensuring
the trust and safety of all involved.