Building Domain Enriched

Deep Learning Algorithms

Srijit Mukherjee
Pennsylvania State University
The Fundamentals
 Introduction to Machine Learning

The main question is

Data Problem
how to build domain
enriched models in a
general way for real number supervised learning
categorical f(x) ~ y
different data types
array finding f
like images, video, tabular
text, graph, images unsupervised learning
hypergraphs, etc text x~F
video finding F
graph

Domain Enriched Solution

 Introduction to Machine Learning

optimize

Data

Preprocessing Model Loss

Problem make the

Inference
model better

In classical machine learning, the goal is to extract hand made features important for the data and problem.
In deep learning, the goal is to make the model learn from a large amount of data, and the problem in hand.
 Introduction to Different Data Types and Problems
What about signals, time
series, social network data?

Here R
means both
real and
categorical age age, weight, height
together. gene 1D signals
1D time series
longitudinal data
text
gene data

colour image
grayscale image colour video
3D grayscale image
3D colour image
grayscale video
 Signals, Time Series, Social Network Data, Knowledge

Here both the

with just the black arrows, it
sequence of Xs is called Markov Property.
and the
relationship Re
between them
will play a You can include causal
significant role in relationships as graphical causal
building a model. models, which is nothing but a
graph relationship.
 Different Machine Learning Problems with Data
Tabular
● Classification
● Clustering

Images
● Classification
● Segmentation
● Object Detection
● Image Enhancement
● Image Reconstruction

Signals and Time Series

● Future Prediction
● Classification
● Clustering
● Change Point Detection
 Why and What of Domain Enriched Models?

optimize

Data

Preprocessing Model Loss

Problem make the

Inference
model better

In classical machine learning, the goal is to extract hand made features important for the data and problem.
In deep learning, the goal is to make the model learn from a large amount of data, and the problem in hand.
The goal of domain learning deep learning is to solve by taking a middle ground between the two.
 Why and What of Domain Enriched Models?

Dash, Tirtharaj, et al. "A review of some techniques for inclusion of domain-knowledge into deep neural networks." Scientific Reports 12.1 (2022): 1040.
 Why and What of Domain Enriched Models?

[1]

[1] Cohen, Joseph Paul, et al. "Problems in the deployment of machine-learned models in health care." Cmaj 193.35 (2021): E1391-E1394.
 Introduction to Deep Learning

optimize

Data

Preprocessing Loss

Problem make the

Inference
model better

In classical machine learning, the goal is to extract hand made features important for the data and problem.
In deep learning, the goal is to make the model learn from a large amount of data, and the problem in hand.
Introduction to Deep Learning
Can we just use this linear
layers with activation to solve
any problem?
Deep Learning is more Domain Enriched than you think!

Recurrent Type Neural Network (time series) expensive Convolutional Neural Network (images)
eg: LSTM (Long and Short Term memory usage in model) eg: spatial/translation invariance

Graphical Neural Network (graph) Transformer Neural Network

eg: permutation of the nodes invariance eg: long range dependency with parallel computing

It should be explained that how each of the network is specifically engineered for the specific purpose.
How to build Domain Enriched solutions?
with examples and applications using images
 Why and What of Domain Enriched Models?

optimize

Data

Preprocessing Model Loss

Problem make the

Inference
model better

In classical machine learning, the goal is to extract hand made features important for the data and problem.
In deep learning, the goal is to make the model learn from a large amount of data, and the problem in hand.
The goal of domain learning deep learning is to solve by taking a middle ground between the two.
 Some Notations to start with.

Dash, Tirtharaj, et al. "A review of some techniques for inclusion of domain-knowledge into deep neural networks." Scientific Reports 12.1 (2022): 1040.
 Preprocessing

background knowledge (BK)

Dash, Tirtharaj, et al. "A review of some techniques for inclusion of domain-knowledge into deep neural networks." Scientific Reports 12.1 (2022): 1040.
 Preprocessing

Shin, Seung Yeon, et al. "Deep vessel segmentation by learning graphical connectivity." Medical image analysis 58 (2019): 101556.
 Preprocessing

Diao, Qi, et al. "Superpixel-based attention graph neural network for semantic segmentation in aerial images." Remote Sensing 14.2 (2022): 305.
 Preprocessing

Word embeddings in Natural Language Processing

 Preprocessing

Higa, Maiki, et al. "Domain knowledge integration into deep learning for typhoon intensity classification." Scientific reports 11.1 (2021): 12972.
 Preprocessing

Yang, Wenkai, et al. "Integrate domain knowledge in training multi-task cascade deep learning model for benign–malignant thyroid nodule classification on ultrasound images." Engineering Applications of Artificial Intelligence 98 (2021): 104064.
 Preprocessing (Data Augmentation)

Remember that CNN is not rotation invariant. We need to teach that to the model.
 Data Augmentation, Bayesian, and Loss Function
What will be the loss function for
● Rotation, and Loss Function (MRI Image) rotation invariance of image?
● Data Distribution and Loss Function (Lasso-Laplace, Ridge-

β ~ N(0, K²I) β ~ Laplace(0, k)

Loss
 To make the model robust, insight
domain knowledge can be helpful Loss
as it exhibits the reasonable ranges
of a normal operation of the target
variable

Gupta, Himel Das, and Victor S. Sheng. "A Roadmap to Domain Knowledge Integration in Machine Learning." 2020 IEEE International Conference on Knowledge Graph (ICKG). IEEE, 2020.
 Loss subgradient

total variation loss

Javanmardi, Mehran, et al. "Unsupervised total variation loss for semi-supervised deep learning of semantic segmentation." arXiv preprint arXiv:1605.01368 (2016).
 Model and Loss

Liu, Ziwei, et al. "Deepfashion: Powering robust clothes recognition and retrieval with rich annotations." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
 Model and Loss

Hafner, Sebastian, Yifang Ban, and Andrea Nascetti. "Unsupervised domain adaptation for global urban extraction using Sentinel-1 SAR and Sentinel-2 MSI data." Remote Sensing of Environment 280 (2022): 113192.
 Inference (Class Activation Map)

Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
 Inference (GradCAM)

You can learn more from here.

Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.
 Inference (GRADCAM)

Higa, Maiki, et al. "Domain knowledge integration into deep learning for typhoon intensity classification." Scientific reports 11.1 (2021): 12972.
 Model and Inference (GRADCAM)

H. Wang, H. Jia, L. Lu and Y. Xia, "Thorax-Net: An Attention Regularized Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography," in IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 2, pp. 475-485, Feb. 2020
 Model and Inference (GRADCAM)

H. Wang, H. Jia, L. Lu and Y. Xia, "Thorax-Net: An Attention Regularized Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography," in IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 2, pp. 475-485, Feb. 2020
Thank you. Questions?