Big Data and Machine Learning in

Healthcare Applications

Oleg S. Pianykh, PhD

Medical Analytics Group
Department of Radiology, Massachusetts General Hospital
Harvard Medical School

DICOM

Oleg Pianykh [email protected]

Oleg Pianykh [email protected] 1

 November 8: Happy Radiography Day!

Oleg Pianykh [email protected]

Digital Hospital
Digital
Hospital

Integration
PACS
HIS (Picture Archiving and
(Hospital Information System ) Communication System)
Text Images
EVN|A08|20060108
0823||||||PID|1||
3175875|1127278|S
EVN|A08|20060108
AMPLE^JOE^^^^||1
0823||||||PID|1||
9901334|M|||5400
3175875|1127278|S
Lake Villa
AMPLE^JOE^^^^||1
Dr^^Metairie^LA^70
9901334|M|||5400
001-1230||(
Lake Villa
Dr^^Metairie^LA^70
001-1230||(

HL7 DICOM
(Health Level 7) (Digital Imaging and
Communications in Medicine)
Oleg Pianykh [email protected]

Oleg Pianykh [email protected] 2

 HL7 ADT (Admission, Discharge,
Transfer) message Pipe

• MSH|^~\&|ADT|N|ADT|MEDSC|200601081527||ADT^A08|RE|P|3.
2||||||ASCII|
• EVN|A08|200601080823||||||PID|1||3175875|1127278|SAMPLE^
JOE^^^^||19901334|M|||5400 Lake Villa Dr^^Metairie^LA^70001-
1230||(405)555-2920|||SINGLE||||||||||||||N|
• MSH|^~\&|ADT|N|ADT|MEDSC|200601081527||ADT^A08|RE|P|3.
2||||||ASCII|EVN|A08|200601080832||||||PID|1||3057088|1051
999|INCOGNITO^MONICA^ANN^^^||19780117|F|||PO Box
1324^^Jefferson^LA^83625-3184||(555)423-1423|||OTHER|||512-
11-1425|||||||||||N|
See http://www.hosinc.com/products/interfaces/interface_documentation.htm
Oleg Pianykh [email protected]

DICOM Read this! Why all these

gibberish characters?

Oleg Pianykh [email protected]

Oleg Pianykh [email protected] 3

 Medical imaging
• Which elements would you need to build a medical imaging
workflow?

• … ???
???
• … ???
???
• … ???
???

Oleg Pianykh [email protected]

PACS Modalities Workstations

• Picture Archiving and

Archive
Communication System
• Connects modalities (image
acquisition devices) to digital
image archive and viewing
workstations
• Uses DICOM as principal
communication/storage
standard Acquire Store View

• Different vendors (GE, Siemens,

Visage, Agfa, Philips…)
Oleg Pianykh [email protected]

Oleg Pianykh [email protected] 4

 Modalities – medical image acquisition

MR CT

Virtually any imaging

device can become a
modality

US VL
Oleg Pianykh [email protected]

Modalities and DICOM

Modality

Study DICOM
data network

DICOM image
Vendor device and formats

Proprietary image
format

DICOM interface !
Oleg Pianykh [email protected]

10

Oleg Pianykh [email protected] 5

 DICOM in medical imaging

Pre-DICOM: Hardcopies
and film
Post-DICOM: Digital data we can
fully explore
Oleg Pianykh [email protected]

11

DICOM History
ACR-NEMA 1.0, 1985
• Digital Imaging and COmmunications in
Medicine
• Proposed in 1985 by American College of
Radiology (ACR) and National Electrical
Manufacturers Association (NEMA) to make
digital medicine manufacturer-independent.
Reviewed in 1988 (ACR-NEMA 1.0, 2.0).
• First time shown at 1992, RSNA.
• Major revision in 1993 (DICOM 3.0). 18
volumes, hundreds of supplements,
annually updated by DICOM workgroups.

Oleg Pianykh [email protected]

12

Oleg Pianykh [email protected] 6

 Evolution of DICOM
• DICOM remains version 3.0 since its inception in 1992.
• Annually updated by DICOM workgroups (around 30); therefore we
have DICOM 3.0 2005, DICOM 3.0 2019, …
• Old functionality is gradually retired.
• New functionality is constantly added. However, the industry is
several years behind the standard, and the hospitals are behind the
industry.
• “Compatible” is always more important than “new.”

Oleg Pianykh [email protected]

13

Secrets of DICOM longevity

• Vendor-independent

• Abstract language: views medical workflow in “object-oriented” sense, as

objects (with attributes) interacting with each other

• Borrows useful things from the other standards (e.g., image compression)

• Makes diagnostic quality a priority

See DICOM updates at David Clunie’s blog: www.dclunie.com/dicom-status/status.html

Oleg Pianykh [email protected]

14

Oleg Pianykh [email protected] 7

 How does DICOM work?

DICOM views any real entity as a collection of its

clinically-relevant attributes
Oleg Pianykh [email protected]

15

DICOM Dictionary
(0008,0020) Study Date
(0008,0021) Series Date
(0008,0022) Acquisition Date (0010,0010) Patient’s Name
(0008,0030) Study Time (0010,0020) Patient ID
(0008,0031) Series Time (0010,0030) Patient's Birth Date
(0008,0032) Acquisition Time (0010,0032) Patient's Birth Time
(0008,0060) Modality (0010,0040) Patient's Sex
(0008,0061) Modalities (0010,0050) Patient's Insurance Plan Code Sequence
(0008,0082) Institution Code Sequence (0010,0101) Patient’s Primary Language Code Sequence
(0008,0090) Referring Physician’s Name (0010,1000) Other Patient IDs
(0008,0092) Referring Physician's (0010,1001) Other Patient Names
(0008,0094) Referring Physician’s Telephone (0010,1005) Patient's Birth Name
(0008,0116) Responsible Organization (0010,1010) Patient's Age
(0008,1010) Station Name (0010,1020) Patient's Size
(0008,1030) Study Description (0010,1030) Patient's Weight
(0010,1040)
DICOM dictionary– a list of standard medical Patient's
data attributes (dataAddress
elements), ordered by (Group,
Element) numbers: (0010,1080) Military Rank
Example: (0008,0020) Study Date (0010,1081) Branch of Service
(0010,2000) Medical Alerts

Oleg Pianykh [email protected]

Around 3000 attributes

16

Oleg Pianykh [email protected] 8

 Data representation with
DICOM dictionary

dictionary
DICOM
(Group, Element) Description
(0010,0010) Patient name
(0010,0030) Patient birth date
Q: Compare to HL7
(0010,1030) Patient weight
… ...
Q: Why do we need Length?
Patient:
Smith, John
DOB: Nov. 08,1968
Weight: 75.50 kg
... (Group, element) Length Value

0010 0010 10 Smith^Joe_

0010 0030 8 19681108
0010 1030 6 75.500
Oleg Pianykh DICOM Data object
[email protected] ...

17

“DICOM Data Object” – very helpful for ML projects

Patient Physician

Hospital
Modality Study data
Oleg Pianykh [email protected]

18

Oleg Pianykh [email protected] 9

 Example: Temporal analysis

AIF Contrast change from

baseline, HU

Using single DICOM

Acquisition Time (0008,0032) tag,
HHMMSS.FFFFFF
Time ti=i sec

Oleg Pianykh [email protected]

19

Example: 3D measurements, rendering

z

Ip
vc

Ds
y

DICOM includes:
-Slice 3D position vr
-Slice orientation
-Slice thickness x
-Distance between Dp
slices
-Time stamp
-Pixel size
…
Oleg Pianykh [email protected]

20

Oleg Pianykh [email protected] 10

 Example: Image depth DICOM
(same image, different
window/level settings)

JPEG

21

21

Example: Image depth

Importing DICOM into non-clinical
DICOM in Photoshop
software implies reducing image quality
(to some color window/level preset)

Oleg Pianykh [email protected]

Q.: What are Window Level and Width?

22

Oleg Pianykh [email protected] 11

 Hands on data
DICOM in Python: PyDICOM

- A pretty good intro:

https://medium.com/@hengloose/a-comprehensive-starter-guide-to-
visualizing-and-analyzing-dicom-images-in-python-7a8430fcb7ed
- Can load individual images, and image series (such as 3D)
- conda install -c conda-forge pydicom

Oleg Pianykh [email protected]

23

Hands on data
Keep it simple:

DICOM image is one

of many elements Deep image
inside of DICOM greyscale requires
object choosing palette for
display

Oleg Pianykh [email protected]

24

Oleg Pianykh [email protected] 12

 Taking full advantage of DICOM attributes

Oleg Pianykh [email protected]

25

What is DICOM file ?

• DICOM file – DICOM Data Object, saved on disk
• Must contain DICM in bytes 129-132:

You need to be able

to recognize DICOM
for your ML/AI
projects

Oleg Pianykh [email protected]

26

Oleg Pianykh [email protected] 13

 Summary: Why DICOM?

• Universal standard and interface of digital medicine

• Excellent image quality

• Full support for numerous image-acquisition parameters and different

data types

• Complete encoding of all image-related data

• Clarity in describing medical devices and their functionality

DICOM is the backbone of many ML/AI
Oleg Pianykh [email protected] projects

27

Getting DICOM in a hospital

• Never assume DICOM.
• Each DICOM device/application should come with a DICOM
Conformance statement, describing DICOM support on this particular
device, and DICOM settings present in its interface.
• If device/app has no DICOM, then everything depends on its image
format:
• If DICOM-compatible, the device/app can be complemented with DICOM
software.
• Otherwise, we need an additional converter (DICOM box).
• Figuring our DICOM availability is the very first step of any PACS-related
project.
Oleg Pianykh [email protected]

28

Oleg Pianykh [email protected] 14

 DICOM Data Types: VRs

• DICOM uses 27 data types, known as VRs (Value Representations)

• Each VR comes with its own format rules
• Each DICOM attribute has its own VR (i.e., format)
• VRs have 2-letter names (PN, UI, …)
• VR names can be used in DICOM files (“explicit” encoding)

Oleg Pianykh [email protected]

29

DICOM Data Dictionary

(Group, Element) Attribute (data Keyword VR VM Retired
tag element) name status
(0008,0001) Length to End LengthToEnd RET
(0008,0005) Specific Character Set SpecificCharacterSet CS 1-n

…
(0010,0010) Patient Name PatientName PN 1
(0010,0020) Patient ID PatientID LO 1
(0010,0021) Issuer of Patient ID IssuerOfPatientID LO 1
(0010,0030) Patient's Birth Date PatientBirthDate DA 1
(0010,0032) Patient's Birth Time PatientBirthTime TM 1
(0010,0040) Patient's Sex PatientSex CS 1
…
(0010,1000) Other Patient IDs OtherPatientIDs LO 1-n
(0010,1001) Other Patient Names OtherPatientNames PN 1-n

…
(FFFE,E00D) Item Delimitation Item ItemDelimitationItem 1

(FFFE,E0DD) Sequence Delimitation SequenceDelimitationItem 1

Item
Oleg Pianykh [email protected]

30

Oleg Pianykh [email protected] 15

 Example: VRs in Adobe Photoshop
VRs in DICOM element list

Oleg Pianykh [email protected]

31

Study: using DICOM with ML/AI projects

Oleg Pianykh [email protected]

32

Oleg Pianykh [email protected] 16

 Great – but how DICOM helps me with ML/AI?
DICOM provides an excellent data source for ML/AI projects:
• Contains high-quality image data
• Provides radiomics features for machine learning
• Contains important non-imaging info (patient age, sex, …)
• Can record AI output as well (such as image heat maps, identified by
AI apps), making them instantly available at any image viewing
workstation
• Integrates well into hospital data and other standards (such as HL7)

Oleg Pianykh [email protected]

33

DICOM and Machine learning framework

• Label the area of interest (for
supervised learning)
• Provide ground truth (often as
true/false classification)
• Convert DICOM data to input pixel
arrays for the model
• Train the model
• Evaluate model quality on unseen
test data

Oleg Pianykh [email protected]

34

Oleg Pianykh [email protected] 17

 Labeling
• 3D Slicer (https://www.slicer.org/) is the
Labeling in 3D Slicer
most popular tool:
• Free
• Most research-oriented
• Includes various plugins

• Integration with PyRadiomics, a

Python library to compute various
measurements of selected areas:
https://pyradiomics.readthedocs.io/en/latest/#

Oleg Pianykh [email protected] Source: https://supervisely.com/blog/how-to-use-3d-slicer-integrated-with-supervisely/

35

From DICOM to Radiomics: Image feature

engineering
Size
Shape

Texture

Entropy

Brightness
Density

Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6837295/
Oleg Pianykh [email protected]

36

Oleg Pianykh [email protected] 18

 Processing with Python
Using PyRadiomics, nii, and nrrd with Python

• 3D Slicer: Saving labeled data in nrrd

files: "nearly raw raster data“, file
format for the representation and
processing of n-dimensional raster
data (nii is another popular format)
• One file per volume, and one file per
selection mask (convenient)
• Loading these files into Python for
processing
• You can implement your own feature
measurements there
Returns a wide
range of radiomics
measurements
Oleg Pianykh [email protected]

37

Models
• Models can be built:
• From the features extracted from the images (such as radiomics)
• From the image pixels directly (such as convolutional neural networks)

• Adding non-imaging features (such as patient age etc) usually

improves model quality. Some of these features are readily available
in DICOM tags.

Oleg Pianykh [email protected]

38

Oleg Pianykh [email protected] 19

 Pitfalls
• GPU hardware limitations: often only 8bpp
unless supported and configured for more
(10 or 12 bpp)
• Unsupervised learning errors (erased
markers and the like)
• Synthetic images with unreal anatomy
• AI sensitivity to image parameters (slice
thickness, contrast, denoising)
• Not easy to integrate with non-DICOM data
(mismatching patient IDs from different
hospitals, pathology data, etc.)
https://lurkertech.com/medai/
Oleg Pianykh [email protected]

39

Oleg Pianykh [email protected] 20