A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

Open in app Sign up Sign In

Search Medium

Published in Towards Data Science

This is your last free member-only story this month.

Sign up for Medium and get an extra one

Hucker Marius Follow

Feb 25, 2020 · 9 min read · · Listen

Save

A table detection, cell recognition and text

extraction algorithm to convert tables in
images to excel files
How to turn screenshots of a table to editable data using OpenCV
and pytesseract

1 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

source: pixabay

Let’s say you have a table in an article, pdf or image and want to transfer it into an
excel sheet or dataframe to have the possibility to edit it. Especially in the field of
preprocessing for Machine Learning this algorithm will be exceptionally helpful
to convert many images and tables to editable data.
In the case that your data exists of text-based PDFs there is already a handful of
free solutions. The most popular ones are tabular, camelot/excalibur, which you
can find under https://tabula.technology/, https://camelot-py.readthedocs.io
/en/master/, https://excalibur-py.readthedocs.io/en/master/.

However, what if your PDF is image-based or if you find an article with a table
online? Why not just take a screenshot and convert it into an excel sheet? Since
there seems to be no free or open source software for image-based data (jpg, png,
image-based pdf etc.) the idea came up to develop a generic solution to convert
tables into editable excel-files.

2 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

But that’s enough for now, let’s see how it works.

Getting started
The algorithm consists of three parts: the first is the table detection and cell
recognition with Open CV, the second the thorough allocation of the cells to the
proper row and column and the third part is the extraction of each allocated cell
through Optical Character Recognition (OCR) with pytesseract.

As most table recognition algorithms, this one is based on the line structure of
the table. Clear and detectable lines are necessary for the proper identification of
cells. Tables with broken lines, gaps and holes lead to a worse identification and
the cells only partially surrounded by lines are not detected. In case some of your
documents have broken lines make sure to read this article and repair the lines:
Click here.

First, we need the input data, which is in my case a screenshot in png-format. The
goal is to have a dataframe and excel-file with the identical tabular structure,
where each cell can be edited and used for further analysis.

The input data for further table recognition and extraction.

Let’s import the necessary libraries.

For more information on the libraries:

cv2 — https://opencv.org/
pytesseract — https://pypi.org/project/pytesseract/

3 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

import cv2
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import csv

try:
from PIL import Image
except ImportError:
import Image
import pytesseract

The first step is to read in your file from the proper path, using thresholding to
convert the input image to a binary image and inverting it to get a black
background and white lines and fonts.

#read your file

file=r'/Users/YOURPATH/testcv.png'
img = cv2.imread(file,0)
img.shape

#thresholding the image to a binary image

thresh,img_bin = cv2.threshold(img,128,255,cv2.THRESH_BINARY
|cv2.THRESH_OTSU)

#inverting the image

img_bin = 255-img_bin
cv2.imwrite('/Users/YOURPATH/cv_inverted.png',img_bin)

#Plotting the image to see the output

plotting = plt.imshow(img_bin,cmap='gray')
plt.show()

The binary inverted image.

4 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

Stay tuned on new articles of Marius Hucker

Stay tuned on new articles of Marius Hucker By signing up, you
will create a Medium account if you don't already have…
medium.com

The next step is to define a kernel to detect rectangular boxes, and followingly the
tabular structure. First, we define the length of the kernel and following the
vertical and horizontal kernels to detect later on all vertical lines and all
horizontal lines.

# Length(width) of kernel as 100th of total width

kernel_len = np.array(img).shape[1]//100

# Defining a vertical kernel to detect all vertical lines of image

ver_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1,
kernel_len))

# Defining a horizontal kernel to detect all horizontal lines of

image
hor_kernel = cv2.getStructuringElement(cv2.MORPH_RECT,
(kernel_len, 1))

# A kernel of 2x2
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))

The next step is the detection of the vertical lines.

#Use vertical kernel to detect and save the vertical lines in a

jpg
image_1 = cv2.erode(img_bin, ver_kernel, iterations=3)
vertical_lines = cv2.dilate(image_1, ver_kernel, iterations=3)
cv2.imwrite("/Users/YOURPATH/vertical.jpg",vertical_lines)

#Plot the generated image

plotting = plt.imshow(image_1,cmap='gray')
plt.show()

5 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

The extracted vertical lines.

And now the same for all horizontal lines.

#Use horizontal kernel to detect and save the horizontal lines in

a jpg
image_2 = cv2.erode(img_bin, hor_kernel, iterations=3)
horizontal_lines = cv2.dilate(image_2, hor_kernel, iterations=3)
cv2.imwrite("/Users/YOURPATH/horizontal.jpg",horizontal_lines)

#Plot the generated image

plotting = plt.imshow(image_2,cmap='gray')
plt.show()

The extracted horizontal lines.

We combine the horizontal and vertical lines to a third image, by weighting both
with 0.5. The aim is to get a clear tabular structure to detect each cell.

# Combine horizontal and vertical lines in a new third image, with

both having same weight.
img_vh = cv2.addWeighted(vertical_lines, 0.5, horizontal_lines,

6 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

0.5, 0.0)

#Eroding and thesholding the image

img_vh = cv2.erode(~img_vh, kernel, iterations=2)
thresh, img_vh = cv2.threshold(img_vh,128,255, cv2.THRESH_BINARY |
cv2.THRESH_OTSU)
cv2.imwrite("/Users/YOURPATH/img_vh.jpg", img_vh)

bitxor = cv2.bitwise_xor(img,img_vh)
bitnot = cv2.bitwise_not(bitxor)

#Plotting the generated image

plotting = plt.imshow(bitnot,cmap='gray')
plt.show()

The extracted tabular structure without cotaining text.

After having the tabular structure we use the findContours function to detect the
contours. This helps us to retrieve the exact coordinates of each box.

# Detect contours for following box detection

contours, hierarchy = cv2.findContours(img_vh, cv2.RETR_TREE,
cv2.CHAIN_APPROX_SIMPLE)

The following function is necessary to get a sequence of the contours and to sort
them from top-to-bottom (https://www.pyimagesearch.com/2015/04/20/sorting-
contours-using-python-and-opencv/).

def sort_contours(cnts, method="left-to-right"):

# initialize the reverse flag and sort index

reverse = False
i = 0

7 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

# handle if we need to sort in reverse

if method == "right-to-left" or method == "bottom-to-top":
reverse = True

# handle if we are sorting against the y-coordinate rather

than
# the x-coordinate of the bounding box
if method == "top-to-bottom" or method == "bottom-to-top":
i = 1

# construct the list of bounding boxes and sort them from top
to
# bottom
boundingBoxes = [cv2.boundingRect(c) for c in cnts]
(cnts, boundingBoxes) = zip(*sorted(zip(cnts, boundingBoxes),
key=lambda b:b[1][i], reverse=reverse))

# return the list of sorted contours and bounding boxes

return (cnts, boundingBoxes)

# Sort all the contours by top to bottom.

contours, boundingBoxes = sort_contours(contours, method=”top-to-
bottom”)

How to retrieve the cells position

The further steps are necessary to define the right location, which means proper
column and row, of each cell. First, we need to retrieve the height for each cell
and store it in the list heights. Then we take the mean from the heights.

#Creating a list of heights for all detected boxes

heights = [boundingBoxes[i][3] for i in range(len(boundingBoxes))]

#Get mean of heights

mean = np.mean(heights)

Next we retrieve the position, width and height of each contour and store it in the
box list. Then we draw rectangles around all our boxes and plot the image. In my
case I only did it for boxes smaller then a width of 1000 px and a height of 500 px
to neglect rectangles which might be no cells, e.g. the table as a whole. These two
values depend on your image size, so in case your image is a lot smaller or bigger
you need to adjust both.

8 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

#Create list box to store all boxes in

box = []

# Get position (x,y), width and height for every contour and show
the contour on image
for c in contours:
x, y, w, h = cv2.boundingRect(c)

if (w<1000 and h<500):

image = cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)
box.append([x,y,w,h])

plotting = plt.imshow(image,cmap=’gray’)
plt.show()

Each cell surrounded by the detected contours/box.

Now as we have every cell, its location, height and width we need to get the right
location within the table. Therefore, we need to know in which row and which
column it is located. As long as a box does not differ more than its own (height +
mean/2) the box is in the same row. As soon as the height difference is higher
than the current (height + mean/2) , we know that a new row starts. Columns are
logically arranged from left to right.

#Creating two lists to define row and column in which cell is

located
row=[]
column=[]
j=0

#Sorting the boxes to their respective row and column

for i in range(len(box)):

if(i==0):

9 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

column.append(box[i])
previous=box[i]

else:
if(box[i][1]<=previous[1]+mean/2):
column.append(box[i])
previous=box[i]

if(i==len(box)-1):
row.append(column)

else:
row.append(column)
column=[]
previous = box[i]
column.append(box[i])

print(column)
print(row)

Next we calculate the maximum number of columns (meaning cells) to

understand how many columns our final dataframe/table will have.

#calculating maximum number of cells

countcol = 0
for i in range(len(row)):
countcol = len(row[i])
if countcol > countcol:
countcol = countcol

After having the maximum number of cells we store the midpoint of each column
in a list, create an array and sort the values.

#Retrieving the center of each column

center = [int(row[i][j][0]+row[i][j][2]/2) for j in

range(len(row[i])) if row[0]]

center=np.array(center)
center.sort()

At this point, we have all boxes and their values, but as you might see in the
output of your row list the values are not always sorted in the right order. That’s
what we do next regarding the distance to the columns center. The proper

10 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

sequence we store in the list finalboxes.

#Regarding the distance to the columns center, the boxes are

arranged in respective order

finalboxes = []

for i in range(len(row)):
lis=[]
for k in range(countcol):
lis.append([])
for j in range(len(row[i])):
diff = abs(center-(row[i][j][0]+row[i][j][2]/4))
minimum = min(diff)
indexing = list(diff).index(minimum)
lis[indexing].append(row[i][j])
finalboxes.append(lis)

Let’s extract the values

In the next step we make use of our list finalboxes. We take every image-based
box, prepare it for Optical Character Recognition by dilating and eroding it and
let pytesseract recognize the containing strings. The loop runs over every cell and
stores the value in the outer list.

#from every single image-based cell/box the strings are extracted

via pytesseract and stored in a list

outer=[]
for i in range(len(finalboxes)):
for j in range(len(finalboxes[i])):
inner=’’
if(len(finalboxes[i][j])==0):
outer.append(' ')

else:
for k in range(len(finalboxes[i][j])):
y,x,w,h = finalboxes[i][j][k][0],finalboxes[i]
[j][k][1], finalboxes[i][j][k][2],finalboxes[i][j][k][3]
finalimg = bitnot[x:x+h, y:y+w]
kernel = cv2.getStructuringElement(cv2.MORPH_RECT,
(2, 1))
border = cv2.copyMakeBorder(finalimg,2,2,2,2,
cv2.BORDER_CONSTANT,value=[255,255])
resizing = cv2.resize(border, None, fx=2, fy=2,
interpolation=cv2.INTER_CUBIC)
dilation = cv2.dilate(resizing,
kernel,iterations=1)

11 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

erosion = cv2.erode(dilation, kernel,iterations=1)

out = pytesseract.image_to_string(erosion)
if(len(out)==0):
out = pytesseract.image_to_string(erosion,
config='--psm 3')
inner = inner +" "+ out

outer.append(inner)

The last step is the conversion of the list to a dataframe and storing it into an
excel-file.

#Creating a dataframe of the generated OCR list

arr = np.array(outer)
dataframe = pd.DataFrame(arr.reshape(len(row),countcol))
print(dataframe)
data = dataframe.style.set_properties(align="left")

#Converting it in a excel-file
data.to_excel(“/Users/YOURPATH/output.xlsx”)

The final dataframe in the terminal.

12 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

The final excel file containing all cells.

That’s it! Your table should now be stored in a dataframe and in an excel-file and
can be used for Nature Language Processing, for further analysis via statistics or
just for editing it. This works for tables with a clear and simple structure. In case
your table has an extraordinary structure, in the sense that many cells are
combined, that the cells size varies strongly or that many colours are used, the
algorithm may has to be adopted. Furthermore OCR (pytesseract) is nearly
perfect in recognizing computer fonts. However, if you have tables containing
handwritten input, the results may vary.

If you use it for your own table(s), let me know how it worked.

Stay tuned on new articles of Marius Hucker

Stay tuned on new articles of Marius Hucker By signing up, you
will create a Medium account if you don't already have…
medium.com

You liked this story?

Support me and my work here

Also Read:

13 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

How to Fix Broken Lines in Table Recognition

14 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

1 import cv2
2 import numpy as np
3 import pandas as pd
4 import matplotlib.pyplot as plt
5 import csv
6
7 try:
8 from PIL import Image
9 except ImportError:
10 import Image
11 import pytesseract
12
13 #read your file
14 file=r'/Users/marius/Desktop/Masterarbeit/Medium/Medium.png'
15 img = cv2.imread(file,0)
16 img.shape
17
18 #thresholding the image to a binary image
19 thresh,img_bin = cv2.threshold(img,128,255,cv2.THRESH_BINARY | cv2.THRESH_OTSU)
20
21 #inverting the image
22 img_bin = 255-img_bin
23 cv2.imwrite('/Users/marius/Desktop/cv_inverted.png',img_bin)
24 #Plotting the image to see the output
25 plotting = plt.imshow(img_bin,cmap='gray')
26 plt.show()
27
28 # countcol(width) of kernel as 100th of total width
29 kernel_len = np.array(img).shape[1]//100
30 # Defining a vertical kernel to detect all vertical lines of image
31 ver_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, kernel_len))
32 # Defining a horizontal kernel to detect all horizontal lines of image
33 hor_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_len, 1))
34 # A kernel of 2x2
35 kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))
36
37 #Use vertical kernel to detect and save the vertical lines in a jpg
38 image_1 = cv2.erode(img_bin, ver_kernel, iterations=3)
39 vertical_lines = cv2.dilate(image_1, ver_kernel, iterations=3)
40 cv2.imwrite("/Users/marius/Desktop/vertical.jpg",vertical_lines)
41 #Plot the generated image
42 plotting = plt.imshow(image_1,cmap='gray')
43 plt.show()
44
45 #Use horizontal kernel to detect and save the horizontal lines in a jpg
46 image_2 = cv2.erode(img_bin, hor_kernel, iterations=3)
47 horizontal_lines = cv2.dilate(image_2, hor_kernel, iterations=3)

15 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

horizontal_lines cv2 dilate(image_2, hor_kernel, iterations 3)

48 cv2.imwrite("/Users/marius/Desktop/horizontal.jpg",horizontal_lines)
49 #Plot the generated image
50 plotting = plt.imshow(image_2,cmap='gray')
51 plt.show()
52
53 # Combine horizontal and vertical lines in a new third image, with both having
same weight.
54 img_vh = cv2.addWeighted(vertical_lines, 0.5, horizontal_lines, 0.5, 0.0)
55 #Eroding and thesholding the image
56 img_vh = cv2.erode(~img_vh, kernel, iterations=2)
57 thresh, img_vh = cv2.threshold(img_vh,128,255, cv2.THRESH_BINARY |
cv2.THRESH_OTSU)
58 cv2.imwrite("/Users/marius/Desktop/img_vh.jpg", img_vh)
59 bitxor = cv2.bitwise_xor(img,img_vh)
60 bitnot = cv2.bitwise_not(bitxor)
61 #Plotting the generated image
62 plotting = plt.imshow(bitnot,cmap='gray')
63 plt.show()
64
65 # Detect contours for following box detection
66 contours, hierarchy = cv2.findContours(img_vh, cv2.RETR_TREE,
cv2.CHAIN_APPROX_SIMPLE)
67
68 def sort_contours(cnts, method="left-to-right"):
69 # initialize the reverse flag and sort index
70 reverse = False
71 i = 0
72 # handle if we need to sort in reverse
73 if method == "right-to-left" or method == "bottom-to-top":
74 reverse = True
75 # handle if we are sorting against the y-coordinate rather than
76 # the x-coordinate of the bounding box
77 if method == "top-to-bottom" or method == "bottom-to-top":
78 i = 1
79 # construct the list of bounding boxes and sort them from top to
80 # bottom
81 boundingBoxes = [cv2.boundingRect(c) for c in cnts]
82 (cnts, boundingBoxes) = zip(*sorted(zip(cnts, boundingBoxes),
83 key=lambda b:b[1][i], reverse=reverse))
84 # return the list of sorted contours and bounding boxes
85 return (cnts, boundingBoxes)
86
87 # Sort all the contours by top to bottom.
88 contours, boundingBoxes = sort_contours(contours, method="top-to-bottom")
89
90 #Creating a list of heights for all detected boxes
91 heights = [boundingBoxes[i][3] for i in range(len(boundingBoxes))]

16 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

91 heights [boundingBoxes[i][3] for i in range(len(boundingBoxes))]

92
93 #Get mean of heights
94 mean = np.mean(heights)
95
96 #Create list box to store all boxes in
97 box = []
98 # Get position (x,y), width and height for every contour and show the contour on
image
99 for c in contours:
100 x, y, w, h = cv2.boundingRect(c)
101 if (w<1000 and h<500):
102 image = cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)
103 box.append([x,y,w,h])
104
105 plotting = plt.imshow(image,cmap='gray')
106 plt.show()
107
108 #Creating two lists to define row and column in which cell is located
109 row=[]
110 column=[]
111 j=0
112
113 #Sorting the boxes to their respective row and column
114 for i in range(len(box)):
Opencv
115 Table Recognition Image Processing Ocr Text Recognition
116 if(i==0):
117 column.append(box[i])
118 previous=box[i]
119
120 else:
1K 32
121 if(box[i][1]<=previous[1]+mean/2):
122 column.append(box[i])

Sign
123 up for The previous=box[i]
Variable
By124
Towards Data Science
125 if(i==len(box)-1):
Every Thursday, the Variable delivers the very best of Towards Data Science: from hands-on tutorials and cutting-
126 row.append(column)
edge research to original features you don't want to miss. Take a look.
127

By128 else:a Medium account if you don’t already have one. Review
signing up, you will create
our Privacy Policy for more information
129 about our privacy practices.
row.append(column)
130 column=[]
131 Get this newsletter
previous = box[i]
132 column.append(box[i])
133
134 print(column)
135 print(row)
136
137 #calculating maximum number of cells

17 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

137 #calculating maximum number of cells

138 countcol = 0
139 for i in range(len(row)):
140 countcol = len(row[i])
141 if countcol > countcol:
142 countcol = countcol
143
144 #Retrieving the center of each column
145 center = [int(row[i][j][0]+row[i][j][2]/2) for j in range(len(row[i])) if row[0]]
146
147 center=np.array(center)
148 center.sort()
149 print(center)
150 #Regarding the distance to the columns center, the boxes are arranged in
respective order
151
152 finalboxes = []
153 for i in range(len(row)):
154 lis=[]
155 for k in range(countcol):
156 lis.append([])
157 for j in range(len(row[i])):
158 diff = abs(center-(row[i][j][0]+row[i][j][2]/4))
159 minimum = min(diff)
160 indexing = list(diff).index(minimum)
161 lis[indexing].append(row[i][j])
162 finalboxes.append(lis)
163
164
165 #from every single image-based cell/box the strings are extracted via pytesseract
and stored in a list
166 outer=[]
167 for i in range(len(finalboxes)):
168 for j in range(len(finalboxes[i])):
169 inner=''
170 if(len(finalboxes[i][j])==0):
171 outer.append(' ')
172 else:
173 for k in range(len(finalboxes[i][j])):
174 y,x,w,h = finalboxes[i][j][k][0],finalboxes[i][j][k][1],
finalboxes[i][j][k][2],finalboxes[i][j][k][3]
175 finalimg = bitnot[x:x+h, y:y+w]
176 kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 1))
177 border = cv2.copyMakeBorder(finalimg,2,2,2,2,
cv2.BORDER_CONSTANT,value=[255,255])
178 resizing = cv2.resize(border, None, fx=2, fy=2,
interpolation=cv2.INTER_CUBIC)
179 dilation = cv2 dilate(resizing kernel iterations=1)

18 of 19 23/12/22, 18:33
A table detection, cell recognition and text extraction ... https://towardsdatascience.com/a-table-detection-cell-...

179 dilation = cv2.dilate(resizing, kernel,iterations=1)

180 erosion = cv2.erode(dilation, kernel,iterations=2)
181
182 out = pytesseract.image_to_string(erosion)
183 if(len(out)==0):
184 out = pytesseract.image_to_string(erosion, config='--psm 3')
185 inner = inner +" "+ out
186 outer.append(inner)

About Help Terms Privacy

Get the Medium app

19 of 19 23/12/22, 18:33