Explaining main algorithm
For a while I’ve been thinking of writing a scientific article. I wanted it to have certain utility.
Morse code is binary: it takes only two values – either dot (short) or hyphen (long). I figured out that short (s) can stand for two-eye blinking whilst long (l) can indicate left-eye blinking. Another question emerged: how to understand when does one-symbol recording stop?
Empty space between two symbols can be presented by right-eye blinking – r. If I input singly symbol of short (dot) and long (hyphen), I will blink my right eye once to indicate the space between two symbols.
To separate independent words, one has to blink her right eye twice and get rr.
Hence, I have collected an ordered set of symbols – r, l, s, - that can be converted into a full-fledged text. Once I accomplish the transformation, I get an answer.
Deciphering Python Code
Let’s take a closer look to the code functions I used.
eye_aspect_ratio(eye). I use dlib library for face detection. Next, out of 68 facial parameters (these are dots that are spread across human face in a sharpened shape of face), I pick 6 that are responsible for eyes location. In Function I determine whether an eye is opened or closed by counting two Euclidean distances between the upper and lower eyelids of the eye, equally offset from the center (parameters A and B), and Euclidean distance between the right and left corners of the eye (parameter C). The bigger the number, the more open the eye is.
def eye_aspect_ratio(eye):
A = dist.euclidean(eye[1], eye[5])
B = dist.euclidean(eye[2], eye[4])
C = dist.euclidean(eye[0], eye[3])
ear = (A + B) / (2.0 * C)
return earbuild_plots(name, value, plot) is responsible for output of an image from camera to computer screen.
def build_plots(name, value, plot):
plot = plot.update(value)
cv2.imshow(name, plot)draw_outline(frame, eye) takes camera frame and eyes coordinates. It then fixates eyes with neon-green rings.
def draw_outline(frame, eye):
eyeHull = cv2.convexHull(eye)
cv2.drawContours(frame, [eyeHull], -1, (0, 255, 0), 1)get_args() gets me prerequisite arguments (shape-predictor) for future execution.
def get_args():
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--shape-predictor", required=True, help="path to facial landmark predictor")
args = vars(ap.parse_args())
return argsopen_video() opens my front camera and returns prerequisite arguments.
def open_video():
args = get_args()
print("[INFO] loading facial landmark predictor")
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(args["shape_predictor"])
print("[INFO] starting video stream thread...")
vs = VideoStream(0).start()
time.sleep(1)
return vs, detector, predictorcalibration(name, flag). Every person’s average ear (eye aspect ratio) is different, I can’t use any universal number for this parameter as former simply doesn’t exist. Thus, the calibration comes at handy. It helps to get numbers for several conditions: both eyes opened, both eyes closed, left eye closed & right eye opened and vice versa. After the first beep video opens and I have to hold my eyes opened till next beep. Afterwards, I close my eyes patiently waiting for last beep. Out of all sampling for eyes opened I return a minimum average value ear for left and right eyes. Out of all sampling for eyes closed I return a maximum average value ear for left and right eyes. Function thus returns optimal values for correct execution.
def calibration(name, flag):
plotLeft = LivePlot(640, 360, [5, 35], invert=True)
plotRight = LivePlot(640, 360, [5, 35], invert=True)
(lStart, lEnd) = face_utils.FACIAL_LANDMARKS_IDXS["left_eye"]
(rStart, rEnd) = face_utils.FACIAL_LANDMARKS_IDXS["right_eye"]
vs, detector, predictor = open_video()
print('\a')
time.sleep(3)
time_start = time.time()
both_eyes_open = []
both_eyes_close = []
while True:
frame = vs.read()
frame = imutils.resize(frame, width=450)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
rects = detector(gray, 0)
for rect in rects:
shape = predictor(gray, rect)
shape = face_utils.shape_to_np(shape)
leftEye = shape[lStart:lEnd]
rightEye = shape[rStart:rEnd]
leftEAR = int(100 * eye_aspect_ratio(leftEye))
rightEAR = int(100 * eye_aspect_ratio(rightEye))
if flag == 0:
earAvg = (leftEAR + rightEAR) / 2.0
both_eyes_open.append(earAvg)
elif flag == 1:
earAvg = (leftEAR + rightEAR) / 2.0
both_eyes_close.append(earAvg)
draw_outline(frame, leftEye)
draw_outline(frame, rightEye)
build_plots("ImagePlotLeft", leftEAR, plotLeft)
build_plots("ImagePlotRight", rightEAR, plotRight)
cv2.imshow(name, frame)
time_dif = time.time() - time_start
if cv2.waitKey(25) == ord("q"):
break
if time_dif > 5:
print('\a')
break
cv2.destroyAllWindows()
vs.stop()
if flag == 0: return min(both_eyes_open)
if flag == 1: return max(both_eyes_close)morse_code_from_eyes(). This is the major function of my code. In real time it monitors and analyzes human eyes. Calibration goes first and is followed by a beep sound, after which the recording starts. Recording goes the similar way as during the calibration, however, now I am comparing the results I get from camera with the ones I got from calibration one step ago. I use counter to trace one symbol per once. If I haven’t utilized counter, I would have stored a large number of symbols per one blink as there wouldn’t be any break. After all symbols are passed, I press the “q” button on a keyboard to finish recording and close front camera. Then function returns the result of symbols recording.
def morse_code_from_eyes():
both_open = calibration("both_eyes_open", 0)
both_close = calibration("both_eyes_close", 1)
plotLeft = LivePlot(640, 360, [5, 35], invert=True)
plotRight = LivePlot(640, 360, [5, 35], invert=True)
(lStart, lEnd) = face_utils.FACIAL_LANDMARKS_IDXS["left_eye"]
(rStart, rEnd) = face_utils.FACIAL_LANDMARKS_IDXS["right_eye"]
vs, detector, predictor = open_video()
print('\a')
time.sleep(3)
counter = 0
points = ""
while True:
frame = vs.read()
frame = imutils.resize(frame, width=450)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
rects = detector(gray, 0)
for rect in rects:
shape = predictor(gray, rect)
shape = face_utils.shape_to_np(shape)
leftEye = shape[lStart:lEnd]
rightEye = shape[rStart:rEnd]
leftEAR = int(100 * eye_aspect_ratio(leftEye))
rightEAR = int(100 * eye_aspect_ratio(rightEye))
earAvg = (leftEAR + rightEAR) / 2.0
draw_outline(frame, leftEye)
draw_outline(frame, rightEye)
build_plots("ImagePlotLeft", leftEAR, plotLeft)
build_plots("ImagePlotRight", rightEAR, plotRight)
if counter == 0:
if earAvg <= both_close + 1:
points += "s"
counter += 1
print(earAvg, "ssssssssssssssssssss")
elif leftEAR - rightEAR >= 0 and earAvg <= both_open - 3:
points += "p"
counter += 1
print(leftEAR - rightEAR, earAvg, "ppppppppppppppppppрр")
elif rightEAR - leftEAR >= 0 and earAvg <= both_open - 3:
points += "l"
counter += 1
print(rightEAR - leftEAR, earAvg, "llllllllllllllllllll")
else:
if counter == 5:
counter = 0
else:
counter += 1
cv2.imshow("Frame", frame)
if cv2.waitKey(25) == ord("q"):
break
cv2.destroyAllWindows()
vs.stop()
return pointstext_from_morse_code(points) is responsible for converting a received stroke of symbols (r,l,s) into a comprehensible text. Firstly, I save a dictionary which keys are a designation in Morse code, and the value is a letter. I split the stroke into “pp” to get separate independent words. Consequently, I go through all the symbols before “p” and convert each one into a letter. After, letters are combined into words and words into sentences. The ultimate result is a returned word.
def text_from_morse_code(points):
alphabet = {"sl": "A", "lsss": "B", "lsls": "C", "lss": "D", "s": "E",
"ssls": "F", "lls": "G", "ssss": "H", "ss": "I", "slll": "J",
"lsl": "K", "slss": "L", "ll": "M", "ls": "N", "lll": "O",
"slls": "P", "llsl": "Q", "sls": "R", "sss": "S", "l": "T",
"ssl": "U", "sssl": "V", "sll": "W", "lssl": "X", "lsll": "Y",
"llss": "Z",
"sllll": "1", "sslll": "2", "sssll": "3", "ssssl": "4",
"sssss": "5", "lssss": "6", "llsss": "7", "lllss": "8",
"lllls": "9", "lllll": "0"}
points = points.split("pp")
answer = ""
for word in points:
letters = word.split("p")
new_word = ""
for letter in letters:
if letter in alphabet:
new_word += alphabet[letter]
else:
new_word += "-"
answer += new_word + " "
return answerConclusion
First you need to run the morse_code_from_eyes() function, and save the result to a variable. After that, pass the resulting string to the text_from_morse_code() function and get the final result.