Pose Coach 🤸‍♀️¶

TL;DR -The project uses deep learning and geometrical techniques to give instructions to perform poses/exercises safely.

Problem Statement:¶

Create an algorithm that can provide instructions to a user performing poses/exercises using geometry and detect body keypoints while doing so for form correction/detection. This application can be utilized in yoga/workout apps if it can provide instructions correctly.

Inspiration¶

Most exercises only work if they are done correctly, i.e in good form. This is why people insist on having personal trainers. However, during the pandemic most gyms were closed and exercises were done at home. This project is a very practical solution to analyse how a person exercises and is he/she doing it right.

Procedure¶

Literature Reviews¶

Two papers were found regarding the topic. A similar implementation with video analysis for body keypoints was done along with open pose model. Also, upon some research it was found that openpose model performs best with a high power GPU. To make the project easy and for it to work on CPUs, Open Pose mobile net thin architecture was decided to be used as it does not require further processing.

The papers can be found here.

Step 1: Detecting body keypoints¶

OpenPose is the first real-time multi-person system to jointly detect human body, hand, facial, and foot key-points (in total 135 key-points) on single images. It was proposed by researchers at Carnegie Mellon University.The original openpose.py from OpenCV example only uses Caffe Model which is more than 200MB while the Mobilenet is only 7MB. Thus, Mobilenet is used in the project. The keypoints detected are as follows:

"Nose"
"Neck"
"RShoulder"
"RElbow"
"RWrist"
"LShoulder"
"LElbow"
"LWrist"
"RHip"
"RKnee"
"RAnkle"
"LHip"
"LKnee"
"LAnkle"
"REye"
"LEye"
"REar"
"LEar"
"Background"

Step 2: Analysing the Pose¶

This was the most interesting step as all the mathematical code was to be written at this stage. At this stage, the keypoints are detected and further sent to be evaluated. On the basis of the pose, the angles, distances, similarities are calculated and whether or not the user is performing the pose correctly is decided.

After that, the mathematical elements are mapped to sentences corresponding to the instructions that can be given on them.

For instance, the warrior 2 pose requires the right leg to be bent at an angle of 90 degree. If it is less or more than that, an instruction regarding the same will be given.

Step 3 : Assembly and User Interface¶

At first both the above steps were done individually, later merging was done. Also, the Plotly Dash user interface which operates on Flask is used for creating the user interface. The "Sketchy" bootstrap theme by plotly is used overall.

In the interface the user can first select an exercise of their preference to be evaluated, after which a prompt to upload a picture of the user performing the exercise is displayed. As most computer devices have cameras installed, the process of clicking a picture is given to the user. Then , the uploaded image is analyzed and body keypoints are detected using open pose mobile net model and they are displayed.

These keypoints are further sent for geometrical evaluation and specific instructions are obtained.

The entire project (including the UI) is done in Python. Also, the project is more on the practical problem solving side and not on the Generic learning side.¶

Future work¶

Improving accuracy of the algorithm. The algorithm fails when the background blends into the picture.
Algorithm used here is openpose mobilenet thin architecture, however larger more precise models can be used.
With high power GPUs, the keypoint detection can be done in real-time and instructions can be provided on the go.
A classification network can be developed to detect the exercise being performed and then the user can be segwayed into the analysis phase.