Robotic Pen Tracker
Overview
For this project, I used an Intel RealSense camera to detect the location of a pen and then had a PincherX100 robotic arm retrieve it. I used the RealSense and OpenCv Python packages for the computer vision algorithm, and the InterbotixManipulatorXS package to control the Pincher robotic arm joint angles.
Computer Vision algorithm
To detect location of the purple pen I first created an HSV mask to filter out all other colors within a threshold. I then found contours around the masked image of the pen, and added them to the original color image. Finally, I calculated the centroid of the pen by using the area of the largest countour currently in frame. To better detect the pen, I added a depth filter to my algorithm so that anything purple detected in the background would not be used for centroid calculation. The centroid was then used as the goal pose for the Pincher robotic arm's end effector.