Using CV and a projector to help me play billiards
By Shrey Birmiwal and Avi Agrawal
Me and my roommates at UT Austin lucked out and got assigned an apartment with a pool table instead of a dining table. Sounds great but I SUCK at the game while my roommates somehow get super lucky and always beat me.
I recently saw someone on YouTube build a system using a camera and a projector to make a real life pool more like a game pigeon, so we decided to build something similar. Also, I wanted to test how autonomous a model like GPT Sol could be given a camera and projector to work in a loop /goal and build the entire project.
First we taped Shrey’s phone to the roof directly above the pool table. The idea was to use the iPhone continuous camera feature to stream to the MacBook. For some reason, GPT struggled a lot when trying to get the stream to work, but I digress. We got this running a model to detect the pool balls and the stick.
Next we taped the projector at an angle to the ceiling. We tried lots of angles and different positions (like putting on a shelf, putting on 3 stacked chairs) but this was really the only way it would project cleanly onto the board.
Then we made the projection normalized. The projector was set to output a grid (which initially was completely off because of the angle and trapezoid features of the projector), but after tuning, we created a virtual mapping from the projector output to the flat pool table.
Finally added the software to predict the next ball and the bounce. In the future we can add stuff like score keeping, game modes (floor is lava), best defense spots, etc. Pretty cool project, even though It's definitely not a novel project. Even cooler to see that GPT basically did all the code even though it needed some coaxing and real world help (aligning, calibration etc).
Check it out: