GCQuad Screen Reader

[andygg1986]

On another post, Ports mentioned the idea of pointing a camera at the screen and using image recognition to read the ball parameters. If you can do that, I don't see why you couldn't fake a COM port to send them to, format the data to look like it comes from a GC2's bluetooth, and then use GSX to play TGC. To be fair, my programming skills are geared towards hacking something together for engineering purposes and not customer-facing products, but I thought I would at least mess around with this for fun. There are already open source image-to-text programs and virtual COM port creators, so I think it is more a matter of piecing stuff together and maybe tweaking a few things here and there.

To that end, can someone with a GCQuad take a video of the screen when they hit a shot, both with and without the backlight on and post it to youtube or some other hosting service? I don't actually know what order the data is displayed (ball in hitting zone, ball data, then club data?) or where on the screen each parameter is located. It is surprisingly hard to find even a still image of the ball data, and the ones that I found are not very good quality due to reflections, small size, and being taken at an angle. With a sample video, I can take screenshots of it and quickly see how difficult extracting the text would be.

[Ports]

Could the same be done thru the mobile app display on an iPad? You may get more information thru something of that nature shown on one screen.

[andygg1986]

Yes I think either one could work. Someone sent me some pictures of the screen on the quad itself and it has all the ball data that you need for GSX. I will see about passing it through some image analysis software and see how it works. If GSX ever passes HMT data though then the iPad app would probably be a better route so you aren’t waiting for the screen to cycle.

[StuartG]

GSX basically converts GC2 interface to something the Protee interface can understand. Are either of those interfaces documented or published anywhere so you could replicate them after getting the numbers? That would be the first concern.

[andygg1986]

You can just use a program like Putty to read what is on the COM port that your GC2 is connected to. You can't have Putty and GSX connected at the same time, but if you look at the serial data in Putty, it is very obvious how the data is presented to GSX. I believe the data is presented at 9600 baud, 8 data bits, 1 stop bit, no parity, and Xon/Xoff flow control, which is standard, but it has been a few months since I have looked at it. It is easy to confirm you are reading it correctly because you can see the data on the GC2 screen at the same time for comparison. I don't know how GSX passes that on to Protee, but it doesn't matter in this case. All you need to do is send it to the COM port in the same format that GC2 does and GSX takes care of the rest.

As a first pass, I tried to read the screen with Tesseract OCR, which is an open source image to text converter. I ran it on a photo of the screen. Unfortunately, it wasn't able to pull all the text in. I'm going to try using OpenCV to do some pre-processing of the image to convert it to 2 bit black and white before trying to read the screen. It will probably involve de-noising and some other processing as well to get it reliable. Then the pre-processed image could be run through Tesseract. The alternative that would be more involved would be to make templates of each letter and number on the screen and do template matching on the image. So basically it just looks for a pattern of pixels and if it is close enough, it says "this must be a number 2 or a letter B". That works well if you have constant lighting and the screen size is is the same in all images, but might not work as well across a variety of setups. Either way, I think it is feasible but it will likely take a little while to get something working since I'm just messing around in the evenings.

[Riedy09]

This idea has great potential. Any progress updates? Thanks.

[andygg1986]

I have been looking at different image preprocessing techniques. The issue is that it needs to be able to read in a variety of lighting conditions so you can't just set a threshold and say everything darker is black and everything lighter is white. I haven't worked a lot on the coding recently though because I got sidetracked designing a TGC2019 course with the tool that ChadGolf developed haha.

[ChadGolf]

Google for ProTee SDK, it’s public and they would grant a license for this. It’s just JSON over TCP.

I would also recommend writing a Python program that is set up to OCR the FSX Window on the computer. The limitation is that it probably can’t have FSX minimized, but eliminates camera hardware differences and inconsistent camera positions and lighting conditions.

If developed correctly, it could work for any launch monitor as well if there’s an easy way to configure where each measurement is in the window.

[andygg1986]

Thanks for the advice!

[WISCOGOLFNUT]

andygg1986 this looks like a great project, thank you for taking time to try and find a solution. I have a GCQuad with FSX2018 so please let me know if I can assist in any way. I’m not the most tech savvy but would be happy to help in some way. Let me know if you still need videos of the screen or if you need anything else. I am on the brink of buying a skytrak as a second monitor just so I can play online TGC but would be happy to contribute to you or others that can find a solution with the Quad.

[Ports]

Now if we could only toggle between two gc2s or skytraks to solve the left right issue I'd be a happy guy too. I wish I had a quad, hope this pans out for you guys. TGC needs more members, adding you quad guys would be great.

[mmlincon]

This is an interesting idea. I see one big problem and that is going to be the delay between the shot and shot appearing in the game. People complain about this all the time. It might be easier to scrape the shot data out of the api data that is being sent to the performance site though. It's not encrypted and all the shot data is there. FSX would need to be running for this though. Bottom line is this is going to me a little messy but if you want it bad enough it would be doable.

[WISCOGOLFNUT]

Thanks for another suggestion @mmlincon.. hoping all you tech guys can crack this mystery.. I would live with 3-4 sec of shot delay if necessary as that’s what my alternative (skytrak) is anyways, but if there is a way for no delay.. even better

[gibbiesmalls]

Unlike the other thread - I bet you Foresight is paying very close attention to this thread. lol

[WISCOGOLFNUT]

Sad but very true!

[andygg1986]

You can get the ball data to show up in FSX right away. Unfortunately as a first pass, the resolution of the text isn’t real high and Tesseract text recognition software can’t read it. It may possible with some sort of template matching where each digit has a template

[Wundej]

Unlike the other thread - I bet you Foresight is paying very close attention to this thread. lol

I’m sure they are. I hope they understand how absolutely idiotic their business strategy is around the Quad/FSX is. They just don’t understand how they’d sell so many more units of the Quad if they opened up communication with the outside world of applications.

Keep up the good work guys.

[WISCOGOLFNUT]

andygg1986 are you throwing in the towel on this since the text recognition isn’t successful or are you thinking of trying to scrape the data as mmlincon suggested or any other method? this is driving me nuts and wish I had the tech background to take this on.. I’m willing to contribute a few hundred bucks to you or anyone that can find a solution to show my gratitude.

[andygg1986]

No I'm still working on it. I am using software called OpenCV to do template matching. Basically I take a screenshot and crop out each digit and save that as a template. The software then tries to find any sections of the image that match that template to "read" a digit. Based on its location I know which field it belongs to (speed, spin, etc). I pretty much have that part working, just need to add more templates to account for variations in background color. I still need to figure out a method for knowing when to grab the screen image though. I'm thinking just taking a capture every second or two and analyzing it. Then I need to throw out any repeats or non valid captures and only send on the good ones. I also need to work on sending the data to the Protee interface, and figure out how to compile it into an executable program. I'm not a programmer so it takes me longer than other people to get stuff done.