Mr Lim

OCR

So I was tasked to develop a C#-based stand-alone testing tool using camera to capture the GUI screen (device-under-test) for image verification and OCR (Optical Character Recognition) comparisons.

I do not have any prior experience in image processing applications so I spent a lot of time researching and reading up on the various libraries and techniques available.

I kicked things off by playing around with Microsoft Office Document Imaging (MODI) which has a OCR component by itself. I also looked into the Universal Windows Platform (UWP) OCR API, but realised that it only supports Windows 10.

After reviewing all the available OCR options, I decided to go with Tesseract, which is no doubt the most mature and robust solution in comparison.

On the hardware aspect, the actual shooting and capturing of the screens, my supervisor had already placed an order for the camera to be used during the testing process, the Canon PowerShot SX720HS.

PowerShot SX720HS

Such a beautiful piece of hardware but there’s only one problem: There is no support for tethered shooting.

Believe me, I’ve tried searching. Very hard. The following list comprises of all leading tethering solutions which did not work for my camera model (but might be of use if you stumble upon this list for yours):

I even emailed PSRemote’s company enquiring about a custom solution and I was willing to pay, no luck;

The only workable tethered shooting option available was Canon’s Camera Connect app. So remote shooting is possible, but only for your iPhone or Android. Considering that I’ll have to run the OCR processing in succession to the capture, NO GO.

My next idea was to run the Camera Connect app via an Android emulator on the PC. Before I got to thinking I’m such a hot shit coming up with such a briliiant idea, I discovered that the method was not workable due to the emulators’ (ALL of them: Android Studio, Bluestacks, Genymotion, Nox) inability to detect the actual WiFi network. NO GO.

What if I look up the PowerShot SDK to see if I’m able to come up with something?

Oops. I downloaded the 9-year old last release SDK anyway but as expected, NO GO.

Imagine my joy and happiness when I discovered CHDK (Canon Hack Development Kit), a software application developed by enthusiasts that enables extra features for ported Canon™ “Point&Shoot” cameras.

The firmware dump for the SX720HS was available but unfortunately, the software has yet to be ported to support my model. Reading up on the porting procedure, I’m clearly ill-equipped in this area and it was frankly just too complex for a project of this size. NO GO.

So I clearly have a huge problem here, the primary intended purpose of capturing the screen, simply cannot be achieved with this model. At this point, I had started looking at alternatives which are within the project budget.

My research led me to the Yi HD Action Camera, which was the cheapest (SGD 119) camera available with generally positive reviews and with support for WiFi tethered shooting, something my supervisor brought up several times which he felt was important to have.

I quickly set up the camera and was able to get the WiFi tethering running. I’m relatively pleased with this option but there were 2 minor issues;

  • Being an Action Camera, there was an in-built fisheye effect. Fisheye correction lens are actually available but the lens themselves are actually more expensive than the camera itself.
  • Barely noticeable delay of screen capture via WiFi, but could be an issue? Testing conducted with a digital clock.

Source code is available here.


Links and resources referred to throughout the entire course of the project (in no particular order).

  1. http://techqa.info/programming/question/20599768/tesseract-ocr:-recognize-complete-dictionary-words-only
  2. http://www.aforgenet.com/articles/shape_checker/
  3. http://www.danvk.org/2015/01/07/finding-blocks-of-text-in-an-image-using-python-opencv-and-numpy.html
  4. http://www.emgu.com/wiki/index.php/License_Plate_Recognition_in_CSharp
  5. http://www.emgu.com/wiki/index.php/Traffic_Sign_Detection_in_CSharp
  6. http://www.itdevspace.com/2012/07/c-crop-white-space-from-around-image.html
  7. http://www.pixel-technology.com/freeware/tessnet2/
  8. http://www.sk-spell.sk.cx/tesseract-ocr-parameters-in-302-version
  9. http://www.win.tue.nl/~aeb/linux/ocr/tesseract.html
  10. https://github.com/charlesw/tesseract
  11. https://github.com/peirick/VS2015_Tesseract/
  12. https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#config-files-and-augmenting-with-user-data
  13. https://github.com/tesseract-ocr/tesseract/blob/master/doc/wordlist2dawg.1.asc
  14. https://github.com/tesseract-ocr/tesseract/issues/403
  15. https://github.com/tesseract-ocr/tesseract/wiki
  16. https://github.com/tesseract-ocr/tesseract/wiki/ControlParams
  17. https://github.com/tesseract-ocr/tesseract/wiki/FAQ#how-do-i-provide-my-own-dictionary
  18. https://github.com/tesseract-ocr/tesseract/wiki/FAQ#wordlist2dawg-doesnt-work
  19. https://github.com/tesseract-ocr/tesseract/wiki/ImproveQuality
  20. https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract
  21. https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract2
  22. https://softwarebydefault.com/2013/05/11/image-edge-detection/
  23. https://sourceforge.net/p/tess4j/discussion/1202294/thread/83066e7c/?limit=25
  24. https://stackoverflow.com/questions/10206526/how-to-detect-text-area-from-image
  25. https://stackoverflow.com/questions/14364662/disable-dictionary-in-tesseract
  26. https://stackoverflow.com/questions/14858514/tesseract-ocr-is-it-possible-to-force-a-specific-pattern
  27. https://stackoverflow.com/questions/17209919/tesseract-user-patterns
  28. https://stackoverflow.com/questions/23506105/extracting-text-opencv
  29. https://stackoverflow.com/questions/2472467/how-to-find-one-image-inside-of-another
  30. https://stackoverflow.com/questions/31375843/detecting-text-regions-from-image-using-emgucv
  31. https://stackoverflow.com/questions/31874393/tesseract-ocr-force-pattern?rq=1
  32. https://stackoverflow.com/questions/37771263/detect-text-area-in-an-image-using-python-and-opencv/38554331#38554331
  33. https://stackoverflow.com/questions/5945156/c-sharp-detect-rectangles-in-image
  34. https://stackoverflow.com/questions/9480013/image-processing-to-improve-tesseract-ocr-accuracy
  35. https://tesseract-ocr.repairfaq.org/tess_variables_all.html
  36. https://www.codeproject.com/Articles/38319/Famous-Otsu-Thresholding-in-C
  37. https://www.codeproject.com/Articles/826377/Rapid-Object-Detection-in-Csharp
  38. https://www.codeproject.com/articles/696537/how-to-use-tessnet2-library
  39. https://www.mail-archive.com/tesseract-ocr@googlegroups.com/msg06330.html
  40. UWP

  41. https://blogs.windows.com/buildingapps/2016/02/08/optical-character-recognition-ocr-for-windows-10/
  42. https://docs.microsoft.com/en-us/uwp/api/windows.media.ocr
  43. https://docs.microsoft.com/en-us/windows/uwp/get-started/universal-application-platform-guide
  44. https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/OCR
  45. https://jeremylindsayni.wordpress.com/2016/03/22/optical-character-recognition-in-c-in-universal-windows-applications-part-2-using-windows-media-ocr/
  46. CHDK

  47. http://chdk.wikia.com/wiki/CHDK_For_Newbies_-_How_To_Install
  48. http://chdk.wikia.com/wiki/CHDK_for_Dummies
  49. http://chdk.wikia.com/wiki/CHDK_in_Brief
  50. http://chdk.wikia.com/wiki/For_Developers
  51. http://chdk.wikia.com/wiki/Prepare_your_SD_card
  52. http://chdk.wikia.com/wiki/USB_Remote
  53. https://chdk.setepontos.com/index.php?action=forum
  54. https://chdk.setepontos.com/index.php?topic=6367.0
  55. Stroke Width Transform

  56. https://sites.google.com/site/roboticssaurav/strokewidthnokia
  57. https://www.codeproject.com/Questions/439783/Stroke-Width-Transform-SWT-in-Csharp
  58. Canny Edge Detection

  59. http://www.gutgames.com/post/Edge-detection-in-C.aspx
  60. https://dzone.com/articles/c-tutorial-how-create-edge
  61. https://en.wikipedia.org/wiki/Canny_edge_detector
  62. https://jacraig.github.io/Craig-s-Utility-Library/html/class_utilities_1_1_media_1_1_swift_bitmap.html
  63. https://www.codeproject.com/Articles/93642/Canny-Edge-Detection-in-C

(070617)