Here we present the results of our preprocessor tests. As it is hard to test the output of the preprocessor objectively, we tested whether the preprocessor improve the efficiency of the OCR module.

Effectiveness preprocessor

Test preprocessor effect

date:Thursday August 17th 2017

Setup:

We generated 1000 images of the calculator-display, with different numbers and a ‘normal’ background (normal lighting and noise). In our first test, these numbers were used as input for the OCR (so without any preprocessing). In our second test, the default preprocessing was applied, executing the functions “gray_scale”, “gaussian_blur”, “filter2d”, “binary_threshold”, “median_blur” and “erosion”. The preprocessed image was further analyzed in the OCR module. The results of the OCR (a string of numbers) was compared with the actual number and the fraction of correct outputs was assessed.

Results:

Without preprocessing the OCR was able to recognize the numbers of 0 of the 1000 images (0%). With preprocessing, the OCR was able to recognize the numbers of 172 of the 1000 images (17%). This number does depend on the test images (which were randomly created and therefore differed between computers), with another set test images the success rate was 24%.

../../_images/pp.jpg

Display before (top) and after (bottom) default preprocessing

Conclusion:

The current preprocessing improves the efficiency of the OCR.

Test backgrounds

date:Tuesday 29th August 2017

Setup:

We generated images on four different backgrounds using the DSEG7Classic font. We generated images with 2 digits, 3 digits and 4 digits with each background. The numbers were used to test the OCR with and without the use of the preprocessor. For the preprocessor we used the following configuration: “gray_scale”, “median_blur”, “binary_threshold” and “erosion”, in that order.

Results:

Without preprocessor, it seems to be able to recognize some with this font, but most are ignored.(No numbers excluded)

  • DSEG7Classic-Regular - clean_bg:
    • 2Digits: Accuracy: 0.0%, correct: 0, number of images: 100
    • 3Digits: Accuracy: 0.0%, correct: 0, number of images: 1000
    • 4Digits: Accuracy: 0.0%, correct: 0, number of images: 951
  • DSEG7Classic-Regular - easy_bg:
    • 2Digits: Accuracy: 0.0%, correct: 0, number of images: 100
    • 3Digits: Accuracy: 0.0%, correct: 0, number of images: 1000
    • 4Digits: Accuracy: 0.0%, correct: 0, number of images: 943
  • DSEG7Classic-Regular - normal_bg:
    • 2Digits: Accuracy: 0.0%, correct: 0, number of images: 100
    • 3Digits: Accuracy: 0.0%, correct: 0, number of images: 1000
    • 4Digits: Accuracy: 0.52%, correct: 5, number of images: 958
  • DSEG7Classic-Regular - hard_bg:
    • 2Digits: Accuracy: 0.0%, correct: 0, number of images: 100
    • 3Digits: Accuracy: 0.0%, correct: 0, number of images: 1000
    • 4Digits: Accuracy: 0.0%, correct: 0, number of images: 954

With preprocessor, it recognizes around 45% of the images.(No numbers excluded)

  • DSEG7Classic-Regular - clean_bg:
    • 2Digits: Accuracy: 53.0%, correct: 53, number of images: 100
    • 3Digits: Accuracy: 50.0%, correct: 500, number of images: 1000
    • 4Digits: Accuracy: 42.8%, correct: 407, number of images: 951
Clean background
  • DSEG7Classic-Regular - easy_bg:
    • 2Digits: Accuracy: 47.0%, correct: 47, number of images: 100
    • 3Digits: Accuracy: 51.3%, correct: 513, number of images: 1000
    • 4Digits: Accuracy: 41.89%, correct: 395, number of images: 943
Easy background
  • DSEG7Classic-Regular - normal_bg:
    • 2Digits: Accuracy: 47.0%, correct: 47, number of images: 100
    • 3Digits: Accuracy: 49.7%, correct: 497, number of images: 1000
    • 4Digits: Accuracy: 40.29%, correct: 386, number of images: 958
Normal background
  • DSEG7Classic-Regular - hard_bg:
    • 2Digits: Accuracy: 35.0%, correct: 35, number of images: 100
    • 3Digits: Accuracy: 50.0%, correct: 500, number of images: 1000
    • 4Digits: Accuracy: 42.03%, correct: 401, number of images: 954
Hard background

Conclusion:

The different backgrounds seem to have little effect on the percentage. Although the amount does differ. A lot of the incorrect numbers are due to the ‘0’ and the ‘8’. In all of the test images not one ‘0’ was recognized out of the 2846. I case of the ‘8’, 220 were recognized in correct numbers out of the 2485. There are incorrect numbers where an ‘8’ is recognized, but failed due to a ‘0’ or an ‘8’. Results of those numbers are for example:

  • Predicted: 8636, Actual: 086.
  • Predicted: 8637, Actual: 087.
  • Predicted: 86363, Actual: 088.
  • Predicted: 1488, Actual: 1480. <- ‘8’ is correct, failed due to ‘0’
  • Predicted: 89633, Actual: 0983.
  • Predicted: 89635, Actual: 0985.
  • Predicted: 896363, Actual: 0988.

Tips for next test:

In case of the ‘8’ the occurrence is a lot less than the ‘0’, but often enough to have a big effect on the total percentage. Tests without the use of those numbers should show a massive difference in accuracy. The resolution is a lot higher than the cropping module returns. This should be about the same for higher realistic accuracies. Also smaller images seem to be easier to read by the OCR module, but tests should prove this statement.

Randomness

date:Monday August 21st 2017

Aim:

To test whether there is some randomness in recognizing the numbers. If the same image is processed 100 times, it should fail 100 times or succeed 100 times, otherwise there is some randomness in the preprocessing/OCR module.

Setup:

Five test images, generated without any adaptations (e.g. no noise added) were tested 100 times (11312555, 11799773, 31195166, 75888955, 89925729). The default preprocessing was applied, executing the functions “gray_scale”, “gaussian_blur”, “filter2d”, “binary_threshold”, “median_blur” and “erosion” and the images were further analyzed in the OCR module. The results of the OCR were compared with the actual numbers on the image.

Results:

The numbers 11312555, 11799773 and 31195166 were recognized 100 times, while the numbers 75888955 and 89925729 were never recognized.

Conclusion:

There is no indication for a randomness in the preprocessing or OCR module.

Effectiveness with different images

date:Thursday August 24th 2017

Aim:

To test the effectiveness of the preprocessor and OCR, but with more variety in images.

Setup:

We created different options in the Image Generation:

  1. Different number of digits
  2. Option to add noise, blur and deform the image, to make it look more realistic
  3. Different backgrounds
  4. Possibility to exclude some numbers - not done yet

Each option was assessed with the default preprocessing option (“gray_scale”, “gaussian_blur”, “filter2d”, “binary_threshold”, “median_blur” ). With each option, we created maximum 1000 random images (or, if the number of digits is <3, the total possible number, i.e. 10 with 1 digit)

Results:

  1. Accuracy depending on the number of digits (# good/total #)
    1. 0% (0/10)
    2. 47% (47/100)
    3. 49% (493/1000)
    4. 42% (398/959)
    5. 31% (304/991)
    6. 24% (244/1000)
    7. 20% (200/1000)
    8. 15% (149/1000)
  2. Accuracy depending on blurring etc. (tested on 1000 3-digit images)
    • If the figure is only blurred: accuracy = 0% (0/1000)
    • If only the size of the figure is decreased: accuracy = 0% (0/1000)
    • If the figure is only deformed: accuracy = 0% (0/1000)
    • If figure is blurred, deformed and with noise: accuracy = 0% (0/1000)
  3. Accuracy depending on different backgrounds (tested on 200 3-digit images)
    • default background: accuracy = 50.55% (92/182)
    • background 1: accuracy = 65.61% (124/189)
    • background 2: accuracy = 42.39% (78/184)
    • background 3: accuracy = 56.18% (100/178)
    • background 4: accuracy = 0% (0/180)