IMSE7140 Assignment 2
Cracking CAPTCHAs
(20 points)
2.1 Brief Introduction
CAPTCHA or captcha is the acronym for “Completely Automated Public Turing test
to tell Computers and Humans Apart.” You must have been already familiar with it
because of its popularity in preventing bot attacks or spam everywhere. This assign ment, however, will guide you in implementing a deep learning model that can crack a
commercial-level captcha!
You deliverables for this assignment should include
1. A single PDF file answers.pdf with answers to all the questions explicitly marked
by “Q” with a serial number in this document, and
2. A train.py file to fulfill the programming task requirements marked by “PT.”
Of course, GPUs can facilitate your experiments—Don’t worry if you don’t have any,
the training requirement is deliberately simplified.
2.2 Training your model
The captchas we will crack is the multicolorcaptcha. Please pip install the exact version
1.2.0 (the current latest one) in case there might be any incompatibility for other releases.
We use the following codes to generate captchas.
1 from multicolorcaptcha import CaptchaGenerator
2
3 generator = CaptchaGenerator (0)
4 captcha = generator . gen_captcha_image ( difficult_level =0)
5 image = captcha . image
6 characters = captcha . characters
7 image . save ( f"{ characters }. png", "PNG")
In this snippet, CaptchaGenerator(0) configures the image size to 256 × 144 pixels,
and the difficult level is set to 0 so that the captchas only contains four 0–9 digits.
Please run the code snippet on your computer. If the captcha is successfully generated,
it should look like Figure 2.1.
1
2.2. Training your model S. Qin
Figure 2.1: Sample captcha with digits 0570
The training and the validation datasets are generated and attached in folders
capts train and capts val. For any machine learning problem, before you start to
devise a solution, it is always a good idea to observe the data and gain some intuition
first. You may immediately recognize some difficulties in this task:
• The digits have a set of random fonts and colors;
• Some certain range of random rotations are applied to the digits;
• Some line segments are randomly added to the image.
Such a task is considered impossible for traditional pattern recognition methods,
which may tackle the problem in a process like this: image thresholding, segmenta tion, handcrafted filter design, and pattern matching. We can conjecture that “filter
design” may fail in capturing useful features and “pattern matching” may have a poor
performance.
Fortunately, in the deep learning era, we can delegate the pattern or feature extrac tion job to deep neural networks. As introduced in the previous lecture “Deep Learning
for Computer Vision,” the slide “Understand feature maps: CAPTCHA recognition”
shows that a typical architecture for the task consists of two parts:
1. A backbone model to extract a feature map from the captcha image, and
2. A certain amount of prediction heads to interpret the feature map to readable
forms.
We will follow this architecture in this assignment. I encourage you to search open source solutions and learn from their experience. Here we follow this Kaggle post by
Ashadullah Shawon.
PT| Use capts train as the training dataset, capts val as the validation dataset, and Keras
as the deep learning framework, referring to Shawon’s solution, provide the training code
train.py that fulfills the following requirements. “Copy and paste” the codes from the
original post is allowed, as well as other AI-generated codes.
2
2.3. Example: A practical model S. Qin
1. The maximal number for epochs should be 10. Considering some students
will train the model by CPU, it is fair to limit the number of epochs, so the training
time for the model should be less than half an hour.
2. The accuracy for one digit should be no less than 30% after training for
10 epochs. The training outputs contain four accuracies respective to the four
digits. Since they are similar, you will only need to examine one of them. Keep in
mind that 30% for one digit indicates that the overall accuracy for the recognition
is only 0.3
4 = 0.81%. Such a low accuracy is not useful for cracking the captcha.
However, on the one hand, you may need a GPU to experiment on a practical
solution; on the other hand, a wild guess for a 0–9 digit has an accuracy of 10%,
so if your model’s accuracy can reach 30% after 10 epochs, it already indicates
the model learns from the training set. Hint: if the accuracy for one digit keeps
wandering around 0.1 but not increasing in the first two or three epochs, it is the
signal that you should modify somewhere in your code and try again.
3. The trained model should be saved as a file my model.keras after training.
Though, this model file my model.keras doesn’t need to be uploaded.
Q1| Can we convert the captcha images to grayscale at the preprocessing stage before train ing? What is the possible advantage by doing that? If any, can you point out the
possible disadvantage?
Q2| After the 10-epoch training, what are your accuracies of one digit, for the training and
the validation datasets respectively?
Q3| Is the accuracy for the validation dataset lower than that for the training dataset? What
are the possible reasons?
Q4| How can we improve the model’s performance on the validation dataset? List at least
three different measures.
2.3 Example: A practical model
To demonstrate that the backbone–heads architecture can actually solve the real-world
captcha, I trained a relatively large model by an Nvidia GeForce RTX 3090 GPU.
You may find in attached the model file 0991-0.9956.keras and the inference code
inference.py. The accuracies versus training epochs are shown in Figure 2.2. The
inference code reads a randomly generated captcha, inferences the model, and compares
the predicted results with the targets. You can press “n” for the next captcha or “q” to
quit the program. You may need to pip install keras cv to run the code.
Q5| What kind of backbone did I use in the model 0991-0.9956.keras?
Q6| The backbone’s pre-trained weights on the ImageNet 2012 dataset were loaded before
training. What is the possible advantage by doing that?
Q7| Why didn’t I use any dropout in the model? Guess the reason.
Q8| In Figure 2.2, you may have noticed that the accuracies rise very fast from 0 to 0.9, but
significantly slow from 0.95 to 0.99. Explain the phenomenon.
Q9| Using the same hardware (which means you can’t upgrade the GPU, for example), how
can we speed up the learning process of the model, i.e. the rate of convergence?
3
2.3. Example: A practical model S. Qin
0 200 400 600 800 1000
Epoch
0.2
0.4
0.6
0.8
1.0
Model Accuracies
digi0
digi1
digi2
digi3
Figure 2.2: Accuracies through 1000 epochs in training
Q10| Since the accuracy for one digit is about 99%, the overall accuracy for a captcha is
0.994 ≈ 96%. This performance would be better than humans. Can you propose some
methods that can even further improve the performance?
Please note that, not all the questions above have a definite answer. You may also
need to do some research as the course doesn’t cover all the details in class. The source
code for training this model and the reference answers will be available on Moodle or
sent by email after all the students completing the submission.


请加QQ：99515681  邮箱：99515681@qq.com   WX：codinghelp