Recognition of handwritten flowcharts with CNNs

Recognition of handwritten flowcharts using convolutional neural networks to generate C source code and reconstructed digital flowchart.

General description

The pipeline implemented in this project in order to solve the problem of recognition of handwritten flowcharts uses preprocessing of images, the image is send to two detectors, the shapes and connectors detector and the text detector. In flow for text, the image is binarize and uses Keras OCR for locate text and an implemented model with CNN + LSTM for classifing characters; besides, the flow to shapes and connectors uses masking unsharp and an implemented model that's called Faster R-CNN with backbone VGG-16.

In order to augment text detector precision the technique called continual learning is used. After, some training with the texts of a certain user, the model will improve.

Finally, like output is generated source code in C and its compilation output, and the reconstructed diagram in image digital.

Note: The flowcharts used for testing are constructed with a defined set of shapes and connectors. You can see here.

How to set up for testing detections

1. Create a virtual enviroment with Conda with name 'tt', the list of requirements is here.
2. Download this repo.
3. Model for shapes and connectors:
   • Download the folder here.
   • Paste it (unzipped) into model/training_results/ (path inside the repo)
4. For text.
   • Download IAM dataset here
     • Inside text_model create a folder with name: 'data_model'.
     • iam.hdf5 (94.1 MB) paste into text_model/data_model/
   • Download trained model here
     • checkpoint_weights.hdf5 (38.5 MB) paste into text_model/output/iam/puigcerver/

Usage

1. Activate your Conda enviroment.
2. Change terminal to repository folder.
3. Type: $ python3 handler.py
4. Use "Recognize flowchart" option for testing detections with handwritten flowcharts.

Some examples of the results

Calculate the nth term of the Fibonacci sequence, compilation output: None.

Hello world, compilation output: None.