• Demonstrate the use of Open Source image processing software for automating the decoding of historic documents which have secret message text embedded within them.
• Identify a valid source of Ground Truth Data.
• Explain how some elements of this application are inherently simple, which bodes well for feasibility and success.

Therefore a CNN makes Predictions, which are probabilistic: not either ‘true’ or ‘false’, but instead a numerical value between 0.0 and 1.0, which corresponds to a Confidence value between zero and 100%.

Some components of the ground truth data for this experiment are available from the same Bodleian Library download page already mentioned in Experiment One:

The previously-used image of the Prologue page from Troilus and Cressida came from the “image PDF” file shown above. The “text PDF” file provides the corresponding Prologue page in text, rather than as an image:

For the project to be successful, there are five attributes for each letter of unencoded text which need to be determinable on request.

The Five Attributes

1. Which of the 24 letters of the Elizabethan English alphabet any given glyph (uncharacterized image blob) is.
2. Whether upper or lower case.
3. Whether ‘a’ or ‘b’ biliteral value.
4. Bounding box coordinates of each glyph.
5. Direct lookup between text position and image (glyph) position.

An example of #5 would be, “letter number 23 of the Prologue is ‘T’, so show the tiny image of that one glyph by itself, using the values of its bounding box”.

The ‘text PDF’ representation is a reliable information source for providing the first two, and will comprise elements of our Ground Truth data.

The third (‘which of ‘a’ or ‘b’ biliteral value’) is already available from Experiment One. The assumption will be made that this data also qualifies as Ground Truth.

The fourth, presumably, can eventually come from custom OpenCV programs. Below is described the first step towards that (more will be needed).

Eurotext.png, grayscale, 450 × 351, 57Kb

OCR Text output

Bounding Boxes

On every execution of the Python program code, the tiny image inside each bounding box gets saved into its own .jpeg file, and will be referred to here as the ‘pictures-of-letters’. This will be important in future Experiments.

Now, though, the Python algorithms can access each picture-of-letter directly as an ordinary program object.

This enables the main CV (Computer Vision) technique which will be used here, Template Matching: a Template image is superimposed over a larger Target image to compare the two for similarity. This is the basis for Recognition, and by extension, Decoding.

For this experiment, the Target is the Bodleian Library facsimile of the T & C Prologue, used without manual preprocessing of any kind.

The Templates would be a set of 96 glyphs (twice 24, the number of letters in the Elizabethan alphabet, with both upper and lower case), each of which is known, with certainty, as being either the ‘a’ or ‘b’ Biliteral variant for that letter.

An existing font, below, is an attempt to duplicate the look of the First Folio’s type. But it can’t suffice for use here, since there is no way to have the bounding boxes of letters overlapping.

It can be seen in the above sample from The Prologue that the bounding boxes of actual First Folio text has overlapping bounding boxes in many places. The letters ‘s’ and ‘f’ are particularly troublesome, as their bounding boxes overlap in two places, both a letter before, and one after. This throws a proverbial wrench into algorithms depending on bounding boxes.

Also, the plentiful noise in the background image is readily visible.

Then from where can our set of 96 non-overlapping glyphs come? These would be the essential Templates used to match against a rectangular region within the Target (in this case, the Folio Prologue page). The project can’t go forward without obtaining these.

The next Experiment will explore using the 1916 decoder (above), which were produced by W.F.F at Riverbank Laboratories. Note how there is ample space between all the letters, so there is no overlapping.

There is also a great deal of noise in the background image, but this might be filterable by our custom Python code.

If each glyph can be saved to its own .jpeg file, a simple naming scheme could provide the crucial requirement of Attribute #5: each letter-picture could have a unique two-character file name, which concisely encodes the letter, its case, and its biliteral value. For example, the upper ‘c’ letter in the decoder card above would be saved in a file, c0.jpeg. The lower ‘c’ letter would be saved in a file, c1.jpeg. For the custom Python code, this would allow ready access to any pair of letter-pictures to match against. Thus is satisfied the requirement, “Direct lookup between text position and image (glyph) position.”

View the Jupiter Notebook: the experimental commands used here, and the results recorded.

View and download the Python source code from Github.

Independently run the program on Google Colab, or create a clone of the program to run on a Colab account of your own.

Notes for this page:

• The downloadable Bodleian file listed above, “XML as PDF” provides a representation of the First Folio as a Text Encoding Initiative document. A more comprehensive abstraction of a book’s content and metadata can hardly be imagined. So from that, we can know that the text transcription in the PDF version has been proofread by human beings, making it suitable as Ground Truth data. We are even given the names of the proofreaders: thank you Pip Willcox, Lucienne Cummings, Judith Siefring, and Emma Stanford! Your work is accepted here as Ground Truth Data!

The Tudor Rose

Do you have experience with Convolutional Neural Networks? Know anyone who does? See opportunity