Performing OCR on Japanese PDFs Using Tesseract and Pytesseract

Extracting Japanese text from a PDF with Tesseract OCR v4 and pytesseract, including the normalization step needed to clean up the output.

Takahiro Iwasa

2022-06-06

4 min read

Tesseract OCR v4 and its Python wrapper pytesseract can extract Japanese text from PDFs.

The source text is Run, Melos! by Osamu Dazai, a work that is now in the public domain.

Requirements

Before we start, ensure the following libraries are installed:

Additionally, Tesseract OCR itself must be installed. In this note, it is set up using Docker. Please refer to the instructions on the official repository for more details.

Building

Writing Python Script

1
import re
2
from datetime import datetime
3

4
import pdf2image
5
import pytesseract
6

7

8
def to_images(pdf_path: str, first_page: int = None, last_page: int = None) -> list:
9
    """ Convert a PDF to a PNG image.
10

11
    Args:
12
        pdf_path (str): PDF path
13
        first_page (int): First page starting 1 to be converted
14
        last_page (int): Last page to be converted
15

16
    Returns:
17
        list: List of image data
18
    """
19

20
    print(f'Convert a PDF ({pdf_path}) to a png...')
21
    images = pdf2image.convert_from_path(
22
        pdf_path=pdf_path,
23
        fmt='png',
24
        first_page=first_page,
25
        last_page=last_page,
26
    )
27
    print(f'A total of converted png images is {len(images)}.')
28
    return images
29

30

31
def to_string(image) -> str:
32
    """ OCR an image data.
33

34
    Args:
35
        image: Image data
36

37
    Returns:
38
        str: OCR processed characters
39
    """
40

41
    print(f'Extract characters from an image...')
42
    return pytesseract.image_to_string(image, lang='jpn')
43

44

45
def normalize(target: str) -> str:
46
    """ Normalize result text.
47

48
    Applying the following:
49
    - Remove new line.
50
    - Remove spaces between Japanese characters.
51

52
    Args:
53
        target (str): Target text to be normalized
54

55
    Returns:
56
        str: Normalized text
57
    """
58

59
    result = re.sub('\n', '', target)
60
    result = re.sub('([あ-んア-ン一-鿐])\s+((?=[あ-んア-ン一-鿐]))', r'\1\2', result)
61
    return result
62

63

64
def save(result: str) -> str:
65
    """ Save the result text in a text file.
66

67
    Args:
68
        result (str): Result text
69

70
    Returns:
71
        str: Text file path
72
    """
73

74
    path = 'result.txt'
75
    with open(path, 'w') as f:
76
        f.write(result)
77
    return path
78

79

80
def main() -> None:
81
    start = datetime.now()
82
    result = ''
83

84
    images = to_images('run-melos.pdf', 1, 2)
85
    for image in images:
86
        result += to_string(image)
87
    result = normalize(result)
88
    path = save(result)
89

90
    end = datetime.now()
91
    duration = end.timestamp() - start.timestamp()
92

93
    print('----------------------------------------')
94
    print(f'Start: {start}')
95
    print(f'End: {end}')
96
    print(f'Duration: {int(duration)} seconds')
97
    print(f'Result file path: {path}')
98
    print('----------------------------------------')
99

100

101
if __name__ == '__main__':
102
    main()

Creating Dockerfile

The run-melos.pdf on line 6 can be downloaded here.

1
FROM python:3.10
2

3
WORKDIR /usr/src/app
4
COPY app.py ./
5
COPY requirements.txt ./
6
COPY run-melos.pdf ./
7
RUN apt update && apt install -y poppler-utils tesseract-ocr tesseract-ocr-jpn \
8
    && pip install -r requirements.txt
9

10
CMD ["python", "app.py"]
11
# CMD ["/bin/sh", "-c", "while :; do sleep 10; done"]

Testing

The script converts the PDF to PNG image data with pdf2image, extracts characters from the images with Tesseract OCR and pytesseract, and saves the result to a text file:

NAME=pytesseract-sample
docker build -t $NAME .
docker run --name $NAME $NAME

# You can see OCR processing result in `result.txt`.
docker cp $NAME:/usr/src/app/result.txt ./
less result.txt

# Clean up
docker container rm $NAME
docker image rm $NAME

Result

The result can be downloaded from:

Conclusion

Running a public-domain Japanese PDF through pdf2image and Tesseract in a Docker container extracted readable text that closely matched the original. The lang='jpn' argument to pytesseract.image_to_string is what activates Tesseract’s Japanese-trained model, but getting readable output also depends on the normalize step’s regex, which strips the whitespace Tesseract tends to insert between Japanese characters — without it, the extracted text reads as fragmented even when the underlying recognition is accurate. Comparing the result against the original text of Run, Melos! is worth doing directly rather than skimming, since misreads between visually similar kanji can slip past a quick check. Because this PDF is a clean, digitally rendered public-domain text rather than a scan, accuracy here represents close to a best case — lower-resolution or scanned source material would likely need image preprocessing before OCR to get comparable results.

Back to posts