PSA-HWT: handwritten font generation based on pyramid squeeze attention

Approach overview

To address the issue of the HWT encoder’s inability to effectively extract stroke details when extracting handwritten font features, inspired by the approach of embedding attention modules into convolutional neural network (CNN) networks to improve performance, we introduce Pyramid Squeeze Attention (PSA) (Zhang et al., 2022) into HWT, proposing a handwritten font generation model called PSA-HWT. We utilize ResNet50 (He et al., 2016) as the backbone network for the encoder. It incorporates Pyramid Squeeze Attention (PSA) into the ResNet50 network for multi-scale feature extraction. The features are then encoded using a Transformer encoder, and subsequently decoded by a Transformer decoder followed by a CNN decoder to generate handwritten fonts. The encoder of PSA-HWT performs multi-scale feature extraction on handwritten fonts using the PSA module. It captures spatial information from multi-scale input feature maps and establishes long-term dependencies between channels with multi-scale channel attention, thereby more fully extracting stroke details of the font. The Transformer encoder utilizes self-attention mechanisms to model features at different positions in the font sequence, capturing the relationships between each position and other positions’ features and enhancing its understanding of contextual information in the sequence, thereby improving the realism of the generated fonts. Additionally, a discriminator is used to distinguish between generated images and real images at both the pixel and global levels, encouraging the generator to produce more realistic images.

PSA-HWT is mainly composed of G (Generator) and D (Discriminator), as shown in Fig. 2.

As illustrated by Fig. 2, generator G consists of two parts: an encoder and a decoder. The encoder is composed of a ResNet50 fused with pyramid squeeze attention and a Transformer encoder. The decoder consists of a Transformer decoder and a CNN decoder. The structure of the CNN encoder is shown in Fig. 3. The CNN encoder consists of the PSA module and ResNet50. In this configuration, one 3 × 3 convolution block in ResNet50 is replaced with the PSA module. The PSA module not only processes multi-scale input tensors using a multi-scale pyramid convolution structure but also compresses the channel dimension of the input tensors. This allows it to effectively extract spatial information of different scales from each channel feature map and place greater emphasis on the weights of the key information in the feature maps. This enables the model to fully extract the important features of the font strokes. First, we use a ResNet50 network incorporating pyramid squeeze attention generate lower-resolution feature maps hij∈Rh×w×d for each style image xij (where i represents a certain style, j is the handwritten font image of the style, the variables h and w denote the height and width of the image, respectively, d is the embedded size). Then, flatten the spatial dimensions hij to produce the feature map sequence of size n×d, where n=h×w. After that, the feature sequence vectors derived from every style image are combined together to create a unified tensor Hi∈RN×d, where N=n×P. P represents the set of the number of handwritten font images. The next step includes entering Hi into the Transformer encoder to model the local and global combinations of feature sequences. The framework of Transformer encoder consists of several layers, encompassing both a module for multi-head self-attention and a module for multi-layer perceptron. The multi-head attention module within each layer is tasked with converting the input sequence from the preceding layer into a triplet, the triplet can be expressed as key K, query Q and value V. The relationship between these three components is depicted in Formula (1).

(1) Q=Hl−1WQ,K=Hl−1WK,V=Hl−1WVwhere WQ∈RN×dq,WK∈RN×dk,WV∈RN×dv are the learnable weight matrix for Q, K and V respectively. The calculation procedure for each attention head is shown in Formula (2).

(2) Oj=softmax(QKTdk)V∈RN×dv,j∈{1,⋯,J}

Connecting the outputs O=[O1,⋯,OJ] of all J heads through a MLP layer yields the output feature sequence Hl for the layer l. The final feature sequence Z∈RN×d is obtained after passing through a total of L layers transformer encoder. Here, dk represents the dimension of k. Dividing by the square root of dk during the computation helps maintain gradient stability during training. We used the softmax function to normalize the results. The decoder consists of multiple Transformer decoder layers. The feature sequence Z undergoes processing by multiple consecutive decoders to obtain feature vectors. In each decoder, embedded queries are processed in parallel. Finally, the feature vectors are concatenated and passed through a linear layer to obtain a vector matrix. This matrix is subsequently input into a CNN decoder comprising of four residual blocks and a Tanh layer, resulting in the production of the ultimate image of the handwritten font. Additionally, we replaced a 7 × 7 convolutional kernel in the ResNet50 network with three 3 × 3 convolutional kernels, which increases the network depth while reducing the number of parameters.

Pyramid Squeeze Attention

Pyramid Squeeze Attention (PSA) efficiently captures and utilizes spatial information from feature maps at different scales. By employing a multi-scale pyramid convolution structure to extract features, it establishes longer-range channel dependencies, enabling the extraction of multi-scale features of the font at a finer granularity. This allows for better extraction of key features such as strokes, curvature, and thickness of the font. PSA is mainly composed of SEWeight (Hu, Shen & Sun, 2018) module and Squeeze and Concat (SPC) module, as shown in Fig. 4.

The SPC module can acquire multi-scale feature maps H×W×C ′, where H and W represent the height and width of the input feature map respectively, and C ′ are input channel dimensions. The channel attention on the feature maps at various scales is extracted utilizing SEWeight module, w1,w2⋯,wS−1 is represented as channel attention vector, which is adjusted using Softmax to obtain the rescaled attention weight vector w1′,w2′⋯,wS−1′. Ultimately, the feature map is achieved through element-wise multiplication of the recalibrated weights and the corresponding feature map, generating a feature map with diverse scale information.

To help readers better understand the details of how pyramid squeeze attention achieves multi-scale feature extraction, Fig. 5 provides the main algorithm code and a detailed explanation.

SEWeight module

The channel attention mechanism can regulate the focus and weight allocation between different channels in the feature map, enhancing the model’s attention to specific channels. This improves the model’s ability to extract and represent important features. Let X∈RC×H×W represent the input of the feature map, where H, W, and C denote height, width, and the channel dimensions of the input, respectively. The SE block consists of two parts: squeeze and excitation, which are used to encode global information and adaptively recalibrate channel relationships, respectively. The SEWeight structure is shown in Fig. 6. Global average pooling is utilized to incorporate global spatial information into channel descriptors. Computation of the global average pooling operation is defined as,

(3) gc=1H×W∑i=1H∑j=1Wxc(i,j)

Equation (4) demonstrates the calculation of the attention weight for the c- th channel in the SE block.

(4) wc=σ(W1δ(W0(gc)))where δ represents ReLU and σ represents Sigmoid. W0∈RC×Cr, W1∈RCr×C. W0 and W1 are both fully connected layers, and the linear relationship between the channels is bound through W0 and W1.

We use the SEWeight module to allow the network to selectively weight the importance of each channel, accurately calculating the importance of each stroke position. This enables the model to better focus on the key features of the fonts, thus improving the quality of the generated fonts.

SPC module

SPC is the key module of PSA to realize multi-scale feature extraction, the structure is shown in Fig. 7.

The multi-branch structure utilized by SPC is designed to extract spatial information from X, where X represents the input feature map. Each individual branch within this structure operates with an input channel dimension of C. X is sliced into S parts, expressed as [X0,X1,⋯,XS−1], and the number of channels in each part is C′=C/S (set S = 4 in Fig. 7). After slicing, the feature map Xi∈RC′×H×W is obtained, where i=0,1,⋯,S−1. To extract this information effectively, multi-scale convolutions are utilized to produce feature maps with diverse spatial scales and depths. Every branch can learn the spatial information at various scales and creates localized interactions across channels. To better handle input tensors at different convolutional kernel scales, group convolution is applied to the kernels. The correlation between the convolutional kernel size at various scales and the group size can be expressed as

(5) G=2K−12where K represents the convolution kernel size, Gdenotes the group size. The computation for feature extraction at various scales is determined as depicted in Eq. (6).

(6) Fi=Conv(ki×ki,Gi)(Xi),i=0,1,2⋯S−1where the i- th convolution kernel size is ki=2×(i+1)+1, and the size of i- th group is Gi=2ki−12. Fi∈RC′×H×W signifies the feature maps at various scales. The finally obtained feature maps spanning multiple scales are connected by Concat way as shown in Eq. (7).

(7) F=Cat([F0,F1,⋯,FS−1])

By deriving the attention weight vector from the feature map encompassing multiple scales, the attention weight vector can be represented as

(8) Zi=SEWeight(Fi),i=0,1,2⋯S−1where Zi∈RC′×1×1 represents the attention weight. The attention weight at various scales is obtained by the SEWeight module, while the PSA unit integrates contextual details from diverse scales, enhancing the pixel-level attention of the high-level feature maps. For the purpose of reinforcing the exchange of attention information, the cross-dimensional vectors are combined in a way that maintains the accuracy. The entire multiscale channel attention vectors are acquired in series as

(9) Z=Z0⊕Z1⋯⊕ZS−1where ⊕ represent the concatenation operator, Zi denotes the attention value Fi, and Z is the vector of attention weights at various scales. Softmax is employed to readjust the weights of channel attention data, aiding in the creation of channel attention connections and improving information exchange with different scales, and the calculation process can be represented as

(10) atti=Softmax(Zi)=exp(Zi)∑i=0S−1exp(Zi)

The weights of multi-scale channels atti are acquired through Softmax, which encapsulates attention weights of the channels and the positional details in space. This facilitates the realization of interaction between global and local channel attention. Following the channel attention is spliced and fused, and finally the complete channel attention vector can be acquired as

(11) att=att0⊕att1⊕⋯⊕attS−1in the attention interaction, att represents the weights assigned to the multi-scale channels. Subsequently, the feature maps Fi are multiplied by the channel attention weight atti, as shown in Eq. (12).

(12) Yi=Fi⊙atti,i=1,2,3,⋯,S−1where ⊙ denotes the channel-wise multiplication, Yi signifies the feature map incorporating channel attention weight with different scales. The features obtained by splicing can be written as

(13) Out=Cat([Y0,Y1,⋯,YS−1])

The PSA module has the ability to incorporate spatial information from multiple scales and different channel attentions into each feature group’s blocks. Utilizing the PSA module enhances the understanding of information interaction between global and local channel attention during feature extraction, which in turn enables us to perform multi-scale feature extraction on handwritten font samples, learn features consistent with the handwriting style of handwritten fonts, and generate more realistic fonts.

Implementation details

We perform experiments on the IAM (Marti & Bunke, 2002) dataset¹ , which comprises 9,862 text lines and over 60,000 English words written by 500 different authors. We select handwritten images from 340 authors for training, and images from the remaining 160 authors for testing. Additionally, we used the CVL (Kleber et al., 2013) dataset² , comprising 311 authors and 101,069 words, to conduct experiments and calculate the FID values. We used 284 authors for training and the remaining 27 authors to test the results. We used the Python and experimented in the Pytorch environment, using A100 GPU server to train and test our model. The images are adjusted to a constant height of 64 pixels while maintaining the aspect ratio of the original image. The number of layers for both the Transformer encoder and decoder attention is set to 3, with each layer having 8 attention heads. Training employs the Adam optimizer and sets the learning rate to 2 × 10⁻⁴, batch size set to 8. In this scenario, training halts after 9.8 k epochs.

Evaluation metrics

The Fréchet inception distance (FID) (Heusel et al., 2017) score and the Geometry Score (GS) (Khrulkov & Oseledets, 2018) are used to evaluate the effectiveness of our model. FID is a metric used to calculate the distance between the features of the generated image and the features of the real image. A lower value of FID means that the features of the two are closer together, indicating that the generated font image is closer to the real font image. GS is a metric that calculates the topological similarity of the images. A lower GS value represents a higher quality of the image generated by the model. SSIM (Wang et al., 2004) and RMSE are metrics used to measure the preservation of pixel details in generated images. A higher SSIM indicates less distortion in the generated image. Additionally, the perceptual similarity of the generated image is quantified using LPIPS (Zhang et al., 2018), the lower LPIPS values indicate that the generated images are more realistic. The Handwriting Distance (HWD) (Pippi et al., 2023) metric operates in a specially trained network feature space, extracting handwriting style features from variable-length input images and comparing the subtle geometric features of handwritten fonts based on perceptual distance. A lower value of the HWD metric indicates a more realistic and authentic generated font image.

Experiments

We evaluate our model using the test set from the IAM dataset, obtained FID and GS scores, and performed quantitative comparisons with the current more advanced HiGAN (Gan & Wang, 2021), ScrabbleGAN (Fogel et al., 2020), TS-GAN (Davis et al., 2020), HWT (Bhunia et al., 2021), CG-GAN (Kong et al., 2022), and VATr (Pippi, Cascianelli & Cucchiara, 2023) models in handwriting font generation research. For comparison, the results are presented in Table 1.

According to Table 1, PSA-HWT obtained the best FID score, while GS score comes second with a slight difference from the optimal GS score. The FID score shows a decrease of 23.29% compared to the classical HWT model. The VATr model has FID score of 17.79 and a GS score of 1.68 × 10⁻². The PSA-HWT model shows a decrease of 16.35% in FID score and 13.09% in GS score compared to the VATr model. Compared to CG-GAN, the FID value decreased by 21.81%. It demonstrates the effectiveness of pyramid squeeze attention in the model and how multi-scale feature extraction enables the generated handwritten font images to be closer to real images. As a result, the quality of generated fonts has been significantly improved.

Introducing pyramid squeeze attention into the model not only enables multi-scale feature extraction during feature extraction but also results in generated fonts that are more similar to real fonts. Table 1 provides a comparison of the font image quality scores. Additionally, this improves the generalization performance of the model significantly. We tested the FID values for the generated fonts under four settings: (1) Generate words from the training set using the style of authors from the training set (IV-S), (2) Generate words from the training set using the style of authors from the test set (IV-U), (3) Generate words using the style of authors from the training set for words that are not appearance in the training set (OOV-S), (4) Generate words using the style of authors from the test set for words that are not appearance in the training set (OOV-U). We compared the results with the GANwriting, TS-GAN, HWT, and CG-GAN models. From the data in Table 2, it can be seen that our model performed the best, particularly in the most challenging OOV-U setting.

In the IV-S and IV-U settings, compared to the HWT model, the FID values decreased by 2.57% and 2.74%, respectively. It can be observed that in the most challenging OOV-U setting, our model achieved better FID scores compared to other models on the IAM dataset. Specifically, the FID score is 5.25% lower than the best-performing CG-GAN. This demonstrates that our model has better performance and generalization.

To demonstrate the performance advantage of our model, we conducted a qualitative comparison with HWT, GANwriting, and Davis. We use the same text content to evaluate the quality of the generated fonts for all four methods. The initial column displays various writers’ examples of stylistic choices (see Figs. 8 and 9). (Note that all images in Figs. 8 and 9 except our model are from Bhunia et al., 2021)

The first column represents examples of different authors’ styles. Although the HWT model can mimic the global and local style patterns of fonts, it cannot capture more fine-grained font features such as the curvature and thickness of strokes. Due to being restricted to a fixed length of query words, GANwriting cannot complete the provided text content, and it has difficulty adhering to the left-tilted top and character style of the examples. The model by Davis et al. can capture the global style but has difficulty imitating the stylistic details of specific characters. This generated font follows the left tilt of a style example but struggles to capture character-level styles and cursive patterns (e.g., the word “the”). Our model not only better simulates global and local style patterns but also captures the fine-grained font features of the examples, generating realistic handwritten text images.

To further validate that introducing pyramid squeeze attention into the HWT model can sufficiently extract multi-scale features. We evaluate the model using evaluation metrics for handwriting font generation quality, including HWD, as well as commonly used metrics such as SSIM, RMSE, and LPIPS, with results compared to the HWT model as shown in Table 3.

As shown in Table 3, compared to the HWT model, the SSIM value increased by 2.46%, the LPIPS value decreased by 3.53%, and the HWD value decreased by 25.41% in terms of font style comparison. This demonstrates that the PSA-HWT model outperforms the HWT model in generating detailed font pixels, aligning with human visual perception, and capturing subtle geometric features of fonts.

To assess the generalization capabilities of the PSA-HWT model, we employed the IAM dataset as the training set and the CVL dataset as the test set. The generated fonts were then evaluated quantitatively using the HWT model, as depicted in Table 4. KID (Bińkowski et al., 2018) is a GAN generation metric used to evaluate the quality of the generated image. A lower KID value indicates better convergence and higher image quality. Comparing our model to the HWT model, our model demonstrates better performance in terms of KID value and HWD value, with a 30.60% decrease in KID score and a 10.69% decrease in HWD score. This highlights the effectiveness of pyramid squeeze attention in our model. Additionally, our model shows a 21.46% decrease in FID value and a 49.27% decrease in GS value compared to the HWT model. This suggests that the handwritten fonts generated by our model closely resemble real handwritten fonts in terms of style and realism. Furthermore, our model demonstrates a significant improvement in generalizability.

We also conducted experiments by training our model using the CVL dataset and calculated the FID values for the test results. The comparative results are shown in Table 5. The FID and GS values decreased by 10.04% and 38.46%, respectively. The HWD and KID scores decreased by 6.35% and 13.88%, respectively.

Analysis

Although the model performs well in generating handwritten fonts, it does not perform well in generating digits (see Fig. 10). This may be due to the following reasons: the content features of the font are extracted from the one-hot representation of the query. Since one-hot encoding can only handle known categories, it cannot effectively encode unknown categories. Therefore, when generating unseen categories in the test set, one-hot encoding encounters issues, resulting in generated content that does not meet expectations.