Source: Academic Headlines

Author: Yan Yimi Editor: Scholar Jun

Today, artificial intelligence (AI) is not only increasingly being applied in the fields of science and business, but is also beginning to make its mark in the field of art.

As people marvel at the ever-emerging AI technologies, a new AI mural restoration technique has appeared.

Recently, in order to solve the "restoration difficulty" of the ancient Chinese mural masterpiece Yongle Palace murals, a research team from Shanxi Datong University, University of Malaya, and Dali University proposed an AI model capable of restoring giant murals - 3M-Hybrid.

It is reported that compared with the best model in four representative Convolutional Neural Network (CNN) models, the model has improved SSIM and PSNR by 14.61% and 4.73% respectively in the restoration of regular-sized murals. In addition, it also achieved good results in the final restoration of giant murals.

The related research paper, titled "A 3M-Hybrid Model for the Restoration of Unique Giant Murals: A Case Study on the Murals of Yongle Palace," has been published on the preprint website arXiv.

How does AI restore large-scale murals?

The Yongle Palace murals are located in the Yongle Palace (also known as Dachunyang Wanshou Palace) in Ruicheng, Shanxi Province. The most valuable artistic feature of the Yongle Palace murals is the exquisite large-scale murals. The entire mural covers more than 1000 square meters and is painted in the Wuji Hall, Sanqing Hall, Chunyang Hall, and Chongyang Hall.

As a precious cultural heritage, the Yongle Palace murals represent artistic masterpieces in the history of Chinese painting. However, due to a lack of maintenance over a long period of time, these unique murals have developed many defects, making mural restoration an urgent task.

Compared with manual restoration techniques, digital restoration methods are more efficient and reversible, especially the image restoration technology based on deep learning has achieved significant success. However, the literature on deep learning-based mural restoration mainly focuses on Dunhuang murals or other regular-sized murals, lacking specific research on the restoration of Yongle Palace murals and similar giant murals.

Compared with other research on mural restoration, the restoration of the giant Yongle Palace murals faces two main challenges: 1) The Yongle Palace murals are scarce and have a unique style; 2) The murals are of a huge size, and the model's proficiency in restoring different types and sizes of defects is limited.

According to the description in the paper, the 3M-Hybrid model proposed in this study can effectively restore the Yongle Palace murals. "3M" refers to three key strategies: multi-frequency, multi-angle, and multi-scale, while "Hybrid" refers to a hybrid CNN-VIT network.

First, the research team divided the giant murals into regular-sized parts for restoration, and then reassembled the restored parts into the original size. In order to enable the regular-sized mural restoration model to effectively handle various types and sizes of defects with limited image data, the research considered two aspects: optimizing training data and improving model structure.

To achieve better restoration results, the study used independent networks specifically designed to learn and extract these high-frequency and low-frequency signals, thereby enhancing the feature learning and restoration capabilities within these specific frequency ranges. The frequency-based training method enables the model to effectively handle defects of different scales and types.

In terms of model structure, the study integrated Convolutional Neural Network (CNN) with pre-trained Vision Transformer (VIT) to enhance the model's feature extraction capability.

In addition, when restoring giant murals, the basic cutting method can lead to gaps and structural distortion when restoring defects of super-large size. To address this issue, the research team adopted a multi-angle strategy to reduce gaps and used a multi-scale approach, combining cutting and scaling methods. This ensured precise restoration while enhancing the overall structure extraction of the murals, solving the problem of multi-scale defects.

From a visual perspective, the model has shown remarkable results in restoring free dust-like, free gel-like, and free linear mask. In addition, the restoration results of free shape block mask demonstrate the preserved structural integrity and credible texture. The 3M-Hybrid model has been proven to be a feasible method for restoring these unique and giant murals.

However, the study is not without its imperfections.

First, the proposed method relies on multiple experiments to select the best values for the fusion weights of the three scales. However, considering the countless possibilities covered by the weight settings and the limited number of experiments, this method may not be precise enough. Therefore, the weight values determined based on experimental results can only guarantee relatively favorable final results.

Second, the evaluation metrics used in the study are not objective enough. The four evaluation metrics currently used do not comprehensively assess the image structure and usually cannot accurately reflect human perception and evaluation of the images.

However, it is undeniable that the study explores the application of deep learning in the restoration of giant murals, particularly focusing on using deep learning techniques to restore the Yongle Palace murals, representing the first attempt to explore deep learning restoration methods for large-scale artworks.

Furthermore, in improving regular-sized image restoration models, the study comprehensively improved from both data and structure perspectives. This provides new insights for future research on restoring unique small datasets.

Helping to Extend the Life of Cultural Relics

In recent years, people have also witnessed the wonderful combination of AI technology and historical relics.

In 2020, a Weibo user named "Dagu Spitzer" used AI technology to restore the 1920 black and white image data of Beijing published by People's Daily four years ago, completing tasks such as coloring, restoring frame rate, and expanding resolution.

In 2021, Tencent Multimedia Laboratory also collaborated with the Dunhuang Academy to use deep learning methods to analyze data on Dunhuang mural diseases, developing an efficient AI mural disease identification tool, and providing immersive remote consultation technology, using 4K ultra-clear image quality to display 360-degree images of cave interiors and details of cultural relics, achieving barrier-free remote relic consultation.

In June of this year, at the main city event of Heritage Day held in Chengdu, the theme forum "Cultural Relics Protection and Utilization, Cultural Confidence and Self-improvement" showcased the AI technology-completed simulation splicing effect of the Sanxingdui site.

The various applications of AI technology in the field of relic restoration are exciting. In the future, it is hoped that AI technology can further advance in the protection and restoration of cultural relics, helping to extend the life of cultural relics.

Paper link:

https://arxiv.org/abs/2309.06194

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。