Can AI really give us a glimpse into lost masterpieces?
In 1945, a fire Three of Gustav Klimt’s most controversial paintings claimed. Commissioned by the University of Vienna in 1894, the “College Paintings” – as they became known – were different from any of the earlier Austrian allegorical works. Once he introduced it, critics erupted about their dramatic departure from the aesthetics of the time. University professors immediately rejected her, and Klimt withdrew from the project. Shortly thereafter, the works found their way into other groups. During World War II, they were placed in the castle north of Vienna for safekeeping, but the castle burned down, presumably because the paintings went with them. All that remains today are some black and white photographs and writings from that time. Yet I stare at them.
Well, the boards are not the same. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to bring to life Klimt’s lost work. It was a painstaking process, starting with these black and white photos and then incorporating artificial intelligence and dozens of information about the painter’s art, trying to recreate what those lost paintings might look like. The results are what Smola and Wallner showed me – and they were even surprised by the colorful, artistic images produced by the AI.
Let’s make one thing clear: No one is saying that this AI is restoring Klimt’s original work. “It’s not actual color re-creation, it’s image recoloring,” Smola is quick to note. “The medium of photography is really an abstraction from the real business.” What machine learning does is provide a glimpse into something that was thought to be lost for decades.
Smola and Wallner find this interesting, but not everyone supports filling in these blanks with AI. The idea of machine learning to recreate lost or destroyed works, such as the faculty boards themselves, is controversial. “My main concern is the ethical dimension of using machine learning in the context of conservation, due to the sheer volume of ethical and moral issues that have plagued the field of machine learning,” says Art Conservative Ben Venou Radin.
To be sure, the use of technology to revitalize human artwork is fraught with thorny questions. Even if there was a perfect AI that could figure out what colors or brush strokes Klimt might have used, no algorithm could generate a composition target. Discussions about this have been raging for centuries. In 1936, before the destruction of Klimt’s paintings, essayist Walter Benjamin argued against mechanical reproductions, even in photographs, saying that “even the most perfect reproduction of a work of art is missing one element: its being in time and space, which is unique. Being in the place in which happen in it.” Benjamin wrote in this Artwork in the age of mechanical reproduction, is what he called “aura” work. For many art lovers, the idea of a computer reproduction of this intangible item is implausible, if not outright impossible.
However, there is still a lot to learn from what AI can do. College paintings were central to Klimt’s development as an artist, and are a crucial bridge between his earlier traditional paintings and more radical later works. But its full-color look has remained shrouded in mystery. This is the mystery that Smola and Wellner were trying to solve. Their project, curated by Google Arts and Culture, wasn’t perfect copies; It was about providing a glimpse of what was missing.
To do this, Wallner developed and trained a three-part algorithm. First, the algorithm was fed hundreds of thousands of art images from the Google Arts and Culture database. This helped in understanding objects, artwork, and composition. Subsequently, she was specifically taught in Klimt’s paintings. “This creates a bias towards its colors and decorations during the time period,” Wallner explains. Finally, the AI was fed color clues to specific parts of the paintings. But with no color cues for the plates, where did these clues come from? Even Smola, an expert on Klimt, was surprised by how much detail the writings of that time revealed. Because the paintings were deemed too sloppy and bizarre, critics tended to describe them in detail, down to the artist’s color choices, he says. “You could call it a mockery of history,” says Simon Rayne, Project Program Manager. The fact that the paintings caused a scandal and were refused puts us in a better position to get them back because there is a lot of documentation. And those kinds of data points, if fed into the algorithm, create a more accurate version of how those plates likely looked at that time.“
The key to this accuracy lies in pairing the algorithm with Smola’s expertise. His research revealed that Klimt’s work during this period tended to have strong and consistent patterns. A study of the paintings found before and after the college paintings provided evidence for the recurring colors and motifs in his work at the time. Even the surprises Smola and Wallner encountered are backed by historical evidence. When Klimt first exhibited his paintings, critics noted his use of red which, at that time, was a rarity in the artist’s painting. but Three ages for a womanPainted shortly after the college paintings, and boldly using red, Smola believes it’s the same color that caused a stir when it was first seen in college paintings. Writings from that time also rave about the stunning green sky in another college painting. Pairing these writings with Smolla’s knowledge of Klimt’s green color palette, when fed into the algorithm, is what produced one of the first surprising images of artificial intelligence.