With more than 500 masks on display and limited space to provide information alongside each piece, MAPI was looking for an alternative that would allow visitors to learn the name, details, and context of what they were viewing, even when they didn’t have a guide.
The solution was to add a new feature to the museum's existing mobile app: visitors can point their phone's camera at a mask, identify it, and access real-time information about the artifact.
“What it does is give users who don’t have a guide the ability to focus the mask with their cell phone and receive that information in real time,” Golubei explained.
The proposal is part of a process of technology integration that MAPI has been developing for several years, which includes initiatives related to the metaverse, a mobile app, and accessibility tools such as audio guides and information available in Spanish, English, and Portuguese.
Artificial intelligence to recognize more than 500 masks
To develop object recognition, Golubei used CLIP, an artificial intelligence model capable of transforming images into numerical representations that can then be compared.
“When people imagine artificial intelligence comparing images, they think it compares them pixel by pixel, just as we would with the naked eye. In this case, the tool converts the photo into a numerical vector,” he explained.
These vectors are compared to determine which images are most similar and, in this way, identify the mask photographed by the visitor.
However, having the technology was only the starting point. The biggest challenge was adapting it to a specific collection of hundreds of pieces.
The process involved photographing the masks, cataloging the images, discarding those that were unusable, and capturing images from different angles and under various lighting conditions. Using this material, it was possible to progressively improve the system’s performance.
Currently, when a visitor takes a photo of a mask, the feature displays the match it considers most likely along with four other alternatives. According to Golubei, the correct piece is among those five options in approximately 90% of cases.
All of that was achieved by training the tool. That was the biggest challenge.
When the user doesn't do what the developer expected
The tests at the museum introduced another variable into the project: people’s actual behavior. The initial evaluations were conducted on a computer, using images to simulate the system’s operation. Next came tests in the MAPI galleries using various devices, including both iOS and Android devices. But it was only when the functionality was put into the hands of visitors that unforeseen situations arose.
The interface displayed a box to guide users when photographing the mask. It seemed natural for the mask to be placed within that space and for the photo to be taken from the front. The visitors did something else.
“Often, what seems intuitive to one person isn’t intuitive to others. People don’t worry about standing in front of the mask and take photos from different angles,” he said.
This observation led to the addition of new photographs to the reference database and to preparing the tool to recognize the parts under conditions other than those initially anticipated. Efforts were also made to better explain how it works within the application itself.
The experience taught us a lesson that Golubei sums up from the perspective of a product developer: “It’s not the user’s fault, but ours for failing to anticipate that use.”
Thus, the tests did not serve merely as a final validation step, but became part of the development process itself: observing how people use a technology made it possible to modify and improve it.
An Unexpected Use
The tests also revealed behavior that had not been originally anticipated. Some visitors, especially children, began using the camera to take pictures of each other instead of pointing it at the exhibits.
The system did exactly what it was designed to do: it converted the image into a numerical representation and searched its database for the mask with the highest similarity. Since the database contains only masks, when a person was photographed, the system found the mask that matched it most closely.
What began as an unintended use ended up becoming a fun activity: discovering which mask in the collection bears the greatest resemblance.
AI as a tool, not as an end in itself
The project also made it possible to apply artificial intelligence to a specific use case. Golubei admits that he was initially skeptical about using it, but now it’s part of various tasks in his daily routine, from programming to staying organized.
In the case of MAPI, it is worth noting that the decision to use it was directly linked to the problem they needed to solve.
I always say that artificial intelligence isn't an end in itself—it's just another tool. If artificial intelligence can solve the problem at hand, then it's worth using.
In this case, the technology made it possible to create a different experience for visitors while also addressing a specific limitation of the museum.
From the Classroom to a Real-World Project
Golubei's first exposure to artificial intelligence came during his studies in systems engineering.
“My first exposure to artificial intelligence was in Machine Learning, a college course. Likewise, the first projects I worked on with the help of AI were also at ORT, and it just kept growing from there,” he recalled.
The MAPI experience allowed him to apply the knowledge he gained during his training to a real-world scenario: starting with a problem, evaluating which technology could solve it, developing a solution, and testing it with real users.
Golubei believes that these types of tools can also open up new possibilities for Uruguayan cultural spaces. Although there are similar initiatives in museums in other countries, he believes that their implementation at the local level still has a long way to go.
Learn more about Systems Engineering