You probably have no trouble recognizing the Confederate flag or the LGBTQ rainbow flag. But the internet’s many subcultures and communication conduits offer an endless supply of new symbols, and you may not know what to make of a Molon Labe patch or an igloo flag when you’re at a protest or watching a news report about one.
That’s why Columbia University researchers have built an app called VizPol that’s designed to recognize such symbols using artificial intelligence and a phone’s camera.
VizPol works along the lines of Cornell University’s Merlin Bird ID, which uses AI to identify birds in your phone’s photos. The underlying AI technology, sometimes called machine learning or deep learning, can’t do everything a human brain can. But it’s really good at recognizing patterns, whether they’re on a Kekistani flag at a far-right rally or a Bohemian waxwing in a tree.
VizPol isn’t for the general public or for content moderators at Facebook or Twitter, at least not yet. It’s geared for journalists who might need to understand obscure or cryptic symbols on people’s flags, T-shirts, signs, tattoos or patches, said Susan McGregor, an assistant professor of journalism and assistant director of the Tow Center for Digital Journalism, a research and teaching center at the Columbia Journalism School.
Understanding those symbols can be important for professionals whose job it is to offer context during a surge in protest activity like armed anti-lockdown protesters entering a statehouse or activists marching against police brutality after the death of George Floyd. An informed assessment can be crucial for making sense of raw videos uploaded to Twitter or weeding out disinformation campaigns on Facebook.
Journalists need to understand what the symbols in a photo or video mean before the images are used. “The problem is that finding out is extremely labor intensive,” McGregor said.
The VizPol app lets users upload a photo and zoom into it, after which it makes its best guess about the symbol. It also includes a reference guide with dozens of symbols.
The project began about a year and a half ago when Nina Berman, a photojournalism professor at the Columbia Graduate School of Journalism, spotted symbols at rallies and protests that could be important. At the Unite the Right 2 rally in 2018, she photographed one soft-spoken woman saying she was protesting threats to her First Amendment rights, McGregor said. Only later did Berman spot the “1488” tattooed on the woman’s arm and realize it was code for a 14-word white supremacist slogan and a reference to the letter H in “Heil Hitler.” (“H” is the eighth letter in the alphabet.)
Now VizPol can recognize lots of other symbols, many of them used by different white-nationalist subgroups. The symbols include the Odal rune, a figure used by a particularly brutal Nazi SS regiment; the black igloo, a drawing used by the extreme-right boogaloo movement; and a colonial era US flag modified to represent the Three Percenters, a gun rights group. VizPol can also recognize the Nazi-stylized flag of Kekistan, a fictional country created by members of the online 4chan forum as a criticism of perceived excessive political correctness.
Brian Levin, director of the Center for the Study of Hate and Extremism at California State University, San Bernardino, said the app will help journalists and researchers, even if it faces tricky issues, such as wrongly identifying or failing to identify a symbol. If the automation works, it will help journalists and researchers keep up with symbols that are changing at internet speed.
“Memes and symbols are constantly being co-opted, regurgitated and abandoned by extremists,” Levin said.
Such symbols are deliberately obscure. “There is an intent to signal to an in-the-know portion of the community,” McGregor said. VizPol also can help clarify which subgroups of a movement a person might belong to, like the Patriot Front, the American Identity Movement or the National Socialist Movement.
VizPol doesn’t focus solely on the imagery of right-wing groups. It also recognizes symbols used by antifa, a loose collection of protest groups that confront neo-Nazis and white supremacists. The symbols used by antifa, short for anti-fascists, are often straightforward, such as three arrows or an image of overlapping red and black flags.
The May 25 killing of George Floyd, an unarmed black man by a white police officer, and subsequent Black Lives Matter protests brought antifa into the news again. That’s made the movement’s symbols relevant to the VizPol database, and spurred the developers to look for more to add, McGregor said.
The researchers have made the app available to more than 100 people for testing, says Ishaan Jhaveri, a research fellow who along with graduate student Bhaskar Ghosh wrote the app. Postdoctoral researchers Svebor Karaman, Xu Zhang and Guangxing Han built VizPol’s machine learning system.
One particular challenge for VizPol is finding enough images to train its AI symbol recognition system, particularly when those symbols in the real world might be distorted by T-shirt folds or partially hidden by another sign.
To get around this problem, the researchers generated synthetic versions of the images using the Blender 3D image generation software. For example, the team mapped symbols onto simulated rippled flags. The team is also working on an update to automatically detect symbols, using training data in which the researchers added them to photos in a UCLA collection of protest photos.
People using the app also contribute to its utility. After a user takes a photo, the app analyzes it and suggests possible identifications. There’s also a web interface useful for photo editors sitting at their PCs. Users can share feedback about whether the app judged correctly. And they can upload images of their own to update VizPol’s symbol recognition abilities, though Columbia’s researchers vet each submission and add descriptions of the symbol’s meaning.
The app is in testing with a relatively small group right now to protect against problems of misuse, like uploading bogus photos or trusting VizPol’s judgment too much in a world where symbols can have lots of meanings, some of them benign.
“We are hoping to work with news projects and organizations,” including photo agencies that distribute a lot of images and write captions, McGregor said. “We want to help journalists do what they’re doing. That’s a benefit to people everywhere.” ♦
(The story originally published by CNET)
