The Hamburg police have been researching facial analysis software for several years, which was then used for the first time after the G20 summit. The technology accesses the nationwide INPOL file for criminal offenders maintained by the Federal Criminal Police Office. The detection rates are meagre, but the system is still to be used permanently in Hamburg for the “processing of major events”.

The face analysis software used by the Hamburg Special Commission “Schwarzer Block” (“Black Block”) has led to the identification of only three people. This was written by the Hamburg Senate in response to a question by Christiane Schneider, a member of parliament. The Special Commission, which was set up after the G20 summit, uses the face recognition software “Videmo360” from the company Videmo, which processes all common image and video formats.

The photos of unknown persons were compared in 95 cases with the face recognition system “GES” by the German Federal Criminal Police Office (BKA). It accesses the INPOL database, which stores people who have been identified and photographed for investigations. As far as we know, around four million photographs are stored there. According to the Senate’s answer, two persons have been identified with the help of the “GES”. It is unclear which database the third person has been named after a query.

170 officials in the Soko “Schwarzer Block”

The police conducted a total of 341 searches with “Videmo360”, 167 of which concerned already known suspects. The software should help them to connect them to further crimes. In 147 cases, further police investigations had been “supported” by the technology. In the case of unknown persons, “Videmo360” is used to search for further visual material in order to identify them with the help of scene-savvy officials. Hundreds of photos of still unknown persons have finally been published by the police in a public manhunt.

In the Special Commission “Schwarzer Block” up to 170 officers are temporarily investigating. Details of the facial analysis software used there were provided by Criminal Director Jan Hieber at a meeting of the Special Committee “Violent rioting around the G20 Summit in Hamburg”. Accordingly, the system has been available since March this year. However, the Hamburg State Criminal Police Office has been researching the semi-automated evaluation of image and video material with the BKA and the Fraunhofer Institute FKIE for several years now. The face recognition software Videmo360 is also used in this project “PERFORMANCE”.

Information portal at the BKA

For the investigations after the G20 summit, the Hamburg police launched an information portal, which is installed on a server at the BKA. Following the example of the attack at the Boston marathon in 2013, the BKA launched such a portal with an upload function to which witnesses handed pictures and videos for the first time. In the case of the Hamburg system, the data is retrieved from the LKA using a VPN connection.

Meanwhile, the police have more than 100 terabytes of image and video material on the G20 summit. In terms of size, most files (94.05 TB) come from public transport, including suburban and underground stations, the main train station, buses and metros. Private individuals and companies have also uploaded tens of thousands of files to the police information portal. Photos and videos backed up by Soko on deliverdthe Internet and in newspapers were also processed. The Soko also used four terabytes of police images and video material. Finally, the officials also visited crime scenes to find witnesses and ask them about possible video evidence. According to Hieber, they have been “quite successful”.

Little helpers from Microsoft and ESRI

Prior to analysis with facial recognition software, the material had to be viewed and stamped with a time and place stamp. “Videmo360” cannot process this geodata, the police used components of the US software manufacturer Environmental Systems Research Institute (ESRI) for which service contracts already exist within the framework of the Hamburg Police GeoPortal (“Portal for ArcGIS”). Further georeferenced search functions were provided by Microsoft via a search tool named “VIDoGIS”. It allows to define an environment to search for other existing image data. According to a presentation at the annual BKA Conference, Microsoft software can index files, compare multiple files and delete duplicates.

According to Criminal Director Hieber, the geolocalization of the files took a lot of time. Initially, it took several months for the infrastructure to be ready for evaluation. The investigations had been supported “by a very high number of persons” from the state and the federal government. Among other things, officers of the state police were seconded to the Soko for several weeks.

Reading took seven weeks

In the end, around 17 TB of usable material (15,157 videos and 16,480 images) were recorded in the “image and video mass data registration” evaluation environment used for face recognition. The server was set up by the police service provider Dataport. According to the Senate, the process took seven weeks, since the software had to “calculate the faces present and recognized in each individual file using an algorithm”.

Despite the extremely poor detection rate, the evaluation with the help of face recognition and geolocalization is highly praised by the police. Hamburg police president Ralf Martin Meyer described the technology at the G20 Special Committee as “a conceptual development of not inconsiderable scale”. This significantly increases the risk of detection of acts of violence. Criminal Director Hieber calls the procedure a “completely new standard in evidence”. In fact, many of the video evidence was brought to court, creating draconian and therefore controversial judgments.

Software with real time function

The police now want to use the face recognition software outside of the G20 summit. The video analysis system set up by the Special Commission “Schwarzer Block” is to be used by the Hamburg State Criminal Police Office for “processing major events”; a corresponding concept is currently being developed there. In addition to prosecution after demonstrations, this is also likely to affect football matches. The system may even be technically extended. “Videmo360″ has a realtime function” when the software can access connected cameras.

The police see the use of “Videmo360” as a “mere tool” for viewing video recordings “lawfully obtained by the police”. A public prosecutor’s order has been requested for each individual search. However, the Hamburg Commissioner for Data Protection and Freedom of Information Johannes Caspar has a problem with this. A month ago, he wrote to the police that he had checked the technology under data protection law and found it illegal. The facial analysis software therefore interferes with the basic right to informational self-determination of those affected.

Image: All rights reserved Videmo