The combination of different techniques will provide police authorities with a powerful surveillance tool. Persons can then be located in intercepted telephone conversations. The technology uses extensive metadata that people leave behind in public places or the Internet.

In “Roxanne”, the European Commission is developing a procedure to trace individuals by their spoken word. The platform is designed to process large amounts of data and combines audio files with other metadata. To analyse networks of suspicious persons, the platform uses facial recognition, for example, to process video from public surveillance or downloaded from providers such as YouTube and Facebook.

In the project, the Commission aims to improve police investigative capabilities, in particular in major criminal cases. The “Roxanne” consortium includes 24 European organisations from 16 countries, half of which are law enforcement agencies and interior ministries. Israel is the only third country by sending experts from the Ministry of Public Security to the EU project. Other participants include Interpol and the arms company Airbus.

Voices of men 

The technology is to be used for telecommunications interception. The police will thus search for specific persons in telephone conversations. IP telephony, for example via messenger services, is also to be intercepted with the help of “Roxanne”.

A fortnight ago, the participants underwent a first test using voice samples from volunteers. According to a blog entry, this was successful, but only men’s voices were tested.

The Czech police have also demonstrated the effectiveness of the technology in drug trafficking investigations. In large amounts of digital data the relationships of members of a criminal network were visualised.

“Breakthrough” project

With the “Speaker Identification Integrated Project” (SIIP), the EU Commission has funded a similar method for identifying “criminals and terrorists” by their voice until 2018. Interpol was also involved in this project at that time, as were the German Federal Criminal Police Office and Airbus.

The project description of “Roxanne” states that the approach used in SIIP, which is based only on the analysis of voice biometrics and does not process any other metadata, has proven to be less successful. Moreover, using the best speaker identification technology currently available would produce too many false alarms at 1%.

The participants therefore expect a “breakthrough” from “Roxanne” and the network analysis used in it. For this purpose, several information that may arise during police investigations will be processed.

Geodata and time stamps

Because suspects often use prepaid SIM cards, “Roxanne” wants to identify the people behind them with the help of previously used telephone and IMEI numbers as well as geodata and time stamps. Content from earlier intercepted calls is also compared in order to improve the identification of the sought-after speaker by means of identical phrases. The audio data are transcribed using voice-to-text software. From this, further metadata is obtained, including places, people and companies. It should also be possible to analyse multilingual audio files.

If it is known that some participants in a communication rarely speak, this characteristic can be searched for specifically. The software should also automatically determine the age, gender and accent of the participants. This should help to limit the targeted speech signal to certain persons. As an example, the project website mentions the identification of a pimp who has directed his calls mainly to people in the age category over 20 years.

As an additional metadata collected in “Roxanne”, the “favourite car” is given as an example. It could be identified by the driving sounds during the call. Suspects might then be identified even if they change their mobile phones.

Complaint from the EU Data Protection Supervisor

The seven million euro project will end in two years, when “Roxanne” wants to present a prototype. The technology should be able to be integrated into crime management systems of the police. The common IBM i2 Analyst Notebook, on which many national systems are based or offer corresponding interfaces, is mentioned.

According to the website of “Roxanne”, the prototype should meet the requirements of European data protection regulations. However, it is doubtful whether the policial dragnetting is allowed by organisations such as Interpol.

Europol is also participating in “Roxanne” as an advisor. The EU Police Agency receives extensive data stockpiles from the Member States via a “Large File Exchange Server”, including from telecommunications surveillance or from confiscated telephones and data carriers. Their analysis inevitably leads to the identification of persons who are not suspected of any crime, which is contrary to the current Europol Regulation. The European Data Protection Supervisor therefore admonished the Police Agency two weeks ago.