Matthias Monroy

High-resolution images from earth observation could help with non-governmental sea rescues in the Mediterranean. However, these have to be purchased from commercial providers, because openly accessible images from EU satellites are of low quality. An initiative now wants to enrich this data with other sources and evaluate it with algorithms.

The EU border agency Frontex uses satellites to stop unwanted migration to Europe. As part of the EUROSUR surveillance system, Frontex has set up various services to automatically detect ships and boats carrying refugees with the help of aircraft, drones and also satellites. Frontex then informs the relevant coast guards of the sighting; North African authorities then return the boat occupants to countries such as Libya or Tunisia. The satellite data comes from the Sentinels of the EU’s „Copernicus“ earth observation programme; Frontex also buys higher-resolution images from private providers. In addition to Frontex, the EU’s maritime safety agency EMSA also maintains a satellite-based monitoring system, „CleanSeaNet“.

The German association Space-Eye is now also experimenting with the use of satellite data. The information is intended to help rescue organisations take on board people in distress at sea and bring them to a safe harbour. The association’s satellite working group consists of a dozen scientists and students. For the interview, i spoke with development engineer Elli Wittmann, Steffen Merseburg and Jonathan.

You want to use data from satellites to detect boats on the Mediterranean. Which images do you use for this, and which services or companies do they come from?

For the development of the system, we work with data from the commercial provider Planet and, in the future, also with data from the „Copernicus“ programme of the European Space Agency. Our project is currently still in a purely scientific stage, and we are working together with various universities and colleges. Our vision is to be able to use as many different data sources as possible. This is also necessary in order to obtain regular images at all.

„Copernicus“ provides images from optical and radar-based satellites, Frontex also uses the service. Can you explain the difference in the technology?

Optical satellites take pictures of the Earth and use sunlight to do so. Radar satellites, on the other hand, emit microwaves, so they actively illuminate the area. This has the advantage of being undisturbed by clouds and being able to take pictures at night. On the other hand, these images have very different characteristics compared to optical images and can be more difficult to interpret. We are also working on integrating radar images into our application.

The „Copernicus“ images are comparatively low resolution. How expensive are the services of commercial providers? And do you order them by the day for a specific region?

Unfortunately, satellite images currently have either a good spatial resolution or a large coverage. But we would need both, which means that in addition to the data from the „Copernicus“ sentinel satellites, we would have to buy further images from several satellites at the same time. But that’s quite expensive, we’re talking five figures here. With the company Planet, for example, you buy a certain quota – that is, a number of square kilometres per month, which you can then download from the archive. These are the images with relatively good resolution (3 by 3 metres per pixel). For even better resolution (0.7 by 0.7 metres per pixel), you would have to task the satellites, i.e. commission a desired region. This is then much more expensive and only possible for very small areas. For comparison: the typical images in Google Maps, which mostly come from aeroplanes, have a resolution of a few centimetres per pixel.

How is the pixel value to be understood? If a rubber dinghy is about ten metres long, is it only visible as one or two small squares on the images taken by the satellites of the EU Earth Observation Programme? How do you then want to determine the probability that it is a boat with refugees?

A value of 3 by 3 metres indicates that one pixel from the image corresponds to an area of three by three metres on the ground. So the ten-metre-long and four-metre-wide inflatable boat can at best be recognised with about three consecutive pixels. There are also satellites such as „World-View-3“ from DigitalGlobe with a resolution of 0.31 by 0.31 metres per pixel. Unfortunately, however, this satellite can only take a small section at a time and cannot scan the entire earth. The inflatable boat would then be about 10 by 30 pixels in size and easily recognisable as such. Boats of this kind are not made for the open sea and are in principle always in distress as soon as they leave the coast. With larger boats, it will not be possible to say exactly. However, our data nevertheless contributes to an overview of the situation.

You talked about the need to observe a large area in order to be able to work carefully. But satellites orbit the Earth, so is such coverage even possible with the current commercial fleet? And wouldn’t geostationary satellites be more useful?

No single provider currently offers the kind of coverage we need. We would like to have several images per day of the entire Mediterranean. The best offer we know of provides a maximum of one complete image per day. Hence the approach of integrating as many different satellites as possible into the system. If we combine the offers of several providers, we can achieve better temporal coverage. Geostationary satellites are not an option for us because they are too far away from the earth and, with several kilometres per pixel, have a too low resolution.

How long does it take for the satellites whose images you want to use to orbit the Earth and arrive back over the Mediterranean?

Normally, a satellite needs several days to be back over the same point. Most providers therefore have a whole fleet consisting of several satellites. This allows them to achieve better temporal coverage.

Another problem is the so-called downlink, i.e. the transmission of the recorded data. What is the challenge and how does this download work when the satellite is out of sight?

Since we don’t have our own satellite yet, we don’t have to worry about the downlink. The respective providers are responsible for that. The only problem for us is that it often takes several hours or even days. This can be the case if the satellites have no contact with ground stations, have too little energy to transmit or cannot transmit and record at the same time. Then, of course, the images have to be pre-processed and loaded into the database so that we can use them. This can be done quickly with some satellites, but unfortunately it takes time with the data we have been using so far.

You also want to automate the detection of boats so that not all satellite images have to be looked through by evaluators. How will this work and what is the „neural network“?

Since we are dealing with such a large amount of data, it is virtually impossible to examine all the data „by hand“ or „by eye“. Therefore, we train various algorithms that can recognise particular anomalies in the images. To do this, we also use neural networks, i.e. algorithms that have learned what a boat looks like on a satellite image based on many example images. These discoveries of the algorithms can then be verified by humans.

At rc3 you reported that the system has already been tested in the Mediterranean. How did that work?

In a first step, we wanted to know whether we could really recognise the small boats in the pictures. To do this, we searched for and found a sailing boat belonging to one of our fellow campaigners on the satellite images. Secondly, we looked for and found rescue missions by other NGOs. Médecins Sans Frontières, for example, has published some of the missions of their rescue ship on an interactive map with location and time information. We then only had to search the archive of the satellite provider to see if pictures were taken at the times. We were able to identify over a hundred rescue missions on the images and use this data, among other things, to verify our algorithms.

To be as accurate as possible, other data can be integrated, including the transponders that large ships regularly use to transmit their identification number, location and destination. What data sources would be useful for you?

We plan to incorporate weather data into our model at various points. For example, in heavy seas, image recognition is less accurate. So to estimate the accuracy of our information, we need to know how high the waves are. In addition, the wind and current conditions of the sea can be used to roughly predict where a boat that is unable to manoeuvre will drift.

You have said that you are currently working mainly scientifically, and you have already carried out initial tests with a sailing boat. How can interested people support you, what are your next steps?

The more people are aware of the conditions at the European borders, the better. We can therefore only call on people to inform themselves about the border policies of Germany and Europe and to become active.

We are all working on a voluntary basis to gather more reliable information. In the near future, we plan to analyse historical data in order to reappraise past cases and to test our system extensively in the process. But to buy and analyse satellite data, for example, we need money. A donation to Space-Eye can directly support the project.

Image: With the help of earth observation, boats with refugees, but also pushbacks in the Mediterranean can be tracked (Bellingcat).