My Master's Thesis: Satellite-Based Detection of Pesticide Overuse

For our master's thesis in Software Engineering at the University of Southern Denmark, Anton Christensen and I worked on a case from the Danish Environmental Protection Agency (Miljøstyrelsen): could satellite imagery help the agency decide which potato fields to inspect for overuse of pre-harvest desiccants?

The problem

Before harvest, seed potato farmers spray a desiccant that kills the plant above ground, so the potatoes stop growing and the skin hardens. The chemicals can leach into groundwater and harm nearby wildlife, so the agency regulates how much and how often they may be applied. Compliance is checked with soil samples, but there are fewer than a thousand inspections a year (746 in 2019), and the sites are picked at random.

The agency's idea was to use the vegetation index NDVI, which can be calculated from Sentinel-2 satellite data. A desiccant makes a field's NDVI drop, and a field whose NDVI drops faster than normal might have been sprayed too heavily. A list of suspicious fields would be a better basis for choosing inspections than chance.

What we built

A Python system that collects satellite data for every field of a given crop type in Denmark and turns it into vegetation index time series:

• A database populator reads the agency's public GIS shapefile of all Danish fields, filters it by crop type, and stores each field's ID, crop type and polygon in MySQL
• Download workers ask the Sentinel Hub Catalog API which dates have imagery for a field, including cloud cover, and download the raw spectral bands as GeoTIFFs from the Process API
• Processing workers read the GeoTIFFs with Rasterio, check the data mask for invalid pixels, calculate NDVI, NDRE, GNDVI and EVI per pixel, and store the results or render heatmaps and KDE plots
• RabbitMQ distributes the work. A work order (for example "download field data for August") is split into tasks of at most ten days, which are spread across the running workers

Engineering decisions

Raw bands instead of ready-made statistics. Sentinel Hub can compute an index or an average for a field on its own servers, which is cheaper. We downloaded the raw reflectance bands instead, because that let us check the data per pixel, draw heatmaps, and calculate new indices later without downloading everything again. We chose Sentinel Hub over openEO because it is a plain REST API and doesn't tie the project to Python.

A relational database. The agency suggested a time series database. We chose MySQL because the data is relational (fields and their readings per date), and because during exploration we kept updating data and changing the schema, which time series databases handle poorly.

Parallel workers. The workload is I/O-bound: most of the time goes to waiting for the API and the database. Python's Global Interpreter Lock rules out simple multithreading, so we ran several worker processes fed by RabbitMQ. In a test with ten fields and 30 days of data, one worker took 119 seconds and four workers took 32 seconds. The time per day of data stayed at about 4.3 seconds, so adding workers did not slow each one down.

Long-running tasks in RabbitMQ. A download task could run for over an hour, but RabbitMQ drops a consumer that runs too long, and our worker blocked the heartbeat while waiting for a task to finish. We moved task execution to a separate thread with a callback, raised the timeout to two hours, and capped each task at ten days of data to stay within it.

Bounding boxes or individual fields. Instead of one request per field, the system can download large bounding boxes and cut the fields out afterwards. On Funen this was about 29 times faster (163 seconds for 40 boxes against 4,824 seconds for 1,993 fields), but the per-field downloads followed the field borders more closely and gave more accurate readings. We kept both modes, so speed can be traded for accuracy.

Results

The data told a clear story. On a typical seed potato field, NDVI starts rising in mid-May, holds just below 0.8 through June and July, falls to about 0.6 between 7 and 15 August, which matches the timing of a desiccant, and drops to near zero at harvest in October.

The heatmaps show the same drop across the whole field, pixel by pixel:

Seed potatoes could also be told apart from starch potatoes, sugar beets and apple orchards by their NDVI curves alone. Starch potatoes are usually left to wilt naturally, so their NDVI falls later and more gradually.

We developed three methods for scoring possible overuse: comparing the decline of NDVI with that of NDRE (which measures deeper into the plant than the leaf canopy), finding each field's steepest NDVI drop, and comparing the NDVI and NDRE gradients. Each method produces a ranked list of fields.

What didn't work

To validate the scores, we needed ground truth: fields where the agency knew whether desiccants had been used correctly or overused. The agency could not share this data because of GDPR, and the farmers we contacted declined. Our contact at the agency also left partway through the project, so the requirements were never validated with the stakeholder.

Clouds were the other problem. Readings with more than 25% cloud cover were too noisy to use, so some fields had gaps of weeks, often around the desiccation date. The scoring methods ranked the same fields very differently, and without ground truth we could not tell which one was right. The steepest-drop NDVI ranking was the most reliable of the three, and we argued it is still a better starting point for inspections than random selection. But the question of whether overuse can be detected remains open.

Next steps

• Get ground truth, either from the agency's internal data or from test data from desiccant manufacturers
• Compare NDVI and NDRE over the whole desiccation period instead of only the steepest drop
• With enough ground truth, train a machine learning model that also includes weather and cloud cover data