Sentinel-2 Deep Resolution 3.0

The article documents the performance of S2DR3 — a major update of our 12-band 10× Single-Image Super-Resolution model that upscales all 12 spectral bands of a single Sentinel-2 L2A (or L1C) scene from the original 10, 20 and 60 m/px spatial resolution to the target 1 m/px.

S2DR3 is specifically optimised to preserve subtle spectral variations of soil and vegetation across all 12 multi-spectral bands of Sentinel-2 L2A and is capable of accurately reconstructing objects and textures with individual spatial features down to 3 m.

The S2DR2 model was released in late 2022 and found multiple useful applications, where the main demand has been driven by the evaluation of real-time data for precision farming including the detection of high-precision field boundaries. The peak demand for S2DR2 data was reached in June 2023, when 12 million sq. km of Sentinel-2 imaging data was processed in a single month.

In my previous S2DR2 post, I reflected on the substantial level of scepticism towards ANN-based super-resolution techniques that I had encountered from the Earth Observation community. I believe such attitudes are slowly changing as AI-based applications and practical solutions, such as ChatGPT, become more commonplace. I also think that a fair dose of healthy scepticism is justified in the presence of purely generative methods such as DALL-E and Midjourney.

The objective of this post therefore is to continue to offer evidence and validation in order to establish S2DR3 as a viable source of high-revisit-rate, radiometrically calibrated, high-resolution satellite imaging data that is suitable for practical analytical applications.

A number of similar solutions have emerged in recent months, suggesting the importance of the subject to the Earth Observation community and the amount of effort that is being applied to the problem by both academic and commercial players. Ultimately, the utility of such methods is confirmed by the emergence of practical applications such as the aforementioned high-precision delineation of agricultural fields.

In our new version S2DR3 we have focused on the spectral fidelity of the super-resolved data. The detailed analysis of the attained performance, both spectral and spatial, is presented in the next section; however I would like to argue that the high spectral fidelity of S2DR3 combined with the superb quality of the source Sentinel-2 L2A data will make S2DR3 an ideal source of remote sensing data for environmental monitoring, as well as Measurement, Reporting and Verification (MRV) applications for the burgeoning sector of carbon credits.

S2DR3 was specifically optimised to preserve subtle variation in spectral characteristics of vegetation across all 12 bands of Sentinel-2 L2A multi-spectral imaging data. Consequently, our model is capable of generating valid results for any spectral analysis pertaining to soil and vegetation at 10× the resolution of the original Sentinel-2 data.

Network architecture

The underlying ANN architecture of our method is not dissimilar to other SISR state-of-the-art methods. To the best of my knowledge, many such solutions invoke a fully convolutional network, such as the excellent RRDBNet proposed in ESRGAN or any of its variants.

It should be said, however, that in my experience four key factors have an impact on the attainable performance of the model, in the following order of decreasing importance:

• Training data
• Loss function
• Hyperparameters
• Network architecture

Qualitative performance and known limitations

Spectral fidelity: The development of the new version of our Sentinel-2 super-resolution model was largely motivated by the objective of further improving the spectral accuracy of the super-resolved data. As described in our previous post, this is particularly challenging because the corresponding high-resolution multi-spectral ground truth imaging data does not exist and had to be artificially synthesised for the purposes of model training. Likewise, the accuracy of the end result is difficult to evaluate exactly due to the lack of proper ground truth. Consequently, our assessment is based on comparing spectral characteristics of the downsampled output to the original input.

Our rationale is that the model is tasked to reconstruct morphologically and semantically coherent spatial details while preserving the spectral characteristics of the source data. Following our comprehensive assessment of the results we concluded that S2DR3 achieves an outstanding spectral fidelity, which makes it suitable for a wide range of important practical analytical applications, as further demonstrated by quantitative results presented in the following section. It should be said, however, that some deviations from the source spectral characteristics have been observed, particularly in datasets with dense and unusual objects that were not well represented in training datasets.

Semantic integrity and hallucination: Throughout our extensive performance evaluation, which included randomised sampling of global Sentinel-2 data, we did not encounter any obvious instances of “hallucination”, namely generation of objects, such as buildings or trees, that are not rooted in ground truth. Nevertheless, the model can often fail to reconstruct small objects, typically < 5 m across, particularly in cases where such objects have a low spectral signature against the background — in other words, their colour is similar to the colour of the background.

Geometric fidelity, artefacts and distortion: Small-scale distortions of objects are common and particularly prominent in urban environments comprised of complex, high-density patterns of regular geometric shapes characteristic of man-made objects. In the context of agricultural fields, the model demonstrates impressive capability to reconstruct regular periodic patterns even where the scale of individual structural features is under 3 m (< 0.3 of a single Sentinel-2 pixel), illustrated in fig. 4. This remarkable capability has, of course, its own limits, and the accuracy of reconstructed textures may vary depending on the scale of the underlying features and their respective spectral characteristics.

Quantitative performance evaluation

We evaluate two separate aspects of the achievable performance. Firstly, the accuracy of the spatial reconstruction is evaluated across the RGB bands using high-resolution ground truth, and RMSE, PSNR and SSIM metrics. Secondly, the preservation of spectral characteristics across the 12 spectral bands of Sentinel-2 is evaluated using the R² score between the original and upscaled pixels. The main purpose of this step is to ensure that the model does not introduce spectral distortion or biases into the original spectral data.

We henceforth present a comprehensive collection of examples generated across a broad range of locations, seasons and types of terrain that demonstrate and document the achievable performance, as well as the expected limitations of the model. All testing samples were selected to be distinctly different from the data that has been used during training. More specifically, the samples were taken from Sentinel-2 scenes that were not used in training.

Each example contains the following:

• Sentinel-2 True Colour Image (TCI: B04, B03, B02) 40 × 40 px at 10 m/px (top, left);
• S2DR3 TCI 400 × 400 px at 1 m/px (top, centre);
• Ground truth RGB 1 m/px (top, right, © Google Earth);
• Sentinel-2 Infra-Red Pseudo-Colour Image (IRPCI: B12, B08, B05) 40 × 40 px at 10 m/px (bottom, left);
• S2DR3 IRPCI 400 × 400 px at 1 m/px (bottom, centre);
• Scatter plot of pixel values across all 12 bands, Sentinel-2 L2A versus S2DR3. Different colours in the scatter plot represent different spectral bands. The plot also contains accuracy evaluation values, including RMSE, PSNR, SSIM and R².