AI Datasets for Space-Based Computer Vision Research

Computer vision datasets for space and satellite imagery: which fit which task, the measured gap from COCO pretraining, quality checks and licence limits.

AI Datasets for Space-Based Computer Vision Research
Written by TechnoLynx Published on 21 Feb 2025

Public computer vision datasets for space and satellite work were each built for specific tasks, so start by matching the dataset to your task. For spacecraft detection and part segmentation there is a 3,117-image dataset of satellites and space stations. SPEED+ covers spacecraft pose estimation. xView, DOTA and SpaceNet cover object detection and mapping in overhead imagery, EuroSAT covers land-use classification from Sentinel-2 imagery and BigEarthNet multi-label Sentinel benchmarking. Ground-level models lost accuracy on a spacecraft segmentation dataset, Earth-observation labels carry their own noise, and a licence study of six widely used general image datasets found potential commercial-use violations for five of them.

Which computer vision datasets fit which space and satellite task

The table below groups the main public datasets by the task they were built for.

Dataset Imagery Task it supports Scale and labels
Spacecraft detection and parts dataset Synthetic and real images of satellites and space stations, 1280×720 Detection, instance segmentation, part segmentation (body, solar panel, antenna) 3,117 images, 3,667 spacecraft, 10,350 part masks; 2,516 train and 600 test images
SPEED+ Tango spacecraft from the PRISMA mission, synthetic plus hardware-in-the-loop Pose estimation that holds up across a domain gap 59,960 labelled synthetic images (80:20 train/validation); lightbox and sunlamp domains unlabelled
xView WorldView-3 satellite imagery at 0.3 m ground sample distance Object detection with horizontal bounding boxes Over 1 million instances, 60 classes, more than 1,400 km²; 847 training images with public labels
DOTA Aerial images from several sensors and platforms, about 4000×4000 pixels Oriented object detection 2,806 images, 188,282 instances, 15 categories, each labelled with an arbitrary quadrilateral
SpaceNet Very high-resolution satellite imagery from many regions Building footprint and road network mapping About 67,000 km², over 11 million building footprints, about 20,000 km of road labels
HLS V2.0 Harmonised 30 m Landsat 8 and Sentinel-2 surface reflectance Atmospherically corrected, cloud-masked input for time-series work Global except Antarctica, record from April 2013
Tarrio cloud references Sentinel-2 Evaluating cloud and cloud-shadow masks Tarrio: 2,681 interpreted points across 28 products

A 2026 survey also names fMoW for overhead scene understanding with metadata, SpaceNet 6 for SAR–optical all-weather mapping, and EuroSAT, BigEarthNet, SEN12MS and So2Sat LCZ42 for Sentinel-based classification and fusion.

A few details in the table change how you can use a dataset. xView was released by the U.S. National Geospatial-Intelligence Agency for the DIUx xView 2018 Challenge and needs a manual download of about 20.7 GB. Its 282-image challenge validation set has no public labels. That means any validation split has to come out of the 847 labelled training images. Over 80% of xView’s classes are fine-grained children of seven parent classes, such as eight distinct truck types.

Box geometry matters too. DOTA labels each object with an oriented quadrilateral, while xView uses horizontal boxes. SpaceNet has more instances than xView but labels only buildings and roads, and the xView authors name fMoW as the closest dataset to their own.

Spacecraft telemetry has fewer public resources. The OPS-SAT anomaly-detection benchmark provides real in-orbit telemetry with curated anomaly labels and baselines. A 2026 survey of on-orbit AI concludes that benchmarks are richer for perception than for constellation-scale collaboration.

Why does a COCO or ImageNet backbone struggle on satellite and orbital imagery?

COCO is made of everyday photos from Flickr, annotated by crowdworkers, and its 2017 detection release has 80 categories. The original paper describes 91 object types “easily recognizable by a 4 year old”, with 2.5 million labelled instances in 328k images. In overhead imagery, objects are far smaller and more densely packed than in ground-level photos, and they vary widely in scale, orientation and shape across the earth’s surface. Orbital imagery adds the problems of the space environment itself: lighting, stray light and shadowing that computer graphics struggles to reproduce.

There are measurements of the gap. On the spacecraft parts dataset, state-of-the-art segmentation models reach an average mIoU 6% to 11% lower than on Earth-based datasets such as Cityscapes and Pascal VOC. Bodies and solar panels segment decently, while antennas, with their irregular shapes, segment poorly. The authors conclude that space-based semantic segmentation “might not be directly solvable by models designed for on-Earth scenarios”.

Performance also drops between climate zones within satellite imagery. The Real-World Distribution Shifts benchmark, built from xView imagery, found every detector’s score on out-of-distribution climate zones fell by more than 35%, 7% and 40% against its in-domain test set for zones A, B and C. In binary flood-damage detection, scores fell by more than 52% on India’s out-of-distribution test set and by more than 62% on the US one. For GLIP and Grounding DINO trained on zone A and tested on zone B, classification errors were the main cause of the drop, followed by background errors. Weaknesses in the pretraining corpus also carry downstream: an expert audit of ImageNet’s wildlife classes notes that its inaccuracies and biases directly affect models fine-tuned from it.

Closing the gap

The spacecraft dataset authors fine-tuned DeepLabV3 from ImageNet weights, and initialised YOLO from ImageNet and EfficientDet from COCO. On xView, training on a multi-resolution version of the data gave 0.2590 total mean average precision, against 0.1456 and 0.1549 for the plain and augmented versions. In the RWDS experiments, training on several source domains improved generalisation to unseen ones, and Grounding DINO gave the best balance of in-domain and out-of-domain performance.

Fine-tune on in-domain imagery and evaluate on a geography or lighting condition the model has never seen. A random split of one region tells you very little about the next region.

Data quality problems that break models trained on satellite and space imagery

Supervised deep learning in Earth observation depends on large, high-quality labelled datasets. Those labels are expensive to get. They often need expert domain knowledge, so teams turn to crowdsourcing, semi-supervised labelling or annotation by ML practitioners, and each of these raises the risk of data quality problems.

Label noise built into how the labels were made

A 2025 Scientific Reports study sorts label noise in Earth observation into four sources: unreliable labelling procedures, difficulty separating signals even for experts, noise inherent to stochastic events, and misalignment when labels come from field or in-situ data rather than the image. BigEarthNet is one of the study’s worked examples. Its labels come from the Corine Land Cover database, which combines national databases built with different methods, and the study cites field comparisons that bring the reliability of its labels into question.

Boundaries and ambiguous classes

In landslide segmentation the model’s uncertainty rises at the edges of landslides, where annotation errors are most likely. In BigEarthNet, inland wetlands had the highest uncertainty of any class, because their spectral overlap with vegetation, forest and bare soil makes even visual identification hard. Labellers disagree too: when several experts label the same sample, their labels are often inconsistent, a problem we cover in why off-the-shelf models fail on edge cases in production.

Atmosphere, clouds and the masks meant to remove them

Atmospheric effects in InSAR imagery mimic the signal itself. In the Hephaestus volcanic dataset, atmospheric contributions in InSAR imagery create patterns that mimic deformation fringes, which complicated annotation even for experts. Rule-based masking fails on snow and ice, buildings, beaches and salt flats that look spectrally like clouds, on wetlands and small water bodies that look like cloud shadows, and on thin cirrus. Even a stronger learned quality score such as Cloud Score+ has known weak spots: cloud shadows over water, moving boats and active ice floes.

Duplicates and train–test leakage

Near-duplicates distort evaluation metrics, especially across train and test splits. In CIFAR-10 and CIFAR-100, 3.3% and 10% of test images had duplicates in the training set, and accuracy fell by 9% to 14% on a duplicate-free test set.

An audit of ten widely used benchmarks estimated label errors averaging at least 3.3%, and at least 6% in the ImageNet validation set.

How to check a space-imagery dataset before you train on it

  1. Write a datasheet for the dataset. Record its motivation, composition, collection process and recommended uses, following the Datasheets for Datasets practice. For satellite data, add the sensor, ground sample distance, processing level, acquisition dates, regions and how cloudy scenes were filtered.
  2. Establish provenance and licence. Find the licence on the download site, then in the dataset files, then by asking the owners. Check that the copy you hold is the creators’ release, since redistributed copies sometimes carry a different licence.
  3. Rebuild the split by geography. Group tiles by scene and location, then hold out whole locations. The volcanic-deformation study used 12 unique locations as its test set.
  4. Search for near-duplicates within and across splits. Near-duplicates cause overfitting and distort evaluation metrics, especially when they sit across the train–test boundary.
  5. Flag likely label errors. Tools such as cleanlab’s Datalab use out-of-sample predicted probabilities and feature embeddings to give each example a label quality score between 0 and 1. Treat a flag as a candidate, not a verdict. In the benchmark audit, 51% of algorithmically flagged candidates were confirmed as errors on human review.
  6. Look for outliers and low-information images. The same tooling flags atypical images, abnormally dark images and low-information images, which can hurt generalisation when they cluster in some classes.
  7. Score scene quality explicitly. Use a continuous quality score rather than a single cloud flag where possible. Cloud Score+ was designed to get around the limits of categorical cloud masks.
  8. Drop or merge classes too thin to learn. RWDS kept only classes present in every domain, with at least 30 samples each, which left 16 classes.
  9. Audit a gold sample. xView’s expert stage compared each batch against expert-labelled gold chips and required 0.75 precision and 0.95 recall at 0.5 IoU. A gold sample of your own gives you the same kind of acceptance threshold.

What licence terms should you check before using an open satellite dataset commercially?

“Publicly available” is not the same as “commercially usable”. A study of six widely used image datasets (CIFAR-10, ImageNet, Cityscapes, MS COCO, FFHQ and VGGFace2) found potential licence violations for five of them if used to build commercial AI software. Except for MS COCO, none of the studied licences explicitly granted the right to commercialise a model trained on the data, or even its output, which also limits the use of models pretrained on them. CIFAR-10, ImageNet and Cityscapes also carry risks if you modify or redistribute them.

ImageNet’s terms limit use to non-commercial research and education, bind a researcher’s for-profit employer, and require the researcher to indemnify the ImageNet team, Princeton and Stanford.

COCO shows a second trap. Its annotations are licensed CC BY 4.0, but the licence on a dataset card usually covers the annotations or the compilation, not the pixels. The COCO consortium states that it does not own image copyright and that image use must follow Flickr’s terms. Commercial use of COCO also comes with obligations: link the licence, keep the copyright notice and do not use COCO to endorse your product. Open Images also licenses annotations and images separately. Its annotations are CC BY 4.0 and its images are listed as CC BY 2.0, but Google makes no warranty about each image’s licence and tells users to check it themselves.

Creative Commons variants differ on commercial use. CC0 imposes no conditions, CC BY and CC BY-SA allow commercial use with attribution (BY-SA adds share-alike terms), and the NC variants exclude commercial use. For satellite data, check the licence of every dataset separately. SpaceNet, for example, states that each of its datasets has a different open data licence. Datasets assembled from several sources need their lineage traced, because the source licences, including the versions in force when the data was collected, can be stricter than the licence on the aggregate.

Split the work the way the licence study does. An engineer can gather the licence, provenance and lineage information, but the compliance risk assessment needs a lawyer. Privacy is a separate question again. Image collections with people in them raise personal-data issues whatever their copyright licence says.

When public datasets stop being enough

The spacecraft parts dataset was published as the first public dataset for spacecraft segmentation, and its authors put the scarcity down to the cost of pixel-level masks and the difficulty of getting images from space. SPEED+ is designed around a typical situation in spaceborne ML development: labelled target-domain images are not available before deployment, so its hardware-in-the-loop lab domains, lit to mimic realistic illumination, are unlabelled and used for testing. Onboard models in orbit have to adapt to changing illumination, seasons and sensor ageing, and to sparse or delayed labels.

Earlier overhead datasets each covered few classes in few regions, and the RWDS drops show how far performance moves between regions and climates.

If your classes, sensor, resolution and geography are all covered by a public dataset, fine-tune on it and spend the effort on the checks above. If any one of those four differs (a new sensor, a region missing from the archive, an object class no benchmark labels), plan for your own labelled target-domain set, even a small one, used at least for validation. SPEED+’s hardware-in-the-loop images capture the stray light, shadows and corner cases that computer graphics struggles to reproduce. Expert annotation is costly and slow, so budget for it from the start rather than discovering it after the first failed evaluation.

TechnoLynx’s computer vision R&D team takes on this engineering: the dataset audit, the domain-adaptation pipeline and the target-domain labelling set-up.

Back See Blogs
arrow icon