Synthetic RGB-trained models and GAN alignment improve thermal weed segmentation

A new framework uses synthetic red-green-blue (RGB) imagery, a small set of real annotations, and generative adversarial network (GAN)-based cross-modality alignment to improve semantic segmentation of crops and weeds in thermal images. By training on more than 1,000 synthetic images and adding as few as 20 labeled field images, the approach can boost segmentation accuracy in complex weedy fields, where low contrast and occlusion make thermal plant isolation difficult. The method aims to enable more reliable extraction of crop canopy temperature for high-throughput field phenotyping while sharply reducing the manual labeling burden.

In weedy outdoor fields, crops and weeds often overlap and share similar colors, shapes, and textures, causing occlusions and false detections. Thermal imagery adds difficulty: plant, weed, and soil temperatures can be similar, producing low contrast, while sunlight, wind, and moisture alter canopy temperatures. Traditional thermal segmentation often relies on temperature thresholds or watershed algorithms, and co-registered red-green-blue (RGB) images are sometimes used to guide masks. However, cross-modal pixel alignment remains difficult without careful camera calibration, and manual annotation of complex plant-weed scenes is costly. Based on these challenges, further research is needed to develop scalable segmentation methods that require minimal labels and can transfer RGB-trained models to thermal imagery without precise calibration.

Researchers at the University of California, Davis (UC Davis), published (DOI: 10.1016/j.plaphe.2026.100214) the study in the journal Plant Phenomics in 2026. The team, from the Department of Biological Systems Engineering and the Department of Plant Sciences, developed a pipeline that combines Helios-generated synthetic RGB images, limited real field annotations, and Cycle-Consistent Generative Adversarial Network (CycleGAN)-Turbo translation for RGB-to-thermal alignment. They evaluated UNet++, Feature Pyramid Network (FPN), and SegFormer models with EfficientNet encoders, then transferred RGB segmentation masks to thermal images for crop temperature extraction.

The study trained models on 1,128 synthetic images generated with Helios, an open-source 3D plant and environment biophysical modeling framework, containing cowpea (Vigna unguiculata) and four weed species: cheeseweed (Malva neglecta), puncturevine (Tribulus terrestris), bindweed (Convolvulus arvensis), and ground cherry (Physalis philadelphica). The team also manually annotated 159 real field images. Three strategies for injecting real data were compared: direct injection, balanced sampling, and fine-tuning. Balanced sampling, which ensured each training batch included at least one real image, performed best. Combining all synthetic images with a few real images produced up to a 25% relative improvement in mean Intersection over Union (IoU) compared with a synthetic-only baseline. With only 20 real images, weed and plant segmentation improved substantially; with about 30 real images, per-class performance surpassed models trained on the full real dataset. Notably, moderate 31-million-parameter models consistently outperformed larger 86-million-parameter versions, suggesting over-parameterization can hinder adaptation when real data are scarce. For cross-modal alignment, CycleGAN-Turbo translated RGB to thermal imagery, and multi-scale template matching using OpenCV mapped RGB masks onto thermal images without geometric calibration, taking about 3 seconds per image on an NVIDIA RTX Titan 24 GB. Fréchet Inception Distance (FID) scores improved in early growth stages but worsened in flowering and late-season canopies.

The authors said the most promising finding was how much a small, carefully chosen real dataset could add. They said balanced sampling helped the model avoid overfitting to synthetic patterns and improved generalization across growth stages. They also said that moderate-sized models often worked better than larger ones in this low-data setting, and that GAN-based translation plus template matching offers a practical way to move RGB-trained segmentation masks into thermal imagery without camera calibration.

The approach could help breeders and phenotyping teams extract canopy temperature more reliably, supporting assessments of drought response, water status, and stomatal behavior. The authors said the same strategy may extend to other low-contrast, multi-sensor imaging tasks, such as light detection and ranging (LiDAR) and multispectral systems. Limitations include a small test set, class imbalance, and weaker alignment in dense late-season canopies. Future work should improve synthetic data diversity and use temporal context or auxiliary sensor cues.

###

References

DOI

10.1016/j.plaphe.2026.100214

Original Source URL

https://doi.org/10.1016/j.plaphe.2026.100214

Funding information

This work was supported by This work was supported by the Bill and Melinda Gates Foundation, Project ID: INV-002830, and USDA NIFA Hatch project 7003146.

About Plant Phenomics

Plant Phenomics publishes breakthrough research that integrates genomics, genetics, physiology, molecular biology, bioinformatics, statistics, mathematics and computer science to advance plant phenotyping—from the cellular level to entire populations. The journal tackles key scientific challenges in phenomics by showcasing cutting‑edge technologies for data acquisition, management, interpretation and modelling. Its scope spans practical applications across all plant sciences, aiming to translate novel phenotyping tools into real‑world solutions for breeding, crop management and fundamental plant biology. By fostering interdisciplinary innovation, Plant Phenomics serves as a hub for researchers developing high‑throughput, accurate and scalable methods to decipher complex plant traits, ultimately accelerating progress in agriculture and plant science.