New directions in weed management and research using 3D imaging

Abstract

Recent innovations in 3D imaging technology have created unprecedented potential for better understanding weed responses to management tactics. Although traditional 2D imaging methods for mapping weed populations can be limited in the field by factors such as shadows and tissue overlap, 3D imaging mitigates these challenges by using depth data to create accurate plant models. Three-dimensional imaging can be used to generate spatiotemporal maps of weed populations in the field and target weeds for site-specific weed management, including automated precision weed control. This technology will also help growers monitor cover crop performance for weed suppression and detect late-season weed escapes for timely control, thereby reducing seedbank persistence and slowing the evolution of herbicide resistance. In addition to its many applications in weed management, 3D imaging offers weed researchers new tools for understanding spatial and temporal heterogeneity in weed responses to integrated weed management tactics, including weed–crop competition and weed community dynamics. This technology will provide simple and low-cost tools for growers and researchers alike to better understand weed responses in diverse agronomic contexts, which will aid in reducing herbicide use, mitigating herbicide-resistance evolution, and improving environmental health.

Introduction

Recent innovations in remote sensing and imaging technology have created unprecedented opportunities for growers to make data-driven decisions about integrated weed management (IWM). The use of digital imaging to map and monitor weeds allows growers to use precision weed control tactics at a field scale while accounting for spatial and temporal variation in weed populations (Andújar et al. 2016; Comba et al. 2019). Until recently, most remote sensing for weed detection and mapping has used 2D approaches such as normalized difference vegetation index (NDVI) and hyperspectral imaging (Andújar et al. 2018; Comba et al. 2019; Smith et al. 2018). However, 2D techniques lack depth information and cannot accurately estimate plant volume or biomass (size), nor can they detect lower layers of the plant canopy (Paturkar et al. 2020; Smith et al. 2018). Moreover, variations in light levels, overlapping leaves and stems, and shadows can make 2D techniques ineffective for distinguishing individual plants in the field (Smith et al. 2018; Zhang et al. 2016).

To mitigate these challenges, 3D imaging techniques are being developed that include the added dimension of depth (Armean et al. 2021). Cost-effective and efficient methods such as structure-from-motion (SfM) and stereo vision are being used to reconstruct 3D models of plants and plant canopies based on many images taken in succession. By incorporating depth data to create models of plant canopies in the field, 3D imaging offers researchers and growers a powerful tool for mapping spatial heterogeneity of weeds in response to management tactics (Armean et al. 2021; Zhang et al. 2016). In this article, we discuss applications of 3D photogrammetric imaging in weed management (e.g., weed detection and mapping for targeted removal), as well as weed research (e.g., modeling of weed–crop competition to predict yield loss). In addition, we discuss applications of 3D imaging for weed management in orchards and grasslands to illustrate the breadth of potential uses of this technology.

Overview of 3D Imaging Techniques

Three-dimensional imaging uses sensor data obtained to generate models of objects based on point clouds, which are large data sets of points in 3D space (Comba et al. 2019). These can be created using laser-based techniques such as light detection and ranging (LIDAR) or by photogrammetric techniques using computer vision algorithms (Andújar et al. 2013; Comba et al. 2019; Paturkar et al. 2020; Table 1). LIDAR, which has been used in agriculture and forestry, emits laser pulses directed toward an object (e.g., a plant), which are reflected back to the sensor to obtain depth information (Andújar et al. 2013). Although LIDAR is extremely accurate and can be used to scan wide areas in the field, it requires sophisticated equipment and is costly (Andújar et al. 2013; Paturkar et al. 2020).

Table 1. Comparison of photogrammetric techniques and light detection and ranging (LIDAR) for 3D imaging.

^a Source: Paturkar et al. 2020.

^b Source: Armean et al. 2021.

^c Source: Dandrifosse et al. 2020.

^d Source: Andújar et al. 2013.

In contrast, photogrammetric techniques such as SfM and stereo vision are low-cost and efficient methods for creating 3D point clouds of objects (Armean et al. 2021; Paturkar et al. 2020; Table 1). In SfM, a single camera takes many 2D images of the object (e.g., a plant) from different viewpoints of known distances with a specified amount of overlap between images (Armean et al. 2021; Figure 1). A 3D point cloud is generated based on features that are matched from different images (i.e., views) of the canopy. The point cloud is a 3D model of the canopy, which can be used to infer canopy structure and biomass based on depth data (Armean et al. 2021; Paturkar et al. 2020; Figures 2 and 3). In stereo vision, a binocular camera takes two images of the plant at once and uses triangulation to create a depth map (Teng et al., 2021). The accuracy of both SfM and stereo vision can be affected by wind conditions in the field, which cause movement and overlap in the canopy (Dandrifosse et al. 2020; Paturkar et al., 2020). However, given their relative simplicity and low cost, photogrammetric techniques have numerous applications in weed management and weed research.

Figure 1. Use of images taken from different angles to create a 3D reconstruction in structure-from-motion (SfM; top) vs. stereo-vision photogrammetry (bottom).

Figure 2. Red, green, and blue (RGB) image of soybeans and weeds (A) and corresponding 3D point cloud reconstruction (B). Lower panels show point cloud reconstructions from different angles, including a top view (C), top view offset 45° from vertical (D), front view (E), under canopy and offset 45° (F), directly under canopy (G), facing canopy from behind (H), facing canopy offset 45° right (I), side view (J), and facing canopy offset 45° left (K).

Figure 3. Data pipeline for calculating canopy height and estimating biomass in the field using red, green, and blue (RGB) images and depth data.

Applications of 3D Imaging in Weed Management

Weed Detection and Mapping

Timely postemergence weed control is critical in crop production (López-Granados et al. 2016). Applying herbicides when weeds are not at the appropriate phenological stage or uniformly without considering spatial variability over time can interfere with weed control (López-Granados et al. 2016; Swanton et al. 1999). For example, if weed control measures are applied too late in the season to control early-emerging weeds, this can result in weed escapes, increased soil weed seedbank, and evolution of herbicide resistance (Neve et al. 2010). Although weeds often occur in spatially aggregated patches of varying size, shape, density, and species composition, weed control tactics have traditionally been applied uniformly in fields due to the difficulty of monitoring populations in situ and in real time (Thornton et al. 1990).

Photogrammetric 3D imaging allows growers to create real-time maps of weeds in the field, rather than monitor weedy patches manually. This can be accomplished early in the season for timely weed control when crops and weeds are growing at different rates (Andújar et al. 2016; Li and Tang 2018; Piron et al. 2011). For example, unmanned aerial vehicles (UAVs) as well as ground vehicles equipped with red, green, and blue (RGB) cameras have been successfully used for implementing management tactics based on within-field variability of weed populations (Castaldi et al. 2017; López-Granados et al. 2016; Wu et al. 2020). By detecting and quantifying weedy patches in real time, growers can make decisions about weed control based on spatial distribution of species types (e.g., broadleaves and grasses) or individual species in the field (Andújar et al. 2016; Wu et al. 2020). Over time, this is likely to lower production costs by reducing labor and herbicide use.

Preventing seed production in late-season weed escapes is crucial for mitigating herbicide resistance (Kutugata et al. 2021). Allowing late-season weed seed production increases the probability of herbicide-resistant mutants, particularly in species with prolific seed production. However, growers often ignore weed escapes, because escapes rarely reduce crop yield (Bagavathiannan and Norsworthy 2012). Nonetheless, detecting and managing late-season weed escapes and minimizing replenishment of the soil seedbank is important for reducing long-term weed persistence (Kutugata et al. 2021). Besides detecting individual plants, RGB-based imaging has the potential to estimate seed output of late-season weed escapes, which can direct precision weed management tactics to reduce the weed seedbank (Kutugata et al. 2021).

Automated Weed Control

The ability to detect and map weed populations in real time using 3D imaging has numerous applications in automated weed control. In addition to UAVs, ground-based robots equipped with RGB cameras have been used for automated weed removal through targeted herbicide spraying or mechanical in-row removal (Castaldi et al. 2017; Li and Tang 2018; Wu et al. 2020). Automated weeding systems allow growers added flexibility for weed control in both conventional and organic production. For example, lightweight robots can be used in the field when it is too wet for conventional equipment, allowing for more timely weed control. These systems can also dramatically reduce the overall cost of weed control, particularly in organic production or in specialty crops where there are fewer herbicide options (Fennimore et al. 2017; Lowenberg-DeBoer et al. 2020).

Three-dimensional imaging can improve the accuracy of automated systems in differentiating weeds by size class in real time. Unlike 2D approaches, which are less likely to differentiate species and capture size differences in field conditions, 3D imaging can be used to detect in-row canopy structure and characteristics of individual weed species at different stages (Wu et al. 2020). This is critical for weed control because of the relatively short window of time during which weed removal by herbicides and cultivation is effective (Kudsk and Streibig 2003; López-Granados et al. 2016). For example, postemergence herbicide labels typically refer to weed size as a critical factor for application timing, outside of which weed escapes are more likely to occur. Using automated systems with 3D imaging, rather than 2D or visual assessments, will enable growers to apply site-specific weed control at the optimum timing, as well as monitor troublesome weed patches in subsequent years.

Monitoring Weed Suppression with Cover Crops

Cover crops offer a promising strategy for weed management, because they compete with weeds for resources and inhibit weed seed germination and seedling growth (Teasdale et al. 1998). However, cover crop performance can be variable in the field, leading to weed escapes. Therefore, monitoring cover crop growth (i.e., biomass) is a crucial factor for predicting weed suppression spatially in the field (Mirsky et al. 2013). Three-dimensional imaging techniques are being developed to measure cover crop performance based on biomass and canopy structure (Cooper et al. 2017; Roth and Streit 2018). By mapping cover crop biomass throughout the season or at least at the moment of termination, growers will be able to better quantify and map spatial heterogeneity in the field, particularly areas where the cover crop is underperforming (i.e., low biomass) and where occurrence of weed escapes is most likely.

In cover crop–based reduced-tillage production systems, 3D imaging can be used to map weed escapes for targeted spraying or mechanical removal. Late-season weed escapes (i.e., weeds that survived early-season weed control or emerged later in the season) are the largest contributors to the soil seedbank. Therefore, detecting them is crucial for preventing replenishment of the seedbank, especially for weeds that pose a high risk for evolving resistance to herbicides (Bagavathiannan and Norsworthy 2012). By using 3D imaging to map areas where cover crops are underperforming, which have higher risk for weed escapes, growers can better prepare and conduct site-specific control actions to prevent the production of new weed seed. Furthermore, detecting weed escapes using 3D imaging will help with selecting the best control or removal tool based on weed size. Finally, monitoring cover crop growth and uniformity can help growers predict crop yield and biomass production and determine the optimal timing of cover crop termination as part of their weed management programs.

Applications of 3D Imaging in Weed Research

Three-dimensional imaging offers weed researchers a potentially transformative tool for understanding spatial variability in weed responses to management tactics. Because weeds are highly heterogeneous in the field, creating spatiotemporal models of weed responses can improve our understanding of the underlying patterns in this variability. Rather than measuring weed responses by point estimates in the field, researchers can use 3D imaging to capture the spatial heterogeneity of weed species, density, and biomass in real time. This will help advance our understanding of weed responses to IWM tactics, such as weed–crop competition and predicted yield loss and weed community dynamics.

Modeling Weed–Crop Competition and Predicting Yield Loss

Accurately modeling weed interference in crops as soon as possible after crop emergence is crucial for predicting yield loss. Early-season weeds affect crop yield much more than weeds that emerge after a critical period (typically a few weeks), as they compete with the crop for a longer period and can cause problems if allowed to produce seed (Kropff and Spitters 1990; Veeranampalayam Sivakumar et al. 2020). Modeling the interactions between weed density, duration of weed competition, and crop yield can help estimate yield loss as a function of weed competition over continuous time, rather than discrete time points (Weaver et al. 1992). This can provide more accurate and well-informed weed thresholds for growers to assess the optimal timing of weed control tactics (Swanton et al. 2015; Weaver et al. 1992).

Many empirical models have been developed for quantifying crop loss based on weed density and relative emergence timing with respect to the crop (Kropff and Spitters 1990). However, because weeds typically emerge in successive flushes, descriptive models based on weed density do not provide a complete picture for predicting yield loss, because weeds of different sizes exhibit different competitive ability (Kropff and Spitters 1990). Another approach has been to create dynamic simulations of competition for light and water by measuring physiological processes. Although these ecophysiological models are helpful at providing information for descriptive models of competition, they can be difficult to derive and are overly detailed for practical management applications (Kropff and Spitters 1990). Competition models that account for plant height and relative leaf cover can provide a more accurate prediction of crop yield loss. Early quantification of relative leaf area of crops and weeds has been shown to be an accurate way to predict crop yield loss and account for differences in weed emergence (Kropff and Spitters 1990). Although predictions of yield loss must be made soon after crop emergence, the competitiveness of weeds and crops largely depends on their share of leaf area in the canopy when the weed–crop canopy closes (Kropff 1988; Spitters and Aerts 1983). Therefore, modeling weed–crop competition requires quantifying the changes in relative leaf area of weeds versus crops between the time of crop emergence and canopy closure (Kropff and Spitters 1990).

Three-dimensional imaging will allow researchers to model weed–crop competition by estimating vegetation cover, particularly leaf area index. Modeling the weed–crop canopy, including each plant’s height, density, and leaf area in space as a function of time, will allow researchers to study changes in canopy structure and compare physiological and morphological changes in response to competition for light, especially before canopy closure. These approaches will be able to provide real-time nondestructive sampling of biomass and leaf cover at any time point, rather than having to harvest and weigh plants at set times. Improved weed–crop competition models will also be useful for growers for monitoring weed interference and predicting yield loss in specific crop cultivars. For example, UAV-derived SfM 3D imaging has been used to compare weed competitiveness in wheat (Triticum aestivum L.) cultivars by measuring morphological and physiological traits associated with early vigor (Aharon et al. 2020). RGB imaging has also been used to compare weed competitiveness in legumes (Travlos et al. 2017).

Weed Community Dynamics

In addition to weed–crop competition, 3D imaging can help researchers better characterize weed community dynamics. Weed community structure can be influenced by environmental factors (e.g., soil type), disturbance events (e.g., tillage), colonization, and competition interactions (Cléments et al. 1994). Understanding changes in weed community structure is crucial for making IWM decisions in order to maintain weed populations below economic thresholds (Swanton and Murphy 1996). Therefore, it is crucial to understand how weed community structure changes in response to environmental factors and management tactics in the short and long term, and how these fluctuations influence succession of weed species and biodiversity (Swanton and Murphy 1996).

Three-dimensional imaging can be used to create descriptive models of weed community dynamics. By using data for leaf texture and density, and differences in plant height as proxies for community diversity and structure, 3D techniques can more accurately describe weed communities in real time. This information can be used to model community changes over many time points in response to IWM tactics or environmental factors. For example, one area of concern in conservation tillage is potential weed interference by perennials, grasses, and wind-borne species (Swanton and Murphy 1996). Three-dimensional quantification of differences in growth habit, plant height, and canopy structure and complexity can be used to describe the spatial and temporal variability of weed species, particularly in perennial systems. By monitoring weed community dynamics over time, researchers can predict changes in community structure and develop appropriate management strategies (Swanton and Murphy 1996).

Applications of 3D Imaging in Other Agricultural Systems

Three-dimensional imaging has the potential to improve weed management in a wide range of agricultural systems besides field crops (Table 1). In this section, we will discuss applications of 3D imaging for weed management in two contrasting systems: orchards and grasslands.

Weed Management in Orchards

Weed management in orchards is crucial for healthy tree growth and fruit yield, as well as soil and water quality. Orchard weeds can compete aggressively with trees for resources, provide habitat for pests, and shade and stunt young trees (Riczu et al. 2015). Poorly controlled orchard weeds can hamper operations of irrigation equipment and automated sprayers and interfere with the accurate placement of water, fertilizers, and pesticides (Brunharo et al. 2020; Liu et al. 2022). Therefore, intensive weed control is necessary in orchards both during establishment and throughout their life span. Although sequential herbicides are widely used in orchards, as are mowing and mulching, variable distribution of weeds necessitates precision control tactics (Brunharo et al. 2020; Rehman et al. 2019; Zeng et al. 2020). Three-dimensional imaging has much potential for improving precision weed control in orchards by accounting for canopy architectural features such as canopy density and depth. For example, LIDAR has been used in orchards to identify weed species, estimate weed cover, and create detailed canopy density maps to inform precision spraying equipment (Riczu et al. 2015; Zeng et al. 2020). However, machine vision using RGB-D cameras is also being developed to map orchards based on canopy height and volume. These systems are being used to guide sensor-driven intelligent spraying of herbicides and pesticides (Rehman et al. 2019; Tagarakis et al. 2022; Zeng et al. 2020).

Weed Management in Grasslands/Rangelands

Timely detection of weeds is crucial for managing grasslands, as weeds can diminish grazing capacity. However, weeds in grasslands tend to be patchy, making monitoring difficult. Therefore, it is important for farmers to have efficient methods for mapping grassland weeds to make decisions about grazing and weed control (Schellberg et al. 2008; Yuba et al. 2020). In a recent study by Yuba et al. (2020), UAV-based RGB imagery with SfM was used to generate 3D point clouds to map patches of the harmful weed fountain grass [Pennisetum alopecuroides (L.) Spreng.] in pastures. This technology has also been used to detect and map weeds in rice (Oryza sativa L.) fields (Kawamura et al. 2020). Despite the relatively few studies using 3D imaging for weeds in grasslands, this technology has considerable potential benefits. For example, 3D imaging can be used to monitor grassland health. Traditionally, “clip and weigh” methods are used to estimate biomass, as it is a key indicator for grassland health and is correlated with canopy height (Lussem et al. 2020). However, these methods are labor-intensive and do not account for variability due to grazing, species composition, and environmental factors (Bareth and Schellberg 2018; Schellberg et al. 2008). Recently, LIDAR has been used to estimate grassland species using 3D point clouds and canopy height and volume data (Paturkar et al. 2020; Schulze-Brüninghoff et al. 2019). Additionally, UAV-based SfM and stereo imaging have been used to quantify canopy height variation, which can be used as a surrogate factor for aboveground biomass (Bareth and Schellberg 2018; Cooper et al. 2017). In a study by Lussem et al. (2020), UAV-based SfM and multiview stereopsis were shown to predict forage biomass based on height with results comparable to ground-truth measurements. Although this technology is still largely untested in grasslands, continuing development of UAV-based imaging holds promise for opening new paths for sustainable weed control and monitoring in grassland systems (Cooper et al. 2017).

Caveats

Several important caveats must be considered when using 3D imaging in weed management and research. First, canopy structure can impact efficiency and accuracy of 3D reconstructions. When the canopy is too dense for light to penetrate the upper layers, the lower layers might not be fully detected by the camera and might be absent from the point cloud. For example, in soybean [Glycine max (L.) Merr.] that have reached canopy closure, only the upper layers are included in the 3D reconstruction (Figure 4). In plants with a sparser canopy, more light penetration allows for the lower layers to be captured in the 3D reconstruction (Figure 4). Second, environmental factors can affect the efficiency and accuracy of 3D reconstructions. Leaf movement due to wind is perhaps one of the most common ways in which leaf shapes are obscured and images blurred, producing noise (Armean et al. 2021; Paturkar et al. 2020). Also, overlapping leaves and shadows can interfere with identification of plant features (Armean et al. 2021; Paturkar et al. 2020). Third, morphological features such as leaf shape and leaf orientation can interfere with 3D reconstruction (Andújar et al. 2018; Armean et al. 2021). For example, it can be more difficult to construct monocotyledonous plants with respect to dicotyledonous plants due to the elongated and thinner leaf morphology (Andújar et al. 2018). In addition, leaves that are oriented more vertically are not always correctly identified, because there is not enough information to distinguish them from stems and branches (Armean et al. 2021). Continued technological developments will likely improve the accuracy of 3D modeling in light of these challenges. It must be highlighted that despite these limitations, 3D imaging provides a much more informative description of vegetation architecture than RGB images alone. As image processing and mapping become increasingly efficient, these techniques will offer a simple yet powerful tool that is widely accessible to growers.

Figure 4. Three-dimensional point cloud reconstructions of soybean (A, top view; B, front view) and cereal rye (Secale cereale L.) (C, top view; D, front view). Note the voids in the soybean point cloud (B) caused by dense canopy cover. Such voids are largely absent in cereal rye (D) due to a more even canopy with greater light penetration.

Future Research and Conclusions

We are at a turning point in weed management where state-of-the-art 3D imaging techniques are making it possible to create spatiotemporal maps of weeds using relatively low-cost equipment. In contrast to traditional 2D imaging techniques, 3D imaging captures depth and canopy structure, allowing more accurate estimates of plant height and volume at the species level. This technology offers growers and researchers alike unprecedented opportunities for real-time, cost-effective mapping and monitoring of weeds and their impact on crop growth and development. By capturing spatial heterogeneity of weeds in the field, 3D imaging will enable growers to create accurate weed population maps for timely site-specific weed control, in addition to improved targeted weed removal and spraying. Moreover, 3D imaging will enable growers to classify individual weeds based on size, which will allow for more targeted weed control tactics that reduce labor and use of herbicides. In addition, 3D imaging will improve our understanding of weed responses to IWM tactics and predict yield losses resulting from weed–crop competition. As 3D imaging technology continues to improve, it will provide new opportunities for growers and researchers with tools to better understand the complexities of weed biology in the field and develop better-informed tactics for weed control. These tools hold great potential to aid in the collective effort of reducing agricultural inputs and improving environmental health.