U.S. patent application number 17/425521 was filed with the patent office on 2022-03-24 for harvester with automated capabilities.
The applicant listed for this patent is Ceres Innovation, LLC. Invention is credited to Frank FAULRING, Jason FAULRING, Jacob GOELLNER, Mark KEYSER, Carlos RAMIREZ.
Application Number | 20220087106 17/425521 |
Document ID | / |
Family ID | |
Filed Date | 2022-03-24 |
United States Patent
Application |
20220087106 |
Kind Code |
A1 |
FAULRING; Frank ; et
al. |
March 24, 2022 |
HARVESTER WITH AUTOMATED CAPABILITIES
Abstract
Systems and methods here may include a vehicle with automated
robotic subcomponents for harvesting delicate agricultural items
such as berries. In some examples, the vehicle includes a targeting
subcomponent and a harvesting subcomponent. Which may utilize
multiple cameras to create three dimensional maps of foliage and
targets. In some examples, the targeting subcomponent includes
automated or semi-automated harvesting targets to be mapped and
passed to the harvesting subcomponent. In some examples, the
harvesting subcomponent includes vacuum features and padded spoons
to detach the target agriculture from the stem.
Inventors: |
FAULRING; Frank; (Salinas,
CA) ; RAMIREZ; Carlos; (Salinas, CA) ;
FAULRING; Jason; (Salinas, CA) ; KEYSER; Mark;
(Salinas, CA) ; GOELLNER; Jacob; (Salinas,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Ceres Innovation, LLC |
Salinas |
CA |
US |
|
|
Appl. No.: |
17/425521 |
Filed: |
January 23, 2020 |
PCT Filed: |
January 23, 2020 |
PCT NO: |
PCT/US2020/014812 |
371 Date: |
July 23, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62796319 |
Jan 24, 2019 |
|
|
|
International
Class: |
A01D 46/30 20060101
A01D046/30; A01D 46/253 20060101 A01D046/253; A01D 46/28 20060101
A01D046/28; A01D 46/00 20060101 A01D046/00 |
Claims
1. A harvesting vehicle system comprising: a vehicle with a
targeting subcomponent and a harvesting subcomponent, the vehicle
including at least one motor in communication with wheels mounted
to traverse planter bed rows; wherein the vehicle includes a
computing device with a processor and a memory, the computing
device in communication with multiple sensors configured to
generate and send sensor data regarding agricultural targets to the
computing device, wherein the computing device configured to map
the agricultural targets using the sensor data; the harvesting
subcomponent in communication with the computing device, the
harvesting subcomponent including a robotic arm with picker head
assembly, the picker head assembly including a vacuum hose in
communication with a compressor, the hose terminating in a bellows
end and spoons configured to pinch together to remove targets using
data received from the computer regarding mapped targets.
2. The system of claim 1 further comprising, a conveyor belt
system, mounted to the harvesting subcomponent, the conveyor belt
system configured to receive and move targets from the harvesting
subcomponent to a packing area of the system.
3. The system of claim 1 further comprising, a turbine and vacuum
assembly mounted to a robotic arm of the harvester subcomponent,
the turbine and vacuum assembly in communication with the computer,
configured to chop and suction material using data from the
computer and map.
4. The system of claim 1 wherein the harvesting subcomponent
includes a picker head assembly with multiple picker heads mounted
to one robotic arm, the picker heads configured to rotate to
harvest and deposit agricultural targets.
5. The system of claim 1 further comprising, a foliage management
subcomponent mounted to the harvester subcomponent, the foliage
management subcomponent including belt drives in communication with
the computing device, the belt drives including a belt configured
to rotate on the belt drives, the belt configured to interact with
foliage of the agricultural targets, and bend the foliage to reveal
agricultural targets for the sensors.
6. The system of claim 5 wherein the belt drives are synchronized
with the at least one motor by the computing device to move the
harvester wheels, such that a relative speed over ground of the
harvester and a portion of the belt that is configured to contact
the foliage, is close to zero when the harvester is in motion.
7. The system of claim 5 wherein the foliage management
subcomponent includes a suction blower mounted above the foliage
management subcomponent, the suction blower configured to suck
ambient air from around the target foliage, up and away from
foliage.
8. The system of claim 5 wherein the foliage management
subcomponent includes a second set of belt drives and a second belt
mounted around the second set of belt drives, configured to rotate
on the belt drives, wherein the two belts configured to squeeze
foliage to reveal agricultural targets.
9. The system of claim 8 wherein the second set of belt drives are
synchronized with the at least one motor by the computing device to
move the harvester wheels, such that a relative speed over ground
of the harvester and a portion of the second belt that is
configured to contact the foliage, is close to zero when the
harvester is in motion.
10. The system of claim 5 further comprising, at least one air
blower mounted on the foliage management subcomponent, the air
blower configured to blow air toward the foliage to clear
debris.
11. The system of claim 1 wherein the robotic arm includes a second
vacuum assembly including a hose, configured to receive the
agricultural targets from the picker head assembly and remove the
targets from the harvester subcomponent to a packing area of the
system.
12. The system of claim 1 further comprising a back end computing
system in communication with the harvester computer, the back end
computing system configured to allow human users to review sensor
data and designate agricultural targets for the harvester
subcomponent to harvest.
13. The system of claim 1 wherein the computer uses neuro network
logic to make preliminary determinations of targets using the
sensor data.
14. The system of claim 12 wherein the computer may allow the human
user to determine an angle of attack for the picker head to harvest
an agricultural target, using the sensor data.
15. A system for harvesting agriculture, comprising: a traversing
machine with at least two robotic articulating arms attached to a
frame of the traversing machine, at least two wheels or tracks
attached to the frame of the traversing machine, and a computing
system with at least a processor and memory attached to the frame
of the traversing machine; the robotic arms including at least one
picker subassembly in communication with the computer system; at
least one sensor attached to the frame of the traversing machine,
configured to capture and send sensor information regarding
potential agricultural targets to the computing system for
analysis; the picker subassembly including a vacuum assembly
coupled to a nozzle with a terminating end, wherein the terminating
nozzle end includes a segmented ported section; the picker
subassembly further including two grappling spoons, the grappling
spoons configured to pinch together toward the vacuum nozzle.
16. The system of claim 15 wherein the picker subassembly vacuum
nozzle is mounted on an extender actuator.
17. The system of claim 15 wherein the baffle section is made of
resilient, pliable material.
18. The system of claim 15 wherein the at least one sensor is at
least one of a camera, laser, and lidar.
19. The system of claim 15 further comprising a foliage management
system, the foliage management system including two sets of belt
drives, and motors for the belt drives in communication with the
computing system, two belts, one each around the respective belt
drive set the belts each including an inside and an outside, the
belt roller sets and belts configured on the traversing machine to
receive foliage between the two belts, on the respective insides of
each, wherein the belt roller sets are synchronized with the at
least two wheels or tracks such that a relative motion of an inside
of the belts and ground on which the traversing machine is moving,
is approximately zero.
20. The system of claim 19 wherein the foliage management system
includes an ambient air vacuum pump in communication with the
computing system, the vacuum pump configured above the two sets of
belt rollers configured to suck air away from plant beds over which
the traversing machine is configured to pass.
Description
CROSS REFERENCE
[0001] This application relates to and claims priority to U.S.
Provisional application 62/796,319 filed Jan. 24, 2019 the entirety
of which is hereby incorporated by reference.
TECHNICAL FIELD
[0002] This application relates to the field of automated
agricultural harvesting equipment using robotic assemblies, mobile
harvesting units, remote harvesting, target tracking, and various
combinations of related technologies.
BACKGROUND
[0003] The agriculture industry is highly reliant on human pickers
to harvest a number of produce, including berries such as
strawberries. The reason human pickers are still used today,
despite the technological advancements available, is because of the
difficulty of identifying a target such as a berry in a field, that
is ready to be picked, reaching through the foliage of the plant to
grasp that berry, and then carefully removing that berry without
damaging it, to package and sell immediately.
[0004] Current automatic harvesting of such delicate and difficult
to grasp agricultural targets such as berries, while operating in a
harsh outdoor environment did not exist before this
application.
SUMMARY
[0005] Systems and methods here may include a vehicle having
various subcomponents for harvesting delicate agricultural items
such as berries. In some examples, the subcomponents may be
automated. In some examples, the vehicle may include a targeting
subcomponent and a harvesting subcomponent. In some examples, the
targeting subcomponent utilizes multiple cameras to create three
dimensional maps of the target and target areas sometimes including
the agricultural foliage. In some examples, the targeting
subcomponent may include any of various cameras, sensors, or other
targeting features to locate and map targets in an automated or
semi-automated manner. The system may then determine coordinates of
the mapped targets to be passed to the harvesting subcomponent. In
some examples, the harvesting subcomponent may include vacuum
features which help a nozzle attach to an agriculture target for
harvesting. In some examples, the harvesting subcomponent includes
padded spoons to aid in removal of the targeted agriculture from
the plant, including in some examples, a stem.
[0006] Systems and methods here may include a harvesting vehicle
system with a vehicle with a targeting subcomponent and a
harvesting subcomponent, the vehicle including at least one motor
in communication with wheels mounted to traverse planter bed rows,
and alternatively or additionally, the vehicle includes a computing
device with a processor and a memory, the computing device in
communication with multiple sensors configured to generate and send
sensor data regarding agricultural targets to the computing device,
wherein the computing device configured to map the agricultural
targets using the sensor data, the harvesting subcomponent in
communication with the computing device, the harvesting
subcomponent including a robotic arm with picker head assembly, the
picker head assembly including a vacuum hose in communication with
a compressor, the hose terminating in a bellows end and spoons
configured to pinch together to remove targets using data received
from the computer regarding mapped targets. In some example
embodiments, alternatively or additionally, a conveyor belt system
is included, mounted to the harvesting subcomponent, the conveyor
belt system configured to receive and move targets from the
harvesting subcomponent to a packing area of the system.
[0007] Alternatively or additionally, some examples include a
turbine and vacuum assembly mounted to a robotic arm of the
harvester subcomponent, the turbine and vacuum assembly in
communication with the computer, configured to chop and suction
material using data from the computer and map. In some examples,
alternatively or additionally, the harvesting subcomponent includes
a picker head assembly with multiple picker heads mounted to one
robotic arm, the picker heads configured to rotate to harvest and
deposit agricultural targets. In some examples, alternatively or
additionally, a foliage management subcomponent is mounted to the
harvester subcomponent, the foliage management subcomponent
including at least two belt drives in communication with the
computing device, the belt drives including a belt configured to
rotate on the belt drives, the belt configured to interact with
foliage of the agricultural targets, and bend the foliage to reveal
agricultural targets for the sensors. In some examples,
alternatively or additionally, the belt drives are synchronized
with the at least one motor by the computing device, to move the
harvester wheels, such that a relative speed of the foliage and a
portion of the belt that is configured to contact the foliage, is
close to zero.
[0008] In some examples, alternatively or additionally, the foliage
management subcomponent includes a suction blower mounted above the
foliage management subcomponent, the suction blower configured to
suck ambient air from around the target foliage, up and away from
the foliage. In some examples, alternatively or additionally,
wherein the foliage management subcomponent includes a second set
of at least two belt drives and a belt mounted around the at least
two belt drives, configured to rotate on the belt drives, wherein
the two combined belts configured to squeeze the foliage to reveal
agricultural targets. In some examples, alternatively or
additionally, the second set of belt drives are synchronized with
the at least one motor by the computing device, to move the
harvester wheels, such that a relative speed of the foliage and a
portion of the belt that is configured to contact the foliage, is
close to zero. In some examples, alternatively or additionally, at
least one air blower is mounted on the foliage management
subcomponent, the air blower configured to blow air toward the
foliage to clear debris. In some examples, alternatively or
additionally, the robotic arm includes a second vacuum assembly
including a hose, configured to receive the agricultural targets
from the picker head assembly and remove the targets from the
harvester subcomponent to a packing area of the system. In some
examples, alternatively or additionally, a back end computing
system is in communication with the harvester computer, the back
end computing system configured to allow human users to review
sensor data and designate agricultural targets for the harvester
subcomponent to harvest. In some examples, alternatively or
additionally, the computer uses neuro network logic to make
preliminary determinations of targets using the sensor data. In
some examples, alternatively or additionally, the computer may
allow the human user to determine an angle of attack for the picker
head to harvest an agricultural target, using the sensor data.
[0009] Systems and methods here include harvesting agriculture,
including a traversing machine with at least two robotic
articulating arms attached to it, at least two wheels attached to
it, and a computing system with at least a processor and memory,
the traversing machine including at least two wheels or tracks
configured below an upper frame, wherein the upper frame supports
the at least two robotic arms, the robotic arms including at least
one seeker subassembly and at least one picker subassembly both in
communication with the computer system, the seeker subassembly
including at least one sensor, configured to send information
regarding potential agricultural targets to the computing system
for analysis, the picker subassembly including a vacuum assembly
coupled to a nozzle with a terminating end, wherein the terminating
nozzle end includes a segmented ported section; the picker
subassembly further including two grappling spoons, the grappling
spoons configured to pinch together toward the vacuum nozzle. In
some examples, alternatively or additionally, the picker
subassembly vacuum nozzle is mounted on an extender actuator. In
some examples, alternatively or additionally, the baffle section is
made of resilient, pliable material. In some examples,
alternatively or additionally, the at least one sensor is at least
one of a camera, laser, and lidar. In some examples, alternatively
or additionally, a foliage management system includes two sets of
belt rollers in communication with the computing system, two belts,
around each of the respective belt roller sets forming an inside
and an outside, the belt roller sets and belts configured on the
traversing machine to receive foliage between the two belts, on the
respective insides of each, wherein the belt roller sets are
synchronized with the at least two wheels or tracks such that a
relative motion of an inside of the belts and a surface on which
the traversing machine is moving, is approximately zero. In some
examples, alternatively or additionally, the foliage management
system includes an ambient air vacuum pump in communication with
the computing system, the vacuum pump configured above the two sets
of belt rollers configured to suck air away from plant beds over
which the traversing machine passes.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] For a fuller understanding of the nature and objects of the
disclosure, reference should be made to the following detailed
description, taken in connection with the accompanying drawings, in
which:
[0011] FIGS. 1A and 1B are diagrams showing example mobile vehicle
examples as described in the embodiments disclosed herein.
[0012] FIG. 2 is a diagram showing example picker head example
details as described in the embodiments disclosed herein.
[0013] FIGS. 3A, 3B and 3C are diagrams showing example extraction
hardware example steps as described in the embodiments disclosed
herein.
[0014] FIG. 4 is a diagram showing example rotating picker head
examples as described in the embodiments disclosed herein.
[0015] FIG. 5 is a diagram showing more example foliage management
features as described in the embodiments disclosed herein.
[0016] FIG. 6 is a diagram of an example foliage management system
as described in the embodiments disclosed herein.
[0017] FIG. 7 is a diagram showing more example foliage management
features as described in the embodiments disclosed herein.
[0018] FIG. 8 is a diagram showing example conveyor belt examples
as described in the embodiments disclosed herein.
[0019] FIG. 9 is a diagram showing additional example conveyor belt
examples as described in the embodiments disclosed herein.
[0020] FIG. 10 is a diagram showing example point-to-point examples
as described in the embodiments disclosed herein.
[0021] FIG. 11 is a diagram showing example pneumatic removal
examples as described in the embodiments disclosed herein.
[0022] FIG. 12 is a diagram showing example sensor examples as
described in the embodiments disclosed herein.
[0023] FIG. 13 is a diagram showing an example networked system
which may be used in the embodiments disclosed herein.
[0024] FIG. 14 is a diagram showing an example computing system
which may be used in the embodiments disclosed herein.
DETAILED DESCRIPTION
[0025] Reference will now be made in detail to embodiments,
examples of which are illustrated in the accompanying drawings. In
the following detailed description, numerous specific details are
set forth in order to provide a sufficient understanding of the
subject matter presented herein. But it will be apparent to one of
ordinary skill in the art that the subject matter may be practiced
without these specific details. Moreover, the particular
embodiments described herein are provided by way of example and
should not be used to limit the scope of the invention to these
particular embodiments. In other instances, well-known data
structures, timing protocols, software operations, procedures, and
components have not been described in detail so as not to
unnecessarily obscure aspects of the embodiments of the
invention.
Overview
[0026] The systems and methods described here include an
automated/semi-automated system with machine(s) that is/are capable
of harvesting agricultural targets such as berries with robotic
assemblies touching the plants or targets themselves. Example
overall systems may include subcomponents such as a seeker or
sensor subsystem to find and locate the targets, that works with
and informs a picking subsystem to harvest the targets. The overall
system(s) may be mounted on wheels or tracks to advance down a row
of targets such as agricultural produce so that the seeker
subassembly may identify and map the targets while the picker
subsystem is used to harvest, gather, and move the targets with
robotic pickers.
[0027] In some example embodiments, additionally or alternatively,
the seeker subassembly includes a camera or multi-camera system
(such as a stereoscopic arrangement) used by a remote operator to
locate targets and three dimensionally map them. In such examples,
these mapped coordinates may then be queued for harvesting.
Additionally or alternatively, in some examples, the harvester
subassembly is then able to follow the seeker subassembly and
harvest the berries whose mapped locations are queued by the seeker
subassembly. In some example embodiments, the harvester subassembly
includes at least one robotic arm with multiple degrees of freedom
capable of reaching into the foliage of a plant and extracting a
target such as a berry. In some examples, the extraction is aided
by a vacuum system to hold the targets. In some examples,
additionally or alternatively, the extractions is augmented by one
or multiple padded spoon graspers, capable of twisting and snapping
a berry stem.
[0028] It should be noted that the examples used here describing
berry harvesting, or even strawberry harvesting in the written
description and/or figures is not intended to be limiting and are
merely used as examples. The agricultural targets to which the
systems here may identify, map, and ultimately harvest may be any
sort including but not limited to berries such as strawberries,
blackberries, blueberries, and raspberries, other examples include
grapes, figs, kiwi, dragon fruit, or other fruits. Vegetables may
be harvested as well, such as Brussel sprouts, tomatoes, peppers,
beans, peas, broccoli, cauliflower, or other vegetable. Any type of
agricultural target may be harvested using the systems described
herein. Additionally or alternatively, the systems and methods here
may be used to target and gather non-agricultural items such as
garbage, or be used to take scientific samples such as rocks or
minerals in environments or situations where it may be advantageous
to avoid human contact or interaction.
Harvester Subassembly Examples
[0029] In some example embodiments, a harvesting subassembly is
included as its own separate vehicle system from the seeker/sensor
subassembly. Additionally or alternatively, in some examples, the
harvesting subassembly may be in communication with or connected to
the seeker subassembly. In some examples, seeker/sensor
subcomponents are integrated into the harvesting assembly and one
machine incorporates all of the features described herein. The
harvesting subassembly may include any number of features that
allow for autonomous, semi-autonomous, or human operable harvesting
of delicate target agriculture such as berries, as described
herein.
[0030] In some examples, either harvesting or seeker/sensor
subassembly may be mounted on its own vehicle subassembly with
wheels and/or tracks or combination of both, to traverse down a row
of agriculture with the seeker subassembly identifying and mapping
the target berries and the harvester subassembly gathering
targets.
[0031] FIGS. 1A and 1B shows figures of example overall traversing
harvesting machine to which any of the various subassemblies
described herein may be attached, mounted on, and/or coupled. Such
a machine may be manned or unmanned, depending on the arrangement
and level of automation programmed into it, as described herein. In
the example, the main traversing subassembly 152 includes various
portions mounted to it including main driving wheels 154 and in
some examples, guide wheels 156. In some examples, guide wheels 156
may be canted outward in order to support traversing a raised mound
101 should a mound be configured as shown. In some examples, tank
treads or tracks may be used instead of wheels 154, 156, and/or a
combination of wheels and tracks may be used.
[0032] In some examples, sensors may be mounted to the frame 153
and/or chassis 155 of the machine 152 as well as or in alternate of
mounting them to a robotic arm. Such sensors may collect sensor
data and send it to a computing system on the harvester assembly or
offboard to a back end system for processing and target
identification. Such target identification may include coordinates
of targets to be picked, and those coordinates sent back to the
system for robotic picking as described herein.
[0033] In some examples, any number of robotic arms 160 may be
mounted to any of various frame portions 153 and/or chassis
portions 155 that comprise the overall traversing subassembly 152.
Such robotic arms 160 may include picker assemblies, sensors, or
combinations of both. It should be noted that many variations of
robotic arms 160 may be used in the systems described here,
including but not limited to robotic wrists with link and joint
combinations with linear and rotational links, gantry robots with
linear joints, cylindrical robots connected to rotary base joints,
polar robots for twisting, and/or jointed-arm or articulating
robots with twisting joints and rotary joints. Any combination of
these or other robotic assemblies 160 may be used on the systems
described herein to manipulate a picker head and/or sensors for
harvesting agricultural targets as described.
[0034] For example purposes, the range 195 of the robotic arm 160
is shown in the FIGS. 1A and 1B to show that the robotic arm 160
may reach different sides of the row mound 101 where targets may be
found and an accumulator for processing targets, such as the
example traversing conveyor 178. In example embodiments, the
robotic arm 160 may include various numbers of joints thereby
allowing for various degrees of freedom to move around and about
the plants and rows, taking different angles for the picker
assembly on the robotic arm 160 to extract a target. In some
examples, the robotic arm 160 may include six degrees of freedom.
In some examples, the robotic arm 160 may include seven degrees of
freedom, or any other number. In various example embodiments, the
robotic arm 160 may be any of various lengths, thereby affecting
the range 195 of the arm, which may be tailored to the needs of the
particular field or mound or target. In some examples, the robotic
arm 160 may include one or more telescoping portions, which may be
elongated and/or retracted, thereby affecting the length of that
portion and the overall reach 195 of the robotic arm 160.
[0035] In some example embodiments, the robotic arms 160 may be
ruggedized in that the tolerances and durability of the arms are
developed for outside, dirty employment. In such examples, the
robotic arms are not to be operated in pristine factory settings.
The systems described here may operate in weather, precipitation,
dirt, mud, heat, cold, and in jarring, rough conditions. As such,
the bearings, tolerances, and actuators may be made of more durable
materials than factory robotic assemblies. In some examples, extra
gaskets may be fitted into the various joints to keep dirt out of
the more delicate metal couplings and pivoting features of the
robotic arms. In such examples, gaskets may be made of rubber,
plastic, or ceramic. The robotic arms may be made with fewer joints
to minimize the number of potential problems that may occur. The
robotic arms may be made of thicker materials, may be heavier, and
be rust-proofed, waterproof, weatherized, and/or otherwise
reinforced.
[0036] It should be noted that the system in FIG. 1A, 1B is shown
straddling one row of plants. In some examples, one system may
straddle two, three, or wider mounds of plants and the example in
FIG. 1A, 1B is merely intended to be an example, and not limiting.
By making the system wider to straddle a second row, two sets of
arms 160 may be used to pick two rows, or three, or four, or
whichever number.
[0037] In some examples, multiple robotic arms 160 may fit onto one
overall traversing vehicle 152. For example, systems may include a
primary picker assembly with a clean-up/redundant picker assembly
which operates behind the primary setup. In those examples, up to
eight picker arms may be employed, four on the primary and four on
the clean-up assembly, with one or two arms operating on each side
of two rows. The clean-up system may operate in the same way that
the primary system operates, to find targets that the primary
system did not harvest, and/or to operate as a redundancy should
one or more arms on the primary system malfunction.
[0038] In examples where targets are fruit plants which are
harvested many multiple times during a single growing season, often
multiple times per week, leaving fruit on a fruit plant may curtail
the productivity of the plant. If the plant senses that it still
has fruit on it, it may not produce more fruit. This would limit
production, so the goal is to remove all of the fruit when ripe. As
the bed rows may allow for some fruit to drape over the side of the
plastic, and become easily exposed to viewing, other fruit may grow
under the foliage, or on top of the bed row tops and be obscured by
foliage. Therefore, to find and harvest as much fruit from each
plant as possible, it may be necessary to maneuver the foliage to
better view and/or harvest fruit targets.
[0039] In some examples, foliage moving arms may be included on the
robotic arms 160 to alter, move, displace, and or otherwise gently
maneuver the foliage of the plant to better expose the targets such
as fruit berries to be picked. In such examples, a bar, or arm, may
be pulled across the top of the foliage in order to temporarily
move it out of the way for the seeker cameras and/or the harvesting
assembly to locate and grapple the target. In some examples, this
foliage moving arm on the robotic arm 160 may be maneuvered
parallel or substantially parallel to the top of the row bed, and
pull across the top of the foliage, bending the plant, but not
breaking the plant leaves. This may reveal targets under the
foliage, those laying on the top of the row bed, or those caught up
in the foliage.
[0040] In some examples, a flexible curtain may be dragged over the
foliage, to avoid damage to the foliage, but still pull it out of
the way for the seeker and/or harvester to operate. In some
examples, this flexible curtain may be a plastic skirt, in some
examples, it may be a fringed or sliced skirt. In some examples, it
may have fringes that drape over the foliage, and yet flex around
the foliage so as not to damage it. As the flexible skirt is pulled
over the plants, it helps the seeker subassembly find the targets
more easily by limiting the area to be targeted with a clean
backdrop. The flexible skirt may be dragged from one side in one
direction during a first harvest and the next time the other
direction, to avoid biasing or pulling the foliage in the same
direction each time.
[0041] In some examples, the overall traversing subassembly 152 may
include a transfer conveyor 178. Such a conveyor may include any
number of conveyor belts, chains, rope, or other mechanism that can
pull materials from one place to another. Such transfer conveyor
may be used to collect harvested targets and move them to a
packaging subassembly, or storage unit.
[0042] In some examples, either or both the picking and/or
harvesting subassemblies or overall harvester 152 may include
location sensing and determining devices. In some examples, GPS
location sensors may be configured on both or either subassemblies
152 for the computing systems to determine locations and/or steer
as discussed in FIG. 13. In some examples, cellular towers and
signals may be used for location sensing by the subassemblies. In
some examples, inertial navigations systems may be used such as a
ring laser gyro, a magnetic gyro and/or any other combination of
such with a computing system. In some examples, additionally or
alternatively, the cameras in the seeker subassembly and/or other
cameras on the harvesting subassembly may be used to identify and
track an agricultural row 101 down which the vehicle 152 may be
steered. The location sensing and/or steering may be fed into any
computing system, either located on the harvesting/seeking systems
or remotely, in order to autonomously, semi-autonomously and/or
allow for human activated remote steering. Any combination of these
or other systems may be used to locate and/or steer the systems
here.
[0043] In some examples, the systems and methods may utilize
self-steering on row 101 with the intention to utilize a human user
droid tender to steer the droid off-row for unloading accumulated
berry containers and reloading empty containers, then finally
steering the droid back onto a new row to be picked. The droid may
have the ability to be converted to full autonomous mode for
turnaround at the head lands as well as unloading and loading
target containers.
Picker Head Examples--Vacuum Point of Contact
[0044] In some example embodiments, the harvesting subassembly may
include at least one picker head that first interacts with the
target in the field to remove or detach the target from the plant
it grows on. Such picker heads may be affixed to or be part of,
attached to, or otherwise in communication with the robotic arms
160 as discussed in FIG. 1A, 1B. In some example embodiments, at
least one picker head may be mounted on the end of a robotic
harvesting arm, alone or in combination with a sensor and/or
lighting system.
[0045] FIG. 2 shows an example picker head assembly, with a front
2A and side 2B view of the same assembly in detail. In the example
shown, the main picker head assembly 202 is mounted with two
actuators, one actuator for a pincers 204 and one actuator for an
extender 206. In some examples, no extender actuator 206 is
utilized. In examples where an extender is utilized, the extender
actuator 206 moves in and down to move the main vacuum nozzle 203
up and down, relative to the robotic arm (not shown).
[0046] The main nozzle 203 may be a hollow tube which may include a
vacuum pump assembly at one end (not shown) and the other end may
be used to secure a coupling suction portion 230 to a target 250
such as a berry. In some examples, the coupling portion 230 may
include one or more bellows or bellow configurations 232 that allow
the coupling portion 230 to stay flexible and malleable to couple
with the target 250. The compression nozzle portion 230 may include
a malleable hood or coupling section 232 which may include one or
more bellow sections, and a rim 234 around an opening 236 to aid in
coupling to a target. In some examples, the compression coupling
portion 230 is or made up of at least one of, or combination of a
neoprene sleeve, a silicone sleeve, a rubber sleeve, or other
natural or synthetic material that is soft and flexible. Such a
malleable coupling section 232 may be configured to deform or
otherwise compress when a target 250 is contacted and may include
baffles or other structure that allows for deformation and
malleability. Such a deformation or compression may allow for the
rim 234 to more easily conform to the target 250 and thereby form a
better suction fit for the opening 236.
[0047] In some examples, the compression coupling portion 230 may
be 1.250 inches in diameter, in some examples, the nozzle may be
between 1.000 and 0.750 inches in diameter, in some examples, the
compression coupling may be 2 inches in diameter. In some examples,
the coupling portion 230 may have an effective ported area of 0.44
sq. in. at rest and an expanded area when stretched over the apex
of a berry of 1.56 sq. in. But in any case, the nozzle could be
customized to any size of intended target.
[0048] In some examples, the compression nozzle portion 230 may
include an internal reverse conical mesh to help capture the target
250 yet be as gentle as possible on them. In such examples, the
mesh creates an environment where the negative vacuum is acting on
a broader surface of the target, thus minimizing the chance of
target damage from localized contact to the grappler edges. This
mesh may form a cradle for the target to lay in even while being
picked, handled, and moved. Such a mesh can be made of silicone
materials for durability and flexibility. Alternate materials may
be used such as a wire mesh, a plastic mesh, or a combination of
wire mesh with plastic coating. Silicon coating may be used on a
wire mesh in some example embodiments as well. In some examples,
the compression nozzle portion 230 and the opening 234 may be sized
for a most average target 250, big enough for the biggest targets
and flexible, but able to grasp and vacuum even a smaller
target.
[0049] Examples may also include an internal spring system, inside
or integrated into the coupling portion 230. Such a spring system
may be made of plastic or metal coil(s) that help return the
coupling portion 230 back to an extended shape after a target is
released by turning off the vacuum and thereby deposited.
Additionally or alternately, an iris or camera lens feature may be
integrated into the nozzle 230. In such examples, the system may be
able to adjust the size of the opening or nozzle end for different
sized targets, opening for larger targets, and constricting for
smaller targets. In such examples, a coil or spring could be wound
tighter for smaller targets and wound looser for larger
targets.
[0050] In some examples, the vacuum hose 203 may be connected with
the main nozzle 230 to impart a suction or lower than ambient
pressure within the nozzle tube 203, and thereby be able to attach
to and secure a target 250. In some examples, a vacuum subsystem
with a vacuum pump may be mounted on the harvesting subassembly and
a vacuum hose may run through or around each harvesting picker
robotic arm. In some examples, vacuum subassemblies may be mounted
on the robotic arm itself, along with a vacuum hose on the picker
head 202.
[0051] In some examples, the amount of suction power that the
pneumatic vacuum system/pump may impart through the tube 203, may
be 35 inches of negative vacuum. In some examples, 50 inches of
negative vacuum may be used. Alternatively or additionally, in some
examples, less than 80 inches negative of vacuum may be used so as
to avoid damage to the target 250. Alternatively or additionally,
in some examples, the amount of suction power may be between 35-50
inches of negative vacuum or between 50-70 inches of negative
vacuum may be used. Additionally or alternatively, the vacuum
system may be able to reverse from suction to blowing air outward,
to clear debris from the bellows, before switching back to a
suction mode for harvesting.
[0052] In some examples, a picker head 202 may include two grappler
spoons, prongs, extenders, arms, or otherwise structures for
grasping a target 212, 214. In some examples, three spoons may be
employed in a similar manner as those examples shown with two as in
FIG. 2. In some examples, four grappler spoons may be configured in
two axes around the picker head 202 assembly. In some examples,
alternatively or additionally, the grappler spoons include a hinged
and/or spring loaded portion at the end to better cushion the
target 250 when pinched. In some examples, the grappler spoons 212,
214 may pivot about the nozzle 203 to impart a twisting motion to
snap a berry or other stem as discussed herein.
[0053] Another portion of the example embodiment of FIG. 2 is the
grappler spoons 212, 214. The grappler spoons 212, 214 may be
configured with the main nozzle 203 between them and be configured
to move in a pincer motion toward the nozzle 203 by a robotic
actuator 216 and a hinge 218 arrangement as discussed in more
detail in FIGS. 3A, 3B and 3C. In some example embodiments, the
grappler spoons include a cushion 220, 222. In some examples, the
cushion 220, 222 may be made of or include closed cell foam,
neoprene, gel filled pads, liquid filled pads, open cell foam,
layers of foam of different densities, a foam backing with a gel
filled pad on top, and/or any combination of the above or other
material that may cushion a target 250 when the grappler spoons
212, 214 pinch the target 250. In some examples, the material
contacting the target 250 is no more than 20-30 durometer in
hardness.
[0054] In some examples, a pneumatic trash cleaning air jet 224 may
be mounted to the end of the grappler spoon 212, 214 in order to
help clear debris. In such examples, air holes may be configured on
the end lip of the spoons and face in various directions to direct
air toward foliage. In some examples, a line of holes may be
configured on the end lip of each grappler spoon 212, 214.
Example Picker Head and Picking Steps
[0055] FIGS. 3A, 3B and 3C show three example snapshots in the
multi-step process of target 350 acquisition and grappling using
the picker head assembly 302 as described, where each step shows
two angles 313, 305 of the same picker head assembly 302, front 313
and side 305. In an example target acquisition, first, 3A, the
picker head 302 is directed to a target 350 by a seeker subassembly
as discussed herein, using passed coordinates and/or manually
steered. Once directed and in place, a robotic arm (860 in FIG. 8)
maneuvers the picker head 302 into close proximity of the target
350 where the compression nozzle portion end 330 may attach to the
target 350 (as described in FIG. 2) resting on the ground or
surface 301 using low pressure imparted by the vacuum nozzle 303.
In some examples, this vacuum nozzle 303 may be maneuvered in an
extended configuration 344 using the extension actuator 306. In
such a configuration, the compression coupling portion 330 may
attach, suction, or otherwise temporarily hold the target 350 and
thereby secure the target 350 with the vacuum suction through the
vacuum tube 303.
[0056] In examples using a pneumatic vacuum, it may be used to
first secure the picker head assembly 302 to a target 350 such as a
strawberry. This vacuum attachment 303 to a target 350 may allow
for the picker assembly 302 to extract a target 350 from foliage,
pick it off a stem and/or off a resting surface 301. By securing a
vacuum attachment 330 to a target 350 first, further even more
grappling, twisting, and/or handling, may be accomplished with
spoons 312, 314, as described herein to aid in harvesting and
moving targets.
[0057] In such examples, pliable bellows 330 may be positioned on
the robotic assembly to within range of a specified target 350 and
the bellows section 330 extended 344 to within vacuum range.
Through the bellows tube 303, a negative pressure may be generated
by a pneumatic vacuum, pump, or other air or pneumatic suction
device (not shown) thereby sucking air through the nozzle end 336,
through the bellows tube 303 and thereby through the various ports,
holes, slots, and/or other shaped voids in the end of the bellows.
As such bellows 330 material may be made of pliable material to be
able to better conform to the shape of a target more easily to
thereby secure a better vacuum hold on the target.
[0058] Such a task is made more difficult by the variety of shapes,
sizes, and orientation of targets needed to be grappled. Such a
task is also made more difficult by the potential of nearby
foliage, other targets, stems, dirt, sticks, etc. which could
interfere with the vacuum suction, and/or reduce the vacuum
pressure that may be applied to a target by the bellows. The
purpose of the bellows vacuum system is to create enough of a
vacuum attachment to the target such as a strawberry to secure the
target from all or as many orientations as possible, including the
apex point in examples of a target such as a strawberry.
[0059] In FIG. 3B, the same picker head assembly 302 is shown with
a front 313 and side 305, the main nozzle tube 303 may be retracted
using the actuator for the extender 306. In some examples, the
actuators 306, 304 may be pistons operated by pneumatic and/or
hydraulic features in communication with a computing system to
operate them. In some examples, the extender actuator 306 may
operate in a generally upward/downward motion 340 away from the
ground 301 or surface and toward the interior of the picker head
assembly 302. Some examples may forego the retraction step and not
utilize the extension actuator 306 and/or may not be configured
with one. In examples where retraction is utilized, with the target
350 attached to the compression nozzle portion 330 which is now
retracted into the picker head assembly 302, the target 350 may be
generally aligned with however many grappler spoons 312, 314, for
example, are fitted on the picker head assembly 302. In some
examples, the compression nozzle portion 330 and thereby the target
350 may be retracted to align with the respective spoon cushion
portions 320, 322 of the grappler spoons 312, 314, no matter how
many grappler spoons are utilized.
[0060] In some examples, as shown through the FIGS. 3A, 3B and 3C,
the spoon actuator 304 may be a piston assembly in communication
with a computer to receive instructions and send data and
configured to raise and lower a bracket assembly 318 on the picker
assembly 302 which may interact with a top end 372, 374 of each
spoon arm 312, 314 to pivot each spoon arm 312, 314 about a pivot
axis 310, 311, thereby opening and closing, or pinching the two
spoons 320, 322 together, and moving them apart. In some examples,
the spoon arms 312, 314 may include springs in the pivot axis areas
310, 311, which may bias the spoons in the closed or pinched
position, and the bracket 318 may move in relation to the top spoon
ends 372, 374 to move down to work against the spring tension to
open the spoons, and up to allow the springs to pinch the spoons
312, 314 together. This actuation may take place due to the
interaction between ramped or angled portions of the top of the
spoon arms 372, 374, and the bracket 318 moving against the spring
tension in the pivot axes 310, 311, pivoting each arm 312, 314
about its respective axes 310, 311.
[0061] As targets 350 may vary in size and shape, the alignment
with the spoons 312, 314, may be obtained by retracting 340 the
compression nozzle portion 330 so that the rim 334 of the
compression nozzle portion 330 is at a place just above the
respective cushion portions 320, 322 of the grappler spoons 312,
314 thereby ensuring that the respective cushion portions 320, 322
of the grappler spoons 312, 314 are able to pinch together to grasp
342 the target 350 without touching the compression nozzle portion
330 when they are in the closed position. Other shapes and sizes of
spoon pads 320, 322 may be used to cushion the targets 350 when the
pincher arms 312, 314 are brought together.
[0062] In some examples, sensors may be placed on or near the
grappler spoons 312, 314 and/or cushions 320, 322, at the hinges
310, 311, under the pads 320, 322, or other areas, to sense the
size and/or shape of the target 350 once gripped. In such examples,
a feedback loop may be used from the sensor data to adjust the
distance the compression nozzle portion 330 is retracted to align
the target 350 with the grappler spoons 312, 314. In some examples,
such a sensor may be a light sensor, a laser sensor, a proximity
sensor, piezoelectric pressure sensors, and/or a camera to align
the target 350 with the grappler spoons 312, 314.
[0063] In some examples, a pneumatic trash cleaning air jet 324 may
be mounted to the end of the grappler spoon 312, 314 in order to
help clear debris in the field, foliage, and other obstructions in
the field to aid in target 350 acquisition. Such an air jet 324 may
include one or more nozzles attached to pneumatic pumps and tubes
that are able to blast jets of air in various directions, thereby
moving, flapping, or otherwise disturbing plants, leaves, dirt,
sticks, stems, or other debris that the system is not trying to
target, but might be in the way of a target. In some examples, the
ends of the grappler spoons 312, 314 themselves may include one or
more nozzles, ports, or holes for air jets to blast debris. In some
examples, the outsides of the grappler spoons 312, 314, opposite
the respective cushion portions 320, 322 may include one or more
nozzles or ports, or holes for air jets to blast debris.
[0064] In the example of the third step, FIG. 3C showing the same
picker head assembly 302, front 313 and side 305, when the
compression nozzle portion 330 and thereby the suctioned target 350
is retracted 340 off the ground or surface 301 and aligned with the
grappler spoon cushions 320, 322, the gripper spoons 312, 314 may
be actuated by the grappler spoon actuator 304 as described, and
squeeze together 342 to grasp the target 350. In some examples, the
retractable end of the baffler 332 may retract above or past the
point where the pincher spoons 312, 314 may pinch together 342 so
as to be able to hold a target 350 with the suction through the
pliable baffles section 332 and the spoons 320, 322 at the same
time. Only by retracting 340 far enough could a target 350 be
secured by both systems at the same time. Such a configuration also
allows for a handoff, for example, the suction may be turned off
once the retracted portion 332 is secured by the pincher spoons
320, 322.
[0065] In this third configuration 3C, the target 350 may be
grasped by the grappler spoons 312, 314, and may still be held by
the compression nozzle portion 330 and the vacuum pressure from the
main nozzle 303. In such examples, a feedback loop to the onboard
or offboard computing systems may be used from the sensor data to
adjust the pressure used to squeeze the target 350. In some
examples, additionally or alternatively, a single expansion spring
(not shown) that may be connected between spoons 312 and 314 may be
used. The spring may set the tension that the grappling spoons 312
and 314 may exert onto the target 350. In some examples,
additionally or alternatively, the sensors may be piezoelectric
pressure sensors on or under the grappler spoon cushions 320, 322.
In some examples, additionally or alternatively, the sensors may
include cameras to visually detect securing the target 350. In some
examples, additionally or alternatively, the sensors may be tension
sensors on the grappler spoon actuator 304 to sense the pressure
exerted on the closure of the grappler spoons 312, 314. In some
examples, additionally or alternatively, the sensors may be tension
sensors on the gripper spoons 312, 314 and/or in a hinged portion
of the gripper spoons 312, 314 to sense the pressure exerted on the
closure of the grappler spoons 312, 314. In some examples, feedback
loops may be analyzed by computer systems in communication with the
grappler spoon sensors 312, 314, and also in communication with the
actuators for the grappler spoons 312, 314 to adjust the pressure
on the target 350, which may be used to secure differently sized
targets 350 while minimizing damage to larger targets 350 and/or
making sure smaller targets 350 are secured.
[0066] In some example embodiments, once the gripper spoons 312,
314 have secured the target 350, the vacuum suction may be turned
off by the computer, reduced, or otherwise cut off from the
compression nozzle portion 330 which would in turn release the
pressure holding the target 350 to the compression nozzle portion
330 but leaving the target 350 in control of the grappler spoons
312, 314. In some examples, the padded grappler spoons 312, 314 may
be configured to rotate 333, to thereby flick and turn the target
350 to remove them from their stems and thereby avoid having to cut
a stem or plant in any way. This removal process of a target 350
from a stem may be advantageous in the shelf life of the target
after harvesting and may be cheaper and easier to accomplish in the
field.
[0067] In some examples, this twisting motion 333 may be a 90
degree twist of the grappler picker head assembly 302. In some
examples, this may be a 180 degree twist. In some examples, this
may be between an 80 degree and 100 degree twist to snap a target
350 stem. In some examples, a snipping element may be used in lieu
of or in addition to the snapping, twisting motion of the grappling
spoons 312, 314. In such examples, a longer target stem may be
desired, and snapping or twisting may remove the stem close to the
target 350.
[0068] In some examples, the snapping of the stem by twisting may
benefit if the stem of the target 350 plant is pulled out and away
from the plant in order to impart a strain on the stem. In such
examples, the pulling of the stem first, and then twisting the stem
may result in cleaner and/or more accurate stem snaps. In some
example embodiments, the extension/retraction actuator 306 may
include a sensor that may sense resistance as the target 350 is
retracted. In some examples, tension sensors may be placed in
joints of the robotic arm(s) in communication with the computing
systems, to make such a determination.
[0069] In some examples, the twisting motion 333 may be imparted
only when the resistance of the retraction of the target 350 meets
a particular threshold, as determined by computing systems in
communication with such sensors, thereby indicating that the stem
of the target 350 is under strain or is otherwise stretched. Such
resistance sensors may include a piezoelectric sensor, a strain
gage, or other sensor mounted in or on the retraction actuator 306.
In some examples, the target may be a berry that includes a calyx
portion where a few leaves and the stem attach to the target berry.
In some examples, this calyx portion may be identified by the
seeker subassembly to help determine which direction to pull the
target, normal to the calyx portion. In such examples, the camera
and computer system may be able to identify a color variation
between the berry itself and the calyx leaves and thereby the
stem.
[0070] In snipping examples, a scissors, saw, clipper, or other
sharp pincher may be secured to the picker head assembly 3C to cut
the stem of the target 350 at a desired length.
Multi Headed Picker Examples
[0071] In some examples, additionally or alternatively, one robotic
arm may include more than one picker head. Such an arrangement may
allow for faster picking, as picker heads move between harvesting
and dropping targets to be packaged and processed.
[0072] FIG. 4 shows an example of a multi-picker, in this example,
a three headed picker rotating assembly mounted on one arm 460
which may be configured to harvest and drop targets into a bucket
accumulator. In the example of FIG. 4, each robotic arm 460
includes three picker heads 402A, 402B, 402C. The picker heads
402A, 402B, 402C may be configured as described in any of the
various example embodiments described herein. In some examples, the
picker heads 402 may be variants of one another, for example,
different sizes that are configured to harvest a certain sized
target. For example, a small, medium, and large picker head 402A,
402B, 402C may be configured on one robotic arm 460 with different
diameters on each nozzle or coupling portion.
[0073] As designed, and in use, when the targeting sub-system
identifies a target and estimates its size, it may assign the
respectively sized one of the three sized picker heads 402A, 402B,
402C to harvest that target. In some examples, the various picker
heads 402A, 402B, 402C have different features on them, for
example, different air jet blasters for different foliage
situations. In some examples, the various picker heads 402A, 402B,
402C may be redundant for maintenance and allow for one or more
heads 402A, 402B, 402C to break or malfunction, and then be
automatically replaced by a new and functioning picker head
assembly 402A, 402B, 402C. In some examples, the functionality of
the picker heads 402A, 402B, 402C may be triaged or placed in a
hierarchy, so the best, most functional picker head 402A, 402B,
402C is used, and if it breaks further, then the next most capable
head 402A, 402B, 402C is used, etc.
[0074] Any of various examples of multiple heads and uses for
multiple heads may be employed by the systems described herein. It
should be noted that the number of three picker heads is not
intended to be limiting and any number of picker heads may be
mounted to a single robotic arm. In some examples, each robotic arm
460 could include two picker heads 402A, 402B. In some examples,
each robotic arm 460 could include four or more picker heads 402.
In such examples, the picker heads 402A, 402B, 402C are configured
onto a rotating portion of the robotic arm 460 to spin and/or
rotate 499 to the target area and to a drop off or packing area as
described. In some examples the rotation is in a direction
perpendicular to the ground, on an axis parallel to the ground as
shown in FIG. 4. In some examples, the rotation may be in a
carousel fashion, parallel to the ground, with an axis
perpendicular to the ground. The example shown is not intended to
be limiting.
[0075] In the example, first 4A, one of the multiple picker heads
402A first couples to a target 450. Next, 4B the picker head
rotates 499 to bring the first picker head 402A with the coupled
target C50 into alignment with whichever accumulator 470 is being
used. In some examples, this rotation also brings another picker
head assembly 402B into approximate alignment with another target
C50. When in position, the first picker head 402A with the target
450 decouples, releases, or otherwise drops off the target 450
while the second picker head 402B couples to another target 450 and
the assembly can then spin or rotate again 499.
Foliage Management Examples
[0076] In some examples, in the field, the targets may grow and
rest not only on the sides of the walls of the planter beds, but
sometimes on the tops of the planter bed crowns and sometimes
within the foliage of the plant itself. In such examples, the goal
of finding and harvesting the targets using the systems and methods
here may be more difficult than if the targets are more easily
presented. In some examples, the foliage may be dense, it may
impede the sensors from finding targets. In some examples,
alternatively or additionally, the foliage may impede the
harvesting pickers from extracting targets, even if they could be
found.
[0077] Thus, in such examples, it may be helpful to finding and
harvesting more targets, to be able to manipulate the foliage on
the plant itself to better reveal any targets that may be growing
and/or resting inside or under the foliage.
[0078] In some example embodiments, as described, an additional
feature may be a trash grinder. In some examples, impediments such
as dead foliage, spoiled targets, or other non-targets may get in
the way of the seeker subassemblies and/or harvesting subassemblies
such as a picker head, from cleanly finding and harvesting targets.
In such examples, the system may employ an additional trash
disposal system.
[0079] FIG. 5 shows an example robotic assembly with an optional
muncher, grinder, or other composting feature head with a target
berry 550 and foliage, trash, dead leaves, or other material 552
that is not to be harvested by can get in the way of the targets
550. Such trash 552 may obscure the sensors from the targets to be
harvested 550 and may clog the vacuum or picker head assemblies as
shown in FIG. 3A, 3B, etc. Therefore, a trash muncher features may
be used to clear debris 552 that is not desired to be harvested and
can impede harvesting.
[0080] In such examples, additionally or alternatively mounted on
and or separately from the picker heads and seeker heads, a
separate hose 510 or hose used for the trash muncher features with
suction capabilities including a separate or shared suction or
vacuum pump (not pictured) may include grinding blades 520 that are
capable of spinning by way of a motor 560. The vacuum pump and/or
motor 560 may be in communication with the computer system as shown
in FIG. 14 etc. to be able to turn spinning blades on, turn
spinning blades off, speed up rotation, slow down rotation, turn
suction on, turn suction off, etc.
[0081] In such a way, the trash material 552 may be suctioned up
and be ground up to be collected or ejected 530. In some examples,
such grinder blades 520 may be located at the mouth of the vacuum
tube 510 as shown. In some examples, additionally or alternatively,
blades 521 may be located further up the vacuum tube 510 instead of
or in addition to blades 520 at the mouth of the tube. A motor 560
may be used to spin the blades 520, 521 inside the tube 510 and be
in communication with he computer to send and receive data
including commands to turn on, off, speed up, low down, the
spinning blades.
[0082] A pump or vacuum system (not shown) may be coupled to the
vacuum tube 510 to suck air, trash, and/or impediments up 522 and
away from the surface 501. Such air and impediments, after being
ground by the spinning blades 520, and/or 521 may be sucked back
and out 530 to the ground, away from the plants, or to a collection
bin (not shown). Such a vacuum pump (not shown) may be in
connection with a computing system onboard to control the vacuum
motor to start, stop, speed up, slow down, or any other kind of
command. In such a way, a clean-up or preparation step may be added
in order to remove impediments to help expose targets before the
sensors look for and map targets, and picker heads harvest the
targets.
[0083] In some examples, additional trash grinding systems may work
in conjunction with the sensor systems to identify impediments for
removal. Such systems may utilize the same targeting and coordinate
mapping system as the target systems, but instead employ the trash
grinding system to remove the impediments instead of harvest the
targets. In such a way, an iterative process may be to utilize the
sensors to first locate and map impediments for the trash grinding
system to remove impediments, then the sensors may be utilized to
locate and map targets from the cleaned up plants, to the pass to
picker heads to harvest. In such a way, additional targets may be
harvested that otherwise might be unseen or too hard to reach.
Foliage Roller and/or Pneumatic Examples
[0084] Additionally or alternatively, FIG. 6 shows an example
system with foliage management features to aid in finding and/or
harvesting such targets. As shown in FIG. 6, three views of the
same scene are shown, a top down view 601, a side view 603 and an
end-on view 605 to the planter bed 660, harvester portions, and
plants 620. It should be noted that in FIG. 6, not all of the
harvesting assembly is shown, only a portion having to deal with
foliage management, and specifically, foliage manipulating rollers
610, 612, pneumatic air jets 630, 632, robotic assembly 662, and
pneumatic vacuum 640 and although they appear in the example FIG. 6
as not connected to anything, would be mounted on the frame of the
harvester, and are shown in FIG. 6 for illustrative purposes only.
It should be noted that all of these features are optional and may
be employed on the harvesting system alone or in any combination
described here or otherwise. They are not reliant upon one another
but may benefit from employment of the other features, meaning any
combination of the above may be used, alone or together.
[0085] In some examples, an arrangement of belts and rollers may
push or bend the main foliage of a plant 620 to allow the robotic
assembly 662 including sensors to more easily observe and/or
harvest the targets or berries 602. Such foliage manipulating
rollers may be more gentle on a plant foliage 620 than just a bar
or pusher, because the speed at which the belts move may be
synchronized with the forward movement of the harvesting machine
down a row (as shown in FIG. 1A, 1B) such that there is imparted no
or little shear force on the foliage 620, but instead only the
intended pushing or bending force to gently move the foliage 620 to
the side.
[0086] In the example views shown in FIG. 6, the harvesting machine
may include belts 610, 612, spun by rollers 614, 615, 616, 617,
618, 619 that include pins or cylinders capable of turning
mechanically by a chain, gear, or motor, as shown in the top-down
view 601 in communication with onboard and/or offboard computer
systems. Between and among these rollers may be a belt, sheet,
ribbon, or other material 610, 612 kept taught or semi-taught
around the rollers 614, 616, 618. The belt 610, 612 may be made of
fabric, plastic, woven fibers, rubber, or plastic coated fabric.
Such material may be of a thickness that is sturdy for field work,
yet gentle on the plant foliage 620.
[0087] In the example of FIG. 6, the rollers 614, 615, 616, 617,
618, 619 are generally arranged perpendicular to the ground 660,
pointing upright such that the outside of the belts 610, 612,
rotate in the same general direction as the movement of the
harvesting machine 650. In this way, the inside of the belts, where
the belts touch the plant 620, is moving the opposite way that the
machine is moving 650 down a plant bed row. In some examples, this
rotation may be synchronized with the forward movement of the
machine down a plant bed row, by computers which receive data
regarding the speed of the harvester, and thereby the speed of the
plants 622 as they are traversed, and the rollers 614, 615, 616,
617, 618, 619 to speed up or slow down at the same time and rate,
such that the interior side of the belts 610, 612, are moving at or
about the same speed as the plants 620 as the machine traverses the
planter rows 660, such that the relative speed between the interior
of the belts 610, 612, and the plants 620 is zero, or close to
zero. This allows the plants 622 to be touched by the belts 610,
612 with as little disturbance as possible, and without shearing,
ripping, or otherwise damaging the foliage 620, and allowing only
the bending or pushing of the main foliage 620 to be effected on
the plant by the main roller system.
[0088] In some examples, the belts 610, 612, and rollers 614, 615,
616, 617, 618, 619 are configured to interact with the plants 620
by squeezing the main foliage into a narrow opening between the two
belts 610, 612. In such a way, the targets or berries 602 may be
more easily seen by and/or harvested by the robotic arms 662 as
described herein. In some examples, only one belt 610 may be
arranged to push aside the foliage on the plant 620 in one
direction. In some examples, instead of the two belts 610, 612,
being aligned so as to squeeze the plant foliage 620 at the same
time, they may be staggered, one after the other such that the
foliage of the plant 620 may be pushed to one side by one belt 610,
then to the other side by the other belt 612 instead of
simultaneously. Any combination of these pushers to one side,
and/or squeezing may be utilized.
[0089] It should be noted that the arrangement of rollers 614, 615,
616, 617, 618, 619 may be more or fewer in number than three per
belt 610, 612. In such examples, only two rollers may be used to
roll a belt, or more such as four rollers may be used to roll a
belt. In such examples, and/or with a three roller example, the
configuration of roller positions on the machine may be changed
such that the shape of the rolling belt may change. In the example
of FIG. 6, the three roller belt configuration includes a generally
wide triangular shape followed by a generally narrow triangular
shape. This allows for the plant 620, when the machine is moving,
to be gathered at a wide end by the first, most widely spaced
rollers 614, 615, and then pinched into a narrower position between
the next rollers 618, 619, and held there through the rest of the
traverse, where the robotic arms identify and/or harvest targets
602, before being released at the last rollers 616, 617 when in
use.
[0090] It should be noted that side view 603 does not show both
belts, but only one belt 612 in either a one belt or staggered belt
embodiment, or without the second belt to show the inside of the
embodiment for purposes of explanation. As mentioned, any
configuration of one, two, staggered, or aligned belts may be
utilized as described herein.
Air Jet Examples
[0091] In some example embodiments, alone or in combination, air
jets 630, 632 could be configured to blow ambient air onto the
foliage 620 and/or the plant beds 660 to remove trash, clear
debris, move foliage 620 or otherwise aid the robotic assembly 662
in finding and harvesting targets 602.
[0092] In such examples, compressors and/or pumps (not shown) may
send compressed air down a tube (not shown) and through a nozzle or
jet 630, 632 to push or maneuver foliage 620 for sensors and/or
harvesting features 662 to better find and harvest targets 602.
Such compressors may be positioned in, on, or around the harvesting
machine. In some embodiments, alternatively or additionally, such
tubes may run down robotic arms 662 and thereby be able to be
maneuvered in the same or similar fashion as the seekers and/or
harvesting picker heads 662. In some examples, such air jets 630,
632 may be configured next to, adjacent, and/or near the seeker
and/or picker head to aid in finding and harvesting targets
602.
[0093] It should be noted that the number of air jets 630, 632,
does not need to be two as shown, and in fact could be any number
of air jets, configured to blow air on the sides of the planter
beds 660 and/or targets 602 or plants 620. In some examples, the
air jets may utilized the same vacuum system as the vacuum 640 if
such a system is used to blow the same air sucked from the top. In
some examples, a filter may be used to filter the air sucked by the
vacuum 640 to blow by the air jets 630, 632. In some examples, the
air jets 630, 632 may include nozzles which concentrate the air
flow into a faster speed. In some examples, the air jets 630, 632
may include diffusers, or other features that push air into
directions which may be able to be aimed or moved by a motor in
communication with a computer.
[0094] More embodiments of Air Jets are described in more detail in
FIG. 7. In the example, similar to FIG. 6, three sides of the same
plant 720 and plant bed 760 are shown with a top-down 701, end-on
705 and side view 703. In the drawing, the plant 702 is being
deflected to the side by a roller setup 710 as described in FIG. 6,
or some other kind of arm or wall and harvested by any number of
robotic assemblies 762 as described herein. In such examples,
various air jets connected to air pumps (not shown) could be
arranged on the harvester system to blow ambient air onto the
plants 720 and/or plant beds 760 to allow for more easily finding
and harvesting the targets 702 by the robotic assembly 762. In the
example of FIG. 7, there are air jets on both sides of the plant
720 with one set of air jets 732 arranged to layover the canopy of
the plant 720 and comb the foliage. In the example shown, three air
jets 732 are so arranged to blow air at the top of the plant canopy
720 and help push the plant to one side. In such examples, any
number of air jets with or without nozzles or diffusers may be
used, for example, three are shown in FIG. 7 but one, two, three,
four, five or more could be used for this purpose. These air jets
732 are generally arranged toward the top of the plant canopy,
aimed to push or deflect the plant 720 in a particular direction.
Either alone or in combination with a belt as shown in FIG. 6, the
purpose of deflecting the plant foliage 720 would be to move debris
and material out of the way for the robotic assembly 762 to find
and/or harvest targets 702 as described herein.
[0095] In some examples as shown in FIG. 7, another air jet 730 or
set of air jets alternatively or additionally, may be configured
for another purpose. In such examples, the second grouping of air
jets 730 either alone or in combination with the others described
herein, may be used as cross-air jets to blow targets off the top
of the plant bed 760 and down onto the side to make it easier for
the robotic assembly 762 to find and/or harvest targets. In the
example of FIG. 7, the cross air jets 730 are arranged lower than
the foliage air jets 732 and instead arranged to blow air down and
across the top of the plant bed 760. This may induce targets 702 to
fall or otherwise move onto the side of the plant bed 760 to make
it easier for the robotic assembly 762 to find and/or harvest
targets 702. The number of cross-air jets could be one as shown, or
more, such as two, three, four, five, or more. The arrangement of
cross-air jets 730 may be configured onto the harvesting machine in
any way, connected to an air pump to blow ambient air onto the
plant bed 760 and/or targets 702.
[0096] In some examples, alone or in combination, alternatively or
additionally, air jets 730, 732, may be configured as comb-air jets
that follow an elliptical path through and around the plant 720. In
such examples, the air jets 730, 732, may comb through the leaves
and canopy of the plant 720, thus releasing targets. Any
arrangement of air jets to manipulate foliage, release targets, or
otherwise blow away debris and dead foliage may be used. And any
combination of nozzles and/or diffusers may be arranged on the air
jets as described herein to either focus the air into a faster
moving stream in a nozzle, or widen the stream of air in a
diffuser. And number of sub nozzles may be configured on the air
jets as well, in order to control the direction of the air jet
streams of air, such sub nozzles may include movable parts,
controlled by a computer to direct air flow in a desired location.
Air pumps pushing air through the nozzles may also be controlled by
computers to turn on, turn off, pulse, or otherwise push air toward
an intended direction and velocity and/or pressure.
Vacuum
[0097] Turning back to FIG. 6, in some examples, a high volume
suction blower, or vacuum 640 may be attached near the belt system.
In such examples, the vacuum system 640 may pull air up and out
away from the plant foliage 620. In some examples, the vacuum
system 640 may come just before the belts 610, 612 such that the
entirety of the foliage 620 may be pulled up or urged upward before
being squeezed between belts 610, 612. In such examples, the vacuum
640 may also remove some dirt, debris, dead leaves, bugs, or other
unwanted materials. In some examples, the material captured from
the vacuum 640 may be blown aside, mulched, or otherwise
recycled.
[0098] In some examples, air jets 630, 632 may be used in
combination with the belt systems 610. In some examples, the air
jets 630, 632 may be staggered to control air positioning at the
plants 620, targets 602, and/or plant beds 660.
Target Accumulator Examples
[0099] In some examples, additionally or alternatively, the
harvester subassembly may include any number of target
accumulators. In some examples, a conveyor belt system may be used
to move accumulated targets from the robotic arm and/or vacuum hose
to a storage area and/or a packaging area. In some examples, a
bucket system may be used to gather and/or move accumulated targets
to a storage area and/or a packaging area.
[0100] FIG. 8 shows an example single headed picker assembly which
is configured to drop targets into a multi-conveyor belt
accumulator assembly from the top down, and from the side. In FIG.
8, the picker head 802 is shown mounted to a robotic arm 860 which
may position the picker head 802 in any of various poses in order
to couple to a target 850. In the example, the targets are on a
mound 801 which the robot arm 860 traverses from above.
[0101] In use, multi-conveyor system as shown in FIG. 8 is
positioned by the robotic arm 860, to extract a target 850 and drop
it, and or/place it onto a first capture conveyor belt 880. In some
examples, this capture conveyor 880 is configured to move
perpendicular to a mound row 801 and/or otherwise away from the
plants and out toward an area which is more open. In some examples,
this capture conveyor 880 is a flat conveyor, and wide enough to
allow for the movement or rolling of targets, such as four inches
wide. In some examples, the speed of this capture conveyor belt 880
may be between 1 in to 5 inches per second. In the example, this
capture conveyor 880 may be configured to move targets 850 to a
second transfer conveyor belt 878. In some examples, the transfer
conveyor 878 is configured to move targets in parallel with a row
mound 801, out and away from the ground to be processed.
[0102] In some examples, the transfer conveyor 878 may include fins
or other compartmentalized portions to reduce rolling of targets
850 as they move up the transfer conveyor 878 at an angle. In some
examples, the transfer conveyor 878 may move targets 850 to a
packaging station or other handling area on a main vehicle, or off
the main vehicle to an accumulator or other receiver section as
described herein. In some examples, the transfer conveyor 878 may
be more narrow than the capture conveyor 880, such as between 3 and
5 inches wide. In some examples, the transfer conveyor 878 revolves
at speeds of between 5 inches and 10 inches per second.
[0103] In some examples, a brush or other skirt assembly 876 is
configured to bridge the capture conveyor 880 and the mound
801.
[0104] FIG. 9 shows an example set of diagrams showing the conveyor
belt arrangement from the side, in three steps of an example pick.
In such examples, a single picker head 902 is attached in a
rotatable manner to the arm 960. In FIG. 9, the first diagram 9A
shows the picker head 902 grappling a target 950 as described
herein. The capture conveyor belt assembly 980 is shown moving the
targets away from the plant 999 and bed and toward another removal
arrangement such as a transfer conveyor 978 or other conveyor or
bucket system.
[0105] 9B shows the picker head assembly 902 retracting the target
950. 9C shows the picker head assembly 902 rotating (arrow) on the
arm 960 and dropping or placing the target 950 onto the capture
conveyor 980 which moves the target to the transfer conveyor 978.
In such examples, the picker head 902 would then rotate back to
another target for grappling and extraction.
[0106] Some examples may employ human packers, that is, the
harvesting system merely uses conveyor systems as shown in FIGS.
1A, 1B, 8, and 10 to transport the targets to a central point on
the system for a human to handle and pack into a box, clam shell
plastic, or other shipping/sales container. But some example
systems may utilize a more automated system to pack targets. In
such examples, similar subsystems to those described for seeking,
targeting, and picking, may be employed to deposit targets into a
container and arrange the targets in a manner that minimizes empty
space and are arranged in a stable arrangement using picker heads.
In such examples, a three dimensional mapping of not only the
target itself but the other targets which are deposited into a
container may be used to maneuver newly placed targets in an
arrangement that is stable and uses the space provided. A three
dimensional puzzle results in each container, placed by the picker
arms or a separate set of picker arms (placer arms) that would
operate by picking targets from the conveyor, and depositing them
in a container, using the same suction and/or pincher system as
described in FIGS. 2, 3A, 3B and 3C. In some examples, the
grappling spoons may not be necessary to pack targets as only the
baffle and nozzle vacuum system may be enough to maneuver them.
[0107] FIG. 10 shows an example which uses either a single transfer
conveyor or a bucket instead of a multi-conveyor system. In FIG.
10, a single headed picker assembly which is configured with the
picker head 1002 to suction couple to a target 1050 as described
herein 10A, then move the target 10B to a bucket, or other
accumulator 1070 where the picker head 1002 may then deposit, drop,
or otherwise place the target 1050 for further removal or
processing. In some examples, the accumulator 1070 is a transfer
conveyor. In some examples, the accumulator 1070 includes a
package.
[0108] FIG. 11 shows an example which uses a vacuum hose instead of
a multi-conveyor system. In FIG. 11, a single headed picker
assembly which is configured with the picker head 1102 to suction
couple to a target 1150 as described herein 11A, includes a second
vacuum assembly 1190 move the target 11B for further removal or
processing. In some examples, the pneumatic assembly 1190 includes
a vacuum transfer tube 1192 which moves the targets 1150 out for
processing. In some examples of the pneumatic transfer system as
shown in FIG. 11, no grappling spoons are necessary if the vacuum
assembly 1190 is positioned relatively below the picker head 1102.
In such examples, the primary picker head vacuum may switch off
when it is retracted and positioned over the mouth 1194 of the
vacuum assembly 1190 and the target 1150 drops into the vacuum
assembly 1190 and transfer tube 1192 by pressure differential
suction.
Seeker/Sensor Subassemblies
[0109] In some examples, the harvesting described herein is
directed by a seeker subassembly that is able to identify targets
for harvesting, pass coordinates for the targets to the picker
subassembly for extraction. Such seeker subassemblies may include
any number of cameras (visible light, thermal, UV or other),
radars, lidars, lasers, acoustic location finders, GPS, inertial
navigation systems, piezoelectric sensors, and/or any combination
of these or other sensors to locate and identify targets.
[0110] In some example embodiments, as discussed herein, the seeker
subassembly may include a camera and/or multiple cameras arranged
so as to be able to view the target foliage and thereby the target
agriculture to be harvested. In some examples, multiple cameras may
be arranged on the seeker subassembly such that images taken from
the multiple cameras may be processed by a computing system to
create three dimensional (3-D) images using machine vision. In some
examples, these images are made of pixels and the computing systems
is able to identify targets represented by pixels to be harvested
and map the targets in three dimensions. In some examples, the
cameras may be configured to acquire multi-spectral or
hyper-spectral imagery to enable the use of advanced analysis
algorithms for evaluating fruit health, quality and ripeness. In
some examples, the images gathered may include those of a thermal
imaging system for evaluating the temperature of the berry to be
harvested. These cameras may comprise of cooled or uncooled sensors
generating area-scanned images of at least 640.times.480 pixels.
Some embodiments may utilize a single thermopile based sensor to
provide an integrated temperature measurement of the mean
temperature of the berry.
[0111] In some examples, the target identification is automated by
Artificial Intelligence (AI) computing systems working in
conjunction with a neural network data base and the camera systems.
In some examples, the target identification is aided by a human who
is analyzing a visual representation of the image data sent by the
camera(s). The human operator may utilize human logic to perform
quality control on the berry quality, correcting errors of
identification by the AI systems, and to determine grade (i.e. #1
grade, #2 grade, spoiled, immature). In addition, the human
operator may determine the best approach angle for the berry
grappler to capture the fruit. In some examples, the processing
computer may provide visual guidance and suggestions to the human
operator as to the quality of the berry using automated algorithms
exploiting the spectral and spatial analysis of the target. In some
examples, the human may be at a remote location, not on or
necessarily near the harvesting machine or field itself, but at a
location networked to the camera systems to analyze image data
coming from the camera(s) and sent by wireless communications. In
some examples, the wireless communications are at least one of
cellular, Private Wireless Network, Bluetooth, satellite, radio,
and/or any other wireless method using antennae and processing
computers. In some examples, a private wireless network will
consist of a fixed or portable base station that forms a mesh
network with other fixed or portable base stations to provide
seamless complete communications coverage of the entire field in
which the harvester is operating. In some examples, the fixed or
portable base station will connect to the Wide Area Network, or
Internet, through a dedicated back-haul connection to an Internet
Service Provider (ISP). The backhaul connection may utilize any
technology capable of carrying the necessary bandwidth to execute
the harvester mission including connections comprising of dedicated
wireless point-to-point links, copper twisted-pair links, coaxial
based communications or optical fiber technologies.
[0112] In operation, the berry grappler, once it reaches a
predetermined off-set distance from the mapped coordinates of the
berry to be picked, control of the movement of the robotic arm is
handed off to an internal guidance system that will lock onto the
targeted berry and fine tune any discrepancies in the logged
coordinates that may occur from the forward movement of the
harvesting platform. The internal guidance system will utilize a
neural network and/or artificial intelligence in conjunction with
accumulated data gathered onto a Neuro Network to make
decisions.
[0113] To deal with leaves, stems and trash obstructing access to
the targeted berry, the berry grappler will have the ability to
recognize obstructions and perform a second operation to clear
access to the targeted berry. The berry grappler will be equipped
with a trash diverter located at its most forward tip of the
grappler spoons. The trash diverter may be equipped with air jets,
or foliage hook device, or rotating paddle device and other
appropriate methods to displace the obstruction. In operation, the
berry grappler trash diverter will move in a progressive diameter
rotation around the targeted berry location, clearing the
obstructions. This second operation will only take place when
obstructions are viewed by the stereo cameras.
[0114] In some examples, the three dimensional image data processed
by and sent from the camera(s) may allow for a virtual reality
environment to be created for a human user. In such examples, a
virtual reality headset or display may be utilized by a user,
remote or close to the harvester, to identify target agriculture
and thereby send the target mapping coordinates to the harvesting
machine for harvesting.
[0115] In some examples, the remote human users may be allowed by
the computer to determine an angle of attack for the picker head to
harvest an agricultural target, using the sensor data. In some
examples, that may include image data such that the user may angle
the picker head assembly using an interface such as a joystick,
touchscreen, or mouse.
[0116] In some examples, data created by the cameras and data
created by the human selection of agriculture is stored by a
computing device. In such examples, the identification data may be
amassed in order to analyze and later create algorithms for neural
network engines to process. In such examples, after much data of
targeting, identification, and harvesting information is gathered,
an neural network engine can be trained may be able to replicate
some or all of the human targeting using the three dimensional
maps.
[0117] Examples of cameras which may be used in the described
systems include stereo vision with resolution of 1920.times.1080
and frame rates of 30 per second. In some examples, different
stereo vision camera systems may be utilized, with different
resolution and frame rates, these being only examples.
[0118] In some examples, a suite of these or other sensors could be
placed on the frame or chassis of the harvester, mounted still, or
on motors to swivel or turn. In some examples, a suite of these or
other sensors could be placed at the end of a robotic arm such as
those shown in FIG. 12. In the example, the sensor 1203 is able to
detect the target 1250 in whichever manner the senor operates
(light, heat, lidar, acoustic, radar, etc.). In some examples,
multiple sensors are placed on a single arm 1260. In some examples,
multiple arms 1260 operate with their own or multiple sensors 1203.
In some examples, the sensors 1203 are mounted on a rotatable
mount, able to move rotate in one, two, three, four, five, six,
seven, or more degrees of freedom. In some examples, a robotic arm
1260 includes sensors, picker heads, and/or multiple sensors and/or
picker heads, and/or a combination of sensors and/or picker heads.
In some examples, the sensors may be stereoscopic sensors, spaced
apart but aimed at a similar focal point to provide data for the
computing system to create three dimensional models of targets and
environment around targets.
[0119] In some examples, a seeker subassembly vehicle may work
independently from the harvester subassembly, and in some examples,
the two subassemblies are on the same traversing machine. In some
examples, the harvesting subassembly has its own wheels and/or
tracks or combination of both, to traverse down a row of
agriculture and harvest the mapped targets it receives from the
seeker subassembly. In some examples, the seeker subassembly
vehicle and harvesting subassembly vehicle are able to mate,
connect, and/or otherwise work in concert by connection. In some
examples, this connection includes a wired connection to allow for
target information to be passed from the seeker subassembly to the
harvesting subassembly. In some examples, the two subassemblies
communicate wirelessly. In some examples, a combination of wired
and wireless communication may be arranged.
Lighting
[0120] In some example embodiments, the seeker subassembly includes
various specialized lighting which may be used to find and identify
targets. Such lights may be configured on the ends of robotic arms,
integrated into robotic arms that include picker heads, or cameras.
Examples are shown in FIGS. 1A and 1B. Such lights may be fixed
onto other sub-assemblies on the seeker assembly and/or harvesting
sub-assembly.
[0121] In some examples, such specialized lighting may be
configured to emit a certain wavelength or spectrum of wavelengths
such as but not limited to visible light, infra-red light, and/or
ultra-violet light. In some examples, the lighting may be at a
wavelength that excites items to fluoresce. In some example
embodiments, light spectrum filters may be used by the cameras
described herein to filter out or delete wave lengths of light that
would otherwise block out any fluorescent properties reflected or
emitted by targets such as berries.
[0122] In some examples, the specialized lighting may be light
emitting diodes which are tuned to emit light at a specific
frequency. In some examples, that frequency may be a combination of
470 nm (blue) and 635 nm (red). In some examples, the lights may be
LED lights. In some examples, the lights may be incandescent
lights. In some examples, the lights may be halogen lights,
fluorescent lights, metal-halide, neon, high-intensity discharge
lamps, or any permutation or combination of any of the above.
Mapping and Passing Target Coordinates
[0123] As described herein, in some example embodiments,
additionally or alternatively, sensors onboard the harvesting
systems such as machine vision camera and computing systems may be
used to map target agriculture in three dimensions and pass the
coordinates to the harvester subassembly for harvesting. In some
examples, these are stereoscopically arranged cameras, with similar
field of views to create image data that may be used by the
computing systems to create three dimensional maps of targets.
These mapping coordinates may be described in a global coordinate
system such as Universal Transverse Mercader (UTM), or a local
coordinate system frame relative to the coordinate system defined
by the three dimensional imaging system on the harvester. In some
examples, an X,Y,Z coordinate system may be employed using an
anchor point in the camera view and/or on the traversing machine
itself.
[0124] The various sensors described herein including but not
limited to visible light cameras, infrared cameras, ultraviolet
cameras, lidars, radars, lasers, or other sensors may be used to
scan the produce plants and identify targets. Using the automated,
semi-automated, or manual selection processes and systems described
herein, the systems could generate coordinates for selected
targets. These mapped target coordinates may then be queued in a
buffer or database, for the harvester subassembly to harvest. In
some examples, after one coordinate is added to the harvesting
coordinate queue, more targets may be added to the queue to be
harvested in turn. In such examples, the targeting subassembly,
machine vision, and target mapping may occur without lag or delay
in the handoff from targeting to harvesting, and not be hampered by
the limitations of the harvesting subassembly itself. In such a
way, in some examples additionally or alternatively, the targeting
subassembly may be mounted on a separate vehicle to travel at its
own speed and send targeting mapped data to the harvesting
subassembly by wireless communications. In some examples, the
targeting subassembly may be a part of the overall machine and
connected to or in communication with the harvesting subassembly to
pass the targeting mapped coordinate queue by wired communications
to the harvesting subassembly. In some examples, a cloud or
distributed computing resource may be utilized so that the
targeting queue may be relayed or sent to the harvesting
subassembly wirelessly.
[0125] In some examples, the mapping may be done early or before a
harvester machine may come down a row. Additionally or
alternatively, in some examples, mapping may be done just before
harvesting, on the same machine in some examples to minimize the
variables of the berries and/or foliage moving. Any time between
target mapping and harvesting may be utilized, depending on the
circumstances of the harvest.
[0126] In some examples, mapping information may be stored in a
remote server as described in FIG. 13, cloud server, or distributed
system, for the purpose of future analysis (post processing) of the
imagery to evaluate the condition of the plant. Post processing
operations may include an evaluation of the plant for disease,
nutrient deficiency, unripe berry inventory, and/or other plant
damage. Data gathering and analysis on all types of agricultural
specifics may be accomplished using the suite of cameras and/or
sensors on the systems described herein. For example, outputs of
post processing operations may be utilized to selectively address
in-field issues at a plant-local scale that may otherwise require
broad remedies using traditional methods. Other outputs of post
processing operations may generate statistical data related to
observations and measurements that are made while the harvester is
operating in the field that can be advantageous to the growers
business efforts.
Planter Subassembly Examples
[0127] In some example embodiments, a subassembly may be utilized
which may be used to automate the bed preparation and planting of
plants in the beds as an additional or alternative embodiment than
the harvesting robotic assemblies described in FIGS. 3A, 3B, 3C,
etc. In some examples, such a subassembly may include an arm with a
propane torch instead of or in addition to a picker assembly. In
some examples, the propane torch includes a circular metal shape
within which the torch is lit. In such examples, when the
circularly shaped torch is brought within range of a plastic sheet
covering a planting bed, the torch may burn a circularly shaped
hole in the plastic, but leave no residual flap or other integrity
diminishing aspects to the plastic. In such examples, the
subassembly may be configured to space the holes for the plantings
in the plastic at predetermined distances from one another. In some
examples, the subassembly may be configured to place one, two, or
three lines of holes on one bed row, depending on how many plants
are configured to be placed on one bed row. In some examples, the
holes may be side-by-side. In some examples, the holes may be
staggered or alternating.
[0128] In some examples, after the robotic arm(s) has burned holes
in the plastic covering the bed rows, another arm may be utilized
to insert a seedling plant into the dirt under the hole in the
plastic. In such examples, camera arrangements may be utilized to
identify the holes in the plastic and align the robotic assembly to
the center of the hole. In such examples, the robotic arm may then
utilize a spade shaped tool to dig into the dirt under the hole a
predetermined distance to make room for roots of a seedling plant.
In some examples, a spoon or pinch mechanism may be used to grasp
and insert the seedling plant in the hole dug by the spade, at a
predetermined distance into the dirt, and inside the hole in the
plastic. In some examples, the hole may be made with the same tool
that plants the seedling. In some examples, this is a two-step
process, to make the hole and then plant the seedling. In some
examples, this is a one-step process, to make the hole and plant
the seedling.
Computer and Computer Network Examples
[0129] Additionally or alternatively, the systems described here
may be used to harvest agricultural targets in an automated,
semi-automated, or even manually controlled manner. In some
examples, the semi-automated manner may be arranged in a remote
setting, allowing for a human to interact with camera views from
the harvester to help target the produce. Such an arrangement may
be made possible by a network and computer arrangement shown in
FIG. 13 and/or FIG. 14.
[0130] The variations on these options depend on how much a remote
or local computing system may be programmed to identify and harvest
a target. For example, in a fully manually controlled system, a
human operator may control the movements of both the seeker system
and the harvesting system. In such examples, by remote control
using a joystick or other computer driven operating device(s) a
human could scan the rows of plants for a target using the camera
systems, and even maneuver the robotic arms that the camera systems
are connected to, to identify targets, and then use a control
system such as a joystick to maneuver the picker head assembly to
the target, and then harvest the target as described herein. Such
examples would allow for remote operation of the systems such as by
wireless control to allow for human controllers to be stationed
anywhere in the world, through some kind of wireless uplink.
[0131] The other extreme of control systems would be a fully
automated system. In such a system, the traversing machines would
move down a row of agricultural targets and the seeker subassembly
would use machine learning/artificial intelligence/neural
networks/and/or other programming to seek out and identify targets
with the seeker subassemblies and then harvest them as described
using the picker heads. Such examples would depend on computer
algorithms and programs to determine using the inputs from the
cameras and sensors, what a target may be and where they are
located. For example, a color camera may be used by the computing
system to detect a red strawberry amongst the green foliage of the
plant it is growing on. Then a laser system could be used to
determine a proximate location and range from the system and the
computers could use that information to triangulate a three
dimensional coordinate system and identify where the target is
located in space, relative to the traversing machine. Next, the
coordinates could be passed to the harvesting subassembly where the
picker heads may attach to and harvest the target strawberry, in
some examples using its own sensors such as cameras and lasers.
[0132] The middle-ground option between the fully automated and the
manually controlled system would be some variant of semi-automated
seeking and harvesting. The degree of semi-autonomy and which
portions were automated and which manually controlled could vary
from separate subassemblies. For example, the seeker subassembly
may be more manually controlled with a human interacting with the
cameras and sensors to help identify targets. In some examples,
that may include a human interacting with a graphical user
interface "GUI" such as a touchscreen to identify a target
displayed on the screen.
[0133] In any of the above examples of automation, the sensors
onboard the harvesting system may be used to create, track and pass
coordinates of the targets for harvesting.
[0134] The computing architecture for the harvester could be
described as a distributed computing system comprising of elements
or processing centers that exist on the harvester, a central server
system which may or may not be a cloud based resource and an
operator processing system. Each of these processing centers are
interconnected through an IP network which may include local
private wireless networks, private wide area networks and/or public
networks such as the Internet as described in FIGS. 13 and 14.
Computational tasks may be divided such that real-time tasks are
executed on the local harvester processor, post-processing
operations and non-real time computation are executed on the
central server and user-interface computation are performed on the
operator processing center.
[0135] In example systems described herein, various computing
components may be utilized to operate the systems. For example, a
communication computing system may allow for remote operation of
the machines, sensors may send information to a computing system to
help differentiate targets from non-targets, target location and
mapping information may be calculated, stored, sent, and utilized
between the seeker systems and harvesting systems, steering and
driving instructions may be calculated and utilized, machine
learning/artificial intelligence/and/or neural networks may be
employed by computing systems to find and harvest targets, and any
of the other computing operations as described herein. In some
examples, alternatively or additionally, a WiFi system/cellular
system/Bluetooth system, or any other communication system, with
the appropriate antenna system and a processor and memory as
described herein, may be used on a subassembly. In some
embodiments, alternatively or additionally, the hardware may
include a single integrated circuit containing a processor core,
memory, and programmable input/output peripherals. In some
examples, various computing components may be used in the seeker
and/or harvesting subassemblies, as well as the communication
systems, control systems, and/or any other portion of the systems
described herein.
[0136] FIG. 13 shows an example networked system which could be
used in the systems and methods here. In FIG. 13, the computer
system 1302 onboard the harvesting system described herein,
including process any images from the various sensors including
cameras taking images of the targets and plants. Such image data
may include pixel data of the captured target images. The computer
1302 could be any number of kinds of computers such as those
included in the camera itself, navigation system, robotic arm
maneuvering systems, traversing or driving systems, sensor systems,
and/or any other another computer arrangement including those
examples are described in FIG. 14.
[0137] As shown in FIG. 13, the various computing systems may be in
communication with a back end computing system 1330 and/or data
storage 1332 to send and receive data regarding the operations of
the harvesting systems described herein. For example, as shown in
FIG. 13, captured image data of targets may be transmitted to a
back end computer system 1330 and associated data storage 1332 for
saving and analysis. In some examples, this may include the remote
operators who are interfacing with the harvesting systems,
selecting targets, providing navigation instruction, steering,
and/or overseeing maintenance of the systems. In some examples, the
communication may be a wireless transmission 1310 by a radio,
cellular or WiFi transmission with associated routers and hubs. In
some examples, the transmission may be through a wired connection
1312. In some examples, a combination of wireless and wired
transmissions may be used to stream data between the back end 1330
and the harvesting system including cameras, robotic pickers,
etc.
[0138] In some examples, the transmission of data may include
transmission through a network such as the internet 1320 to the
remote operators, back end server computers 1330, and associated
data storage 1332. Once at the back end server computer servers
1330 and associated data storage 1332, the pixelated image data may
be acted upon by the remote operators to choose targets to harvest.
In some examples, the data may be useful to train the neural
network, and/or artificial intelligence models as to good targets
versus targets to pass up. In such examples, the image and target
data may be stored, analyzed, used to train models, or any other
kind of image data analysis. In some examples, the storing,
analyzing, and/or processing of image data may be accomplished at
the computer 1302 which is involved in the original image capture.
In some examples, the local computer 1302 and a back end computing
system 1330 may split or share the data storing, modeling,
analyzing, and/or processing. Back end computer resources 1330 may
be more powerful, faster, or be able to handle more data than may
be otherwise available at the local computers 1302 on the
harvesting machines. In some examples, the networked computer
resources 1330 may be spread across many multiple computer
resources by a cloud infrastructure. In some examples, the
networked computer resources 1330 may be virtual machines in a
cloud infrastructure.
[0139] FIG. 14 shows an example computing device 1400 that may be
used in practicing example embodiments described herein. FIG. 14
could describe computers such as 1302, 1330 or other systems as
described in FIG. 13. Such computing device 1400 may be the back
end server systems use to interface with the network, receive and
analyzed data, as well as generate remote operator GUIs,
additionally or alternatively, it could be a computing system
aboard the harvesting system as described, to instruct the robotic
assemblies, cameras, communications, etc. Such computer 1400 may be
a device used to create and send data, as well as receive and cause
display of GUIs representing data such as back end interfaces for
remote operators, camera image analysis, etc. In FIG. 14, the
computing device could be a smartphone, a laptop, tablet computer,
server computer, or any other kind of computing device. The example
shows a processor CPU 1410 which could be any number of processors
in communication via a bus 1412 or other communication with a user
interface 1414. The user interface 1414 could include any number of
display devices 1418 such as a screen. The user interface also
includes an input such as a touchscreen, keyboard, mouse, pointer,
buttons, joystick or other input devices. Also included is a
network interface 1420 which may be used to interface with any
wireless or wired network in order to transmit and receive data.
Such an interface may allow for a smartphone, for example, to
interface a cellular network and/or WiFi network and thereby the
Internet. The example computing device 1400 also shows peripherals
1424 which could include any number of other additional features
such as but not limited to cameras, sensors 1425, and/or antennae
1426 for communicating wirelessly such as over cellular, WiFi, NFC,
Bluetooth, infrared, or any combination of these or other wireless
communications. The computing device 1400 also includes a memory
1422 which includes any number of operations executable by the
processor 1410. The memory in FIG. 14 shows an operating system
1432, network communication module 1434, instructions for other
tasks 1438 and applications 1438 such as send/receive message data
1440 and/or SMS text message applications 1442. Also included in
the example is for data storage 1458. Such data storage may include
data tables 1460, transaction logs 1462, user data 1464 and/or
encryption data 1470. The computing device 1400 also include one or
more graphical processing units (GPUs) for the purposes of
accelerating in hardware computationally intensive tasks such as
execution and or evaluation of the neural network engine and
enhanced image exploitation algorithms operating on the multi-modal
imagery collected. The computing device 1400 may also include one
or more reconfigurable hardware elements such as a field
programmable gate array (FPGA) for the purposes of hardware
acceleration of computationally intensive tasks.
CONCLUSION
[0140] As disclosed herein, features consistent with the present
inventions may be implemented by computer-hardware, software and/or
firmware. For example, the systems and methods disclosed herein may
be embodied in various forms including, for example, a data
processor, such as a computer that also includes a database,
digital electronic circuitry, firmware, software, computer
networks, servers, or in combinations of them. Further, while some
of the disclosed implementations describe specific hardware
components, systems and methods consistent with the innovations
herein may be implemented with any combination of hardware,
software and/or firmware. Moreover, the above-noted features and
other aspects and principles of the innovations herein may be
implemented in various environments. Such environments and related
applications may be specially constructed for performing the
various routines, processes and/or operations according to the
invention or they may include a computer or computing platform
selectively activated or reconfigured by code to provide the
necessary functionality. The processes disclosed herein are not
inherently related to any particular computer, network,
architecture, environment, or other apparatus, and may be
implemented by a suitable combination of hardware, software, and/or
firmware. For example, various machines may be used with programs
written in accordance with teachings of the invention, or it may be
more convenient to construct a specialized apparatus or system to
perform the required methods and techniques.
[0141] Aspects of the method and system described herein, such as
the logic, may be implemented as functionality programmed into any
of a variety of circuitry, including programmable logic devices
("PLDs"), such as field programmable gate arrays ("FPGAs"),
programmable array logic ("PAL") devices, electrically programmable
logic and memory devices and standard cell-based devices, as well
as application specific integrated circuits. Some other
possibilities for implementing aspects include: memory devices,
microcontrollers with memory (such as 1PROM), embedded
microprocessors, Graphics Processing Units (GPUs), firmware,
software, etc. Furthermore, aspects may be embodied in
microprocessors having software-based circuit emulation, discrete
logic (sequential and combinatorial), custom devices, fuzzy
(neural) logic, quantum devices, and hybrids of any of the above
device types. The underlying device technologies may be provided in
a variety of component types, e.g., metal-oxide semiconductor
field-effect transistor ("MOSFET") technologies like complementary
metal-oxide semiconductor ("CMOS"), bipolar technologies like
emitter-coupled logic ("ECL"), polymer technologies (e.g.,
silicon-conjugated polymer and metal-conjugated polymer-metal
structures), mixed analog and digital, and so on.
[0142] It should also be noted that the various logic and/or
functions disclosed herein may be enabled using any number of
combinations of hardware, firmware, and/or as data and/or
instructions embodied in various machine-readable or
computer-readable media, in terms of their behavioral, register
transfer, logic component, and/or other characteristics.
Computer-readable media in which such formatted data and/or
instructions may be embodied include, but are not limited to,
non-volatile storage media in various forms (e.g., optical,
magnetic or semiconductor storage media) and carrier waves that may
be used to transfer such formatted data and/or instructions through
wireless, optical, or wired signaling media or any combination
thereof. Examples of transfers of such formatted data and/or
instructions by carrier waves include, but are not limited to,
transfers (uploads, downloads, e-mail, etc.) over the Internet
and/or other computer networks by one or more data transfer
protocols (e.g., HTTP, FTP, SMTP, and so on).
[0143] Unless the context clearly requires otherwise, throughout
the description and the claims, the words "comprise," "comprising,"
and the like are to be construed in an inclusive sense as opposed
to an exclusive or exhaustive sense; that is to say, in a sense of
"including, but not limited to." Words using the singular or plural
number also include the plural or singular number respectively.
Additionally, the words "herein," "hereunder," "above," "below,"
and words of similar import refer to this application as a whole
and not to any particular portions of this application. When the
word "or" is used in reference to a list of two or more items, that
word covers all of the following interpretations of the word: any
of the items in the list, all of the items in the list and any
combination of the items in the list.
[0144] Although certain presently preferred implementations of the
invention have been specifically described herein, it will be
apparent to those skilled in the art to which the invention
pertains that variations and modifications of the various
implementations shown and described herein may be made without
departing from the spirit and scope of the invention. Accordingly,
it is intended that the invention be limited only to the extent
required by the applicable rules of law.
[0145] The foregoing description, for purpose of explanation, has
been described with reference to specific embodiments. However, the
illustrative discussions above are not intended to be exhaustive or
to limit the invention to the precise forms disclosed. Many
modifications and variations are possible in view of the above
teachings. The embodiments were chosen and described in order to
best explain the principles of the invention and its practical
applications, to thereby enable others skilled in the art to best
utilize the invention and various embodiments with various
modifications as are suited to the particular use contemplated.
[0146] The foregoing description, for purpose of explanation, has
been described with reference to specific embodiments. However, the
illustrative discussions above are not intended to be exhaustive or
to limit the invention to the precise forms disclosed. Many
modifications and variations are possible in view of the above
teachings. The embodiments were chosen and described in order to
best explain the principles of the invention and its practical
applications, to thereby enable others skilled in the art to best
utilize the invention and various embodiments with various
modifications as are suited to the particular use contemplated.
Etc.
* * * * *