U.S. patent application number 13/454588 was filed with the patent office on 2013-05-23 for mobile and one-touch tasking and visualization of sensor data.
This patent application is currently assigned to RAYTHEON COMPANY. The applicant listed for this patent is Christopher J. Graham, Paul C. Hershey, Steve Kearsley, Leslie A. Ledda, Niraj N. Shah, Sylvia A. Traxler, Mu-Cheng Wang. Invention is credited to Christopher J. Graham, Paul C. Hershey, Steve Kearsley, Leslie A. Ledda, Niraj N. Shah, Sylvia A. Traxler, Mu-Cheng Wang.
Application Number | 20130128041 13/454588 |
Document ID | / |
Family ID | 48426233 |
Filed Date | 2013-05-23 |
United States Patent
Application |
20130128041 |
Kind Code |
A1 |
Hershey; Paul C. ; et
al. |
May 23, 2013 |
MOBILE AND ONE-TOUCH TASKING AND VISUALIZATION OF SENSOR DATA
Abstract
The technology described herein includes a system and/or a
method for data tasking and visualization of data. The method
includes receiving, by a computing device, a screening policy
selection from a user associated with the computing device;
transmitting, by the computing device, the screening policy
selection to one or more sensor platform devices; receiving, by the
computing device, one or more data sets from the one or more sensor
platform devices in response to the transmission of the screening
policy selection; and displaying, by the computing device, the one
or more data sets to the user.
Inventors: |
Hershey; Paul C.; (Ashburn,
VA) ; Graham; Christopher J.; (Bethesda, MD) ;
Traxler; Sylvia A.; (Zephyr Cove, NV) ; Wang;
Mu-Cheng; (Acton, MA) ; Kearsley; Steve;
(Laguna Niguel, CA) ; Ledda; Leslie A.; (Heath,
TX) ; Shah; Niraj N.; (Cerritos, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Hershey; Paul C.
Graham; Christopher J.
Traxler; Sylvia A.
Wang; Mu-Cheng
Kearsley; Steve
Ledda; Leslie A.
Shah; Niraj N. |
Ashburn
Bethesda
Zephyr Cove
Acton
Laguna Niguel
Heath
Cerritos |
VA
MD
NV
MA
CA
TX
CA |
US
US
US
US
US
US
US |
|
|
Assignee: |
RAYTHEON COMPANY
Waltham
MA
|
Family ID: |
48426233 |
Appl. No.: |
13/454588 |
Filed: |
April 24, 2012 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61560918 |
Nov 17, 2011 |
|
|
|
61615702 |
Mar 26, 2012 |
|
|
|
Current U.S.
Class: |
348/143 ;
348/E7.085; 702/189 |
Current CPC
Class: |
G06K 9/00 20130101; G01S
7/003 20130101 |
Class at
Publication: |
348/143 ;
702/189; 348/E07.085 |
International
Class: |
G06F 15/00 20060101
G06F015/00; H04N 7/18 20060101 H04N007/18 |
Claims
1. A data tasking and visualization system, the system comprising:
a computing device comprising: a screening module configured to
receive a screening policy selection from a user associated with
the computing device, the screening policy selection being selected
from a plurality of screening policies and each of the plurality of
screening policies being indicative of a mission event, a
transmitter configured to transmit the screening policy selection
to one or more sensor platforms, a receiver configured to receive
one or more data sets from the one or more sensor platforms in
response to the transmission of the screening policy selection, and
a display device configured to display the one or more data sets to
the user.
2. The system of claim 1, further comprising the computing device
further comprising: a identification module configured to associate
one or more annotations received from the user with the one or more
data sets, and the transmitter configured to transmit the one or
more annotations to a screening policy server.
3. The system of claim 1, further comprising: the one or more
sensor platforms, each sensor platform of the one or more sensor
platforms comprising: a receiver configured to: receive the
screening policy selection from the computing device, and receive
sensor data from one or more sensors, a data screening module
configured to generate a data set from the sensor data based on the
screening policy selection, and a transmitter configured to
transmit the data set to the computing device.
4. The system of claim 3, further comprising each of the one or
more sensor platforms further comprising: a multimedia pattern
matching module configured to identify one or more mission-relevant
objects in the sensor data based on the screening policy selection,
the sensor data comprises image data, video data, or any
combination thereof, a multimedia tagging module configured to
associate one or more data tags with the sensor data associated
with the identified one or more mission-relevant objects based on
the mission event, and the transmitter further configured to
transmit the sensor data associated with the identified one or more
mission-relevant objects and the associated one or more data tags
to the computing device.
5. The system of claim 1, further comprising: a screening policy
server comprising: a receiver configured to: receive sensor data
and data sets from the one or more sensor platforms, and receive
screening policy selections from a plurality of computing devices,
a data repository module configured to store the sensor data, the
data sets, and the screening policy selections, and a screening
policy generation module configured to generate the plurality of
screening policies based on the stored sensor data, the stored data
sets, and the stored screening policy selections.
6. The system of claim 5, further comprising the screening policy
generation module of the screening policy server further configured
to modify at least one of the plurality of screening policies based
on the stored sensor data, the stored data sets, and the stored
screening policy selections.
7. The system of claim 1, further comprising the screening module
of the computing device further configured to determine a set of
screening policies from the plurality of screening policies based
on one or more mission parameters.
8. The system of claim 7, wherein the one or more mission
parameters comprises a real-time mission event, a planned mission
event, a user preference, a data type associated with the sensor
data, or any combination thereof.
9. The system of claim 1, wherein the plurality of screening
policies comprise a model-based policy, a statistics-based policy,
a learning-based policy, a event-driven policy, or any combination
thereof.
10. The system of claim 1, wherein the computing device is a mobile
computing device.
11. A method for data tasking and visualization of data, the method
comprising: receiving, by a computing device, a screening policy
selection from a user associated with the computing device, the
screening policy selection being selected from a plurality of
screening policies and each of the plurality of screening policies
being indicative of a mission event; transmitting, by the computing
device, the screening policy selection to one or more sensor
platform devices; receiving, by the computing device, one or more
data sets from the one or more sensor platform devices in response
to the transmission of the screening policy selection; and
displaying, by the computing device, the one or more data sets to
the user.
12. The method of claim 12, further comprising: associating, by the
computing device, one or more annotations received from the user
with the one or more data sets; and transmitting, by the computing
device, the one or more annotations to a screening policy
server.
13. The method of claim 11, further comprising: receiving, by a
sensor platform device of the one or more sensor platform devices,
the screening policy selection from the computing device;
receiving, by the sensor platform device, sensor data from one or
more sensors, generating, by the sensor platform device, a data set
from the sensor data based on the screening policy selection; and
transmitting, by the sensor platform device, the data set to the
computing device.
14. The method of claim 13, further comprising: identifying, by the
sensor platform device, one or more mission-relevant objects in the
sensor data based on the screening policy selection, the sensor
data comprises image data, video data, or any combination thereof;
associating, by the sensor platform device, one or more data tags
with the sensor data associated with the identified one or more
mission-relevant objects based on the mission event; and
transmitting, by the sensor platform device, the sensor data
associated with the identified one or more mission-relevant objects
and the associated one or more data tags to the computing
device.
15. The method of claim 11, further comprising determining, by the
computing device, a set of screening policies from the plurality of
screening policies based on one or more mission parameters.
16. The method of claim 15, wherein the one or more mission
parameters comprises a real-time mission event, a planned mission
event, a user preference, a data type associated with the sensor
data, or any combination thereof.
17. A computer program product, tangibly embodied in an information
carrier, the computer program product including instructions being
operable to cause a data processing apparatus to: receive a
screening policy selection from a user associated with the
computing device, the screening policy selection being selected
from a plurality of screening policies and each of the plurality of
screening policies being indicative of a mission event; transmit
the screening policy selection to one or more sensor platform
devices; receive one or more data sets from the one or more sensor
platform devices in response to the transmission of the screening
policy selection; and display the one or more data sets to the
user.
Description
RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application No. 61/560,918, filed on Nov. 17, 2011 and U.S.
Provisional Application No. 61/615,702, filed on Mar. 26, 2012. The
entire teachings of the above applications are incorporated herein
by reference.
BACKGROUND
[0002] Current sensor resources, such as land-based, sea-based, and
airborne sensors resources, transmit sensor information back to
analysts and/or computer system. Currently, service analysts who
monitor intelligence feeds from sensor platforms watch mundane
video footage for hours on end without ever seeing anything of
value, a waste of both time and resources. Thus, a need exists in
the art for improved sensor data management.
SUMMARY
[0003] One approach to sensor data management is a system for
policy-based data management. The system includes one or more
sensor platforms, each sensor platform of the one or more sensor
platforms comprising: a receiver configured to receive sensor data
from one or more sensors, a data screening policy module configured
to select one or more screening policies from a plurality of
screening policies based on one or more mission parameters and a
platform type associated with the sensor platform, a data screening
module configured to generate a data set from the sensor data based
on the selected one or more screening policies, and a transmitter
configured to transmit the data set to one or more computing
devices.
[0004] Another approach to sensor data management is a method for
policy-based data management. The method includes receiving, by a
sensor platform device, sensor data from one or more sensors;
selecting, by the sensor platform device, one or more screening
policies from a plurality of screening policies based on one or
more mission parameters and a platform type associated with the
sensor platform device; generating, by the sensor platform device,
a data set from the sensor data based on the selected one or more
screening policies; and transmitting, by the sensor platform
device, the data set to one or more computing devices.
[0005] Another approach to sensor data management is a computer
program product that is tangibly embodied in an information
carrier. The computer program product includes instructions being
operable to cause a data processing apparatus to: receive sensor
data from one or more sensors; select one or more screening
policies from a plurality of screening policies based on one or
more mission parameters and a platform type associated with the
sensor platform device; generate a data set from the sensor data
based on the selected one or more screening policies; and transmit
the data set to one or more computing devices.
[0006] Another approach to sensor data management is a system for
data tasking and visualization. The system includes a computing
device including: a screening module configured to receive a
screening policy selection from a user associated with the
computing device, the screening policy selection being selected
from a plurality of screening policies and each of the plurality of
screening policies being indicative of a mission event, a
transmitter configured to transmit the screening policy selection
to one or more sensor platforms, a receiver configured to receive
one or more data sets from the one or more sensor platforms in
response to the transmission of the screening policy selection, and
a display device configured to display the one or more data sets to
the user.
[0007] Another approach to sensor data management is a method for
data tasking and visualization of data. The method includes
receiving, by a computing device, a screening policy selection from
a user associated with the computing device, the screening policy
selection being selected from a plurality of screening policies and
each of the plurality of screening policies being indicative of a
mission event; transmitting, by the computing device, the screening
policy selection to one or more sensor platform devices; receiving,
by the computing device, one or more data sets from the one or more
sensor platform devices in response to the transmission of the
screening policy selection; and displaying, by the computing
device, the one or more data sets to the user.
[0008] Another approach to sensor data management is a computer
program product. The computer program product is tangibly embodied
in an information carrier. The computer program product includes
instructions being operable to cause a data processing apparatus
to: receive a screening policy selection from a user associated
with the computing device, the screening policy selection being
selected from a plurality of screening policies and each of the
plurality of screening policies being indicative of a mission
event; transmit the screening policy selection to one or more
sensor platform devices; receive one or more data sets from the one
or more sensor platform devices in response to the transmission of
the screening policy selection; and display the one or more data
sets to the user.
[0009] In other examples, any of the approaches above can include
one or more of the following features.
[0010] In some examples, the system further includes the one or
more computing devices, each computing device of the one or more
computing devices includes: a receiver configured to receive a data
set from at least one of the one or more of the sensor platforms, a
display device configured to display the received data set, a
mission parameter module configured to generate one or more
modified mission parameters based on input from a user associated
with the computing device, and a transmitter configured to transmit
the one or more modified mission parameters to at least one of the
one or more of the sensor platforms.
[0011] In other examples, each of the one or more sensor platforms
further includes: the receiver further configured to receive second
sensor data from one or more sensors, the data screening policy
module further configured to select a second set of one or more
screening policies from the plurality of screening policies based
on the one or more modified mission parameters and the platform
type associated with the sensor platform, the data screening module
further configured to generate a second data set from the second
sensor data based on the selected second set of one or more
screening policies, and the transmitter further configured to
transmit the second data set to the one or more computing
devices.
[0012] In some examples, the sensor data includes image data, video
data, audio data, or any combination thereof; and each of the one
or more sensor platforms further includes: a data link module
configured to determine a data link parameter associated with a
data link between the respective sensor platform and at least one
of the one or more computing devices; and a multimedia chipping
module configured to generate one or more image chips from the
sensor data based on the data link parameter and one or more image
parameters, the one or image parameters are received from one or
more users associated with the at least one of the one or more
computing device.
[0013] In other examples, the one or image parameters include an
image chipping technique, a user preference, a user selected focus
area, or any combination thereof.
[0014] In some examples, each of the one or more sensor platforms
further includes: a data link module configured to determine a data
link parameter associated with a data link between the respective
sensor platform and at least one of the one or more computing
devices; and a data reduction module configured to dynamically
reduce the data set before transmission by the transmitter based on
the data link parameter and the one or more mission parameters.
[0015] In other examples, the one or more mission parameters
include a real-time mission event, a planned mission event, a user
preference, a data type associated with the sensor data, or any
combination thereof.
[0016] In some examples, the plurality of screening policies
includes a model-based policy, a statistics-based policy, a
learning-based policy, a event-driven policy, or any combination
thereof.
[0017] In other examples, each of the sensor platforms comprises
the one or more sensors.
[0018] In some examples, the method further includes receiving, by
a computing device, data sets from a plurality of sensor platform
devices; generating, by the computing device, one or more modified
mission parameters based on input from a user associated with the
computing device; and transmitting, by the computing device, the
one or more modified mission parameters to one or more of sensor
platform devices.
[0019] In other examples, the method further includes receiving, by
the sensor platform device, the one or more modified mission
parameters; selecting, by the sensor platform device, one or more
second screening policies from the plurality of screening policies
based on the one or more modified mission parameters and the
platform type associated with the sensor platform device;
generating, by the sensor platform device, a second data set from
the sensor data based on the selected one or more second screening
policies; and transmitting, by the sensor platform device, the
second data set to the computing device.
[0020] In some examples, the method further includes receiving, by
the sensor platform device, the one or more modified mission
parameters; receiving, by the sensor platform device, second sensor
data from the one or more sensors; selecting, by the sensor
platform device, one or more second screening policies from the
plurality of screening policies based on the one or more modified
mission parameters and the platform type associated with the sensor
platform device; generating, by the sensor platform device, a
second data set from the second sensor data based on the selected
one or more second screening policies; and transmitting, by the
sensor platform device, the second data set to the computing
device.
[0021] In other examples, the method further includes determining,
by the sensor platform device, a data link parameter associated
with a data link between the sensor platform device and at least
one of the one or more computing devices; generating, by the sensor
platform device, one or more image chips from the sensor data based
on the data link parameter and one or more image parameters, the
one or image parameters are received from one or more users
associated with the at least one of the one or more computing
devices and the sensor data comprises image data, video data, audio
data, or any combination thereof; and transmitting, by the sensor
platform device, the one or more image chips to the least one of
the one or more computing devices.
[0022] In some examples, the method further includes determining,
by the sensor platform device, a data link parameter associated
with a data link between the respective sensor platform and the one
or more computing devices; and dynamically reducing, by the sensor
platform device, the data set before transmitting based on the data
link parameter and the one or more mission parameters, the dynamic
reduction of the data set comprises a compression technique, a data
correlation technique, or any combination thereof.
[0023] In other examples, the one or more mission parameters are
pre-determined based on a planned mission associated with the
sensor platform device.
[0024] In some examples, the method further includes dynamically
generating, by the sensor platform device, the one or more mission
parameters based on one or more real-time mission events.
[0025] In other examples, the computing device further includes: a
identification module configured to associate one or more
annotations received from the user with the one or more data sets,
and the transmitter configured to transmit the one or more
annotations to a screening policy server.
[0026] In some examples, the system further includes the one or
more sensor platforms, each sensor platform of the one or more
sensor platforms includes: a receiver configured to: receive the
screening policy selection from the computing device, and receive
sensor data from one or more sensors, a data screening module
configured to generate a data set from the sensor data based on the
screening policy selection, and a transmitter configured to
transmit the data set to the computing device.
[0027] In other examples, each of the one or more sensor platforms
further includes: a multimedia pattern matching module configured
to identify one or more mission-relevant objects in the sensor data
based on the screening policy selection, the sensor data comprises
image data, video data, or any combination thereof, a multimedia
tagging module configured to associate one or more data tags with
the sensor data associated with the identified one or more
mission-relevant objects based on the mission event, and the
transmitter further configured to transmit the sensor data
associated with the identified one or more mission-relevant objects
and the associated one or more data tags to the computing
device.
[0028] In some examples, the system further includes a screening
policy server including: a receiver configured to: receive sensor
data and data sets from the one or more sensor platforms, and
receive screening policy selections from a plurality of computing
devices, a data repository module configured to store the sensor
data, the data sets, and the screening policy selections, and a
screening policy generation module configured to generate the
plurality of screening policies based on the stored sensor data,
the stored data sets, and the stored screening policy
selections.
[0029] In other examples, the system further includes the screening
policy generation module of the screening policy server further
configured to modify at least one of the plurality of screening
policies based on the stored sensor data, the stored data sets, and
the stored screening policy selections.
[0030] In some examples, the screening module of the computing
device further configured to determine a set of screening policies
from the plurality of screening policies based on one or more
mission parameters.
[0031] In other examples, the one or more mission parameters
includes a real-time mission event, a planned mission event, a user
preference, a data type associated with the sensor data, or any
combination thereof.
[0032] In some examples, the plurality of screening policies
includes a model-based policy, a statistics-based policy, a
learning-based policy, a event-driven policy, or any combination
thereof.
[0033] In other examples, the computing device is a mobile
computing device.
[0034] In some examples, the method further includes associating,
by the computing device, one or more annotations received from the
user with the one or more data sets; and transmitting, by the
computing device, the one or more annotations to a screening policy
server.
[0035] In other examples, the method further includes receiving, by
a sensor platform device of the one or more sensor platform
devices, the screening policy selection from the computing device;
receiving, by the sensor platform device, sensor data from one or
more sensors, generating, by the sensor platform device, a data set
from the sensor data based on the screening policy selection; and
transmitting, by the sensor platform device, the data set to the
computing device.
[0036] In some examples, the method further includes identifying,
by the sensor platform device, one or more mission-relevant objects
in the sensor data based on the screening policy selection, the
sensor data comprises image data, video data, or any combination
thereof; associating, by the sensor platform device, one or more
data tags with the sensor data associated with the identified one
or more mission-relevant objects based on the mission event; and
transmitting, by the sensor platform device, the sensor data
associated with the identified one or more mission-relevant objects
and the associated one or more data tags to the computing
device.
[0037] In other examples, the method further includes determining,
by the computing device, a set of screening policies from the
plurality of screening policies based on one or more mission
parameters.
[0038] In some examples, the method further includes the one or
more mission parameters include a real-time mission event, a
planned mission event, a user preference, a data type associated
with the sensor data, or any combination thereof.
[0039] The sensor data management techniques described herein can
provide one or more of the following advantages. An advantage to
the technology is that each sensor platform can transmit only that
information relevant to the respective mission of the computing
device so that corresponding user assigned to the mission can
receive a reduced set of information resulting in reduced overall
system bandwidth and processing requirements. Another advantage of
the technology is that the reduction of data enables the user to
respond to mission events of interest, thereby decreasing
processing time and increasing data throughput. Another advantage
of the technology is that the automated, real-time data reduction
based on the mission parameters reduces data overload from
intelligence, surveillance, and reconnaissance assets and reduces
the users' inability to review the data due to the data
overload.
BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The foregoing and other objects, features and advantages
will be apparent from the following more particular description of
the embodiments, as illustrated in the accompanying drawings in
which like reference characters refer to the same parts throughout
the different views. The drawings are not necessarily to scale,
emphasis instead being placed upon illustrating the principles of
the embodiments.
[0041] FIG. 1 illustrates an exemplary block diagram of a data
management environment;
[0042] FIG. 2 is a flow diagram of an exemplary sensor platform in
a policy-based data management system;
[0043] FIG. 3 is a flow diagram of an exemplary computing device in
a data tasking and visualization system;
[0044] FIGS. 4A-4E illustrate exemplary graphical user interfaces
of the technology;
[0045] FIG. 5 is a block diagram of an exemplary sensor
platform;
[0046] FIG. 6 is a block diagram of an exemplary sensor
platform;
[0047] FIG. 7 is a block diagram of an exemplary screening policy
server;
[0048] FIG. 8 is a flow diagram of an exemplary policy
environment;
[0049] FIG. 9 is a block diagram of an exemplary image access and
storage process;
[0050] FIG. 10 illustrates an exemplary data process; and
[0051] FIG. 11 is a flowchart of an exemplary method for
policy-based data management.
DETAILED DESCRIPTION
[0052] The sensor data management includes a multi-factor
information engine (MIE) that, generally, provides processing of
sensor data on sensor platforms and/or computing devices. The
technology can automatically select screening algorithms and/or
compression algorithms based on the mission plan (e.g., desert
environment, water environment, etc.) and events that occur during
the mission (e.g., different types of targets, new environmental
information, etc.). The on-sensor platform data screening can
greatly improve operations center and analyst productivity by
identifying areas of interest in big data sets, thereby decreasing
the processing time and effort to identify and/or track a target.
For example, in operation, the technology enables a plurality of
sensor platforms to dynamically and in real-time screen and/or
compress sensor data for transmission
[0053] The screening algorithms can include one or more tunable
parameters/sensitivities: nominal parameters derived based on
essential elements of information (EEI), data type, analyst/user
preferences, and dynamic/adaptive parameters for the increase of
the tactical relevance of the data. The technology can include
multiple sets (e.g., catalogues) of screening algorithm
configuration files on-board the sensor platform. The sets can
includes tasking assignments for the sensor chosen based on the
mission plan (e.g., water environment to identify fast moving
boats, urban environment to identify cargo vans, etc.) and the most
likely mission events to occur during that specific mission (e.g.,
cargo vans near a shipping facility, movement of the camera, etc.).
The technology can automatically switch between configuration files
(also referred to as screening policies) based on mission events.
For example, if occupants of a boat move to a truck, the technology
can dynamically and automatically switch from a boat screening
policy to a truck screening policy. The technology can provide
secondary screening algorithm catalogues, the number of which can
depend on the storage capacity of the platform. The subsequent
catalogues can, for example, accommodate unplanned mission events,
thereby providing flexibility in the sensor data being communicated
to the computing devices.
[0054] If the mission completely changes so that unexpected
information is now required, then new catalogues (also referred to
as screening policies) can be installed from the operations and
analysts centers and/or from the tactical end-users. In other
words, new screening policies can be transmitted to the sensor
platforms in real-time based on changing environments (e.g.,
automatically based on location, automatically based on sensor
data, in response to a user request, etc.). This adaptive changing
of screening policies can enable sensor tasking directly from the
tactical end-user, which advantageously increases the chance for
detection of a target by the end-user and decreases the time to
organize and/or sort through the data. The end-users have the
ability to change the tasking of the sensor through the
tasking/screening manager at the ground segment, thereby decreasing
the network link bandwidth to the end-user and decreasing the
processing and storage needed at the end-user's computing
device.
[0055] FIG. 1 illustrates an exemplary block diagram of a data
management environment 100. The environment 100 includes sensor
platforms 112, 114, and 116, a screening policy server 120, and
computing devices 130 and 140. Users 135 and 145 interact with the
computing device 130 and 140, respectively, to view and/or request
sensor data. The screening policy server 120 transmits screening
policies to the sensor platforms 112, 114, and 116 (e.g., on
demand, based on updates, etc.).
[0056] For example, in operation, the sensor platforms 112, 114,
and 116 are deployed to a desert environment for detection of
trucks in the desert (e.g., border crossing mission, truck tracking
mission, etc.). Each of the sensor platforms 112, 114, and 116
include one or more sensors (e.g., video camera, radar, etc.). The
one or more sensors on each of the sensor platforms 112, 114, and
116 receive sensor data from target objects. Each of the sensor
platforms 112, 114, and 116 selects one or more screening policies
based on the mission parameters (e.g., desert, night, etc.) and/or
the platform type (e.g., aircraft, satellite, etc.). Each of the
sensor platforms 112, 114, and 116 generates a data set from the
sensor data based on the selected one or more screening policies
and transmits the data set to the computing devices 130 and 150.
The users 135 and 145 of the computing devices 130 and 150,
respectively, can view and/or modify the data set.
[0057] Table 1 illustrates exemplary sensor data, screening policy,
and data set. As illustrated in Table 1, the screening policy on
the sensor platform reduces the data sent to the computing devices
130 and 140 (i.e., the data set), thereby reducing the transmission
bandwidth between the sensor platform and the computing device and
reducing the data for review by the user 135 and 140,
respectively.
TABLE-US-00001 TABLE 1 Exemplary Sensor Data, Screening Policy, and
Data Set Data Set Sensor Platform Sensor Data Screening Policy
Destination Data Set Sensor Platform Parts A, B, C, D, Desert
Computing Parts B, D, F 112 E, F, G, H Environment; Device 130
Truck Target Objects Sensor Platform Parts L3, L4, L5, Truck Target
Computing Parts L4, L28, 114 L6 . . . L99 Objects Device 130 L56
Desert Computing Parts L6, L9, L10 Environment Device 140 Sensor
Platform Parts Z4, Z9, Fast Moving Computing Parts Z9, Z98 116 Z13,
Z45, Z56, Objects Device 130 Z98 Slow Moving Computing Parts Z45
Objects Device
[0058] Table 2 illustrates exemplary screening policy selection. As
illustrated in Table 2, each sensor platform can select a screening
policy based on the sensor platform type, the available screening
policies (e.g., available from a remote server, available in local
memory, etc.), and a selection from the computing device (e.g., the
user selects a desert screening policy, the user selects a boat
screening policy, etc.).
TABLE-US-00002 TABLE 2 Exemplary Screening Policy Selection
Screening Policy Selection From Sensor Available Screening
Computing Selected Sensor Platform Platform Type Policies Device
Screening Policy Sensor Platform Aircraft Low Altitude Desert;
Desert Medium Altitude 114 Medium Altitude Desert Desert; High
Altitude Desert . . . Sensor Platform Aircraft with Desert with
Thermal Black/White Desert with 114 Video Camera Imaging; Desert
with Imaging Black/White Video Imaging; Desert Imaging with
Black/White Imaging Sensor Platform Satellite with Low Altitude
Desert; Desert High Altitude 116 Optical Medium Altitude Desert
Lenses Desert; High Altitude Desert . . . Sensor Platform Unmanned
Low Velocity Desert; Desert Low Velocity 112 Aerial Medium Velocity
Desert Vehicle Desert; High Velocity Desert . . .
[0059] In some examples, each of the sensor platforms 112, 114, and
116 selects different screening policies and/or generates different
data sets from the sensor data based on the different screening
policies. In other examples, each of the sensor platforms 112, 114,
and 116 selects the same screening policies and generates different
data sets from the sensor data based on the same screening policies
but different sensor data (e.g., video sensor data, radar sensor
data, etc.).
[0060] Although FIG. 1 illustrates three sensor platforms 112, 114,
and 116, the technology can utilize any number and/or types of
sensor platforms. For example, the technology can utilize one
hundred sensor platforms. Although FIG. 1 illustrates two computing
devices 130 and 140, the technology can utilize any number and/or
types of computing devices. For example, the three sensor platforms
112, 114, and 116 can transmit sensor data and/or receive screening
policy requests from four hundred different computing devices.
Although FIG. 1 illustrates the sensor platforms 112, 114, and 116
as devices in the air and/or space, the sensor platforms can be any
type of sensor device and/or in any location (e.g., a video camera
on a mobile handheld device, a land-mounted video camera, an audio
sensor on a vehicle, etc.). In others, the technology can utilize
sensor data from any type of sensor located in any location (e.g.,
ground, air, space, water, etc.).
[0061] In other words, the technology enables the sensor platforms
to do some of the analysis before transmitting the data to the
operations and/or analyst centers (e.g., the computing devices at
these locations), thereby saving analysis time and computer
resources (e.g., network bandwidth, processing time, etc.). In some
examples, the technology sends back only the important information
for human viewing, thereby reducing end-user time on data analysis.
In other examples, extraneous data can be available for viewing at
a later time, if needed, for example if the analyst computer made a
mistake. The technology is more responsive to a greater range of
threats or opportunities across the spectrum due to the ability to
scale down the data being sent to an end-user while also saving all
of the data for later viewing by the end-user.
[0062] In some examples, the technology can provide accelerated and
improved information feeds by processing data at earliest possible
point for intelligence, surveillance, and reconnaissance
processing, exploitation, and dissemination (ISR PED) through
screening policies and/or automatic in-line processing. The
technology can provide definition of the events and can initiate
automated swapping of screening policies on the sensor platform
based on mission conditions and/or mission parameters.
[0063] The technology can provide air to ground transmission of the
output from the sensor platform into the downlink pathway (i.e., to
one or more computing devices). The technology can match
appropriate or more likely screening policies with mission plans
(e.g., water environment, city environment, etc.), objectives
(e.g., detection of people, detection of ships, etc.), and/or
typical mission scenarios (e.g., crowded city, empty farmland,
etc.). The technology can provide direct user control and
extraction for relevant intelligence, surveillance, and
reconnaissance (ISR) information and can support mission needs
through dynamic re-tasking of the sensor platforms and/or screening
policies. The technology can be used for any type and/or number of
plans, objectives, and/or scenarios (e.g., manned and unmanned;
air, water and land platforms that are sending ISR data over
communication links to various operation centers, etc.).
[0064] FIG. 2 is a flow diagram of an exemplary sensor platform 210
in a policy-based data management system (not shown). The sensor
platform 210 includes one or more sensors 220 and/or a receiver
(not shown). The sensor platform 210 further includes a data
screening policy module 230, a data screening module 240, a
transmitter, a multimedia chipping module 260, a data reduction
module 270, and a multimedia pattern matching module 280. The
multimedia chipping module 260, the data reduction module 270,
and/or the multimedia pattern matching module 280 are optional
components of the sensor platform 210. The sensor platform receives
sensor data 225 about one or more target objects, in this example,
a tank 251, a car 252, a tanker truck 253, and a truck 254.
[0065] The sensors 220 and/or the receiver receive sensor data 225.
The sensor data 225 can include video data, radar data, and/or any
other type of sensor data. The data screening policy module 230
selects one or more screening policies 234 from a plurality of
screening policies 232 based on one or more mission parameters
(e.g., water mission, desert mission, truck detection mission,
etc.) and a platform type (e.g., satellite, aircraft, video camera,
radar, etc.) associated with the sensor platform.
[0066] The data screening module 240 generate a data set 245 from
the sensor data 225 based on the selected one or more screening
policies 234 and/or one or more received screening policies 236.
The data screening module 240 can receive the one or more received
screening policies 236 from another system and/or user. The
transmitter 250 transmits the data set 245 to one or more computing
devices (not shown).
[0067] In some examples, the sensors 220 and/or the receiver
receive second sensor data (not shown). The second sensor data can
include video data, radar data, and/or any other type of sensor
data. The data screening policy module 230 selects one or more
second screening policies (not shown) from the plurality of
screening policies 232 based on one or more modified mission
parameters (e.g., water mission, desert mission, truck detection
mission, etc.) and a platform type (e.g., satellite, aircraft,
video camera, radar, etc.) associated with the sensor platform. The
data screening module 240 generate a second data set (not shown)
from the second sensor data based on the selected one or more
screening policies 234 and/or one or more received screening
policies 236. The transmitter 250 transmits the second data set to
one or more computing devices (not shown).
[0068] Table 3 illustrates modified mission parameters and the
resulting data set. As illustrated in Table 3, the sensor platform
can select a new screening policy based on modifications to the
mission parameters and transmit a new data set (e.g., additional
data, revised data, etc.) to the computing device for review by the
user. The modification and new data set process advantageously
enables the user to change the viewable data in real-time to
increase efficiency and reduce time spend reviewing data. The
mission parameter selection enables the users to fine-tune the
sensor data (e.g., boat size, relationships, behaviors, events,
etc.) to identify and track target objects, thereby increasing the
user's efficiency and decreasing the processing time for the
computing device.
TABLE-US-00003 TABLE 3 Exemplary Modified Mission Parameters
Initial Modified Revised Sensor Screening Initial Data Mission
Screening Revised Data Platform Policy Set Parameters Policy Set
Sensor Fast Moving Data Parts Reduce Speed Slow Moving Data Parts
Platform 114 Boat in Water AZ, BD, LM of Boats Boat in Water HG,
PN, JM Sensor Slow Moving Data Parts JK, Move to Slow Moving Data
Parts JK, Platform 116 Boat in Water L4, L8 Coastal Area Boat Close
to L4, L8, U8, Land in U3 Water & Slow Moving Boat in Water
Sensor Trucks in Data Parts Increase Trucks in Data Parts Platform
116 Urban U8, U9, U10 Truck Size Urban U8, U9, U10, Landscape &
Landscape & U12, U56 Tanks in Tanks in Urban Urban Landscape
Landscape & Trains in Urban Landscape
[0069] In other examples, the sensor data 225 includes image data,
video data, and/or audio data. In some examples, a data link module
(not shown) determines a data link parameter (e.g., low bandwidth,
high bandwidth, etc.) associated with a data link between the
respective sensor platform and at least one of the one or more
computing devices.
[0070] In other examples, the multimedia chipping module 260
generates one or more image chips from the sensor data based on the
data link parameter and one or more image parameters. The one or
image parameters can be received from one or more users associated
with at least one of the one or more computing devices. In some
examples, the one or image parameters comprise an image chipping
technique, a user preference, and/or a user selected focus
area.
[0071] In other examples, the data reduction module 270 dynamically
reduces the data set before transmission by the transmitter to form
a reduced data set 275 based on the data link parameter and the one
or more mission parameters.
[0072] In some examples, the multimedia pattern matching module 280
identifies one or more mission-relevant objects in the sensor data
based on the screening policy selection. In other examples, the
sensor data includes image data and/or video data. The multimedia
tagging module (not shown) associates one or more data tags 285
(e.g., location, time, suspected target object type, etc.) with the
sensor data associated with the identified one or more
mission-relevant objects based on the mission event. The
transmitter 250 transmits the data tags 285 associated with the
identified one or more mission-relevant objects and the associated
one or more data tags to the computing device.
[0073] In other examples, the one or more mission parameters
include a real-time mission event (e.g., course variation,
detection of a target object, etc.), a planned mission event (e.g.,
arrival at a waypoint, desert environment, etc.), a user preference
(e.g., no low resolution images, thermal images only, etc.), and/or
a data type associated with the sensor data (e.g., video, images,
audio, etc.). In some examples, the plurality of screening policies
232 includes a model-based policy, a statistics-based policy, a
learning-based policy, and/or a event-driven policy. The plurality
of screening policies 232 can be pre-loaded onto the sensor
platform 210 based on the expected mission and/or can be
dynamically uploaded to the sensor platform 210 based on a changed
mission. Each of the plurality of screening policies 232 can
include image filters, compression techniques, image analysis
techniques, computing device capabilities (e.g., screen resolution,
bandwidth restrictions, etc.), and/or any other type of policy to
reduce/change the sensor data.
[0074] FIG. 3 is a flow diagram of an exemplary computing device
310 in a data tasking and visualization system (not shown). The
computing device 310 includes a receiver 320, a display device 330,
a mission parameter module 340, a screening module 350, and a
transmitter 360. The screening module 350 receives a screening
policy selection 355 from a user associated with the computing
device. The screening policy selection 355 is selected from a
plurality of screening policies and each of the plurality of
screening policies is indicative of a mission event (e.g.,
detection of a target object, desert environment, water
environment, boat detection mission, etc.). As illustrated in the
GUI 335, the user can select one or more different policies (in
this example, a receive boat policy and a receive water policy).
The transmitter 360 transmits the screening policy selection 355 to
one or more sensor platforms 304, 306, and/or 308.
[0075] The receiver 320 receives one or more data sets 315 from the
one or more sensor platforms 304, 306, and/or 308 in response to
the transmission of the screening policy selection. The display
device 330 displays, via a graphical user interface (GUI) 335, the
one or more data sets to the user.
[0076] The mission parameter module 340 generates one or more
modified mission parameters 345 (e.g., new environment, new image
resolution threshold, etc.) based on input from a user associated
with the computing device. As illustrated in the GUI 335, the user
can select one or more modified mission parameters (in this
example, a non-moving parameter and a black/white parameter). The
transmitter 360 transmits the one or more modified mission
parameters 345 to at least one of the one or more of the sensor
platforms 304, 306, and/or 308. The sensor platforms 304, 306,
and/or 308 can utilize the one or more modified mission parameters
345.
[0077] In other examples, the computing device 310 includes a
identification module (not shown) and the identification module
associates one or more annotations received from the user with the
one or more data sets. The transmitter 360 transmits the one or
more annotations to a screening policy server (not shown) for
storage with the associated data sets.
[0078] In some examples, the screening module 350 determines a set
of screening policies from the plurality of screening policies
based on one or more mission parameters. In other examples, the one
or more mission parameters include a real-time mission event, a
planned mission event, a user preference, and/or a data type
associated with the sensor data. In some example, the plurality of
screening policies includes a model-based policy, a
statistics-based policy, a learning-based policy, and/or an
event-driven policy. In other examples, the computing device is a
mobile computing device.
[0079] FIG. 4A illustrates an exemplary graphical user interface
(GUI) 435a displayed on a display device 430a on a computing device
410a. The computing device 410a can access stored images and
associated information (e.g., name, tags, etc.) from an image
server for viewing. The image server can store the sensor data from
the one or more sensor platforms for later viewing by a user
associated with the computing device 410a. As illustrated in the
GUI 435a, the user of the computing device 410a can show, edit, or
destroy each of the images. Further, the user of the computing
device 410a can view the tags of the image. The image interface
into the image server advantageously enables users to access all of
the available sensor data at a later time after the transmission of
the reduced data set to the computing device of the user.
[0080] FIG. 4B illustrates an exemplary graphical user interface
(GUI) 435b displayed on a display device 430b on a computing device
410b. The GUI 435b illustrates a data set from a sensor platform
(in this example, a Global Hawk unmanned aerial vehicle). The user
can select four different missions profiles (that is, mission 1,
mission 2, mission 3, or mission 4) and the computing device 410b
transmits the policy selection to the sensor platform. The sensor
platform screens the sensor data based on the policy selection and
transmits the screened data (also referred to as a data set) to the
computing device 410b. The computing device 410b displays the
screened data on the GUI 435b. For example, mission 1 is directed
to the detection of boats in the water and mission 2 is directed to
aircraft flying below one thousand feet.
[0081] FIG. 4C illustrates an exemplary graphical user interface
(GUI) 435c displayed on a display device 430c on a computing device
410c. The GUI 435c illustrates a data set from a sensor platform
(in this example, an unmanned aerial vehicle). The user can select
different mission parameters (also referred to a screening catalog)
and the computing device 410b transmits the mission parameters to
the sensor platform. The sensor platform screens the sensor data
based on the mission parameters and transmits the screened data
(also referred to as a data set) to the computing device 410c. The
computing device 410c displays the screened data on the GUI
435c.
[0082] FIG. 4D illustrates an exemplary graphical user interface
(GUI) 435d displayed on a display device 430d on a computing device
410d. The GUI 435d illustrates a data set from a sensor platform.
The user can select different mission parameters (e.g., parameters
within a small boat screening policy) and the computing device 410d
transmits the mission parameters to the sensor platform. The sensor
platform screens the sensor data based on the mission parameters
and transmits the screened data (also referred to as a data set) to
the computing device 410d. The computing device 410d displays the
screened data on the GUI 435d.
[0083] FIG. 4E illustrates an exemplary graphical user interface
(GUI) 435e displayed on a display device 430e on a computing device
410e. The GUI 435e illustrates a data set from a sensor platform
(in this example, a Global Hawk unmanned aerial vehicle). As
illustrated in the GUI 435e, the user can view behaviors,
annotations about the image (e.g., who, what, when, where, why,
etc.), object information, and/or other relevant information about
the data set. In other words, the technology advantageously enables
the annotation of sensor data and/or sharing of the annotations
among the computing devices. To this end, a user utilizing a
computing device can share annotations with other users via this
process, thereby advantageously sharing intelligence information
without overwhelming the other users.
[0084] FIG. 5 is a block diagram of an exemplary sensor platform
510. The sensor platform 510 includes a receiver 511, a data
screening policy module 512, a data screening module 513, a
transmitter 514, a data link module 515, a multimedia chipping
module 516, one or more sensors 517 (e.g., video camera, radar,
audio, etc.), a data reduction module 518, a multimedia pattern
matching module 519, a multimedia tagging module 520, a processor
594, and a storage device 595. The modules and devices described
herein can, for example, utilize the processor 594 to execute
computer executable instructions and/or include a processor to
execute computer executable instructions (e.g., an encryption
processing unit, a field programmable gate array processing unit,
etc.). It should be understood that the sensor platform 510 can
include, for example, other modules, devices, and/or processors
known in the art and/or varieties of the illustrated modules,
devices, and/or processors.
[0085] The sensors 517 and/or the receiver 511 receive sensor data.
The data screening policy module 512 selects one or more screening
policies from a plurality of screening policies based on one or
more mission parameters (e.g., water mission, desert mission, truck
detection mission, etc.) and a platform type (e.g., satellite,
aircraft, video camera, radar, etc.) associated with the sensor
platform 510.
[0086] The data screening module 513 generates a data set from the
sensor data based on the selected one or more screening policies
and/or one or more received screening policies. The data screening
module 513 can receive the one or more received screening policies
from another system and/or user. The transmitter 514 transmits the
data set to one or more computing devices (not shown).
[0087] The data link module 515 determines a data link parameter
associated with a data link between the sensor platform 510 and at
least one of the one or more computing devices. The multimedia
chipping module 516 generates one or more image chips from the
sensor data based on the data link parameter and one or more image
parameters. The data reduction module 518 dynamically reduces the
data set before transmission by the transmitter to form a reduced
data set based on the data link parameter and the one or more
mission parameters.
[0088] The multimedia pattern matching module 519 identifies one or
more mission-relevant objects (e.g., people, faces, men, women,
mean with beards, men without beards, specific individuals,
mountain backdrop, flat terrain backdrop, vehicles, weapons, roads,
trails, power lines, landing strips, building, mobile antenna,
etc.) in the sensor data based on the screening policy selection.
The multimedia tagging module 520 associates one or more data tags
(e.g., location, time, suspected target object type, etc.) with the
sensor data associated with the identified one or more
mission-relevant objects based on the mission event. The
transmitter 514 transmits the data tags associated with the
identified one or more mission-relevant objects and the associated
one or more data tags to the computing device.
[0089] The processor 594 executes the operating system and/or any
other computer executable instructions for the sensor platform 510
(e.g., executes applications, etc.). The storage device 595 stores
the received data (e.g., actual sensor data, processed radar data,
etc.), the identifications, and/or any other data associated with
the sensor platform 510. The storage device 595 can store image
information and/or any other data associated with the sensor
platform 510. The storage device 595 can include a plurality of
storage devices and/or the sensor platform 510 can include a
plurality of storage devices (e.g., a radar storage device, an
identification storage device, etc.). The storage device 595 can
include, for example, long-term storage (e.g., a hard drive, a tape
storage device, flash memory, etc.), short-term storage (e.g., a
random access memory, a graphics memory, etc.), and/or any other
type of computer readable storage.
[0090] FIG. 6 is a block diagram of an exemplary computing device
610. The computing device 610 includes a receiver 611, a display
device 612, a mission parameter module 613, a transmitter 614, a
screening module 615, an identification module 616, a processor
694, and a storage device 695. The modules and devices described
herein can, for example, utilize the processor 694 to execute
computer executable instructions and/or include a processor to
execute computer executable instructions (e.g., an encryption
processing unit, a field programmable gate array processing unit,
etc.). It should be understood that the computing device 610 can
include, for example, other modules, devices, and/or processors
known in the art and/or varieties of the illustrated modules,
devices, and/or processors.
[0091] The receiver 611 receives one or more data sets from the one
or more sensor platforms in response to the transmission of the
screening policy selection. The display device 612 displays, via a
graphical user interface (GUI), the one or more data sets to the
user. The mission parameter module 613 generates one or more
modified mission parameters (e.g., new environment, new image
resolution threshold, etc.) based on input from a user associated
with the computing device. The transmitter 614 transmits the one or
more modified mission parameters to at least one of the one or more
of the sensor platforms.
[0092] The screening module 615 determines a set of screening
policies from the plurality of screening policies based on one or
more mission parameters. The identification module 616 associates
one or more annotations received from the user with the one or more
data sets. The transmitter 614 transmits the one or more
annotations to a screening policy server for storage with the
associated data sets.
[0093] The processor 694 executes the operating system and/or any
other computer executable instructions for the computing device 610
(e.g., executes applications, etc.). The storage device 695 stores
the received data (e.g., actual sensor data, processed radar data,
etc.), the identifications, and/or any other data associated with
the computing device 610. The storage device 695 can store image
information and/or any other data associated with the computing
device 610. The storage device 695 can include a plurality of
storage devices and/or the computing device 610 can include a
plurality of storage devices (e.g., a radar storage device, an
identification storage device, etc.). The storage device 695 can
include, for example, long-term storage (e.g., a hard drive, a tape
storage device, flash memory, etc.), short-term storage (e.g., a
random access memory, a graphics memory, etc.), and/or any other
type of computer readable storage.
[0094] FIG. 7 is a block diagram of an exemplary screening policy
server 710. The screening policy server 710 includes a receiver
711, a data repository module 712, a screening policy generator
module 713, a transmitter 714, a processor 794, and a storage
device 795. The modules and devices described herein can, for
example, utilize the processor 794 to execute computer executable
instructions and/or include a processor to execute computer
executable instructions (e.g., an encryption processing unit, a
field programmable gate array processing unit, etc.). It should be
understood that the screening policy server 710 can include, for
example, other modules, devices, and/or processors known in the art
and/or varieties of the illustrated modules, devices, and/or
processors.
[0095] The receiver 711 receives sensor data and data sets from the
one or more sensor platforms. The receiver 711 receives screening
policy selections from a plurality of computing devices. The data
repository module 712 stores the sensor data, the data sets, and
the screening policy selections (e.g., stored in the storage device
795). The screening policy generation module 713 generates the
plurality of screening policies based on the stored sensor data,
the stored data sets, and the stored screening policy selections.
The transmitter 714 transmits the plurality of screening policies
to the plurality of computing devices and/or the one or more sensor
platforms.
[0096] In other examples, the screening policy generation module
713 modifies at least one of the plurality of screening policies
based on the stored sensor data, the stored data sets, and the
stored screening policy selections.
[0097] The processor 794 executes the operating system and/or any
other computer executable instructions for the screening policy
server 710 (e.g., executes applications, etc.). The storage device
795 stores the received data (e.g., actual sensor data, processed
radar data, etc.), the identifications, and/or any other data
associated with the screening policy server 710. The storage device
795 can store image information and/or any other data associated
with the screening policy server 710. The storage device 795 can
include a plurality of storage devices and/or screening policy
server 710 can include a plurality of storage devices (e.g., a
radar storage device, an identification storage device, etc.). The
storage device 795 can include, for example, long-term storage
(e.g., a hard drive, a tape storage device, flash memory, etc.),
short-term storage (e.g., a random access memory, a graphics
memory, etc.), and/or any other type of computer readable
storage.
[0098] FIG. 8 is a flow diagram of an exemplary policy environment
800. The environment 800 includes a policy decision point system
830 and a policy enforcement point system 840. The policy decision
point system 830 (can be the policy screening server and/or a part
of the policy screening server) receives (801) a new policy. The
policy decision point system 830 integrates (802) the new policy
with existing policies. The policy decision point system 830
retrieves (803) the existing policies from the repository (e.g.,
storage device, database, etc.). The policy decision point system
830 forwards (804) the integrated policy to a rules engine. The
policy decision point system 830 saves (805) the integrated policy
into the repository. The policy decision point system 830 reports
(806) the integration of the new policy.
[0099] The rules engine generates (807) actions (e.g., email alert,
documentation acceptance, etc.) if conditions are met for the new
policy being integrated into the policy. A user (i.e., person in
the loop) can, in some examples, decide (808) on whether to take
the actions. The policy decision point system 830 can send (809)
the decision on whether to accept or reject the new policy and/or
actions to user transmitting the new policy. The policy decision
point system 830 sends (810) actions to the policy enforcement
point system 840.
[0100] The policy enforcement point system 840 (can be the policy
screening server and/or a part of the policy screening server)
retrieves (811) the specific command syntax from a database of
screening techniques. The policy enforcement point system 840
readies (812) the new screening policies for execution. The policy
enforcement point system 840 retrieves (813) the specific screening
techniques from a storage device and reports (814) the runtime
status of the screening policy. The generation and verification of
the screening policies utilizing the process described herein
advantageously enables the integration of new policies within the
existing screening policies in an efficient and cost-effective
manner.
[0101] FIG. 9 is a block diagram of an exemplary image access and
storage process 900. A sensor 910 senses raw image data (also
referred to as sensor data) and transmits the raw image data to a
data store 920. The data store 920 stores the raw image data and
transits the processed image to an image server 970 upon request.
The storage of the raw and processed images in the data store 920
advantageously enables long-term storage and retrieval of the
images by users upon request without overburdening the users with
all of the images.
[0102] An image processing and screening algorithm 940 receives the
raw image data from the data store 920. The image processing and
screening algorithm 940 processes and screens the raw image data
utilizing one or more screening policies (e.g., selected by a user
and transmitted from a computing device 980, selected by a context
switching engine 930, etc.). If the image processing and screening
algorithm 940 detects any anomalies with the raw image data, then
the image processing and screening algorithm 940 reports the
anomaly to the context switching engine 930. The context-switching
engine 930 can manage the one or more screening policies based on
the anomalies and/or user criteria received from the computing
device 980. The image processing algorithm 940 transmits the
resultant processed image data to the data store 920. In addition,
the image processing algorithm 940 transmits meta-data derived from
the processing of the image and other context sensitive input data
to the image store 920 for later query.
[0103] During the processing and screening algorithm 940 activity,
if the image processing and screening algorithm 940 detects any
interesting objects, then the processing and screening algorithm
940 can decide to transmit the screened image data to an image
reduction algorithm 950 in order to filter, resulting in higher
quality end-user imagery yield, i.e., reduction in non-interesting
images based on user criteria. Upon completion of processing and
screening, the image processing and screening algorithm 940
transmits the screened image data to an image reduction algorithm
950. The image reduction algorithm 950 processes the image data for
reduction based on user criteria received from the computing device
980. Based upon satisfactory reduction test passage, i.e., user
criteria quality thresholds (e.g., image size, image quality, image
scale, etc.) being met, the image reduction algorithm 950 sends a
notification to the image compression algorithm 960. After the
processing, the image reduction algorithm 950 sends the resulting
image data to the image compression algorithm 960. The image
compression algorithm 960 compresses the reduced image based on
user criteria received from the computing device 980 for
compression quality and other compression processing algorithms
settings. The image compression algorithm 960 transmits all reduced
image data and associated meta-data derived from the compression
process to the image store 920. The compression of the image
advantageously enables the technology to automatically adjust the
image quality/compression to different data links between the
sensor platform and the computing device, while at the same time
allowing user criteria received from the computing device 980 to
override automatic settings and processing. For example, if the
computing device and the sensor platform both utilize a high
bandwidth data link, then the image can be transmitted in an
uncompressed format. As another example, if the computing device
and the sensor platform both utilize a low bandwidth data link, the
image can be minimized for transmission and the user, utilizing the
computing device, can access the uncompressed image later via the
image server 970 process described herein.
[0104] The computing device 980 can receive images from the image
reduction algorithm 950 and/or the image server 970. For example, a
user utilizing the computing device 980 can view a partial image
from the image reduction algorithm 950 and then query the image
server 970 for all related images. A query can be based up on
meta-data stored in the image store 920 and processed by the image
server 970. Meta-data includes geo-location data, temporal data,
sensor data, image processing and screening algorithm 940 data and
contextual information, and image compression data 960. The image
server 970 returns the requested images or textual list of imagery
information to the computing device 980 for display to the user.
The user of the computing device can manually or automatically
utilize the image screening and reduction process described in the
process 900 to reduce the network bandwidth to the computing device
980 and/or reduce the images for review. The user of the computing
device 980 can also augment an image's meta-data via submittal of
deleted, modified or new information to the imagery server 970.
[0105] FIG. 10 illustrates an exemplary data process 1000
utilizing, for example, the sensor platform 114 of FIG. 1. The
sensor platform 114, via a processor and/or storage device,
receives (1010) a new image. The sensor platform 114 determines
(1020) a context for the image. In a first example (i.e., a first
new image for a first screening policy), the detected context is
land. In a second example (i.e., the first new image for a second
screening policy), the detected context is water. In this example,
the first new image includes a part with land and another part with
water. In other words, the sensor platform 114 can perform the same
process to the same image with different screening policies and
thus, different parameters for the process (as illustrated in FIG.
10). In other examples, a context switching engine determines the
screening policy based on the determined context.
[0106] In the first example, the sensor platform 114 segments
(1030) the new image. The sensor platform 114 reduces (1040) the
noise for the segmented image (e.g., utilizing any type of standard
noise reduction). The sensor platform 114 groups (1060) the
segmented images (e.g., grouping based on number of target objects,
etc.). The sensor platform 114 filters (1070) (e.g., white noise
filter, etc.) and labels (1080) (e.g., associated annotations with
the image) the image. The sensor platform 114 compresses (e.g.,
utilizing any type of standard image compression, etc.) and
distributes (1090) (e.g., file transfer protocol, etc.) the image
to the computing device(s) associated with the first screening
algorithm.
[0107] In the second example, the sensor platform 114 segments
(1040) the new image. The sensor platform 114 reduces (1040) the
noise for the segmented image. The sensor platform 114 groups
(1060) the segmented images. The sensor platform 114 filters (1070)
and labels (1080) the image. The sensor platform 114 compresses and
distributes (1090) the image to the computing device(s) associated
with the first screening algorithm.
[0108] FIG. 11 is a flowchart of an exemplary method 1100 for
policy-based data management utilizing, for example, the sensor
platform 114 of FIG. 1 (also referred to as the sensor platform
device) and/or the computing device 130 of FIG. 1. The sensor
platform 114 receives (1110) sensor data from one or more sensors.
The sensor platform 114 selects (1120) one or more screening
policies from a plurality of screening policies based on one or
more mission parameters and a platform type associated with the
sensor platform device. The sensor platform 114 generates (1130) a
data set from the sensor data based on the selected one or more
screening policies. The sensor platform 114 transmits (1140) the
data set to one or more computing devices.
[0109] In some examples, the computing device 130 receives (1150)
data sets from a plurality of sensor platform devices. The
computing device 130 generates (1152) one or more modified mission
parameters based on input from a user associated with the computing
device. The computing device 130 transmits (1154) the one or more
modified mission parameters to one or more of sensor platform
devices.
[0110] In other examples, the sensor platform 114 receives (1160)
the one or more modified mission parameters. The sensor platform
114 selects (1162) one or more second screening policies from the
plurality of screening policies based on the one or more modified
mission parameters and the platform type associated with the sensor
platform device. The sensor platform 114 generates (1164) a second
data set from the sensor data based on the selected one or more
second screening policies. The sensor platform 114 transmits (1166)
the second data set to the computing device.
[0111] In some examples, the sensor platform 114 receives the one
or more modified mission parameters. The sensor platform 114
receives second sensor data from the one or more sensors. The
sensor platform 114 selects one or more second screening policies
from the plurality of screening policies based on the one or more
modified mission parameters and the platform type associated with
the sensor platform device. The sensor platform 114 generates a
second data set from the second sensor data based on the selected
one or more second screening policies. The sensor platform
transmits the second data set to the computing device.
[0112] In other examples, the sensor platform 114 determines a data
link parameter (e.g., 50 mega bytes per second, 256 bytes per
second, etc.) associated with a data link between the sensor
platform device and at least one of the one or more computing
devices. The sensor platform 114 generates one or more image chips
from the sensor data based on the data link parameter and one or
more image parameters (e.g., an image chip is a frame of a video,
an image chip is an image that shows an entire truck, etc.). The
one or image parameters are received from one or more users
associated with the at least one of the one or more computing
devices and the sensor data includes image data, video data, and/or
audio data. The sensor platform 114 transmits the one or more image
chips to the least one of the one or more computing devices.
[0113] In some examples, the sensor platform 114 determines a data
link parameter associated with a data link between the respective
sensor platform and the one or more computing devices. The sensor
platform 114 dynamically reduces the data set before transmitting
based on the data link parameter and the one or more mission
parameters. The dynamic reduction of the data set includes a
compression technique, and/or a data correlation technique.
[0114] In other examples, the one or more mission parameters are
pre-determined based on a planned mission associated with the
sensor platform device. In some examples, the sensor platform 114
dynamically generates the one or more mission parameters based on
one or more real-time mission events.
[0115] FIG. 12 is a flowchart of an exemplary method 1200 for data
tasking and visualization of data utilizing, for example, the
sensor platform 114 of FIG. 1 and/or the computing device 130 of
FIG. 1. The computing device 130 receives (1210) a screening policy
selection from a user associated with the computing device. The
screening policy selection is selected from a plurality of
screening policies and each of the plurality of screening policies
being indicative of a mission event. The computing device 130
transmits (1220) the screening policy selection to one or more
sensor platform devices. The computing device 130 receives (1230)
one or more data sets from the one or more sensor platform devices
in response to the transmission of the screening policy selection.
The computing device 130 displays (1240) the one or more data sets
to the user.
[0116] In some examples, the computing device 130 associates (1250)
one or more annotations received from the user with the one or more
data sets. The computing device 130 transmits (1252) the one or
more annotations to a screening policy server.
[0117] In other examples, the sensor platform 114 receives (1260)
the screening policy selection from the computing device. The
sensor platform 114 receives (1262) sensor data from one or more
sensors. The sensor platform 114 generates (1264) a data set from
the sensor data based on the screening policy selection. The sensor
platform 114 transmits (1266) the data set to the computing
device.
[0118] In some examples, the sensor platform 114 identifies one or
more mission-relevant objects in the sensor data based on the
screening policy selection. The sensor data includes image data
and/or video data. The sensor platform 114 associates one or more
data tags with the sensor data associated with the identified one
or more mission-relevant objects based on the mission event. The
sensor platform 114 transmits the sensor data associated with the
identified one or more mission-relevant objects and the associated
one or more data tags to the computing device.
[0119] In other examples, the computing device 130 determines a set
of screening policies from the plurality of screening policies
based on one or more mission parameters. In some examples, the one
or more mission parameters include a real-time mission event, a
planned mission event, a user preference, and/or a data type
associated with the sensor data.
[0120] The above-described systems and methods can be implemented
in digital electronic circuitry, in computer hardware, firmware,
and/or software. The implementation can be as a computer program
product. The implementation can, for example, be in a
machine-readable storage device, for execution by, or to control
the operation of, data processing apparatus. The implementation
can, for example, be a programmable processor, a computer, and/or
multiple computers.
[0121] A computer program can be written in any form of programming
language, including compiled and/or interpreted languages, and the
computer program can be deployed in any form, including as a
stand-alone program or as a subroutine, element, and/or other unit
suitable for use in a computing environment. A computer program can
be deployed to be executed on one computer or on multiple computers
at one site.
[0122] Method steps can be performed by one or more programmable
processors executing a computer program to perform functions of the
invention by operating on input data and generating output. Method
steps can also be performed by and an apparatus can be implemented
as special purpose logic circuitry. The circuitry can, for example,
be a FPGA (field programmable gate array) and/or an ASIC
(application-specific integrated circuit). Subroutines and software
agents can refer to portions of the computer program, the
processor, the special circuitry, software, and/or hardware that
implement that functionality.
[0123] Processors suitable for the execution of a computer program
include, by way of example, both general and special purpose
microprocessors, and any one or more processors of any kind of
digital computer. Generally, a processor receives instructions and
data from a read-only memory or a random access memory or both. The
essential elements of a computer are a processor for executing
instructions and one or more memory devices for storing
instructions and data. Generally, a computer can be operatively
coupled to receive data from and/or transfer data to one or more
mass storage devices for storing data (e.g., magnetic,
magneto-optical disks, or optical disks).
[0124] Data transmission and instructions can also occur over a
communications network. Computer program products suitable for
embodying computer program instructions and data include all forms
of non-volatile memory, including by way of example semiconductor
memory devices. The computer program products can, for example, be
EPROM, EEPROM, flash memory devices, magnetic disks, internal hard
disks, removable disks, magneto-optical disks, CD-ROM, and/or
DVD-ROM disks. The processor and the memory can be supplemented by,
and/or incorporated in special purpose logic circuitry.
[0125] To provide for interaction with a user, the above described
techniques can be implemented on a computer having a display
device. The display device can, for example, be a cathode ray tube
(CRT) and/or a liquid crystal display (LCD) monitor. The
interaction with a user can, for example, be a display of
information to the user and a keyboard and a pointing device (e.g.,
a mouse or a trackball) by which the user can provide input to the
computer (e.g., interact with a user interface element). Other
kinds of devices can be used to provide for interaction with a
user. Other devices can, for example, be feedback provided to the
user in any form of sensory feedback (e.g., visual feedback,
auditory feedback, or tactile feedback). Input from the user can,
for example, be received in any form, including acoustic, speech,
and/or tactile input.
[0126] The above described techniques can be implemented in a
distributed computing system that includes a back-end component.
The back-end component can, for example, be a data server, a
middleware component, and/or an application server. The above
described techniques can be implemented in a distributing computing
system that includes a front-end component. The front-end component
can, for example, be a client computer having a graphical user
interface, a Web browser through which a user can interact with an
example implementation, and/or other graphical user interfaces for
a transmitting device. The components of the system can be
interconnected by any form or medium of digital data communication
(e.g., a communication network). Examples of communication networks
include a local area network (LAN), a wide area network (WAN), the
Internet, wired networks, and/or wireless networks.
[0127] The system can include clients and servers. A client and a
server are generally remote from each other and typically interact
through a communication network. The relationship of client and
server arises by virtue of computer programs running on the
respective computers and having a client-server relationship to
each other.
[0128] Packet-based networks can include, for example, the
Internet, a carrier internet protocol (IP) network (e.g., local
area network (LAN), wide area network (WAN), campus area network
(CAN), metropolitan area network (MAN), home area network (HAN)), a
private IP network, an IP private branch exchange (IPBX), a
wireless network (e.g., radio access network (RAN), 802.11 network,
802.16 network, general packet radio service (GPRS) network,
HiperLAN), and/or other packet-based networks. Circuit-based
networks can include, for example, the public switched telephone
network (PSTN), a private branch exchange (PBX), a wireless network
(e.g., RAN, bluetooth, code-division multiple access (CDMA)
network, time division multiple access (TDMA) network, global
system for mobile communications (GSM) network), and/or other
circuit-based networks.
[0129] The transmitting device can include, for example, a
computer, a computer with a browser device, a telephone, an IP
phone, a mobile device (e.g., cellular phone, personal digital
assistant (PDA) device, laptop computer, electronic mail device),
and/or other communication devices. The browser device includes,
for example, a computer (e.g., desktop computer, laptop computer)
with a world wide web browser (e.g., Microsoft.RTM. Internet
Explorer.RTM. available from Microsoft Corporation, Mozilla.RTM.
Firefox available from Mozilla Corporation). The mobile computing
device includes, for example, a Blackberry.RTM..
[0130] Comprise, include, and/or plural forms of each are open
ended and include the listed parts and can include additional parts
that are not listed. And/or is open ended and includes one or more
of the listed parts and combinations of the listed parts.
[0131] One skilled in the art will realize the invention may be
embodied in other specific forms without departing from the spirit
or essential characteristics thereof. The foregoing embodiments are
therefore to be considered in all respects illustrative rather than
limiting of the invention described herein. Scope of the invention
is thus indicated by the appended claims, rather than by the
foregoing description, and all changes that come within the meaning
and range of equivalency of the claims are therefore intended to be
embraced therein.
* * * * *