U.S. patent application number 12/144057 was filed with the patent office on 2008-10-16 for people searches by multisensor event correlation.
Invention is credited to LISA MARIE BROWN, Arun Hampapur, Zuoxuan Lu, Sharathchandra U. Pankanti, Andrew William Senior, Chiao-Fe Shu, Ying-Li Tian.
Application Number | 20080252727 12/144057 |
Document ID | / |
Family ID | 38862696 |
Filed Date | 2008-10-16 |
United States Patent
Application |
20080252727 |
Kind Code |
A1 |
BROWN; LISA MARIE ; et
al. |
October 16, 2008 |
PEOPLE SEARCHES BY MULTISENSOR EVENT CORRELATION
Abstract
A data indexing system and method includes acquiring activity
data in a context and indexing the activity data in accordance with
contextual conditions. The activity data is stored in accordance
with indices. An event is correlated with the activity data by
using the indices to review the activity data in the context.
Inventors: |
BROWN; LISA MARIE;
(Pleasantville, NY) ; Hampapur; Arun; (Norwalk,
CT) ; Lu; Zuoxuan; (Yorktown Heights, NY) ;
Pankanti; Sharathchandra U.; (Manhasset, NY) ;
Senior; Andrew William; (New York, NY) ; Shu;
Chiao-Fe; (Scarsdale, NY) ; Tian; Ying-Li;
(Yorktown Heights, NY) |
Correspondence
Address: |
KEUSEY, TUTUNJIAN & BITETTO, P.C.
20 CROSSWAYS PARK NORTH, SUITE 210
WOODBURY
NY
11797
US
|
Family ID: |
38862696 |
Appl. No.: |
12/144057 |
Filed: |
June 23, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11454819 |
Jun 16, 2006 |
|
|
|
12144057 |
|
|
|
|
Current U.S.
Class: |
348/143 ;
348/E7.085; 707/999.002; 707/E17.004 |
Current CPC
Class: |
G06F 16/784 20190101;
G06F 16/7335 20190101; G07G 1/0036 20130101 |
Class at
Publication: |
348/143 ; 707/2;
707/E17.004; 348/E07.085 |
International
Class: |
H04N 7/18 20060101
H04N007/18; G06F 17/30 20060101 G06F017/30; G06F 7/00 20060101
G06F007/00 |
Claims
1. A data indexing method, comprising: acquiring activity data in a
context; indexing the activity data in accordance with contextual
conditions; storing the activity data in accordance with indices;
and correlating an event with the activity data by using the
indices to review the activity data in the context.
2. The method as recited in claim 1, wherein acquiring activity
data includes employing at least one of sensors and cameras.
3. The method as recited in claim 1, wherein the activity data
include activities of an individual and the context includes a
retail establishment, the method further comprising: associating
separate observations of the individual using the indices.
4. The method as recited in claim 3, wherein associating separate
observations is performed while the individual remains in the
retail establishment.
5. The method as recited in claim 1, further comprising enabling
searches of a database storing the indices and conducting the
searches to prevent one of returns fraud by an individual and
insurance claim fraud.
6. The method as recited in claim 1, wherein the indices include
one or more of: facial appearance, speed, time of observation,
color, anthropometric measurements, gait, biometric
identifiers.
7. The method as recited in claim 1, wherein the indices include
data from one or more of an RFID tag, a pressure pad, a motion
sensor, a beam breaker, a local positioning system.
8. The method as recited in claim 1, further comprising tracking an
individual in the context to provide a history of the individuals
activities within the context.
9. The method as recited in claim 1, further comprising archiving
the activity data in accordance with the indices.
10. A computer program product for data indexing comprising a
computer useable medium including a computer readable program,
wherein the computer readable program when executed on a computer
causes the computer to perform the steps of: acquiring activity
data in a context; indexing the activity data in accordance with
contextual conditions; storing the activity data in accordance with
indices; and correlating an event with the activity data by using
the indices to review the activity data in the context.
11. A data indexing system, comprising: a plurality of acquisition
devices configured to acquire activity data in a context; a
surveillance engine configured to process the activity data and
derive search indices for the activity data; an index association
engine to correlate different observations using the indices; a
user-interface configured to conduct searches and present search
results at a user station, wherein the searches correlate stored
activity data with newly acquired activity data at the user station
based on the indices.
12. The system as recited in claim 11, wherein the acquisition
devices include at least one of sensors and cameras.
13. The system as recited in claim 11, wherein the activity data
include activities of an individual and the context includes a
retail establishment, and the index association engine is
configured to match the indices for the activity data in the
context at an earlier time with the newly acquired indices at the
user station.
14. The system as recited in claim 13, wherein the user-interface
conducts the searches while the individual remains in the retail
establishment.
15. The system as recited in claim 11, wherein fraud is detected by
locating video, based on indices, of a customer acting in a manner
inconsistent with an action made by the customer at a different
time.
16. The system as recited in claim 11, wherein the indices include
one or more of: facial appearance, speed, time of observation,
color, anthropometric measurements, gait, biometric
identifiers.
17. The system as recited in claim 11, wherein the indices include
data from one or more of an RFID tag, a pressure pad, a motion
sensor, a beam breaker, a local positioning system.
18. The system as recited in claim 11, wherein the index
association engine is configured to store indexed activities.
19. The system as recited in claim 11, further comprising a
database configured to archive activity data in accordance with the
indices.
20. The system as recited in claim 11, further comprising an index
server configured to satisfy queries from the user-interface by
conducting searches.
Description
RELATED APPLICATION INFORMATION
[0001] This application is a continuation of co-pending U.S. patent
Ser. No. 11/454,819, filed Jun. 16, 2006, which is incorporated by
reference herein in its entirety.
BACKGROUND
[0002] 1. Technical Field
[0003] The present invention relates to video camera surveillance,
and more particularly, to fraud prevention using video and other
devices to understand, index and replay human activity.
[0004] 2. Description of the Related Art
[0005] Fraud and shoplifting continue to persist as a main issue in
retail businesses. One component of the problem is fraud by
customers, specifically claiming a refund for a returned item, when
they use a genuine receipt with an item that is just off the
shelves, while keeping the item that they bought at home. Current
solutions are very limited. Current solutions include store
detectives or "human video surveillance".
[0006] Other problems, such as fraudulent injuries and compensation
claims also remain a problem for retail establishments.
Surveillance systems can provide some protection against these
problems, but in many instances prove to be inadequate. Many
consumers are aware of camera systems and take steps to avoid or
defeat them.
SUMMARY
[0007] A surveillance system and method includes acquiring human
activity data in a context and indexing the activity data in
accordance with contextual conditions. The activity data is stored
in a database in accordance with indices. Searches of the database
are enabled using the indices to obtain previous human activity in
the context.
[0008] A data indexing method includes acquiring activity data in a
context, indexing the activity data in accordance with contextual
conditions, storing the activity data in accordance with indices,
and correlating an event with the activity data by using the
indices to review the activity data in the context.
[0009] A data indexing system includes a plurality of acquisition
devices configured to acquire activity data in a context. A
surveillance engine is configured to process the activity data and
derive search indices for the activity data. An index association
engine correlates different observations using the indices, and a
user-interface configured to conduct searches and present search
results at a user station. The searches correlate stored activity
data with newly acquired activity data at the user station based on
the indices.
[0010] These and other objects, features and advantages will become
apparent from the following detailed description of illustrative
embodiments thereof, which is to be read in connection with the
accompanying drawings.
BRIEF DESCRIPTION OF DRAWINGS
[0011] The disclosure will provide details in the following
description of preferred embodiments with reference to the
following figures wherein:
[0012] FIG. 1 is an illustrative depiction of a retail
establishment floor plan showing an embodiment in accordance with
present principles;
[0013] FIG. 2 is a block/flow diagram showing an illustrative
surveillance method performed by the system in accordance with
present principles; and
[0014] FIG. 3 is a block diagram illustratively showing components
and flow of information between the components in a surveillance
system in accordance with present principles.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
[0015] Embodiments in accordance with present principles are
directed to the use of video cameras and other sensing devices to
record activities in locations such as stores, public transport,
streets, etc. Using video analysis software to index the video,
events of interest can be retrieved using a variety of cues. In
illustrative examples, some possible "events of interest" may
include the following exemplary scenarios:
[0016] 1) A customer enters a store, with a receipt for a product
bought earlier, proceeds to pick up a new product of the same type,
and then goes to "return" it at customer service. In this instance,
the system semi-automatically or automatically finds video from the
time that the customer entered the store, and can be used to show
the employee that no refund should be given. The video can be
presented to the customer, or saved for future use.
[0017] 2) An employee or shopper claims to have had an accident in
a certain location. Video is found of the claimed time/location and
it can be verified whether such an accident took place.
[0018] 3) When an accident is claimed, prior video may be found
that indicates dangerous behavior on the part of a claimant or
subsequent video may be found indicating behavior inconsistent with
the claimed injury.
[0019] 4) Other scenarios may benefit from the systems and methods
as disclosed herein.
[0020] Current video surveillance systems may record video clips
that are sought in the above examples, but one advantage of the
present principles is that automatic processing, indexing and
association of video makes finding these clips a relatively simple
(and thus cost-effective) task that can quickly be carried out by a
harried store clerk when a return is requested/claim is made.
[0021] The speed and simplicity mean at least that: (1) claims can
be defeated immediately by demonstration of the video; (2) the
density of video recording can be increased because it becomes
feasible to search many more feeds; (3) the cost of searching is
reduced, as the search is performed by computer, rather than by a
person as in the present case; (4) the accuracy of the search is
increased because human error and loss of attention are
avoided.
[0022] Embodiments of the present invention can take the form of an
entirely hardware embodiment, an entirely software embodiment or an
embodiment including both hardware and software elements. In a
preferred embodiment, the present invention is implemented in a
combination of hardware and software. The software includes but is
not limited to firmware, resident software, microcode, etc.
[0023] Furthermore, the invention can take the form of a computer
program product accessible from a computer-usable or
computer-readable medium providing program code for use by or in
connection with a computer or any instruction execution system. For
the purposes of this description, a computer-usable or
computer-readable medium can be any apparatus that may include,
store, communicate, propagate, or transport the program for use by
or in connection with the instruction execution system, apparatus,
or device. The medium can be an electronic, magnetic, optical,
electromagnetic, infrared, or semiconductor system (or apparatus or
device) or a propagation medium. Examples of a computer-readable
medium include a semiconductor or solid state memory, magnetic
tape, a removable computer diskette, a random access memory (RAM),
a read-only memory (ROM), a rigid magnetic disk and an optical
disk. Current examples of optical disks include compact disk--read
only memory (CD-ROM), compact disk--read/write (CD-R/W) and
DVD.
[0024] A data processing system suitable for storing and/or
executing program code may include at least one processor coupled
directly or indirectly to memory elements through a system bus. The
memory elements can include local memory employed during actual
execution of the program code, bulk storage, and cache memories
which provide temporary storage of at least some program code to
reduce the number of times code is retrieved from bulk storage
during execution. Input/output or I/O devices (including but not
limited to keyboards, displays, pointing devices, etc.) may be
coupled to the system either directly or through intervening I/O
controllers.
[0025] Network adapters may also be coupled to the system to enable
the data processing system to become coupled to other data
processing systems or remote printers or storage devices through
intervening private or public networks. Modems, cable modem and
Ethernet cards are just a few of the currently available types of
network adapters.
[0026] Referring now to the drawings in which like numerals
represent the same or similar elements and initially to FIG. 1, a
floor plan for a retail store 100 is depicted to illustrate aspects
in accordance with present principles. Major cost burdens in the
retail industry come from theft, returns fraud and false
injury/workman's compensation claims. Each of these problems can be
addressed by a sophisticated video surveillance system 102.
[0027] In one embodiment, a retail store 100 is fitted with wired
or wireless cameras 105, 106 and 107 at strategic locations, in
particular at entrances/exits 104, returns counters 103, observing
high value merchandise in merchandise display areas 102 and at cash
registers 101. The video from the cameras 105, 106 and 107 is
automatically processed by system 102 to extract salient features
that can be used for indexing customer actions.
[0028] One system that can be adapted in accordance with present
principles may include an IBM.RTM. Smart Surveillance System.TM.,
which extracts information such as the number of people visible,
their actions, locations and paths or trajectories over time.
System 102 employs cameras 105, 106 and 107 to extract appearance
information (including e.g., view-normalized area and height),
clothing color (e.g. a positional color histogram), facial
appearance or any other feature or aspect needed for a given
application. For example, other features that may be extracted from
images may include, e.g., facial profile, ear or gait biometrics,
behavior patterns, etc.
[0029] Additional sensors 110, 112 may be placed at the same
locations covered by the cameras 105, 106 and 107 or other
locations to augment the data that can be gathered about customers.
Additional sensors 110, 112 may detect the presence of people using
pressure pads, beam breakers, infra-red, PIR motion detection, or
the position of shopping carts with RFID, magnetic sensors, active
bat technologies, or any other technology which can provide useful
information, etc.
[0030] The data from the sensors 110, 112 is communicated to a
central, or distributed, index server in system 102, using
mechanisms similar to those described in A. Hampapur, L. Brown, J.
Connell, S. Pankanti, A. W. Senior, and Y. -L. Tian, Smart
Surveillance: Applications, Technologies and Implications, IEEE
Pacific-Rim Conference on Multimedia, Singapore, December 2003,
which is incorporated herein by reference in its entirety.
[0031] This data is stored in a database and may be used for a
variety of other purposes, including purposes other than those in
accordance with this disclosure, such as store usage and traffic
statistics.
[0032] Where the cameras' fields of view are contiguous, it may be
desirable to track activity/people from one camera to another,
using knowledge of the relative orientations of the cameras (for
example, using ground-plane homographies, etc.) and/or knowledge of
the entry and exit points from the fields of view and connections
between them. In this way, it may be possible to maintain a
continuous track of a person, to know that a person observed in a
camera must necessarily be the same person observed in another
camera, and knowing that the person has remained in view
throughout. Even unmonitored areas 108 may be tracked based on
correlated features entering and/or exiting such areas.
[0033] In accordance with present principles, employees of the
retail store 100 preferably have access to displays 120 and
interfaces 122 so as to permit access to the stored data and
surveillance information that has been collected and indexed. In a
particularly useful embodiment, a camera 106 present at the return
counter 103 or a camera 105 at the checkout 101 may be employed to
gather data of a next person on line. For example, clothing, face
or other criteria can be gathered and compared to the database in
system 102 such that information on this person is retrieved in
advance or retrieved at the time service is provided.
[0034] For example, an individual enters a store with a red shirt
on and dark sunglasses and not carrying anything. System 102
employs camera 106 at the entrance 104 to record this information.
The person proceeds to pick up a large item from a merchandise
display 102 and proceeds to the return desk 103, where the
individual's appearance is again recorded. Additionally the
individual would be recorded whenever in the field of view of any
other store camera which might give continual recording throughout
the individuals time in the store. When this individual gets to the
return counter 103, a comparison of his features are employed to
locate some or all of the other recordings of him available at the
time he presents the merchandise for return. In this way, the clerk
can easily determine that this individual entered the store empty
handed and that an attempt to defraud the retail store is being
made.
[0035] Referring to FIG. 2, an illustrative scenario is presented
to demonstrate aspects in accordance with present principles. A
usage scenario is depicted for an illustrative embodiment in a
scenario for returns fraud prevention. In block 200, a system (102)
is provided with cameras placed throughout a location (e.g., a
retail store). When a customer enters the store in block 210, the
cameras record the event in block 220, and extract indices such as
a location and appearance of the customer. The customer is tracked
while in the field of view of the camera, and the index data is
updated continuously in block 230.
[0036] In block 240, a decision is made as to the whether a
customer enters another camera's field of view. If the customer
enters another camera's field of view, the new camera captures the
customer's appearance and indices. Then, the customer is again
tracked and index information is updated in block 245. A customer
may be tracked by zero, one or more cameras simultaneously.
[0037] In block 250, it is determined whether the customer
approaches a service location (e.g., the returns desk). If the
customer does not approach the returns desk then this portion of
the program can terminate. If the customer does approach the
returns desk, the customer's presence and appearance are captured
at the returns desk and entered into the index in block 260. To
help determine if a fraud is taking place, the store clerk can then
use the system database to search, in block 270, for other indices
that relate to the same person.
[0038] An index association engine 330 (FUG. 3), returns a set of
matches in block 275 to the store clerk. Present matching customer
observations are provided to the returns clerk for review. The
matching process may use any available indices to find the best
matches, using information such as the time, together with known
temporal distributions (e.g. distribution of times taken to walk
from a door to the returns counter), appearance (an image of the
customer, face appearance or clothing color histogram), tracked
path (to estimate route taken) plus any other sensor data, to limit
the observed events at entrances, or other camera views, that might
correspond to the current person at the returns counter.
[0039] In block 280, the store clerk can then go through the
matches using a variety of techniques, such as through a graphical
browser interface. In one embodiment, the matching results are
summarized through the presentation of a keyframe accompanied by a
small amount of textual information.
[0040] The keyframe may be automatically chosen at recording time
to "best represent" the previous event--including but not limited
to the clearest face image (according to criteria such as highest
quality, most frontal, most neutral expression, best lighting) or
the shot where the subject is portrayed at highest resolution.
[0041] A number of other criteria for choosing keyframes can be
devised, including a combination of criteria. The keyframe may also
be altered to highlight or augment the information included within
it, for example to highlight or outline the subject of interest,
draw the path of the person over time, inset an enlargement of the
face, stamp a timecode or a digital watermark. The accompanying
textual information may be null (to save screen real-estate) or
include such useful information as the timestamp, camera location
or orientation, identifying features of the individual, etc., and
any of these features could be written into the frame or
dynamically displayed, e.g., by `mouse-over` when the cursor passes
over the image.
[0042] The results could be reviewed in other manners, for example,
by presenting the keyframes sequentially instead of spatially, or
presenting an automatically extracted summary video of events of
interest, or presenting a verbal description textually or aurally.
While skimming the "browsing" representation, further information
may be made available, by clicking or "mouse-over" of certain
parts, to help the clerk determine a match. In one embodiment,
clicking a keyframe opens a small popup window that shows the
original video of the event and additional information.
[0043] Whatever the method used to examine the results, the clerk
seeks a match in the returned results in block 280. If a match is
found then, this can be verified by reviewing the additional
information available. The additional information can also be used
to support a determination, in block 290, of whether the
transaction requested might be fraudulent or not, and appropriate
action taken. Most of the action depends on store procedures.
[0044] In block 295, the system may support the archiving of the
relevant information (to prevent a timed expiry that may
automatically clean old, unimportant data from the database). This
information may be stored in the existing database, specially
packaged, or written out to another device (e.g. USB memory key,
DVD-R, another computer, printout, video tape, etc.).
[0045] In the event that the clerk does not find any relevant
events in block 280, a search may be extended in block 285 by
reweighting the search/query criteria, changing the criteria (e.g.
extending the time period searched), reordering the results, or
narrowing the criteria with additional information. Additional
information may become available because of movements of the
customer (e.g., a better face shot) or by asking questions of the
customer ("Which door did you come through?", "When did you get to
the store?").
[0046] In many cases, the information included in the system may be
of a sensitive, privacy-intrusive nature, and naturally the system
may be enhanced by the deployment of privacy protecting
technologies, such as encryption, access control privileges, data
unbundling (separating different components of the information
acquired into different streams that are stored, encrypted and
accessed separately), and information masking, as described in the
U.S. Patent Application No. 2003/0231769 A1, entitled, "Application
Independent System, Method, and Architecture for Privacy
Protection, Enhancement, Control, and Accountability in Imaging
Service Systems", incorporated herein by reference in its
entirety.
[0047] For use as evidence, the data captured may also need to be
watermarked, timestamped, encrypted or cryptographically signed. In
one embodiment, the system would incorporate these known
technologies.
[0048] Referring to FIG. 3, a video surveillance system 300 is
illustratively shown in greater detail in accordance with one
illustrative embodiment. System 300 can receive any input data from
e.g., cameras, streaming content sensors etc. In the embodiment
shown, cameras 301 and sensors 302 are placed at strategic
locations, e.g., at entrances, exits, returns counters, observing
high value merchandise, at cash registers, etc. The video from the
cameras 301 is automatically processed by one or more smart
surveillance engines 305 to extract salient features or indices 310
that can be used for indexing customer activity.
[0049] Surveillance engines 305 extract information such as the
number of people visible, their locations and path over time, etc.
In addition, appearance information is recorded including
view-normalized area, height, clothing color, e.g. a positional
color histogram, facial appearance, facial profile, ear or gait,
biometrics, speed, time of observation, color, anthropometric
measurements, biometric identifiers, etc.
[0050] Additional sensors 302 may be placed at the same locations
covered by the cameras 301 or other locations to augment the data
that can be gathered about people. Additional sensors 302 may
include pressure pads, beam breakers, infra-red, PIR motion
detection, RFID, magnetic sensors, active bat technologies,
etc.
[0051] Index data 310 from the cameras 301 and sensors 302 is
communicated to a central, or distributed, index server 320. The
data is stored by the index server 320, and may be used for a
variety of other purposes or applications 335, e.g., store usage,
traffic statistics, forensic analysis, etc.
[0052] A continuous track of a person is preferably maintained to
know that a person observed in a camera must necessarily be the
same person observed in another camera, and knowing that the person
has remained in view throughout. A user can make searches on the
data at an employee terminal 340. Searches are entered at an input
device 370, e.g., a mouse and/or keyboard, and are sent in the form
of queries 345 to the index server 320. Index server 320 uses an
index association engine 330 to carry out the search, and return
results 350, which are displayed on an output device or display 360
at the user's terminal 340. Index association engine 330 may
associates separate observations (e.g., in real-time while the
individual remains in the retail establishment). These observations
are correlated with other observation and events by using the
indices in the context.
[0053] It should be appreciated that numerous modifications and
other embodiments may be devised by those skilled in the art.
Additionally, feature(s) and/or element(s) from any embodiment may
be used singly or in combination with other embodiment(s).
[0054] Having described preferred embodiments for multisensor event
correlation for retail fraud prevention (which are intended to be
illustrative and not limiting), it is noted that modifications and
variations can be made by persons skilled in the art in light of
the above teachings. It is therefore to be understood that changes
may be made in the particular embodiments disclosed which are
within the scope and spirit of the invention as outlined by the
appended claims. Having thus described aspects of the invention,
with the details and particularity required by the patent laws,
what is claimed and desired protected by Letters Patent is set
forth in the appended claims.
* * * * *