U.S. patent number 7,623,115 [Application Number 10/759,782] was granted by the patent office on 2009-11-24 for method and apparatus for light input device.
This patent grant is currently assigned to Sony Computer Entertainment Inc.. Invention is credited to Richard L. Marks.
United States Patent |
7,623,115 |
Marks |
November 24, 2009 |
Method and apparatus for light input device
Abstract
An input device for interfacing with a computing device is
provided. The input device includes a body configured to be held
within a human hand. The input device includes a light emitting
diode (LED) affixed to the body and a power supply for the LED. A
mode change activator is integrated into the body, where the mode
change activator is configured to cause a change of a color of a
light originating from the LED. The color change is capable of
being detected to cause a mode change at the computing device.
Methods for detecting input commands from an input source within a
field of sight of an image capture device, and a computing system
which includes the input device are provided.
Inventors: |
Marks; Richard L. (Foster City,
CA) |
Assignee: |
Sony Computer Entertainment
Inc. (Tokyo, JP)
|
Family
ID: |
34826444 |
Appl.
No.: |
10/759,782 |
Filed: |
January 16, 2004 |
Prior Publication Data
|
|
|
|
Document
Identifier |
Publication Date |
|
US 20040207597 A1 |
Oct 21, 2004 |
|
Related U.S. Patent Documents
|
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
Issue Date |
|
|
10207677 |
Jul 27, 2002 |
7102615 |
|
|
|
Current U.S.
Class: |
345/156 |
Current CPC
Class: |
G06F
3/017 (20130101); G06F 3/03542 (20130101); G06F
3/0304 (20130101); A63F 2300/1093 (20130101); G06F
2203/0331 (20130101) |
Current International
Class: |
G09G
5/00 (20060101) |
Field of
Search: |
;345/156-158,7,163,419,169,168,175 ;341/31 ;463/30 ;340/425.5
;348/143,211.6,211.9,171 ;351/211,205,221
;382/104,87,131,145,149,154 ;701/45 ;349/39 |
References Cited
[Referenced By]
U.S. Patent Documents
Foreign Patent Documents
|
|
|
|
|
|
|
0353200 |
|
Jan 1990 |
|
EP |
|
0652686 |
|
May 1995 |
|
EP |
|
0750202 |
|
Dec 1996 |
|
EP |
|
0 823 683 |
|
Feb 1998 |
|
EP |
|
1435258 |
|
Jul 2004 |
|
EP |
|
2814965 |
|
Apr 2002 |
|
FR |
|
2206716 |
|
Nov 1989 |
|
GB |
|
2376397 |
|
Nov 2002 |
|
GB |
|
2388418 |
|
Nov 2003 |
|
GB |
|
1284897 |
|
Nov 1989 |
|
JP |
|
6102980 |
|
Apr 1994 |
|
JP |
|
9128141 |
|
May 1997 |
|
JP |
|
9185456 |
|
Jul 1997 |
|
JP |
|
1138949 |
|
Feb 1999 |
|
JP |
|
2000-172431 |
|
Jun 2000 |
|
JP |
|
2001-166676 |
|
Jun 2001 |
|
JP |
|
2004-145448 |
|
May 2004 |
|
JP |
|
WO 98/48571 |
|
Oct 1998 |
|
WO |
|
WO 99/26198 |
|
Oct 1999 |
|
WO |
|
WO 02/27456 |
|
Feb 2002 |
|
WO |
|
WO 3/079179 |
|
Sep 2003 |
|
WO |
|
WO 2005/073838 |
|
Aug 2005 |
|
WO |
|
Other References
"The Tracking Cube: A Three-Dimensional Input Device," IBM
Technical Disclosure Bulletin, Aug. 1, 1989, pp. 91-95, vol. 32,
No. 3B, IBM Corp. New York, US. cited by other .
K. B. Shiivioga, et al., "Touch and Force Reflection for
Telepresence Surgery", Engineering in Medicine and Biology
Opportunities of the IEEEE, Baltimore, MD, USA, Nov. 3, 1994, New
York, New York, USA, pp. 1049-1050. cited by other .
Iddan, et al., "3D Imaging in the Studio (And Elsewhere . . . )",
Proceedings of the SPIE, SPIE, Bellingham, VA, US, vol. 4298, Jan.
24, 2001, pp. 48-55, XP008005351. cited by other .
Jojic, et al., "Tracking Self-Occluding Articulated Objects in
Dense Disparity Maps", Computer Vision, 1999, the Proceedings fo
the Seventh IEEE International Conference on Kerkyra, Greece Sep.
20-27, 1999, Los Alamitos, CA, US, IEEE Computer Society, US, Sep.
20, 1999, pp. 123-130. cited by other .
Klinker, et al., "Distributed User Tracking Concepts for Augmented
Reality Applications", pp. 37-44, Augmented Reality, 2000, IEEE and
ACM Int'l Symposium, Oct. 2000, XP010520308, ISBN: 0-7695-0846-4,
Germany. cited by other .
Nakagawa, et al., "A Collision Detection and Motion Image Synthesis
Between a Background Image and a Foreground 3-Dimensional Object",
TVRSJ Bol. 4, No. 2, pp. 425-430, 1999, Japan. cited by other .
Mihara, et al., "A Realtime Vision-Based Interface Using Motion
Processor and Applications to Robotics", vol. J84-D-11, No. 9, pp.
2070-2078, Sep. 2001, Japan. cited by other .
Nakamura, et al., "A Consideration on Reconstructing 3-D Model
Using Object Views", 2004-01601-003, pp. 17-21, Kokkaido
University, Japan, nakamura @gmedia.eng.hokudai.ac.jp. cited by
other .
Nishida, et al., "A Method of Estimating Human Shapes by Fitting
the Standard Human Model to Partial Measured Data", D-II vol.
J84-D-II, No. 7, pp. 1310-1318, Jul. 2001. cited by other .
Wilson & Darrell, "Audio-Video Array Source Localization for
Intelligent Environments", 2002 IEEE Dept. of Electrical Eng and
Computer Science, MIT, Cambridge, MA 02139. cited by other .
Fiala, et al., "A Panoramic Video and Acoustic Beamforming Sensor
for Videoconferencing", 2004 IEEE, Computational Video Group,
National Research Council, Ottawa, Canada KlA 0R6. cited by other
.
Hemmi, et al., "3-D Natural Interactive Interface-Using Marker
Tracking from a Single View",Sep. 9, 1991, Systems and Computers in
Japan. cited by other .
Kanade, et al., "A Stereo Machine for Video-rate Dense Depth
Mapping and Its New Application" 1996, CVPR 96, IEEE Computer
Society Conference, pp. 196-202 (022). cited by other .
Gvili, et al., "Depth Keying", SPIE vol. 5006 (2003), 2003
SPIE-IS&T, pp. 564-574 (031). cited by other.
|
Primary Examiner: Dharia; Prabodh M
Attorney, Agent or Firm: Martine Penilla & Gencarella,
LLP
Parent Case Text
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a continuation-in-part claiming priority under
35 U.S.C. .sctn. 120 of U.S. patent application Ser. No. 10/207,677
filed on Jul. 27, 2002, now U.S. Pat. No. 7,102,615 and entitled
"MAN-MACHINE INTERFACE USING DEFORMABLE DEVICE." This application
is herein incorporated by reference in its entirety for all
purposes.
Claims
What is claimed is:
1. A method for triggering input commands of a program run on a
computing system, comprising: defocusing an image capture device
that is placed in a direction of a display screen; monitoring a
field of view in front of the image capture device; identifying a
light emitting diode (LED) light source within the field of view,
the (LED) light source being pointed in the direction of the image
capture device, the identifying being done from a defocused image
produced by the image capture device having a diffuser, such that
defocusing the image capture device produces an expanded region of
pixels around captured light of the (LED) light source; tracking
the (LED) light source as it is moved, the expanded region of
pixels around the (LED) light source increasing pixel data usable
to identify the (LED) light source during the tracking, and the
display screen illustrating objects that can be interacted with;
detecting a change in light emitted from the (LED) light source;
and in response to detecting the change, triggering an input
command at the program run on the computing system, the input
command acting to interface with one or more objects illustrated on
the display screen.
2. The method of claim 1, wherein the change in light is one of a
color change, and a light variation change.
3. The method of claim 1, wherein the (LED) light source is emits
multiple colors of light.
4. The method of claim 1, wherein the method operation of
identifying a light source within the field of view includes,
masking background light effects within the field of view.
5. The method of claim 4, wherein the method operation of masking
background light effects within the field of view includes,
reducing an amount of light allowed into an aperture of the image
capture device.
6. The method of claim 1, wherein the input command causes a mode
change linked to a cursor displayed on the display screen
associated with the computing system.
7. The method of claim 1, wherein the method operation of
identifying the (LED) light source within the field of view
includes, defining an area representing the (LED) light source
within a grid associated with the image capture device and the
defocusing is configured to expand the area representing the (LED)
light source within the grid.
8. The method of claim 7, wherein the method operation of
identifying a light source within the field of view includes,
calculating a centroid of an image representing the (LED) light
source through a grid associated with the image capture device,
such that the centroid is of the expanded area representing the
(LED) light source within the grid.
9. The method of claim 8, further comprising: translating
coordinates of the centroid to a location on a display screen
associated with the computing system; detecting movement of the
(LED) light source within the field of view; and correlating the
movement of the (LED) light source to movement of a cursor on the
display screen.
10. A method for detecting input commands from an input source
within a field of sight of an image capture device positioned near
a display screen, comprising: defocusing the image capture device
using a diffuser; minimizing an amount of light entering the image
capture device; detecting a first LED color light from the input
source through the image capture device, the first LED color light
being directed toward the display screen and the image capture
device, such that directing the first LED color light toward the
display screen enables interactive movement of a cursor on the
display screen; detecting a change from the first LED color light
to a second LED color light; identifying a position of the light
entering the image capture device from the defocused image capture
device, the defocusing causing the light entering the image capture
device to expand and produce an expanded region of pixels around
the light to facilitate identification and tracking of the light;
and presenting a mode change in response to changing to the second
LED color light, the mode change operating to cause an action on
one or more objects being illustrated on the display screen.
11. The method of claim 10 wherein the method operation of
minimizing an amount of light entering an image capture device
includes, reducing an aperture size of the image capture device to
enhance a signal representing light from the input source relative
to other captured image data.
12. The method of claim 11 wherein the method operation of reducing
an aperture size of the image capture device results in filtering
background light capable of interfering with light received from
the input device.
13. The method of claim 10 wherein the method operation of
detecting a first LED color light signal from the light source
through the image capture device includes, determining a location
of a center of the first LED color light signal on a coordinate
system associated with the image capture device; and mapping the
location to a corresponding location on a display screen.
14. The method of claim 10 wherein the method operation of
detecting a change from the first color light to a second color
light includes, detecting the second LED color light from the input
source; and comparing pixel values associated with the first LED
color light to pixel values associated with the second LED color
light.
15. The method of claim 10 further comprising: reverting to the
first color light from the second color light; and in response to
reverting to the first color light, terminating the mode
change.
16. The method of claim 10 wherein the mode change is associated
with one of a click and drag operation and a highlighting
operation.
17. The method of claim 10 wherein the first color light and the
second color light originate from one of a single light emitting
diode or multiple light emitting diodes.
18. A computing system, comprising: (a) a hand held input object
having a body, the body having an end with a light emitting diode
(LED), the body including a button; (b) an image capture device
being positioned in a direction of a display screen, the image
capture device having a diffuser to produce a defocused image; (c)
a computing device configured for interfacing with the hand held
input object and the image capture device, the computing device
including, logic for monitoring a field of view associated with the
image capture device; logic for tracking a position of a light
source of the LED, the defocused image producing an expanded region
of pixels around captured light from the LED light source to
increase pixel data usable to identify the LED light source during
tracking, the LED light source being pointed in a direction that is
toward the display screen, the monitoring and tracking being of the
LED light source from the defocused image; logic for detecting a
change in the light source; and logic for triggering a mode change
command at a main program run through the computing device, in
response to the detected change in the LED light source as caused
by a press in the button of the hand held input object, the mode
change triggering an action for interfacing with one or more
objects illustrated on the display screen.
19. The computing system of claim 18, wherein the computing device
is one of a game console, a general computer, networked computer,
and a distributed processing computer.
20. The computing system of claim 18, wherein the logic for
detecting a color change in the LED light source includes, logic
for detecting a change in a pixel value associated with the LED
light source; and logic for detecting a change in a position of the
LED light source relative to the image capture device.
21. The computing system of claim 19, wherein each logic element is
one or a combination of hardware and software.
22. The computing system of claim 20, wherein the logic for
detecting a change in a position of the LED light source relative
to the image capture device includes, logic for calculating a
centroid of an image representing the LED light source through a
grid associated with the image capture device. logic for
translating coordinates of the centroid to a location on a display
screen associated with the computing system; logic for detecting
movement of the LED light source within the field of view; and
logic for correlating the movement of the LED light source to
movement of a cursor on the display screen.
23. The computing system of claim 22, wherein the logic for
correlating the movement of the LED light source to movement of a
cursor on the display screen includes, logic for adjusting a scale
associated with translation of the movement of the LED light source
to the movement of the cursor according to a distance of a user
relative to the image capture device.
24. The computing system of claim 18, further comprising: logic for
minimizing an amount of light entering the image capture device in
order to mask background light not associated with the LED light
source.
25. An system including input device for interfacing with a
computing device, the system comprising: (a) the input device
including, (i) a body; (ii) a light emitting diode (LED) affixed to
the body of the input device; (iii) a power supply for the LED;
(iv) a mode change activator integrated into the body of the input
device, the mode change activator configured to cause a variation
of a light originating from the LED, wherein the variation is
capable of being detected to cause a mode change at the computing
device; (b) a display screen configured to illustrate objects; (c)
an image capture device having a diffuser, the diffuser configured
to render defocused images by the image capture device so as to
produce an expanded region of pixels around a light of the LED, the
expanded region of pixels increasing pixel data to identify the
light of the LED, the image capture device being placed at a
location of the display screen, the image capture device being
configured to capture the LED of the input device when directed
toward the display screen to enable interaction with illustrated
objects as a result of the mode change, the mode change being a
result of the LED of the input device changing from one color to
another color, and the mode change being discontinued when the
changing in light reverts back to an original color or another
color, as triggered by the mode change activator of the input
device.
26. The system of claim 25, wherein the mode change activator is
configured to cycle between two variations of the light.
27. The system of claim 25, wherein the body includes a first end
and a second end, a first LED affixed to the first end, a second
LED affixed to the second end.
28. The system of claim 27, wherein the first LED emits a first
variant of light and the second LED emits a second variant of
light.
29. The system of claim 25, further comprising: an infrared
emitter.
30. The system of claim 25, wherein the body includes a first LED
adjacently located to a second LED, the first LED emitting a first
variant of light and the second LED emitting a second variant of
light.
31. The system of claim 25, wherein the body is configured to be
held within a human hand.
32. The system claim 25, wherein the body is a ring configured to
fit over a human finger.
33. The system of claim 25, wherein the body is thimble shaped and
the LED is affixed to a base of the thimble shaped body.
34. The system of claim 25, wherein the mode change activator is
configured to cycle between at least three light variant changes.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
This invention relates generally to alternative input mechanisms to
computer systems, and more particularly to visually tracking a
light, wherein a change in the light triggers an action on the part
of the computer system.
2. Description of the Related Art
There has been a great deal of interest in searching for
alternatives to input devices for computing systems. The keyboard
and mouse for the desktop computing system are taken for granted at
this time. However, for interactive entertainment applications in a
"living room" environment, the keyboard and the mouse have failed
to gain widespread acceptance.
Tracking of moving objects using digital video cameras and
processing the video images for producing various displays has been
attempted also. However, these systems tend to rely on having a
plurality of video cameras available for developing positional
information about the object based on triangulation. The cost for
these systems becomes prohibitive for their introduction into the
"living room" environment.
However, in spite of the above knowledge and techniques, problems
continue to hinder successful object tracking, and a particularly
difficult problem is extracting precisely only those pixels of a
video image which correspond unambiguously to an object of
interest. For example, although movement of an object having one
color against a solid background of another color, where the object
and background colors vary distinctly from one another, can be
accomplished with relative ease, tracking of objects, even if
brightly colored, is not so easy in the case of multi-colored or
non-static backgrounds. Changes in lighting also dramatically
affect the apparent color of the object as seen by the video
camera, and thus object tracking methods which rely on detecting a
particular colored object are highly susceptible to error or
require constant re-calibration as lighting conditions change. The
typical home use environment for video game programs demands much
greater flexibility and robustness than possible with conventional
object tracking computer vision systems.
Thus, an alternative input device must be able to be tracked under
the home use environment by a single relatively inexpensive camera
in order to become widely accepted. Additionally, the alternative
input device must be convenient to use. While a glove worn on the
hand of a user, where the glove includes sensors that are tracked
by a camera to capture input, has been trialed, users have not
embraced the glove. One of the reasons for the lack of enthusiasm
for a glove is the inconvenience of having to continually remove
and put on the glove. Furthermore, the alternative input devices
described above do not easily adapt themselves to being able to
indicate a mode change, functionally similar to a mouse click.
Thus, there is a need to solve the problems of the prior art to
provide an input device capable of being tracked by a single video
camera, wherein the input device is convenient for the user and
capable of indicating a mode change.
SUMMARY OF THE INVENTION
Broadly speaking, the present invention fills these needs by
providing a method and system that provides a device capable of
producing one or more colors of light (or variations of a single or
multiple light shades) where a change or variation triggers a mode
change to the system controlling a display of image data. As used
herein, a mode change refers to a discrete event or action
triggered through the detection of a change in light emitted from
an input device. It should be appreciated that the present
invention can be implemented in numerous ways, including as a
process, a system, or a device. Several inventive embodiments of
the present invention are described below.
In one embodiment a method for triggering input commands of a
program run on a computing system is provided. The method initiates
with monitoring a field of view in front of an image capture
device. Then, a light source within the field of view is
identified. Next, a change in the light emitted from the light
source is detected. In response to detecting the change, an input
command is triggered at the program run on the computing
system.
In another embodiment, a method for detecting input commands from
an input source within a field of sight of an image capture device
is provided. The method initiates with minimizing an amount of
light entering the image capture device. Then, a first color light
signal is detected from the input source through the image capture
device. Next, a change from the first color light signal to a
second color light signal is detected. Then, a mode change is
triggered in response to the change in the first color light
signal.
In yet another embodiment, a computer readable medium having
program instructions for triggering input commands of a program run
on a computing system is provided. The computer readable medium
includes program instructions for monitoring a field of view in
front of an image capture device. Program instructions for
identifying a light source within the field of view and program
instructions for detecting a change in light emitted from the light
source are included. Program instructions for triggering an input
command at the program run on the computing system in response to
detecting the change are included.
In still yet another embodiment, a computer readable medium having
program instructions for detecting input commands from an input
source within a field of sight of an image capture device is
provided. The computer readable medium includes program
instructions for minimizing an amount of light entering the image
capture device and program instructions for detecting a first color
light signal from the input source through the image capture
device. Program instructions for detecting a change from the first
color light signal to a second color light signal and program
instructions for triggering a mode change in response to the change
in the first color light signal are provided.
In another embodiment, a computing system having an input detection
system, that determines when to trigger input commands of a main
program run through the computing system is provided. The computing
system includes an image capture device. Logic for monitoring a
field of view associated with the image capture device and logic
for tracking a position of a light source associated with an input
object are included. Logic for detecting a color change in the
light source and logic for triggering an input command at the main
program run through the computing system, where the triggering is a
result of the detected color change in the light source are
included.
In yet another embodiment, an input device for interfacing with a
computing device is provided. The input device includes a body
configured to be held within a human hand. The input device
includes a light emitting diode (LED) affixed to the body and a
power supply for the LED. A mode change activator is integrated
into the body, where the mode change activator is configured to
cause a change of a color of a light originating from the LED. The
color change is capable of being detected to cause a mode change at
the computing device.
Other aspects and advantages of the invention will become apparent
from the following detailed description, taken in conjunction with
the accompanying drawings, illustrating by way of example the
principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention, together with further advantages thereof, may best
be understood by reference to the following description taken in
conjunction with the accompanying drawings.
FIG. 1A is a simplified schematic diagram of a system having the
capability of input detection from a light source in order to
trigger a mode change in accordance with one embodiment of the
invention.
FIG. 1B is an alternative representation of the system depicted in
FIG. 1A.
FIG. 2 is a simplified schematic diagram illustrating the capture
of light from a light source through an image capture device in
accordance with one embodiment of the invention.
FIG. 3 is a schematic diagram illustrating the determination of the
location of a light source and the subsequent translation of that
location to control movement of a corresponding cursor on a display
screen in accordance with one embodiment of the invention.
FIG. 4 is a schematic diagram illustrating a scheme for enhancing a
tracking and translation methodology in accordance with one
embodiment of the invention.
FIG. 5 is a simplified schematic diagram illustrating a scheme for
setting a scale dependent upon the distance a user is from the
image capture device in accordance with one embodiment of the
invention.
FIG. 6 represents alternative configurations of an input device in
accordance with one embodiment of the invention.
FIG. 7 illustrates an alternative embodiment for the input devices
illustrated in FIG. 6.
FIG. 8A is a simplified schematic diagram of a pair of input
devices configured to communicate with a computing device in
communication with a display monitor in accordance with one
embodiment of the invention.
FIG. 8B is a schematic diagram of an alternative light input device
configuration to the rings of FIG. 7.
FIG. 9 is a flow chart diagram illustrating the method operations
for triggering input commands for a program running on a computing
system in accordance with one embodiment of the invention.
FIG. 10 is a flow chart diagram illustrating the method operations
for detecting input commands from an input source within a field of
sight of an image capture device in accordance with one embodiment
of the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
An invention is disclosed for an input device that is capable of
emitting multiple colors/frequencies of light in order to trigger
an event. In the following description, numerous specific details
are set forth in order to provide a thorough understanding of the
present invention. It will be apparent, however, to one skilled in
the art that the present invention may be practiced without some or
all of these specific details. In other instances, well known
process steps have not been described in detail in order not to
unnecessarily obscure the present invention.
The embodiments of the present invention provide a user input
device that is capable of emitting multiple colors of light which
are captured through an image capture device. The changing from one
light color to another initiates an event or action that can be
displayed on a monitor. For example, the change from one light
color to another can signify a mouse click, i.e., a mode change, to
open a file or window for a personal computer, grab, drag or
manipulate an image for a computer game, or even start a smart
appliance. It should be appreciated that the input device includes
some type of button, or actuator, which is used to change between
light colors being emitted from a light emitting diode (LED). Thus,
the input device will include some sort of power supply for the LED
also. As will be explained by the embodiments described herein the
user input device allows for the introduction of an experience that
can be related to a mouse click while being convenient for the
user. It should be appreciated that the light device is a "dumb"
device. That is, no active communication takes place between the
input device and the computer as the computer, in essence, watches
for the light input device. In one embodiment, the light input
device is configured as a flashlight. Of course, the light input
device may be cordless, e.g., battery powered for the generation of
light.
FIG. 1A is a simplified schematic diagram of a system having the
capability of input detection from a light source in order to
trigger a mode change in accordance with one embodiment of the
invention. Image capture device 104 is in communication with
computing device 106, which in turn is in communication with
display screen 108. Input device 100 includes a light source 102.
For example, light source 102 may be a suitable light emitting
diode (LED). Light emanating from LED 102 is detected by image
capture device 104. It should be appreciated that image capture
device 104 may be any suitable image capture device capable of
detecting light from LED 102. For example, any suitable web cam or
other camera may be used as image capture device 104. The location
in space of LED 102 is used to control movement of cursor 110 on
display screen 108 in one embodiment. That is, as input device 100,
and associated LED 102, is moved in space, that movement is
translated in order to move cursor 110. Additionally, as will be
explained in more detail below, input device 100, through LED 102,
may cause a mode change, similar to the mode change initiated by
clicking a mouse for a desktop computer. As mentioned above, a mode
change refers to a discrete event or action triggered through the
detection of a change in light emitted from an input device. In
essence, input device 100 may be used as a computer mouse through
color changes associated with LED 102. Exemplary mode changes
include mode changes associated with mouse-type operations where
the movement features of the mouse in conjunction with the
selection features, i.e., clicking, double clicking, and right
clicking, are incorporated into the system described below. That
is, the tracking features of the system capture the movement of the
input device while the light color changes provide the means to
capture selection features. While a mode change has been described
in relation to mouse-type operations, it should be appreciated that
the embodiments are not limited to mouse-type operations. That is,
any suitable discrete operation or action is included here.
FIG. 1B is an alternative representation of the system depicted in
FIG. 1A. Here, user 112 is holding input device 100 within his
hand. LED 102 which is affixed to an end of input device 100, is
within a detection region of image capture device 104. Image
capture device 104 is in communication with computing device 106
which is in communication with display screen 108. In one
embodiment, computing device 106 is any suitable video game
console, e.g., the PLAYSTATION 2 console. Controller 116 is also in
communication with computing device 106. Thus, user 112 may move
input device 100 from an initial location, thereby changing the
position of LED 102 relative to camera 104. This relative movement
is then translated in order to move a cursor on display screen 108.
Additionally, a mode change associated with the cursor being moved
on display screen 108 is triggered through a change in the color of
light emitted from LED 102. It should be appreciated that the
embodiments described herein provide for a mouse-like device to be
introduced in the living room for interactive entertainment and any
other suitable application.
FIG. 2 is a simplified schematic diagram illustrating the capture
of light from a light source through an image capture device in
accordance with one embodiment of the invention. Here, user 112 is
holding input device 100 which includes a light source thereon.
Image capture device 104 monitors a field of view 118 through which
light from the light source of input device 100 is detected. The
light source associated with input device 100 is within plane 117,
which corresponds to digitized screen 120. Here, an image of the
light source associated with input device 100 is illustrated by
region 122 of screen 120. It should be appreciated that the
resolution of screen 120 may be associated with any suitable
resolution typical of a web cam or other suitable camera. In one
embodiment, screen 120 is defined by a screen size of
320.times.240. Thus, as user 112 moves input device 100, the
associated movement is captured through capture device 104 to
determine a location of the light source within screen 120. It
should be appreciated that the screen size and the imaging device
size are decoupled. However, the screen and image device size are
mapped in order to determine corresponding positions between the
two. In one embodiment, the image device is mapped to a region of
the screen. Here, most of the screen is used for displaying a
scene, game image, etc., and there is a relatively small input
palette in a corner or some other suitable region of the
screen.
FIG. 3 is a schematic diagram illustrating the determination of the
location of a light source and the subsequent translation of that
location to control movement of a corresponding cursor on a display
screen in accordance with one embodiment of the invention. Here,
screen 120 defines an image of a light source as region 122. Region
122 includes portions of pixel P.sub.a, P.sub.b, P.sub.c, P.sub.d,
P.sub.e, and P.sub.f. The remainder of each of the pixels in screen
120, i.e., all pixels except pixels P.sub.a-P.sub.f, are black. In
one embodiment, ensuring that the remainder of the pixels is black
is achieved through a masking operation. The masking operation
includes reducing the size of an aperture of image capture device
104 in order to minimize an amount of light allowed into the image
capture device. In one embodiment, the aperture size may be
adjusted electronically by adjusting the sensor game and exposure
time. This scheme enhances the ability to detect a light source
while reducing interference effects associated with background
lighting. It should be appreciated that since the characteristics
of the light input device and the image capture device are known,
then the image capture device parameters (white balance, gain,
exposure, saturation, etc.) may be set explicitly to track a
particular pre-determined pixel value, i.e., no calibration is
required. As the input device is a light, the room lighting is not
a factor here. Thus, an active method for detecting a light change
is provided.
Still referring to FIG. 3, center 124 of region 122 is calculated
through a centroid calculation in which the center's of each of
pixels P.sub.a-P.sub.f are calculated and then weighted according
to the associated pixel value in order to determine the coordinates
of center 124. The coordinates of center 124 are then mapped to
display screen 128, which corresponds to the display screen being
viewed by the user. Thus, movement of the light source will cause
movement of region 122 on grid 120, which may also be referred to
as a screen associated with the image capture device. The
corresponding movement of region 122 will be associated with the
calculation of a new center. The new center will then be mapped to
a location on screen 128 in order to move cursor 130 on screen 128
so that the user is given a feeling of control, over the movement
of cursor 130 through the LED input device. As will be explained in
more detail below, the input device may have a button or some other
suitable activation device which when pressed will cause the
respective LED to change to a different color from a previous
color. This different color is then captured by image capture
device 104. The detection of the different color results in
different pixel values being associated with the change in color.
For example, the pixels corresponding to region 122 will be
associated with different values by the color change. The different
pixel values will then signal the mode change similar to a mode
change signal associated with a mouse click. Thus, a user may click
and drag, highlight, etc., images on the display screen. That is,
the user may perform any functionality achieved through a mouse
associated with a computer.
In one embodiment, the centroid calculation is performed as
described hereafter. The pixels not associated with pixels
P.sub.a-P.sub.f, are assigned a value of 0 as no light is detected,
i.e., the pixels are black. It should be appreciated that the
masking technique described above may be used to ensure that the
image capture device can lock in on a light emanating from an input
device by reducing interference from background lighting. Each of
pixels P.sub.a-P.sub.f are assigned a value corresponding to the
amount of area of the pixel intersecting with region 122. In one
embodiment, where pixel values are assigned from 0-255, 0
corresponding to no light, pixel P.sub.e is assigned the highest
value while pixel P.sub.f is assigned the lowest value. For
exemplary purposes the pixel values of pixels P.sub.a, P.sub.b,
P.sub.c, P.sub.d, P.sub.e, and P.sub.f are 121, 230, 80, 123, 240,
and 10, respectively. Each of pixels P.sub.a-P.sub.f is associated
with a respective pixel center point. Each of the two dimensional
coordinates of each of the pixel centers is multiplied by the value
of the respective pixel. These weighted values for one of the two
dimensional coordinates are then summed together. In one
embodiment, the summation of the weighted values for each
coordinate is then divided by the summation of the pixel values
associated with region 122 in order to provide the coordinates for
the center of region 124. This technique may be described
mathematically as:
.times..SIGMA..times..times..times..times..times..times..times..times..ti-
mes..times..times..times..times..times..SIGMA..times..times..times..times.-
.times..SIGMA..times..times..times..times..times..times..times..times..tim-
es..times..times..times..times..times..SIGMA..times..times.
##EQU00001##
Here, (x,y)center represent the two coordinates of center 124,
xpixel center represents the x coordinate for each of pixels
P.sub.a-P.sub.f, and ypixel center represents the y coordinate for
each of pixels P.sub.a-P.sub.f. Thus, center 124 corresponds to a
certain location of the image of the capture device. This position
corresponds to a location on screen 128. With reference to video
frames, (x,y)center may be calculated for each frame of the video
and the location of (x,y)center is used to set a position of cursor
130 on screen 128. In one embodiment, a resolution associated with
grid 120 is less than the resolution associated with screen 128,
thereby enabling smooth movement of cursor 130 across screen 128.
It will be apparent to one skilled in the art that a non-weighted
centroid may also be determined, especially if the background is
not known, e.g., the background is not all black. Here, the
location of the centroid may not be as accurate as when the
background is known, however, the accuracy is still suitable for
the embodiments described herein. In one embodiment, the
non-weighted centroid is calculated when the user is in a dark room
or with an infrared LED and camera. It will be apparent to one
skilled in the art that while FIGS. 1A, 1B, 2, and 3 refer to a
cursor, the embodiments are not limited to use with a cursor. In
essence any suitable indicator that provides feedback on the second
location of the input device may be used. For example, effects like
distortion, brightening, darkening, telescope windowing, etc. may
be employed to provide feedback on the second location of the input
device.
FIG. 4 is a schematic diagram illustrating a scheme for enhancing a
tracking and translation methodology in accordance with one
embodiment of the invention. It should be appreciated that where a
light source is captured through image capture device 104 and
subsequently located within screen 120, the corresponding region
associated with the light source is contained within one pixel.
Thus, the subsequent translation to a cursor may cause the cursor
movement to appear jumpy, due to the quantization effect of the
discrete sampling of the image capture device. In order to
alleviate the jumpiness, image capture device may be defocused to
blossom or expand the region associated with the light source. For
example, region 132 represents an initial capture of a
corresponding light source. As can be seen, region 132 is contained
within one block of grid 120, which represents a single pixel. In
order to expand or blossom region 132 the image capture device is
defocused where regions 134 and 136 represent different defocusing
parameters. Thereafter, the centroid of the expanded region may be
calculated as discussed above. In one embodiment, a diffuser is
placed over the LED to defocus the light source. For example, the
diffuser may be a piece of tape that causes the light to
diffuse.
FIG. 5 is a simplified schematic diagram illustrating a scheme for
setting a scale dependent upon the distance a user is from the
image capture device in accordance with one embodiment of the
invention. Here, user 102a is at a first distance associated with
image plane 117a while user 102b is at a second distance
corresponding to image plane 117b. It is not necessary that image
capture device 104 has depth capability, as the corresponding
scales from images captured at image plane 117a and image plane
117b may be used to provide a relative degree of distance
corresponding to respective image areas occupied by user 102a and
user 102b. According to this relative degree of distance, the
amount of movement for input device 100 to cause a corresponding
movement of a cursor on display screen 108 may be adjusted. For
example, if the user is closer to image capture device 104, then
larger movements may be used to correspond to a movement of a
cursor as compared to smaller movements when the user is at a
farther distance.
FIG. 6 represents alternative configurations of an input device in
accordance with one embodiment of the invention. Input device 100a
includes LED 102a-1 and LED 102a-2 located at opposing ends of the
input device. Button 103 is included on the body of input device so
that a user may press the button in order to trigger a light change
for a corresponding LED. More than one button may be incorporated
into input device 100a in order to accommodate multiple LED's in
one embodiment. Input device 100b includes a single LED 102b at an
end of the input device. Here, LED 102b is capable of emanating
multiple colors as button 103 is pressed. Input device 100c
illustrates multiple LED's located adjacent to each other. Here,
LED 102c-1 is adjacent to LED 102c-2. As an alternative to input
device 100c, the input device may have a fork-type configuration
where an LED is affixed to each of the ends of the prongs (tines)
of the fork. Button 103 may also be used to trigger one of the
LED's to emit light while another is off. Button 103 may also be
referred to as a mode change activator. A mode change activator is
broadly defined as any suitable mechanism that may be used to cause
the LED to switch between colors of light and/or variations of
colors of light. For example, the mode change activator may be a
button, a switch, a rotary dial, etc. In addition, the LED's may be
located elsewhere on the body of the input devices shown in FIG. 6.
For example, LED's may be incorporated on the sides of the input
device. Alternatively, a line of LED's along the side of the input
device may be provided. In another embodiment, a large LED at an
end of the input device is provided, thereby enabling a capture
device to detect a change in shape when the input device is tilted.
That is, the input device may be configured to enable the capture
device to detect a change in angle of the input device relative to
the capture device. For example, a user may angle the input device
up, down or to the side in order to cause a certain mode change or
response to the particular angle change. One skilled in the art
will appreciate that numerous other suitable configurations are
possible for the input device besides the configurations
illustrated in FIG. 6. Thus, the embodiments described herein are
not limited to the exemplary configurations of FIG. 6.
FIG. 7 illustrates an alternative embodiment for the input devices
illustrated in FIG. 6. Here, input device 100 is configured as a
remote control device which includes LED 102 and infrared
capability indicated by light 140. Thus, the input device may be
incorporated into a suitable remote control commonly used for
television sets. In one embodiment, an LED capable of toggling
between at least three colors is provided. Here, a third color may
be used to provide functionality corresponding to a "right click"
on a computer mouse.
FIG. 8A is a simplified schematic diagram of a pair of input
devices configured to communicate with a computing device in
communication with a display monitor in accordance with one
embodiment of the invention. External input devices 100-1 and 100-2
are configured to fit over a finger or thumb of a user's hands 142a
and 142b. As shown, each of input devices 100-1 and 100-2 is
capable of emanating light that is detected by image capture device
104 which is in communication with computing device 106. While
image capture device 104 is shown outside of the frame of computing
device 106, it should be appreciated that the image capture device
may be integrated into the computing device in one embodiment of
the invention. Input devices 100-1 and 100-2 transmit different
color light signals in one embodiment. Computing device 106 is in
communication with display monitor 108. Computing device 106
transmits digital data to display monitor 108 so that the digital
data can be viewed. Display monitor 108 may display text 146a, menu
146b and/or graphics 146c. Of course, it should be noted that any
suitable digital data may be displayed. In one embodiment, where
computing device 106 is a game console, display monitor 108
displays graphics or a user interface associated with a game being
played.
Each of input devices 100-1 and 100-2, of FIG. 8A, is configured to
generate different colors of light. In one embodiment, input
devices 100-1 and 100-2 may emanate a common color of light and at
least one different color of light. It should be appreciated that
button 144 may be used to toggle between the different colors of
light. The light may be emitted through a LED on the side of the
ring band of input devices 100-1 and 100-2. Alternatively, input
devices 100-1 and 100-2 may be configured as thimbles, where a LED
is affixed to the base of the thimble. In another embodiment, a
single input device is employed rather than multiple devices. It
should be appreciated that input devices 100-1 and 100-2 are
configured to fit comfortably over a user's finger or thumb. Thus,
different sizes can be made available based upon a user's age,
gender, etc. Alternatively, the input devices may be made to be
adjustable. That is, elastic straps, or even VELCRO straps, may be
incorporated to secure the input device to the user's finger or
thumb in one embodiment of the invention.
FIG. 8B is a schematic diagram of an alternative light input device
configuration to the rings of FIG. 7. Here, thimble body 100-3
includes button 144 which is configured to change the light or
frequency of light being emitted through LED 151. Of course, more
than on LED may be located on the base of thimble 100-3, as
discussed with reference to FIG. 6.
One skilled in the art will appreciate that the image capture
device and the computing device include logic capable of providing
the functionality described herein. The logic may include software
elements and/or hardware elements. For example, the software
elements may include software code processed through a central
processing unit. The hardware elements include logic gates and
circuitry configured to achieve the functionality described herein.
It will be apparent to one skilled in the art that the hardware
elements, e.g., logic gates, may be synthesized to execute the
functionality described herein. Of course, the system may include a
combination of software and hardware elements interacting to
provide the desired outcome also.
FIG. 9 is a flow chart diagram illustrating the method operations
for triggering input commands for a program running on a computing
system in accordance with one embodiment of the invention. The
method initiates with operation 150 where a field of view in front
of an image capture device is monitored. Here, the field of view
may capture a light source emanating from an input device having an
LED capable of emitting multiple colors of light, as described with
reference to FIGS. 1A through 2. The method then advances to
operation 152 where a light source within the field of view is
identified. As described above, the light source emanates from the
LED. The method then proceeds to operation 154 where a change in
light emitted from the light source is detected. Here, a button may
be pressed in order to change the color of light emitted from an
LED which is then captured by the image capture device. That is,
pixel values will change when the change in the color of light
occurs. As used herein, the change in the color of light may refer
to an actual color change, e.g. from red to green. However, the
change of color may also refer to a variation of the color, e.g.,
one yellow variant to another yellow variant. In essence, any
suitable change that will cause a corresponding change in the pixel
values associated with the colors or variants may be used here. The
method then moves to operation 156 where an input command at the
program run on the computing system is triggered in response to the
change in the color of light. For example, a mode change associated
with a mouse click on a desktop computer may be triggered here.
Thus, click and drag functionality, highlighting functionality, and
any other suitable functionality achieved through a mouse click is
capable of being introduced in the "living room" environment
without the traditional mouse and keyboard hardware. In addition,
the input command triggered by the light change can be an action, a
movement cue, a modality change, etc.
FIG. 10 is a flow chart diagram illustrating the method operations
for detecting input commands from an input source within a field of
sight of an image capture device in accordance with one embodiment
of the invention. The method initiates with operation 160 where an
amount of light entering the image capture device is minimized.
That is, an aperture of the image capture device may be reduced in
order to mask background lighting. Thus, the minimization of the
background lighting will enhance the tracking capability of a light
source such as the input device described herein. For example, the
reduction in the amount of light entering the image capture device
may be achieved as described with reference to FIG. 3. The method
then advances to operation 162 where a first color light from the
input source is detected through the image capture device as
described with reference to FIGS. 2-4. Here, the first color light
is detected and correlated to an image screen.
The method of FIG. 10 then proceeds to operation 164 where the
first color light being changed to a second color light is
detected. Here, the change in the color of light may be triggered
through pressure applied to a button or mode change activator on
the input device as described above. One skilled in the art will
appreciate that the color change, or frequency change, may be
detected by examining corresponding pixel values associated with an
image capture device. The method then moves to operation 166 where
a mode change is presented in response to the change in the first
color light. The mode change may act to enable a user to perform
click-and-drag operations, highlighting operations, or any other
suitable operations associated with a mode change such as enabled
through a computer mouse. Additionally, when the second color light
is changed back to the first color light, the end of a
click-and-drag operation or highlighting operation is signified.
Alternatively, a third color light may be used to enable
functionality associated with a "right" mouse click. It should be
noted that the mode change is displayed on a display screen similar
to the mode change for a computer mouse in one embodiment of the
invention. Additionally, one skilled in the art will appreciate
that while FIG. 10 is described with respect to color changes, the
invention is not limited to color changes as variations of the same
color of light may be used, e.g., shades of colors. Furthermore,
different frequencies of light may be used instead of different
colors. For example, infrared light may be used with a visible
wavelength of light. As discussed above, any change to a light
source that is capable of causing a change in pixel values
associated with an image capture device may be used for the
embodiments described herein.
In summary, an input device capable of triggering a mode change
through a multi-color, or multi-frequency, light generator is
provided. Of course, multiple collocated lights having different
colors or frequencies may be used to provided the same effect.
Thus, any light generating structure can be used, whether it is a
solid-state device such as a diode, or one or more conventional
light bulbs. In one embodiment, the light is provided by one or
more LEDs that can be coupled or installed onto a pen-like object.
The light emitted from the input device is tracked by an image
capture device. The image capture device can take on any number of
forms, including a camera, an array of charged coupled devices
(CCDs), a digital camera, a conventional camera that is coupled to
a digitizer, or a webcam. In general, the image capture device
should be able to detect light from the input device (or any object
capable of emitting light). Preferably, the input device will
generate enough light so as to enable the camera to detect a spot
of light, which may be bright relative to other surrounding
light.
Thus, in one embodiment, similar to a computer mouse input device,
the light from the input device may be set to always be on when it
is detected by the camera. Thereafter, to cause control, the user
may press a button, or some other suitable triggering mechanism, on
the input device to change a color of light (or a frequency) being
emitted. The webcam captures the color/frequency change, thereby
enabling the color change to act as a button press for a mouse,
i.e., mouse click. It should be noted that any suitable number of
colors may be tracked and that the input device may have multiple
buttons for the multiple colors or the input device may have a
single button that cycles through the multiple colors. In another
embodiment, instead of changing visible color, the change may be
from a first color that changes to general light, as is typical of
standard flash lights. In still another embodiment, the input
device can flicker between a first light and then to an off state,
where not light is emitted at all. For example, an infrared LED may
alternate between a first light and an off state. In such a case,
when the light is off, the light may stay off for a set period of
time. The timing of the off state can then cause one or more same,
similar or different states or changes to occur on a display
screen.
Furthermore, the input device described herein enables enhanced
functionality for interactive entertainment applications. For
example, with respect to sports video games, a user may use the
input device to design a play, e.g., a play for a football game,
basketball game, etc. Strategy games, such as chess and war games,
which were previously limited due to the difficulty in adopting the
joystick controller to function as a mouse-type device become more
user friendly when the input device described herein is used for
mouse-type input, instead of the joystick. With respect to a video
game that incorporates strategy, a user may be on a hillside
directing armies to attack an enemy. In another embodiment, the
input device may be used for television programming. The ability to
incorporate a mode change associated with the change of a light
color in conjunction with the capability to track the movement of
the light source enables the accomplishment of these features.
The invention may employ various computer-implemented operations
involving data stored in computer systems. These operations are
those requiring physical manipulation of physical quantities.
Usually, though not necessarily, these quantities take the form of
electrical or magnetic signals capable of being stored,
transferred, combined, compared, and otherwise manipulated.
Further, the manipulations performed are often referred to in
terms, such as producing, identifying, determining, or
comparing.
Any of the operations described herein that form part of the
invention are useful machine operations. The invention also relates
to a device or an apparatus for performing these operations. The
apparatus may be specially constructed for the required purposes,
or it may be a general purpose computer selectively activated or
configured by a computer program stored in the computer. In
particular, various general purpose machines may be used with
computer programs written in accordance with the teachings herein,
or it may be more convenient to construct a more specialized
apparatus to perform the required operations.
Although the foregoing invention has been described in some detail
for purposes of clarity of understanding, it will be apparent that
certain changes and modifications may be practiced. For example,
although specific examples have been provided for use in relation
to video gaming, the applications can be applied to any computer or
computing device that will require some interaction. The computing
device can be a single stand alone unit or can be interconnected to
other computing devices over a local or global network, such as the
Internet. Accordingly, the present embodiments are to be considered
as illustrative and not restrictive, and the invention is not to be
limited to the details given herein, but may be modified within the
scope and equivalents of the description.
* * * * *