U.S. patent application number 11/833908 was filed with the patent office on 2008-07-31 for multi-touch sensing through frustrated total internal reflection.
Invention is credited to Jefferson HAN.
Application Number | 20080179507 11/833908 |
Document ID | / |
Family ID | 39028233 |
Filed Date | 2008-07-31 |
United States Patent
Application |
20080179507 |
Kind Code |
A2 |
HAN; Jefferson |
July 31, 2008 |
MULTI-TOUCH SENSING THROUGH FRUSTRATED TOTAL INTERNAL
REFLECTION
Abstract
High-resolution, scalable multi-touch sensing display systems
and processes based on frustrated total internal reflection employ
an optical waveguide that receives light, such as infrared light,
that undergoes total internal reflection and an imaging sensor that
detects light that escapes the optical waveguide caused by
frustration of the total internal reflection due to contact by a
user. The optical waveguide may be fitted with a compliant surface
overlay to greatly improve sensing performance, minimize the affect
of contaminants on and damage to the contact surface, to generally
extend system life and to provide other benefits. The systems and
processes provide true multi-touch (multi-input) and high-spatial
and temporal resolution capability due to the continuous imaging of
the frustrated total internal reflection that escapes the entire
optical waveguide. Among other features and benefits, the systems
and processes are scalable to large installations and are well
suited for use with rear-projection and other display devices.
Inventors: |
HAN; Jefferson; (New York,
NY) |
Correspondence
Address: |
PATENT DOCKET CLERK;COWAN, LIEBOWITZ & LATMAN, P.C.
1133 AVENUE OF THE AMERICAS
NEW YORK
NY
10036
UNITED STATES
212-790-9200
212-575-0671
|
Prior
Publication: |
|
Document Identifier |
Publication Date |
|
US 20080029691 A1 |
February 7, 2008 |
|
|
Family ID: |
39028233 |
Appl. No.: |
11/833908 |
Filed: |
August 3, 2007 |
Current U.S.
Class: |
250/224; 250/221;
345/175 |
Current CPC
Class: |
G06F 3/0425 20130101;
G06F 2203/04109 20130101; G06F 2203/04808 20130101; G06F 3/04883
20130101 |
Class at
Publication: |
250/224; 250/221;
345/175 |
International
Class: |
H01J 40/14 20060101
H01J040/14; G06F 3/042 20060101 G06F003/042 |
Claims
1. A multi-touch sensing display, comprising: an optical waveguide
adapted to receive light and adapted to cause at least some of the
received light to undergo total internal reflection within the
optical waveguide, the optical waveguide adapted to allow the total
internal reflection to be frustrated upon occurrence of a physical
phenomena, the optical waveguide adapted to allow some of the light
undergoing total internal reflection to escape when the total
internal reflection is frustrated; and an imaging sensor adapted to
detect light escaping the optical waveguide.
2. The multi-touch sensing display of claim 1, further comprising a
plurality of infrared light emitting diodes disposed directly
against a polished edge of the optical waveguide for emitting the
light received by the optical waveguide.
3. The multi-touch sensing display of claim 1, wherein the optical
waveguide is a sheet of acrylic with edges treated to admit
light.
4. The multi-touch sensing display of claim 1, wherein the physical
phenomena comprises contacting the optical waveguide with an object
so that the total internal reflection is frustrated upon contacting
the optical waveguide with the object.
5. The multi-touch sensing display of claim 4, wherein the object
is a finger of a user of the multi-touch sensing display.
6. The multi-touch sensing display of claim 1, wherein the optical
waveguide is adapted to allow the total internal reflection to be
frustrated simultaneously at a plurality of positions within the
optical waveguide upon occurrence of a plurality of physical
phenomenon, each of the positions disposed apart from one
another.
7. The multi-touch sensing display of claim 6, wherein the
plurality of physical phenomenon include at least contacting a
first location of a surface of the optical waveguide with a first
object and contacting a second location of the surface of the
optical waveguide with a second object, the first and second
locations being disposed apart from one another, the optical
waveguide being adapted to cause total internal reflection to be
frustrated within the optical waveguide at first and second
positions corresponding to the first and second locations upon
contacting the optical waveguide with the first and second
objects.
8. The multi-touch sensing display of claim 7, wherein the first
and second objects are first and second fingers of a user of the
multi-touch sensing display.
9. The multi-touch sensing display of claim 1, wherein the imaging
sensor is adapted to discriminately detect light output from
positions along a two-dimensional surface of the optical
waveguide.
10. The multi-touch sensing display of claim 1, wherein the imaging
sensor is adapted to discriminately detect light simultaneously
escaping from a plurality of positions along a two-dimensional
surface of the optical waveguide.
11. The multi-touch sensing display of claim 1, further comprising
a light source adapted to emit light of predetermined wavelengths;
wherein the optical waveguide is adapted to receive the light
emitted by the light source, and the imaging sensor is adapted to
detect light only substantially at the predetermined
wavelengths.
12. The multi-touch sensing display of claim 1, further comprising
a light source adapted to emit light of first and second
predetermined wavelengths; wherein the optical waveguide is adapted
to receive the light emitted by the light source, and the imaging
sensor is adapted to detect light only substantially at the first
predetermined wavelength; the multi-touch sensing display further
comprising a second imaging sensor adapted to detect light only
substantially at the second predetermined wavelength escaping the
optical waveguide.
13. The multi-touch sensing display of claim 1, further comprising
a light source adapted to emit pulses of light at a rate
substantially synchronized to a shutter of the imaging sensor;
wherein the optical waveguide is adapted to receive the light
emitted by the light source.
14. The multi-touch sensing display of claim 1, wherein the optical
waveguide has a non-planar surface, and the physical phenomena
comprises contacting the non-planar surface of the optical
waveguide so that the total internal reflection is frustrated upon
contacting the non-planar surface of the optical waveguide.
15. The multi-touch sensing display of claim 1, further comprising
a video projector adapted to project a two-dimensional video image
onto the optical waveguide.
16. The multi-touch sensing display of claim 15, further comprising
a diffuser onto which the video image is projected.
17. The multi-touch sensing display of claim 16, wherein the
optical waveguide includes a non-contact surface and a contact
surface; the diffuser being disposed on the non-contact surface and
the physical phenomena comprising contacting the contact surface of
the optical waveguide so that the total internal reflection is
frustrated upon contacting the contact surface of the optical
waveguide.
18. The multi-touch sensing display of claim 17, wherein the video
projector is disposed on a non-contact side of the optical
waveguide corresponding to a side on which the non-contact surface
is disposed, the video projector adapted to project the video image
toward the optical waveguide for viewing by a user disposed on a
contact side of the optical waveguide corresponding to a side on
which the contact surface is disposed.
19. The multi-touch sensing display of claim 1, further comprising
a video projector adapted to project a two-dimensional video image;
and a diffuser disposed adjacent to a surface of the optical
waveguide, the video image being projected onto the diffuser.
20. The multi-touch sensing display of claim 19, wherein a small
gap is disposed between a substantial portion of the diffuser and
the optical waveguide so that the frustration of the total internal
reflection by the diffuser is minimized.
21. The multi-touch sensing display of claim 19, wherein the
diffuser is an electro-switchable diffuser screen synchronized to
projection of the video projector and to the shutter of the imaging
sensor.
22. The multi-touch sensing display of claim 19, wherein the
diffuser is an electro-switchable diffuser screen synchronized to
projection of the video projector and to the shutter of the imaging
sensor.
23. The multi-touch sensing display of claim 1, wherein the imaging
sensor includes a plurality of imaging sensors adapted to detect
light escaping from respectively different portions of the optical
waveguide.
24. The multi-touch sensing display of claim 23, further comprising
a plurality of video projectors, each of the video projectors being
adapted to project a respective two-dimensional video image onto a
respective one of said different portions of the optical
waveguide.
25. The multi-touch sensing display of claim 1, further comprising
an LCD display panel disposed between the imaging sensor and the
optical waveguide for displaying a video image, the LCD display
panel being transparent to light escaping the optical waveguide so
as to not prevent the escaping light from being detected by the
imaging sensor.
26. A multi-touch sensing display, comprising: an optical waveguide
adapted to receive light and adapted to cause some of the received
light to undergo total internal reflection within the optical
waveguide; a compliant surface overlay disposed sufficiently close
to the optical waveguide to enable depression of the compliant
surface overlay by an external force to cause the compliant surface
overlay to contact the optical waveguide, the optical waveguide and
the compliant surface overlay adapted to cause the total internal
reflection within the optical waveguide to be frustrated upon
contact of the optical waveguide by the compliant surface overlay,
the optical waveguide adapted to allow some of the light undergoing
total internal reflection to escape when the total internal
reflection is frustrated; and an imaging sensor adapted to detect
light escaping the optical waveguide.
27. The multi-touch sensing display of claim 26, wherein a small
gap is disposed between a substantial portion of the compliant
surface overlay and the optical waveguide so that the frustration
of the total internal reflection is minimized during non-depression
of the compliant surface overlay.
28. The multi-touch sensing display of claim 26, wherein the
compliant surface overlay is adapted to be depressed by at least
one finger of a user of the multi-touch sensing display, a stylus
and another object applying force.
29. The multi-touch sensing display of claim 26, wherein the
compliant surface overlay is adapted to contact the optical
waveguide at a plurality of locations simultaneously in response to
simultaneous depression of the compliant surface overlay at a
plurality of corresponding locations, the optical waveguide adapted
to cause the total internal reflection to be frustrated
simultaneously within the optical waveguide at said plurality of
locations.
30. The multi-touch sensing display of claim 29, wherein at least
two of the plurality of corresponding locations at which the
compliant surface overlay are simultaneously depressed are disposed
apart from one another.
31. The multi-touch sensing display of claim 26, wherein the
imaging sensor is adapted to discriminately detect light output
from positions along a two-dimensional surface of the optical
waveguide.
32. The multi-touch sensing display of claim 26, wherein the
imaging sensor is adapted to discriminately detect light
simultaneously escaping from a plurality of positions along a
two-dimensional surface of the optical waveguide.
33. The multi-touch sensing display of claim 26, further comprising
a light source adapted to emit light of a predetermined wavelength;
wherein the optical waveguide is adapted to receive the light
emitted by the light source, and the imaging sensor is adapted to
detect light only substantially at the predetermined
wavelength.
34. The multi-touch sensing display of claim 26, further comprising
a light source adapted to emit light of first and second
predetermined wavelengths; wherein the optical waveguide is adapted
to receive the light emitted by the light source, and the imaging
sensor is adapted to detect light only substantially at the first
predetermined wavelength; the multi-touch sensing display further
comprising a second imaging sensor adapted to detect light only
substantially at the second predetermined wavelength escaping the
optical waveguide.
35. The multi-touch sensing display of claim 26, further comprising
a light source adapted to emit pulses of light at a rate
substantially synchronized to the shutter of the imaging sensor;
wherein the optical waveguide is adapted to receive the light
emitted by the light source.
36. The multi-touch sensing display of claim 26, wherein the
compliant surface overlay includes a non-planar contact surface,
the compliant surface overlay being adapted to contact the optical
waveguide upon depression by the external force of the non-planar
contact surface.
37. The multi-touch sensing display of claim 26, further comprising
a video projector adapted to project a two-dimensional video image
onto the optical waveguide.
38. The multi-touch sensing display of claim 37, comprising a
diffuser onto which the video image is projected.
39. The multi-touch sensing display of claim 37, wherein the
compliant surface overlay is adapted to diffuse the video image
projected by the video projector.
40. The multi-touch sensing display of claim 26, wherein the
compliant surface overlay includes a component that rejects ambient
light of the same wavelengths as the light source, a component that
reduces friction at the interaction surface, a component that
reduces glare, and a component that provides a comfortable cushion
for the user to depress.
41. The multi-touch sensing display of claim 26, further comprising
a video projector adapted to project a two-dimensional video image
through the optical waveguide onto the compliant surface overlay,
the compliant surface overlay being adapted to diffuse the video
image projected by the video projector.
42. The multi-touch sensing display of claim 26, further comprising
an LCD display panel disposed between the imaging sensor and the
optical waveguide for displaying a video image, the LCD display
panel being transparent to light escaping the optical waveguide so
as to not prevent the escaping light from being detected by the
imaging sensor, and the compliant surface adapted not to
diffuse.
43. A method of multi-touch sensing, comprising the steps of:
receiving light within an optical waveguide; internally reflecting
the received light within the optical waveguide; frustrating the
internally reflected light within the optical waveguide to cause
some of the reflected light to escape the optical waveguide; and
imaging the escaped light.
44. The method of claim 43, comprising emitting the light by a
light source having a plurality of infrared light emitting diodes,
the emitted light being received within the optical waveguide.
45. The method of claim 43, comprising providing a sheet of acrylic
with edges treated to admit light as the optical waveguide.
46. The method of claim 43, wherein frustrating the internally
reflected light comprises contacting the optical waveguide with an
object.
47. The method of claim 43, wherein frustrating the internally
reflected light comprises contacting the optical waveguide with a
finger of a user.
48. The method of claim 43, wherein frustrating the internally
reflected light includes frustrating the internally reflected light
simultaneously at a plurality of positions within the optical
waveguide, each of the positions disposed apart from one
another.
49. The method of claim 43, wherein frustrating the internally
reflected light comprises contacting simultaneously first and
second locations of a surface of the optical waveguide to cause
reflected light to escape simultaneously from first and second
positions of the optical waveguide, the first and second positions
being disposed apart from one another and corresponding
respectively to the first and second locations.
50. The method of claim 49, wherein frustrating the internally
reflected light comprises contacting simultaneously the first and
second locations of the surface of the optical waveguide by first
and second fingers of a user.
51. The method of claim 43, wherein imaging the escaped light
comprises imaging discriminately light output from positions along
a two-dimensional surface of the optical waveguide.
52. The method of claim 43, wherein imaging the escaped light
comprises imaging discriminately light simultaneously escaping from
a plurality of positions along a two-dimensional surface of the
optical waveguide.
53. The method of claim 43, wherein receiving light within the
optical waveguide comprises receiving light of a predetermined
wavelength, and imaging the escaped light comprises imaging light
only substantially at the predetermined wavelength.
54. The method of claim 43, wherein receiving light within the
optical waveguide comprises receiving light of first and second
predetermined wavelengths, and imaging the escaped light comprises
imaging light only substantially at the first and second
predetermined wavelengths.
55. The method of claim 54, wherein imaging the escaped light
comprises imaging light only substantially at the first
predetermined wavelength by a first imaging sensor and imaging
light only substantially at the second predetermined wavelength by
a second imaging sensor.
56. The method of claim 43, wherein receiving light comprises
receiving pulses of light within the optical waveguide; and imaging
the escaped light comprises imaging the escaped light by an imaging
sensor having a shutter substantially synchronized to the rate of
the pulses of light received within the optical waveguide.
57. The method of claim 43, comprising providing an optical
waveguide having a non-planar surface.
58. The method of claim 43, comprising projecting a two-dimensional
video image onto the optical waveguide.
59. The method of claim 58, comprising diffusing the projected
video image by the optical waveguide.
60. The method of claim 43, comprising projecting a two-dimensional
video image, diffusing the projected video image on a non-contact
surface of the optical waveguide; and contacting a contact surface
of the optical waveguide to frustrate the internally reflected
light.
61. The method of claim 60, comprising projecting the
two-dimensional video image from a non-contact side of the optical
waveguide corresponding to a side on which the non-contact surface
is disposed.
62. The method of claim 43, comprising projecting a two-dimensional
video image and diffusing the projected video image adjacent to the
optical waveguide.
63. The method of claim 62, comprising diffusing the projected
video image by a diffuser disposed adjacent to the optical
waveguide, and providing a small gap between a substantial portion
of the diffuser and the optical waveguide so that frustration of
the internally reflected light by the diffuser is minimized.
64. The method of claim 63, comprising providing as the diffuser an
electro-switchable diffuser screen synchronized to a rate of
projection of the video image.
65. The method of claim 43, wherein imaging the escaped light
comprises imaging by a plurality of imaging sensors light escaping
from respectively different portions of the optical waveguide.
66. The method of claim 65, comprising projecting a plurality of
two-dimensional video images onto said respectively different
portions of the optical waveguide.
67. The method of claim 43, comprising displaying through the
optical waveguide a video image by an LCD display panel, and
passing through the LCD display panel light escaping the optical
waveguide, and wherein imaging the escaped light comprises imaging
the escaped light passing through the LCD display panel.
68. A method of multi-touch sensing, comprising the steps of:
receiving light within an optical waveguide; internally reflecting
the received light within the optical waveguide; depressing a
compliant surface overlay disposed adjacent to the optical
waveguide; contacting the optical waveguide by the compliant
surface overlay upon depressing the compliant surface overlay;
frustrating the internally reflected light within the optical
waveguide upon contacting optical waveguide by the compliant
surface overlay to cause some of the internally reflected light to
escape the optical waveguide; and imaging the escaped light.
69. The method of claim 68, comprising disposing a small gap
between a substantial portion of the compliant surface overlay and
the optical waveguide so that frustrating the internally reflected
light is minimized when the compliant surface overlay is not
depressed.
70. The method of claim 68, wherein depressing the compliant
surface overlay comprises depressing the compliant surface overlay
by a finger of a user.
71. The method of claim 68, wherein depressing the compliant
surface overlay comprises depressing simultaneously the compliant
surface overlay at a plurality of locations, contacting the optical
waveguide comprises contacting simultaneously the optical waveguide
by the compliant surface at a plurality of corresponding locations,
and frustrating the internally reflected light comprises
frustrating simultaneously the internally reflected light within
the optical waveguide at said plurality of corresponding
locations.
72. The method of claim 71, wherein at least two of said plurality
of corresponding locations are disposed apart from one another.
73. The method of claim 68, wherein imaging the escaped light
comprises imaging discriminately light output from positions along
a two-dimensional surface of the optical waveguide.
74. The method of claim 68, wherein imaging the escaped light
comprises imaging discriminately light simultaneously escaping from
a plurality of positions along a two-dimensional surface of the
optical waveguide.
75. The method of claim 68, wherein receiving light within the
optical waveguide comprises receiving light of a predetermined
wavelength, and imaging the escaped light comprises imaging light
only substantially at the predetermined wavelength.
76. The method of claim 68, wherein receiving light within the
optical waveguide comprises receiving light of first and second
predetermined wavelengths, and imaging the escaped light comprises
imaging light only substantially at the first and second
predetermined wavelengths.
77. The method of claim 76, wherein imaging the escaped light
comprises imaging light only substantially at the first
predetermined wavelength by a first imaging sensor and imaging
light only substantially at the second predetermined wavelength by
a second imaging sensor.
78. The method of claim 68, wherein receiving light comprises
receiving pulses of light within the optical waveguide; and imaging
the escaped light comprises imaging the escaped light by an imaging
sensor having a shutter substantially synchronized to the rate of
the pulses of light received within the optical waveguide.
79. The method of claim 68, comprising providing the compliant
surface overlay with a non-planar contact surface, and depressing
the compliant surface overlay comprises depressing the non-planar
contact surface of the compliant surface overlay.
80. The method of claim 68, comprising projecting a two-dimensional
video image onto the optical waveguide.
81. The method of claim 80, comprising diffusing the projected
video image by the optical waveguide.
82. The method of claim 80, comprising diffusing the projected
video image by the compliant surface overlay.
83. The method of claim 68, wherein the compliant surface overlay
includes a component that rejects ambient light of the same
wavelengths as the light source, a component that reduces friction
at the interaction surface, a component that reduces glare, and a
component that provides a comfortable cushion for the user to
depress.
84. The method of claim 68, comprising projecting a two-dimensional
video image through the optical waveguide onto the compliant
surface overlay; and diffusing the projected video image by the
compliant surface overlay.
85. The multi-touch sensing display of claim 16, wherein the
diffuser is an electro-switchable diffuser screen synchronized to a
rate of projection of the video projector, the diffuser being
adapted to be non-diffusive in accordance with shutter periods of
the imaging sensor.
86. The multi-touch sensing display of claim 85, further comprising
at least one additional imaging sensor adapted to image through the
diffuser so as to image the physical phenomena causing the
frustration of the total internal reflection.
87. The multi-touch sensing display of claim 85, wherein the
physical phenomena includes contacting the optical waveguide with
an object, the multi-touch sensing display further comprising at
least one additional imaging sensor adapted to image through the
diffuser so as to image the object contacting the optical
waveguide.
88. The multi-touch sensing display of claim 16, wherein the
diffuser is a directional diffuser adapted to diffuse for
predetermined angles of incidence, and adapted to not diffuse for
other angles of incidence.
89. The multi-touch sensing display of claim 38, wherein the
diffuser is an electro-switchable diffuser screen synchronized to a
rate of projection of the video projector, the diffuser being
adapted to be non-diffusive in accordance with shutter periods of
the imaging sensor.
90. The multi-touch sensing display of claim 89, further comprising
at least one additional imaging sensor adapted to image through the
diffuser so as to image an object applying the external force to
cause the compliant surface overlay to contact the optical
waveguide.
91. The multi-touch sensing display of claim 38, wherein the
diffuser is a directional diffuser adapted to diffuse for
predetermined angles of incidence, and adapted to not diffuse for
other angles of incidence.
92. The multi-touch sensing display of claim 26, wherein the
compliant surface overlay includes a plurality of layers, at least
one of the layers being a wavelength selective shield to mitigate
interference by external ambient light of the detection of light
escaping the optical waveguide by the imaging sensor.
93. The multi-touch sensing display of claim 92, wherein at least
one of the layers of the compliant surface overlay is adapted to
contribute to the frustration of the total internal reflection
within the optical waveguide upon contact of the optical waveguide
by the compliant surface overlay.
94. The method of claim 58, comprising diffusing the projected
video image by an electro-switchable diffuser screen synchronized
to a rate of projection of the video image; wherein imaging the
escaped light is carried out by an imaging sensor, the diffuser
being non-diffusive in accordance with shutter periods of the
imaging sensor.
95. The method of claim 94, comprising imaging through the diffuser
by a second imaging sensor an object causing the frustration of the
internally reflected light.
96. The method of claim 58, comprising diffusing the projected
video image by a directional diffuser adapted to diffuse for
predetermined angles of incidence, and adapted to not diffuse for
other angles of incidence.
97. The method of claim 80, comprising diffusing the projected
video image by an electro-switchable diffuser screen synchronized
to a rate of projection of the video image; wherein imaging the
escaped light is carried out by an imaging sensor, the diffuser
being non-diffusive in accordance with shutter periods of the
imaging sensor.
98. The method of claim 97, comprising imaging through the diffuser
by a second imaging sensor an object causing the frustration of the
internally reflected light.
99. The method of claim 80, comprising diffusing the projected
video image by a directional diffuser adapted to diffuse for
predetermined angles of incidence, and adapted to not diffuse for
other angles of incidence.
100. The method of claim 68, comprising providing the compliant
surface overlay with a plurality of layers, at least one of the
layers being a wavelength selective shield to mitigate interference
by external ambient light of the imaging of light escaping the
optical waveguide.
101. The multi-touch sensing display of claim 100, comprising
providing the compliant surface overlay with at least one layer
adapted to contribute to the frustration of the internally
reflected light within the optical waveguide upon contact of the
optical waveguide by the compliant surface overlay.
Description
REFERENCE TO RELATED APPLICATION
[0001] This application claims priority to U.S. patent application
No. 60/821,325, filed Aug. 3, 2006, the disclosure of which is
incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates to techniques for enabling
high-resolution multi-touch sensing displays based on frustrated
total internal reflection.
[0004] 2. Description of the Related Art
[0005] Touch sensing is commonplace for single points of contact,
but it is relatively difficult to sense multiple points of contact
simultaneously or "multi-touch sensing."
[0006] One fairly straightforward approach for multi-touch sensing
is to utilize multiple discrete sensors, with each sensor sensing a
respective point of contact. For example, Tactex Control Inc. has a
line of array sensors for use as floor sensors, security devices
and other applications. As another example, the publication Lee,
S., Buxton, W., and Smith, K. C., "A Multi-Touch Three Dimensional
Touch-Sensitive Tablet," Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (San Francisco, Calif., United
States), CHI '85. ACM Press, New York, N.Y., 21-25 (1985),
incorporated herein by reference, describes the use of sensors
arranged in a matrix configuration with an active element (diode)
disposed at each node. The Fingerworks iGesturePad is another
example of a device that employs multiple discrete sensors in a
matrix configuration with active transistors at each node. U.S.
Pat. No. 6,323,846 to Westerman et al., incorporated herein by
reference, discloses additional examples of using an array of
proximity sensors in a multi-touch surface system.
[0007] Multi-touch sensing may be achieved by carefully employing a
purely passive matrix of force-sensitive-resistors (FSRs), as
discussed in Hillis, W. D., "A High Resolution Imaging Touch
Sensor," International Journal of Robotics Research, pages 1, 2,
33-44 (1982), incorporated herein by reference. U.S. Pat. No.
4,134,063 to Nicol et al., incorporated herein by reference,
discloses the use of capacitive electrodes for this purpose. And
more recently discussed in Rekimoto, J., "SmartSkin: An
Infrastructure for Freehand Manipulation on Interactive Surfaces,"
Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, CHI '02, ACM Press, New York, N.Y., 113-120 (2002),
incorporated herein by reference. Such systems, while less complex
than systems that employ multiple active sensors, still entail
numerous electrical connections and thus disadvantageously limit
their application to uses that require relatively low resolution
(e.g., under 100.times.100). Furthermore, such systems are visually
opaque and thus require the use of top-projection if to be
integrated with a graphic display. Finally, such systems have had
problems with robustness given the feeble nature of the electrical
signals they utilize.
[0008] The use of video cameras has been proposed to acquire
high-resolution datasets at rapid rates. However, these video based
techniques are quite imprecise and are not able to determine if
true touch contact has been made, a disparity that can be quite
disturbing to the user. Recent approaches include estimating depth
from intensity as discussed in Matsushita, N. and Rekimoto, J.,
"HoloWall: Designing a Finger, Hand, Body, and Object Sensitive
Wall," Proceedings of the 10th Annual ACM Symposium on User
Interface Software and Technology (Banff, Alberta, Canada, Oct.
14-17, 1997), UIST '97, ACM Press, New York, N.Y., 209-210 (1997);
estimating depth from stereo as disclosed in Wilson, A. D.,
"TouchLight: An Imaging Touch Screen and Display for Gesture-Based
Interaction," Proceedings of the 6th International Conference on
Multimodal Interfaces (State College, Pa., USA, Oct. 13-15, 2004),
ICMI '04, ACM Press, New York, N.Y., 69-76 (2004); Malik, S. and
Laszlo, J., "Visual Touchpad: A Two-Handed Gestural Input Device,"
Proceedings of the 6th International Conference on Multimodal
Interfaces (State College, Pa., USA, Oct. 13-15, 2004), ICMI '04,
ACM Press, New York, N.Y., 289-296 (2004); and tracking markers
embedded within a deformable substrate as disclosed in Kamiyama,
K., Vlack, K., Mizota, T., Kajimoto, H., Kawakami, N., and Tachi,
S., "Vision-Based Sensor for Real-Time Measuring of Surface
Traction Fields," IEEE Comput. Graph. Appl. 25, 1 (January 2005),
68-75. Each of these references is incorporated herein by
reference.
[0009] Another group of touch sensing techniques is to employ
frustrated total internal reflection (FTIR). When light encounters
an interface to a medium with a lower index of refraction (e.g.
glass to air), the light becomes refracted to an extent which
depends on its angle of incidence, and beyond a certain critical
angle, it undergoes total internal reflection (TIR). Fiber optics,
light pipes, and other optical waveguides rely on this phenomenon
to transport light efficiently with very little loss. However,
another material at the interface can frustrate this total internal
reflection, causing light to escape the waveguide there
instead.
[0010] Frustrated total internal reflection is well known and has
been used in the biometrics community to image fingerprint ridges
since at least the 1960s. U.S. Pat. No. 3,200,701 to White,
incorporated herein by reference, issued in 1965 and describes
using FTIR to optically detect the ridge pattern of a skin
surface.
[0011] U.S. Pat. No. 3,673,327 to Johnson et al., incorporated
herein by reference, issued in 1972 and discloses an early version
of a touch actuable device in which a binary device detects the
attenuation of light through a platen waveguide caused by a finger
in contact.
[0012] U.S. Pat. No. 3,846,826 to Mueller, incorporated herein by
reference, issued in 1974 and describes an imaging touch sensor
that allows a user to "paint" onto a display using free-form
objects, such as brushes, styli and fingers. In that device, light
from the flying spot of a CRT is totally internally reflected off
the face of a large prism and focused onto a single photo detector,
thereby generating an updating bitmap of areas that are being
contacted. In 1985, this method was updated in an optically
inverted configuration, with a video camera and a broad light
source replacing the CRT and photodetector, as disclosed in Greene,
R., "The Drawing Prism: A Versatile Graphic Input Device,"
Proceedings of the 12th Annual Conference on Computer Graphics and
Interactive Techniques SIGGRAPH '85, ACM Press, New York, N.Y.,
103-110 (1985), incorporated herein by reference.
[0013] U.S. Pat. No. 4,346,376 to Mallos, incorporated herein by
reference, discloses a CRT-based touch sensor, which replaced the
bulky prism with a thin platen waveguide and operates by detecting
the light scattered away by an object in optical contact. More
recent fingerprint sensors use this approach, as disclosed in
Fujieda, I., Haga, H., "Fingerprint Input based on Scattered-Light
Detection," Applied Optics-IP, 36, 35, 9152-9156 (1997),
incorporated herein by reference.
[0014] The robotics community also has used this approach since
1984 in the construction of tactile sensors for robot grippers, but
with a compliant surface overlay. Various publications include:
Mott, D. H., Lee, M. H., and Nicholls, H., "An Experimental Very
High Resolution Tactile Sensor Array," Robot Sensors Vol. 2:
Tactile and Non-Vision, Pugh, A., Ed. Springer-Verlag, Berlin,
179-188 (1986); Tanie, K., Komoriya, K., Kaneko, M., Tachis, S.,
and Fujikava, A., "A High Resolution Tactile Sensor," Robot Sensors
Vol. 2: Tactile and Non-Vision, Pugh, A., Ed. Springer-Verlag,
Berlin, 189-198 (1986); and U.S. Pat. No. 4,668,861 to White, each
of which is incorporated herein by reference.
[0015] With the use of a compliant surface overlay, a structured
flexible membrane, normally kept apart from the waveguide by an
air-gap, makes optical contact with the waveguide when depressed.
U.S. Pat. No. 4,484,179 to Kasday, incorporated herein by
reference, discloses this approach in the context of a touch
sensitive display.
[0016] Additional publications that set forth various interaction
techniques utilizing multi-touch sensing include: Buxton, W., Hill,
R., and Rowley, P., "Issues and Techniques in Touch-Sensitive
Tablet Input," Proceedings of the 12th Annual Conference on
Computer Graphics and Interactive Techniques SIGGRAPH '85, ACM
Press, New York, N.Y., 215-224 (1985); Dietz, P. and Leigh, D.,
"DiamondTouch: A Multi-User Touch Technology," Proceedings of the
14th Annual ACM Symposium on User Interface Software and Technology
(Orlando, Fla., Nov. 11-14, 2001), UIST '01. ACM Press, New York,
N.Y., 219-226 (2001); Westerman, W., Elias, J. G., and Hedge, A.,
"Multi-Touch: A New Tactile 2-D Gesture Interface for
Human-Computer Interaction," Proceedings of the Human Factors and
Ergonomics Society 45th Annual Meeting (Minneapolis/St. Paul,
Minn., October 2001), 632-636 (2001); and Wu, M. and Balakrishnan,
R., "Multi-Finger and Whole Hand Gestural Interaction Techniques
for Multi-User Tabletop Displays," Proceedings of the 16th Annual
ACM Symposium on User Interface Software and Technology (Vancouver,
Canada, Nov. 2-05, 2003), UIST '03, ACM Press, New York, N.Y.,
193-202 (2003), each of which is incorporated herein by
reference.
OBJECTS AND SUMMARY OF THE INVENTION
[0017] In view of the foregoing, it is seen that there has only
limited development in the field of multi-touch sensing displays.
Hence, there remains the need for a multi-touch sensing display
that is high resolution and high precision, relatively simple,
inexpensive and scalable.
[0018] It is therefore an object of the present invention to
provide multi-touch sensing display systems/processes that are
relatively simple, inexpensive and scalable for providing
high-resolution multi-touch sensing.
[0019] It is a further object of the present invention to provide
multi-touch sensing systems/processes that are based on frustrated
total internal reflection.
[0020] It is another object of the present invention to provide
multi-touch sensing systems/processes suitable for use with
graphical display without resorting to top projection.
[0021] In accordance with one embodiment of the present invention,
a multi-touch sensing display comprises an optical waveguide
adapted (i.e., designed) to receive light and adapted to cause some
of the received light to undergo total internal reflection within
the optical waveguide, the optical waveguide adapted to allow total
internal reflection to be frustrated upon occurrence of a physical
phenomena and adapted to allow some of the received light to escape
when total internal reflection is frustrated, and an imaging camera
adapted to detect light escaping the optical waveguide.
[0022] As an aspect of the invention, the sensor includes infrared
light emitting diodes disposed directly against the edge of the
optical waveguide.
[0023] As a further aspect of the invention, the optical waveguide
is a sheet of acrylic with edges treated to admit light.
[0024] As another aspect of the invention, the physical phenomena
entails contacting the optical waveguide with an object so that
total internal reflection is frustrated upon such contact.
[0025] As a feature of this aspect, the object is a finger of a
user of the multi-touch sensing display.
[0026] As another aspect of the invention, the optical waveguide is
adapted to allow total internal reflection to be frustrated
simultaneously at multiple positions upon occurrence of multiple
physical phenomenon, and some of those positions are disposed apart
from one another.
[0027] As a feature of this aspect, the physical phenomenon include
at least contacting a first location of a surface of the optical
waveguide with a first object and contacting a second location of
the surface of the optical waveguide with a second object, the
first and second locations being disposed apart from one another,
and the optical waveguide is adapted to cause total internal
reflection to be frustrated at corresponding first and second
positions.
[0028] As a further feature of this aspect, the first and second
objects are first and second fingers of a user of the multi-touch
sensing display.
[0029] As another aspect of the invention, an imaging sensor is
adapted to discriminately detect light output from positions along
a two-dimensional surface of the optical waveguide.
[0030] As an additional aspect of the invention, the imaging camera
is adapted to discriminately detect light simultaneously escaping
from multiple positions along a two-dimensional surface of the
optical waveguide.
[0031] As yet a further aspect of the invention, the sensor
includes a light source adapted to emit light of a predetermined
wavelength that is received by the optical waveguide, and the
imaging camera is adapted to detect light only substantially at the
predetermined wavelength.
[0032] As yet another aspect of the invention, the sensor includes
a light source adapted to emit light of first and second
predetermined wavelengths that are received by the optical
waveguide, and the imaging camera is adapted to detect light only
substantially at the first predetermined wavelength, and the
multi-touch sensing display further includes a second imaging
camera adapted to detect light only substantially at the second
predetermined wavelength escaping the optical waveguide.
[0033] As yet an additional aspect of the invention, the sensor
includes a light source adapted to emit pulses of light at a rate
substantially synchronized to the shutter of the imaging
camera.
[0034] As another aspect of the invention, the optical waveguide
has a non-planar surface, and the physical phenomena comprises
contacting the non-planar surface of the optical waveguide so that
total internal reflection is frustrated upon such contact.
[0035] As a further aspect of the invention, the sensor includes a
video projector adapted to project a two-dimensional video image
onto the optical waveguide.
[0036] As a feature of this aspect, the multi-touch sensing display
includes a diffuser onto which the video image is projected.
[0037] As another feature of this aspect, the optical waveguide
includes non-contact and contact surfaces, the diffuser is disposed
on the non-contact surface and the physical phenomena comprises
contacting the contact surface of the optical waveguide so that
total internal reflection is frustrated upon such contact.
[0038] As a further feature of this aspect, the video projector is
disposed on a non-contact side of the optical waveguide
corresponding to a side on which the non-contact surface is
disposed, and the video projector is adapted to project the video
image toward the optical waveguide for viewing by a user disposed
on a contact side of the optical waveguide.
[0039] As another aspect of the invention, the sensor includes a
video projector adapted to project a two-dimensional video image
and a diffuser disposed adjacent to a surface of the optical
waveguide, and the video image is projected onto the diffuser.
[0040] As a feature of this aspect, a small gap is disposed between
a substantial portion of the diffuser and the optical waveguide so
that frustration by the diffuser is minimized.
[0041] As a further feature of this aspect, the diffuser is an
electro-switchable diffuser screen synchronized to a rate of
projection of the video projector, and is made non-diffusive
according to the shutter periods of the imaging sensor.
[0042] As a further feature of this aspect, the diffuser is a
directional diffuser (e.g. holographic, Lumisty, etc.) that
diffuses for certain angles of incidence, and is non-diffuse for
others.
[0043] As a further aspect of the invention, additional cameras are
used to view through the diffuser when it is made non-diffusive to
observe the touching object.
[0044] As another aspect of the invention, the imaging camera
includes multiple imaging cameras adapted to detect light escaping
from respectively different portions of the optical waveguide.
[0045] As a feature of this aspect, the sensor includes multiple
video projectors and each video projector is adapted to project a
respective two-dimensional video image onto a respective one of the
different portions of the optical waveguide.
[0046] As an additional aspect of the invention, the sensor
includes an LCD display panel disposed between the imaging camera
and the optical waveguide, and the LCD display panel is transparent
to light escaping the optical waveguide so as to not prevent the
escaping light from being detected by the imaging camera.
[0047] In accordance with another embodiment of the present
invention, a multi-touch sensing display comprises an optical
waveguide adapted to receive light and adapted to cause some of the
received light to undergo total internal reflection within the
optical waveguide, a compliant surface overlay disposed
sufficiently close to the optical waveguide to enable depression of
the compliant surface overlay by an external force to cause the
compliant surface overlay to contact the optical waveguide, the
optical waveguide and the compliant surface overlay adapted to
cause total internal reflection within the optical waveguide to be
frustrated upon contact of the optical waveguide by the compliant
surface overlay, the optical waveguide adapted to allow some of the
light undergoing total internal reflection to escape when total
internal reflection is frustrated, and an imaging camera adapted to
detect light escaping the optical waveguide.
[0048] As an aspect of this embodiment of the present invention, a
small gap is disposed between a substantial portion of the
compliant surface overlay and the optical waveguide so that
frustration of total internal reflection is minimized during
non-depression of the compliant surface overlay.
[0049] As a further aspect of this embodiment, the compliant
surface overlay is adapted to be depressed by a finger of a user of
the multi-touch sensing display.
[0050] As a further aspect of this embodiment, the compliant
surface overlay is adapted to be depressed by passive styluses,
gloved hands, and arbitrary objects.
[0051] As another aspect of this embodiment, the compliant surface
overlay is adapted to contact the optical waveguide at multiple
locations simultaneously in response to simultaneous depression of
the compliant surface overlay at multiple corresponding locations,
while maintaining a gap in locations that are not depressed, and
the optical waveguide is adapted to cause total internal reflection
to be frustrated simultaneously within the optical waveguide at
those locations.
[0052] As a feature of this aspect, at least two of the depressed
locations are disposed apart from one another.
[0053] As an additional aspect, the imaging camera is adapted to
discriminately detect light output from positions along a
two-dimensional surface of the optical waveguide.
[0054] As yet a further aspect, the imaging camera is adapted to
discriminately detect light simultaneously escaping from multiple
positions along a two-dimensional surface of the optical
waveguide.
[0055] As yet another aspect, the sensor further includes a light
source adapted to emit light of a predetermined wavelength that is
received by the optical waveguide, and the imaging camera is
adapted to detect light only substantially at the predetermined
wavelength.
[0056] As an additional aspect, the light source emits light of
first and second predetermined wavelengths, and the imaging camera
is adapted to detect light only substantially at the first
predetermined wavelength, and the multi-touch sensing display
further includes a second imaging camera adapted to detect light
only substantially at the second predetermined wavelength escaping
the optical waveguide.
[0057] As yet a further aspect, the light source emits pulses of
light at a rate substantially synchronized to the shutter of the
imaging sensor.
[0058] As yet another aspect, the compliant surface overlay
includes a non-planar contact surface.
[0059] As a further aspect, the sensor includes a video projector
adapted to project a two-dimensional video image onto the optical
waveguide.
[0060] As a feature of this aspect, the multi-touch sensing display
includes a diffuser onto which the video image is projected.
[0061] As a feature of this feature, the diffuser is an
electro-switchable diffuser screen synchronized to a rate of
projection of the video projector, and is made non-diffusive
according to the shutter periods of the imaging sensor.
[0062] As a further feature, the diffuser is a directional diffuser
(e.g. holographic, Lumisty, etc.) that diffuses for certain angles
of incidence, and is non-diffuse for others.
[0063] As yet another feature, additional cameras are used to view
through the diffuser when it is made non-diffusive to observe the
touching object.
[0064] As another feature of this aspect, the compliant surface
overlay is adapted to diffuse the video image projected by the
video projector.
[0065] As yet a further aspect of this embodiment, the compliant
surface overlay includes a component that rejects ambient light of
the same wavelengths as the light source, a component that reduces
friction at the interaction surface, a component that reduces
glare, and a component that provides a comfortable cushion for the
user to depress.
[0066] As yet another aspect, the sensor includes a video projector
adapted to project a two-dimensional video image through the
optical waveguide onto the compliant surface overlay, and the
compliant surface overlay has a thin layer of rubber adapted to
diffuse the video image projected by the video projector.
[0067] In accordance with a further embodiment of the present
invention, a method of multi-touch sensing comprises the steps of
receiving light within an optical waveguide, internally reflecting
the received light within the optical waveguide, frustrating the
internally reflected light within the optical waveguide to cause
some of the reflected light to escape the optical waveguide, and
imaging the escaped light.
[0068] As an aspect of this embodiment of the present invention,
the light is emitted by a light source having multiple infrared
light emitting diodes.
[0069] As a further aspect, the method includes providing a sheet
of acrylic with edges treated to admit light as the optical
waveguide.
[0070] As an additional aspect, the optical waveguide is contacted
with an object to frustrate the internally reflected light.
[0071] As yet another aspect, one or more fingers of a user contact
the optical waveguide to frustrate the internally reflected
light.
[0072] As yet a further aspect, the internally reflected light is
frustrated simultaneously at multiple positions within the optical
waveguide, and some of those positions are disposed apart from one
another.
[0073] As yet an additional aspect, first and second locations of
the optical waveguide disposed apart from one another are
simultaneously contacted to cause reflected light to escape
simultaneously from corresponding positions of the optical
waveguide.
[0074] As a feature of this aspect, frustration at such multiple
positions is carried out by contacting the optical waveguide by two
fingers of a user.
[0075] As yet a further aspect, the light escaping the optical
waveguide is imaged discriminately along a two-dimensional
surface.
[0076] As yet another aspect, discriminate imaging of light
simultaneously escaping from multiple positions along a
two-dimensional surface of the optical waveguide is carried
out.
[0077] As yet an additional aspect, light of a predetermined
wavelength is received within the optical waveguide, and light only
substantially at the predetermined wavelength escaping the optical
waveguide is imaged.
[0078] As yet a further aspect, light of two different wavelengths
is received within the optical waveguide, and light only
substantially at those wavelengths escaping the optical waveguide
is imaged.
[0079] As a feature of this aspect, two different cameras image
light at the two different wavelengths.
[0080] As yet a further aspect, pulses of light are received within
the optical waveguide, and an imaging camera having a shutter
synchronized to the rate of the pulses detects the escaped
light.
[0081] As yet another aspect, an optical waveguide having a
non-planar surface is provided.
[0082] As yet an additional aspect, the method includes projecting
a two-dimensional video image onto the optical waveguide.
[0083] As a feature of this aspect, a diffuser is provided to
diffuse the projected video image.
[0084] As yet a further aspect, a two-dimensional video image is
projected on and diffused by a diffuser disposed on a non-contact
surface of the optical waveguide, and a contact surface of the
optical waveguide is contacted to cause the internally reflected
light to be frustrated.
[0085] As a feature of this aspect, the video image is projected
from a non-contact side of the optical waveguide.
[0086] As yet another aspect, the projected video image is diffused
adjacent to the optical waveguide.
[0087] As a feature of this aspect, a small gap between a
substantial portion of the diffuser and the optical waveguide is
provided so that frustration of the internally reflected light by
the diffuser is minimized.
[0088] As a further feature of this aspect, an electro-switchable
diffuser screen synchronized to a rate of projection of the video
image is provided, the electro-switchable diffuser screen being
non-diffusive according to the shutter periods of a sensor imaging
the escaped light.
[0089] As yet another feature, the provided electro-switchable
diffuser screen is a directional (e.g. holographic, Lumisty, etc.)
that diffuses for certain angles of incidence, and is non-diffuse
for others
[0090] As another aspect, additional cameras are provided to view
through the diffuser when the diffuser is made non-diffusive to
observe the touching object.
[0091] As yet a further aspect, multiple imaging cameras image
light escaping from respectively different portions of the optical
waveguide.
[0092] As a feature of this aspect, multiple video images are
projected onto the respectively different portions of the optical
waveguide.
[0093] As yet another aspect, a video image is displayed through
the optical waveguide by an LCD display panel that allows light
escaping the optical waveguide to pass through it.
[0094] In accordance with another embodiment of the present
invention, a method of multi-touch sensing comprises the steps of
receiving light within an optical waveguide, internally reflecting
the received light within the optical waveguide, depressing a
compliant surface overlay disposed adjacent to the optical
waveguide, contacting the optical waveguide by the compliant
surface overlay upon depressing the compliant surface overlay,
frustrating the internally reflected light within the optical
waveguide upon contacting the optical waveguide by the compliant
surface overlay to cause some of the internally reflected light to
escape the optical waveguide, and imaging the escaped light.
[0095] As an aspect of this embodiment of the present invention,
the method further includes the steps of disposing a small gap
between a substantial portion of the compliant surface overlay and
the optical waveguide so that frustrating the internally reflected
light is minimized when the compliant surface overlay is not
depressed.
[0096] As a further aspect of this embodiment, the compliant
surface overlay is depressed by a finger of a user.
[0097] As yet a further aspect, the compliant surface overlay is
depressed by passive styluses, gloved hands, and/or arbitrary
objects.
[0098] As another aspect, the compliant surface overlay is
depressed simultaneously at multiple locations, the optical
waveguide is contacted simultaneously by the compliant surface
overlay at multiple corresponding locations, and the internally
reflected light within the optical waveguide is frustrated
simultaneously at those corresponding locations.
[0099] As a feature of this aspect, at least two of the
corresponding locations are disposed apart from one another.
[0100] As an additional aspect, light output from positions along a
two-dimensional surface of the optical waveguide is imaged
discriminately.
[0101] As yet a further aspect, light simultaneously escaping from
multiple positions along a two-dimensional surface of the optical
waveguide is imaged discriminately.
[0102] As yet another aspect, light of a predetermined wavelength
is received within the optical waveguide, and light escaping the
optical waveguide at the predetermined wavelength is imaged.
[0103] As yet an additional aspect, light of two different
predetermined wavelengths is received within the optical waveguide,
and light substantially only at those wavelengths escaping the
optical waveguide is imaged.
[0104] As a feature of this aspect, first and second cameras are
provided and each camera images light at a different
wavelength.
[0105] As yet a further aspect, pulses of light are received within
the optical waveguide, and an imaging camera having a shutter
substantially synchronized to that images the escaped light.
[0106] As yet another aspect, a compliant surface overlay having a
non-planar contact surface is provided.
[0107] As yet an additional aspect, a two-dimensional video image
is projected onto a diffuser disposed on the optical waveguide.
[0108] As a further object, an electro-switchable diffuser screen
synchronized to a rate of projection of the video image is
provided, the electro-switchable diffuser screen being
non-diffusive according to the shutter periods of a sensor imaging
the escaped light.
[0109] As yet another feature, the provided electro-switchable
diffuser screen is a directional (e.g. holographic, Lumisty, etc.)
that diffuses for certain angles of incidence, and is non-diffuse
for others
[0110] As another aspect, additional cameras are provided to view
through the diffuser when the diffuser is made non-diffusive to
observe the touching object.
[0111] As a feature of this aspect, the projected video image is
diffused by a diffuser disposed on the optical waveguide.
[0112] As a further feature of this aspect, the compliant surface
overlay diffuses the projected video image.
[0113] As yet another aspect, the compliant surface overlay
includes a component that rejects ambient light of the same
wavelengths as the light source, a component that reduces friction
at the interaction surface, a component that reduces glare, and a
component that provides a comfortable cushion for the user to
depress.
[0114] As yet an additional aspect, a video image is projected
through the optical waveguide onto the compliant surface overlay,
and the projected video image is diffused by the compliant surface
overlay.
[0115] Various other objects, advantages and features of the
present invention will become readily apparent to those of ordinary
skill in the art, and the novel features will be particularly
pointed out in the appended claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0116] The following detailed description, given by way of example
and not intended to limit the present invention solely thereto,
will best be appreciated in conjunction with the accompanying
drawings, wherein like reference numerals denote like elements and
parts, in which:
[0117] FIGS. 1A-1D show several examples of multi-touch sensing in
accordance with the present invention;
[0118] FIG. 2 is a schematic illustration of a multi-touch sensing
display based on frustrated total internal reflection in accordance
with the present invention;
[0119] FIG. 3 is a schematic illustration of utilizing two fingers
with the multi-touch sensing display of the present invention;
[0120] FIG. 4 is a schematic illustration of a non-planar optical
waveguide that may be utilized in accordance with the present
invention;
[0121] FIG. 5 is a schematic illustration of a multi-touch sensing
display employing a compliant surface overlay in accordance with
the present invention;
[0122] FIGS. 6A and 6B respectively show images from outputs of a
contaminated surface and when employing a compliant surface in
accordance with the present invention;
[0123] FIG. 7 is a schematic illustration of a non-planar optical
waveguide and a non-planar compliant surface overlay that may be
utilized in accordance with the present invention;
[0124] FIG. 8 is a schematic illustration of a system employing
multiple imaging sensors and projectors in accordance with the
present invention;
[0125] FIG. 9 is a schematic illustration of a multi-touch sensing
display employing an LCD in accordance with the present invention;
and
[0126] FIG. 10 is a schematic illustration of a multi-touch sensing
display employing multiple wavelengths of light and multiple
imaging sensors in accordance with the present invention.
DETAILED DESCRIPTION OF THE PRESENT INVENTION
[0127] Multi-touch sensing enables a user to interact with a system
with more than one finger at a time, as in chording and bi-manual
operations. Multi-touch sensing may accommodate multiple users
simultaneously, which is especially useful for larger
shared-display systems such as interactive walls and tabletops.
FIGS. 1A through 1D of the drawings show several simple examples of
multi-touch sensing of the present invention.
[0128] Multi-touch sensing in accordance with the present invention
is based on frustrated total internal reflection (FTIR). When light
encounters an interface to a medium with a lower index of
refraction, such as glass to air, the light becomes refracted to an
extent which depends on its angle of incidence. Beyond a certain
critical angle, the light undergoes total internal reflection
(TIR). But, if another material is placed at the interface, total
internal reflection is frustrated, causing light to escape the
waveguide. Since the concept of FTIR is well known and understood
in the art, further technical description of FTIR is omitted herein
except where necessary for an understanding of the present
invention.
[0129] The present invention employs FTIR to produce a system that
acquires true touch image information at high spatial and temporal
resolutions. As discussed herein, by such use, the multi-touch
system of the present invention is scalable to large installations,
and is well suited for use with display technologies including
rear-projection.
[0130] FIG. 2 of the drawings is a simplified schematic
illustration of a multi-touch sensing display 10 in accordance with
the present invention. As shown, multi-touch sensing display 10
also called, for convenience, "sensor 10") includes an optical
waveguide 12, a light source 14, light absorbing surfaces (or
"baffle") 16, and an imaging sensor 20 (also called imaging camera
herein)(other elements in FIG. 2 described below).
[0131] Light source 14 preferably includes multiple high-power
infrared LEDs, which are placed directly against a polished edge of
optical waveguide 12 so as to maximize coupling into total internal
reflection. For example, the LEDs can provide a total optical
output of 460 mW at 880 nm, although other optical outputs can be
employed.
[0132] Optical waveguide 12 may be a sheet of acrylic whose edges
have been polished to admit light, but other suitable materials may
be utilized. In one example of a manufactured prototype, a
one-quarter inch (6.4 mm) thick acrylic sheet having the dimensions
of 16 inches by 12 inches (406 mm.times.305 mm) is utilized as the
optical waveguide. Common glass generally is not preferred due to
its poor optical transmittance. However, clearer glass formulations
(e.g. "water white") may be employed. Though more expensive, such
glass is structurally stiffer and is far less easily scratched than
acrylic.
[0133] The light emitted from light source 14 undergoes total
internal reflection within optical waveguide 12, thus causing the
light to remain trapped within optical waveguide 12. When an object
is placed in contact with a contact surface 12a of the optical
waveguide, such as a finger 30 shown in FIG. 2, total internal
reflection is frustrated thus causing some light to scatter from
the optical waveguide ("scattered light"), as represented by arrows
"A" in the figure.
[0134] While sensor 10 of the present invention may be employed for
single contact (or touch) applications, it is particularly well
suited in accordance with the present invention for multi-touch
applications. FIG. 3 schematically illustrates when two fingers of
a user simultaneously contact the optical waveguide of the
multi-touch sensing display of the present invention. As shown,
first and second fingers 30a, 30b contact the optical waveguide 12
at two different locations, thus resulting in the frustration of
the total internal reflection within the optical waveguide in two
regions. Thus, light escapes from such different regions of the
optical waveguide, as represented by arrows "B" and "C" in the
figure.
[0135] Referring back to FIG. 2, the imaging sensor is mounted
orthogonally to detect the light scattered through the optical
waveguide. In another arrangement, various optical components
(e.g., mirrors) may be employed to redirect the scattered light to
allow the imaging sensor to be disposed at another location. In
either case, the imaging sensor preferably is equipped with a
band-pass filter matched to the output of the light source (e.g.,
14) to minimize background signal. The imaging sensor may be of
different types, e.g., CCD, CMOS.
[0136] The function of the imaging sensor can also be provided by
any other means for sensing multiple points of light, such as a 2D
array of photodiodes or phototransistors or other light-sensing
elements.
[0137] In accordance with the present invention, imaging sensor 20
continuously images the light escaping from non-contact surface 12b
of optical waveguide 12. Accordingly, imaging sensor 20
discriminately senses, for each successive instant of time, all
points of contact of optical waveguide 12. Hence, for a "single"
point of contact, such as contact by one finger of a user as shown
in FIG. 2, a single "area" of contact corresponding to the surface
of the finger actually contacting the optical waveguide is
discriminately sensed by the imaging sensor. Likewise, when two or
more objects (e.g., two or more fingers of a user) contact the
optical waveguide, multiple areas of contact are discriminately
(and simultaneously) sensed by the imaging sensor. As used herein,
a point of contact, contacting a "location" or other similar phase
is understood to mean an area of contact, unless it is clear within
the context of the description that another meaning is
intended.
[0138] In addition, even for a single point of contact (i.e., area
of contact), the sensor of the present invention discriminates
between a relatively small point of contact and a larger point of
contact. For example, a finger contacting the optical waveguide
with a relatively small amount of pressure provides an area of
contact that generally is smaller than the area of contact when
greater pressure is applied (i.e., more of the finger contacts the
surface of the optical waveguide when greater pressure is
applied).
[0139] By employing an imaging sensor with a sufficiently high
frame capture rate (e.g., 60 frames per second) and a sufficiently
high imaging resolution, one, two or more points (areas) of contact
of the optical waveguide are continuously sensed to sufficiently
track all initial contact, movement and discontinuation of contact
by one or more objects (including simultaneous and/or sequential
contact/movement), such as by the fingers of a user of the
multi-touch sensing display of the present invention.
[0140] The output of imaging sensor 20 preferably is supplied to a
suitable computer (not shown) or other electronic device capable of
handling various well-known image-processing operations, such as
rectification, background subtraction, noise removal, and analysis
for each video frame. Well-known machine vision tracking techniques
then may be employed to translate the video sequences into discrete
touch events and strokes. An imaging sensor that captures the light
at 8-bit monochrome at 60 frames per second at a resolution of
640.times.480 (corresponding to 1 mm.sup.2 precision on the
surface) is suitable for many multi-touch sense applications. Of
course, an imaging sensor having greater resolution, a different
frame capture rate and/or other characteristics may be employed.
Processing may be carried out by any suitable computing system.
[0141] Multi-touch sensing in accordance with the present invention
provides full imaging touch information without occlusion or
ambiguity issues. The touch sense is zero-force and true, that is,
it accurately discriminates touch from a very slight hover. The
multi-touch sensing display of the present invention is capable of
sampling at both high temporal and spatial resolutions. The
multi-touch sensing display is scalable to relatively large
surfaces, such as a wall-sized touch display, although various
factors including sensor/camera resolution and amount of
illumination should be taken into account for the multi-touch
sensing display to cover relatively large areas.
[0142] In one particular variation of the present invention, the
optical waveguide has a non-flat contact surface, i.e., non-planar.
The contact surface may be concave, convex or other non-flat
design. As one example, FIG. 4 shows an optical waveguide 32 having
a hemispherical shape suitable, for example, for terrestrial body
mapping control applications.
[0143] The multi-touch sensing display of the present invention can
be used standalone, but because it is completely visually
transparent, it is particularly well suited for use in combination
with rear-projection. For example, such a combination avoids the
disadvantages of occlusion and shadowing associated with top/front
projection. In accordance with another embodiment of the present
invention, a video projector 22 as shown in FIG. 2 may be employed
within the multi-touch sensing display of the present invention.
Although FIG. 2 shows projector 22 arranged alongside imaging
sensor 20, projector 22 may be disposed at other locations and/or
away from imaging sensor 20, generally with the aid of suitable
optics.
[0144] Along with projector 22, a suitable diffuser 18 is disposed
on the rear (non-contact) side of optical waveguide 12. Diffuser 18
is disposed alongside optical waveguide 12 with a small gap 24
between the two so that diffuser 18 does not frustrate the total
internal reflection of the light output by light source 14.
Moreover, diffuser 18 does not appreciably affect the IR image seen
by imaging sensor 20 since diffuser 18 is relatively close to the
sources of light (e.g. the user's fingers) being imaged. While this
scheme introduces a disparity between the display and interaction
surfaces, corresponding to the thickness of the waveguide
(one-quarter inch in the example), an optical waveguide having a
smaller thickness may be employed if necessary. In such case,
rigidity of a relatively large optical waveguide can be increased
by employing another layer of transparent material stacked to the
rear of the diffuser to add structural support without increasing
disparity.
[0145] Preferably, optical waveguide 12 includes an anti-reflective
coating on the non-contact side (projector side) to minimize
reduction in the brightness of the display output by projector
22.
[0146] The response of the multi-touch sensing display of the
present invention may be dependant on the optical qualities of the
object being sensed. For example, an article, such as a coffee mug,
lying on the contact side of the optical waveguide may not be
detected if such contact does not frustrate the total internal
reflection of the light. The present invention, however, may be
designed (further discussed below) so that FTIR is not dependent on
the type of material contacting the contact side of the optical
waveguide. In such case, the multi-touch sensing display of the
present invention sufficiently detects contact by, for example,
gloved hands (or dry skin), passive styluses and arbitrary
objects.
[0147] In accordance with another embodiment of the present
invention, a compliant surface overlay may be employed with the
multi-touch sensing display of the present invention. FIG. 5 of the
drawings is a simplified schematic illustration of a multi-touch
sensing display 40 employing a compliant surface overlay 48 (or
"compliant surface"). As shown, compliant surface 48 is disposed
adjacent the contact surface of optical waveguide 42. A small gap
54 is disposed between compliant surface 48 and optical waveguide
42 so that total internal reflection of the light output by light
source 44 is not frustrated (or is negligibly frustrated) when
there is no contact with the compliant surface by, for example, a
user's finger 60. On the other hand, when compliant surface 48 is
depressed at one location or simultaneously depressed at multiple
locations by, for example, one or more fingers of a user, the
compliant surface contacts the optical waveguide immediately below
(i.e., adjacent) such contacted locations thus frustrating total
internal reflection at such points of contact which, in turn,
causes (simultaneously) light to escape from the optical waveguide
at locations that correspond to where the compliant surface was
contacted. Imaging sensor 50 thereafter detects the escaped
light.
[0148] The compliant surface may be made of various plastic films
and other materials, including common vinyl rear-projection screen
material (e.g., Rosco Gray #02015). Various other compliant
surfaces in accordance with the present invention are discussed
further below.
[0149] The multi-touch sensing display employing a compliant
surface in accordance with the present invention advantageously is
immune to contaminants, such as oil and perspiration, which may be
deposited on the sensor over extended usage. That is, the existence
of oils, dirt, perspiration and other materials on the contact
surface of the compliant surface does not degrade or otherwise
impact frustrated total internal reflection upon depression of the
compliant surface. Likewise, scratches and nicks on the contact
surface of the compliant surface do not impact the sensing
capability of the multi-touch sensing display of the present
invention. For example, FIG. 6A shows an image of an output of a
contaminated surface in the absence of a compliant surface, whereas
FIG. 6B shows an image of an output when employing a compliant
surface. As shown, noise resulting from the contaminants is
completely removed in the case of when a compliant surface is
employed in the multi-touch system of the present invention.
[0150] The multi-touch sensing display employing a compliant
surface in accordance with the present invention advantageously
also now functions based on true force information rather than the
effectiveness of the touching object with respect to FTIR. This
allows the sensor to indiscriminately detect any object depressing
the surface. Thus, a user may utilize passive styluses, or use
gloved hands, pens, etc.
[0151] In an alternative embodiment, as further discussed below,
multiple infrared wavelengths are employed to better discriminate
the desired signal from background sources and noise.
[0152] The multi-touch sensing display employing a compliant
surface in accordance with the present invention, as discussed
above, may be utilized without a rear projector. In accordance with
another embodiment of the present invention, the multi-touch
sensing display employing a compliant surface also employs a rear
projector (e.g., projector 52 shown in FIG. 5). Preferably, and
advantageously, the compliant surface operates also as a diffuser
for the rear-projection. Hence, any disparities that result from
the use of multiple surfaces in a combined multi-touch sensor and
display system are eliminated by the present invention. Thus, this
embodiment of the present invention is a fundamentally unified
system for graphical display and for sensing.
[0153] In accordance with another embodiment of the present
invention, the compliant surface is comprised of a composite of
multiple materials, each generally contributing to one or more of
the following desired characteristics: i) FTIR effectiveness; ii)
function as an optical diffuser for rear-projection; iii)
wavelength selective shielding to mitigate interference from
external ambient light; iv) anti-glare to enhance visibility of the
display; v) the tactile "feel" for a human; and vi) durability--a
"hardcoat" wear layer preferably replaceable in the field. The
various layers employed may be affixed to one another using
well-known index-of-refraction matched optical adhesives.
[0154] As one example of a compliant surface comprised of a
composite of multiple materials, a stack includes (1) a thin layer
of rubber, (2) a thin-film PET (polyethylene terephthalate) film
with a metal coating, and (3) a thin PET film chemically treated to
have a matte surface. The thin layer of rubber provides for FTIR
contact, operates as the diffuser for rear-projection, and also
provides a comfortable tactile response. The thin-film PET
(polyethylene terephthalate) film with the metal coating
reflects/absorbs ambient infrared light. The thin PET film treated
to have a matte surface provides for a comfortable surface on which
a user's finger or fingers can easily glide across, and for
durability.
[0155] The compliant surface may be non-flat, i.e., non-planar. It
may concave, convex or have another non-flat design. Similar to the
non-flat optical waveguide shown in FIG. 4, FIG. 7 shows an
exemplary non-flat optical waveguide 62 on which a non-flat
compliant surface overlay 64 is disposed. As another variation, a
non-flat compliant surface overlay may be disposed over a flat
optical waveguide.
[0156] In accordance with a further embodiment of the present
invention, any of the herein-described embodiments and variations
may employ a light source (e.g., LED 14 shown in FIG. 2 or LED 44
shown in FIG. 5) that is pulsed and synchronized to the shutter of
the imaging sensor (e.g., video sensor 20 shown in FIG. 2 or video
sensor 50 shown in FIG. 5), to beneficially reduce the amount of
ambient light received by the imaging sensor. That is, the imaging
sensor's shutter is only exposed to the pulse period of the light
source. As a feature of the present invention, the light source can
be pulsed at a brighter intensity to increase the signal to noise
ratio of the system. Other types of light sources (e.g. lasers) may
be used as well.
[0157] In another embodiment of the present invention, multiple
imaging sensors and multiple projectors are employed, generally to
allow for the implementation of a relatively large multi-touch
system (e.g., for use by multiple users simultaneously). FIG. 8
shows a schematic illustration of an exemplary system 70 employing
three imaging sensors 72a, 72b and 72c, along with three projectors
74a, 74b and 74c. In one version, as shown in FIG. 8, each sensor
images light escaping a different portion of optical waveguide 76
and each projector projects a respective video image onto one of
those portions. The imaging sensors and projectors may be spaced
apart from one another along a single axis, multiple axes, along a
grid system, or other suitable manner. In the exemplary arrangement
shown in FIG. 8, a compliant surface 78 is employed, but multiple
sensors and/or multiple projectors may be employed in a system
without a compliant surface overlay, or in any other embodiment
described herein.
[0158] In a further embodiment of the present invention, an LCD
display panel is used in place of a projector. FIG. 9 shows a
simplified schematic illustration of a system 80 with a multi-touch
sensor employing an LCD panel 82 disposed between an optical
waveguide 84 and an imaging sensor 86. LCD panel 82 is
adapted/designed to be transparent to infrared light so that
infrared light that escapes optical waveguide 84 is imaged by
imaging sensor 86. Since LCD panels are well known, further
description thereof is omitted herein except where otherwise
necessary for an understanding of the present invention. An LCD
backlight 90 may be disposed behind LCD panel 82. A compliant
surface 88 also may be utilized and, in such case, the compliant
surface preferably is adapted/designed to not have optical diffuser
properties. Generally, use of an LCD display panel advantageously
reduces volume and increases portability.
[0159] In yet a further embodiment of the present invention, one or
more additional image sensors are also placed behind the LCD panel.
The images from these sensors are processed by computing machines
to determine extra information about the touch points and/or the
user--e.g. the pose or identification of the user's finger
associated with each touch.
[0160] In yet a further embodiment of the present invention, two
wavelengths of light, preferably infrared light, are output from
the light source and two imaging sensors are employed, such as
schematically illustrated in FIG. 10. In the illustrative
embodiment, two sets of LEDs 102a and 102b are employed in system
100. LED 102a, which may be an array of LEDs or another type of
light source, emits light at, for example, 880 nm. Similarly, LED
102b, which also may be an array of LEDs or another type of light
source, emits lights at, for example, 950 nm. Imaging sensors 104a
images light at the first wavelength (e.g., 880 nm) and imaging
sensor 104b images light at the second wavelength (e.g., 940 nm).
Suitable filters may be employed. As an optional feature of the
present invention, light must be received by both imaging sensors
(e.g., at the same time and location) for the system to acknowledge
the occurrence of a contact (i.e., FTIR response) at such
time/location. Well-known processing methods may be employed to
process both video streams in this manner. Thus, by employing
multiple wavelengths and multiple imaging sensors, FTIR response is
further discriminated from background light. Moreover, a live
finger is discriminated from latent residues in the event a
compliant surface is not utilized. Three or more wavelengths may be
employed. In a variation, a single imaging sensor is employed and
designed to image light at multiple selective frequencies or
frequency ranges. Multiple wavelengths of light, along with one or
more imaging sensors, may be employed in the various systems
described herein, including systems that employ a compliant surface
overlay as well as those that do not.
[0161] In yet another embodiment of the present invention, an
electro-switchable diffuser screen is employed with a projector. An
LC (liquid crystal) privacy glass window capable of being
electrically switchable from a transparent state to a frosty
appearance under electronic control is employed. Such an
electro-switchable screen, is disclosed in Kunz, A. M. and Spagno,
C. P., "Technical System for Collaborative Work. In Proceedings of
the Workshop on Virtual Environments" (Barcelona, Spain, May 30-31,
2002); and in W. Sturzlinger and S. Muller, Eds. ACM International
Conference Proceeding Series, vol. 23. Eurographics Association,
Aire-la-Ville, Switzerland, 73-80, each of which is incorporated
herein by reference.
[0162] The LC privacy glass window is switched between the two
states at a relatively rapid rate (e.g., 60 times a second) and is
synchronized to switch with the imaging sensor (e.g., infrared
camera) so that a second (or third) sensor view see through the
screen, such as to observe the touching object, when the LC privacy
glass window is momentarily transparent. If a compliant surface is
employed, it is chosen/engineered to not be diffusive. The images
from the additional sensors are processed by computing hardware to
determine other information about the touches--e.g. the orientation
or pose of the user's hand. In a variation, the diffuser can be a
directional-type diffuser that is designed to diffuse for selected
angles of incidence, and to not diffuse for other angles of
incidence.
[0163] In yet a further embodiment of the present invention,
various other diffusers may be employed to allow additional sensors
to view through the screen clearly while simultaneously diffusing
light from rear-projection. An exemplary screen includes the
holographic film employed by TouchLight, discussed in Wilson, A.
D., "TouchLight: An Imaging Touch Screen and Display for
Gesture-Based Interaction," Proceedings of the 6th International
Conference on Multimodal Interfaces (State College, Pa., USA, Oct.
13-15, 2004), ICMI '04, ACM Press, New York, N.Y., 69-76 (2004),
incorporated herein by reference. A less expensive, directionally
scattering film is discussed in Matsushita, M., Iida, M., Ohguro,
T., Shirai, Y., Kakehi, Y., and Naemura, T., "Lumisight Table: A
Face-to-face Collaboration Support System That Optimizes Direction
of Projected Information to Each Stakeholder," Proceedings of the
2004 ACM Conference on Computer Supported Cooperative Work
(Chicago, Ill., USA, Nov. 6-10, 2004), CSCW '04, ACM Press, New
York, N.Y., 274-283 (2004), which is incorporated herein by
reference. A Rayleigh-scattering material may be employed that that
diffuses visible wavelengths, but is substantially transparent to a
convenient infrared band.
[0164] The present invention has been described in the context of a
number of embodiments, and for various ones of those embodiments, a
number of variations and examples thereof. It is to be understood,
however, that other expedients known to those skilled in the art or
disclosed herein may be employed without departing from the spirit
of the invention.
* * * * *