U.S. patent application number 12/691506 was filed with the patent office on 2017-07-06 for method, device and program for browsing information on a display.
This patent application is currently assigned to MOTIONIP, LLC. The applicant listed for this patent is Manne Hannula, Johannes Vaananen. Invention is credited to Manne Hannula, Johannes Vaananen.
Application Number | 20170192729 12/691506 |
Document ID | / |
Family ID | 8561217 |
Filed Date | 2017-07-06 |
United States Patent
Application |
20170192729 |
Kind Code |
A9 |
Vaananen; Johannes ; et
al. |
July 6, 2017 |
METHOD, DEVICE AND PROGRAM FOR BROWSING INFORMATION ON A
DISPLAY
Abstract
In one embodiment, a method and program for browsing information
on a hand-held device having a display is provided. The device
includes (1) showing on the display a portion of the page residing
around the predefined point and having a shape similar to the shape
of the display, (2) generating a mirror line by mirroring the
reference line in relation to a line that is perpendicular to the
display surface and travels via the reference point in response to
tilting of the hand-held device in relation to the spatial initial
state, (3) defining a hit point (xn,yn) where the mirror line hits
the virtual surface and the page containing information, and (4)
showing on the display at least a portion of the page around the
hit point, said portion to have a shape similar to the shape of the
display, the position of the hit point on the page to correspond to
the position of the reference point on the display.
Inventors: |
Vaananen; Johannes; (Oulu,
FI) ; Hannula; Manne; (Kempele, FI) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Vaananen; Johannes
Hannula; Manne |
Oulu
Kempele |
|
FI
FI |
|
|
Assignee: |
MOTIONIP, LLC
Morrisville
NC
|
Prior
Publication: |
|
Document Identifier |
Publication Date |
|
US 20100125818 A1 |
May 20, 2010 |
|
|
Family ID: |
8561217 |
Appl. No.: |
12/691506 |
Filed: |
January 21, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12569797 |
Sep 29, 2009 |
|
|
|
12691506 |
|
|
|
|
11159786 |
Jun 23, 2005 |
7607111 |
|
|
12569797 |
|
|
|
|
10071172 |
Feb 8, 2002 |
|
|
|
11159786 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/1407 20130101;
G09G 2320/0261 20130101; G06F 2200/1614 20130101; G06F 1/1686
20130101; G06F 1/1626 20130101; G06F 1/1694 20130101; G06F
2200/1637 20130101 |
International
Class: |
G06F 3/14 20060101
G06F003/14; G09G 5/00 20060101 G09G005/00; G08B 21/00 20060101
G08B021/00 |
Foreign Application Data
Date |
Code |
Application Number |
May 16, 2001 |
FI |
20011039 |
Claims
1. A hand-held device for browsing information, the handheld device
comprising and being adapted to: show on the display a portion of
the page residing around the predefined point and having a shape
similar to the shape of the display, generate a mirror line by
mirroring the reference line in relation to a line that is
perpendicular to the display surface and travels via the reference
point in response to tilting of the hand-held device in relation to
the spatial initial state, define a hit point (xn,yn) where the
mirror line hits the virtual surface and the page containing
information, and show on the display at least a portion of the page
around the hit point, said portion to have a shape similar to the
shape of the display, the position of the hit point on the page to
correspond to the position of the reference point on the
display.
2. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to set the initial state at a certain
predefined angle in relation to the earth surface.
3. The hand-held device according to claim 1, wherein the
predefined angle is between 20-30 degrees.
4. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: browse the information on the display
at the speed which depends on the location and orientation of the
display surface with reference to the virtual surface.
5. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: filter out unintentional movements
from a coordinate axial and/or tilting movements before showing the
portion of the page on the display.
6. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: keep the orientation of the
information displayed unchanged when rotating the hand-held device
around an axis being essentially perpendicular to the display
surface.
7. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: lock/unlock the portion of the page
displayed in response to the pressing of a button.
8. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: set the hand-held device into a zoom
mode and zoom in or out the displayed information when rotating the
hand-held device around the axis, an axis being essentially
perpendicular to the display surface.
9. The hand-held device according to claim 1, wherein the hand-held
device is further adapted to: set the hand-held device to a zoom
mode; and zoom in or out the displayed information based on the
tilting of the hand-held device.
10. The hand-held device according to claim 1, wherein the
hand-held device is further adapted to: determine with a camera a
distance between the display surface and a second reference point
related to a user; and zoom in or out the displayed information
based on the distance.
11. The hand-held device according to claim 1, wherein the
hand-held device is further adapted to: measure the orientation and
location of the hand-held device in relation to a user of the
hand-held device with a video camera, seek a certain point related
to the user on a video image to be set as a second reference point;
and change the size of the information according to the movements
of the second reference point in relation to the hand-held
device.
12. The hand-held device according to claim 11, wherein the
hand-held device is further adapted to: use heuristic algorithms
and/or a neural network to seek and define the location of the
point to be used as the second reference point.
Description
RELATED APPLICATION
[0001] This application is a continuation of U.S. Pat. No.
7,607,111, issued Oct. 20, 2009, which is a continuation-in-part of
abandoned U.S. patent application Ser. No. 10/071,172, filed Feb.
8, 2002, which claims the benefit of and priority to Finnish Patent
Application Ser. No. 2001/1039, filed May 16, 2001, the contents of
all of which are incorporated by reference herein in their
entirety.
BACKGROUND
[0002] The present invention relates to display devices where
information can be browsed. In particular, the present invention
relates to a novel and improved method and system for browsing
information with hand-held devices with a display device.
[0003] Various electronic mobile devices, e.g. mobile phones,
computers, Personal Digital Assistants (PDA, comprise displays. The
transfer of the information to be viewed on the display is executed
at least partially by a processor. A device typically comprises
also a keypad with which the user of the device enters various
commands. There are also touch-sensitive displays (touch screens).
There a separate keypad is not needed. A device is controlled by
touching the touch screen.
[0004] The display of a mobile device is capable of showing only
limited amount of information at a time. Because of the size of the
display, e.g. a large image must be viewed part by part. In order
to view such an image, the user of the device controls the display,
e.g. by scrolling the display with a mouse etc.
[0005] Devices equipped with a display have different kinds of user
interfaces with which the user interacts with the device. There are
graphical user interfaces and speech controlled user interfaces. A
graphical user interface can be controlled with various control
devices including, for example, keypad, touch screen, different
kinds of cursor controlling methods, etc.
[0006] There are, however, drawbacks in the prior-art devices in
the usability of the device, especially in the browsing of
information with the device. When the information to be viewed on
the display must be viewed by parts, it is difficult and slow to
browse the whole information part by part. It is, for example,
difficult to display a wide panorama picture on the display, while
at the same time quickly and easily browsing the picture.
[0007] For the user of a mobile hand-held device it is difficult to
perceive visual entireties that can not be displayed at a time on
the display. Therefore the browsing of the information should be
carried out as naturally and logically as possible. A user of a
mobile hand-held device must be able to learn and use the device
easily and efficiently.
[0008] From prior-art solutions it is known to use location
detectors for browsing information with a device. Reference
publication WO 9918495 (Telefonaktiebolaget L M Ericsson) describes
a method where the display device is moved essentially in the plane
of the display device, whereby different parts of a complete screen
image are shown on said display device. When the display device is
moved essentially in a direction perpendicular to the plane of the
display device, the magnification of the screen image changes. The
movement in the plane is a bit problematic. In the plane movement
the necessary movements may be quite remarkable/large, and it may
be difficult to maintain the display device in a proper position
for reading or browsing.
[0009] Another prior-art solution is to use tilt detectors for
moving, or to be more specific, for scrolling the view on the
display device. One solution of this kind is described in WO
9814863 (Philips). When the screen image is moved by scrolling
(tilting the display device), the result is better than in moving
the display device in the plane of the display device, as described
above. However, to move the screen image fluently and to return
from some point to the initial point of browsing is difficult
because controlling a discontinuous motion requires continuous and
precise handling of the display device. The controlling of the
scrolling movement can be compared to a movement of a ball on a
plane surface by tilting the plane. In order to stop the rolling of
the ball, the plane surface must be perpendicular against the
gravity of the earth. In other words, the control of the movements
and usability are not at an acceptable level so that the use of
such a device would be natural and logical.
[0010] There are also various kinds of motion and/or location
controlled display devices used in, e.g. in virtual helmets. There
the display device focuses like a virtual camera. The display
device displays an object to which the device (camera) points in
the modeled virtual environment. To use a virtual camera model in a
hand-held device is not so straightforward because displaying
peripheries of a large screen image results in a disadvantageous
viewing angle. Therefore, the adjustment and zooming of a display
image must be implemented in a most natural and logical manner. In
prior-art solutions the browsing of information on the display
device is slow and awkward because the solutions are based on
artificial logic.
SUMMARY
[0011] An objective of the present invention is to adjust the view
on a display device in a manner as natural as possible so that the
user of the hand-held device can concentrate on the information
displayed on the display device and not on the adjustment of the
displayed information.
[0012] The objective is achieved by a method, hand-held device and
computer program for browsing information on a display device of a
hand-held device. In the present invention, the display device is
coupled to a processor mapping the information content generated by
the processor into the virtual data object suitable for conveying
the information to the user of the hand-held device. The display
device displays a portion of the virtual data object at a time on
the display device. The virtual data object comprises e.g.
characters, pictures, lines, links, video or pixels that can be
conveniently displayed on the display device at a time.
[0013] The idea of the present invention is to browse information
on the display device of a hand-held device naturally and
logically. Characteristic of the invention is that information is
browsed on the display device essentially in a mirror-like way. In
other words, the portion of the virtual data object displayed on
the display device is moved at the same direction as the hand-held
device is tilted. In other words, the movements of the portion of
the virtual data object displayed on the display device depend on
the orientation of the hand-held device. An important feature of
the invention is also that a certain orientation of the hand-held
device always displays the same portion of the virtual data object
on the display device. The browsing method described above is
extremely logical, and the movements and responses to the movements
are natural.
[0014] The core functions of the browsing can be explained by means
of the following example. The information is browsed with the
hand-held device essentially in the same way as looking at a view
from a hand mirror. The hand mirror is typically held in hand quite
close to the viewer. The hand mirror represents the display device
and the view behind the viewer the virtual data object. When the
hand mirror is tilted, the view behind the viewer moves in response
to the changes in the orientation of the hand mirror.
[0015] When approaching the functionality of a hand mirror the
browsing of information on a display device of a hand-held device
is made natural and logical.
[0016] The present invention is most applicable with hand-held
devices with a display when a large data object is displayed by
parts on the display. With the present invention, a large data
object can be browsed naturally and logically from the user's
perspective. The position memory of the muscles of a human body
makes it easier to return to previously browsed points and to the
starting point.
[0017] The present invention also reduces the need to use exterior
mechanical switches, keypad or other known control mechanisms for
browsing information on the display device. Therefore the use of a
hand-held device is easier and simpler. The basic functionalities
of the present invention can be implemented with mass production
components, and with moderate processing power. Thus, the features
described in the present invention can be taken in use in consumer
products without notable expense increase.
[0018] Other objects and features of the present invention will
become apparent from the following detailed description considered
in conjunction with the accompanying drawings. It is to be
understood, however, that the drawings are designed solely for
purposes of illustration and not as a definition of the limits of
the invention, for which reference should be made to the appended
claims. It should be further understood that the drawings are not
necessarily drawn to scale and that, unless otherwise indicated,
they are merely intended to conceptually illustrate the structures
and procedures described herein.
DRAWINGS
[0019] The above-mentioned features and objects of the present
disclosure will become more apparent with reference to the
following description taken in conjunction with the accompanying
drawings wherein like reference numerals denote like elements and
in which:
[0020] FIG. 1 illustrates how the hand-held device is operated
according to the present invention,
[0021] FIGS. 2a, 2b and 2c illustrate more specific examples of how
the hand-held device of FIG. 1 is handled,
[0022] FIG. 3 illustrates an exemplary viewing setup of the present
invention,
[0023] FIG. 4 illustrates an example of how a view on the display
device can be formed and calculated according to the viewing setup
of FIG. 3,
[0024] FIG. 5 is a block diagram illustrating an embodiment of the
hand-held device in accordance with the present invention,
[0025] FIG. 6 is a block diagram illustrating another embodiment of
the hand-held device in accordance with the present invention,
[0026] FIGS. 7a, 7b, 7c and 7d illustrate the view change of the
display of the hand-held device in response to user actions,
[0027] FIGS. 8a, 8b and 8c illustrate different ways of browsing
information,
[0028] FIG. 9 is a flow diagram illustrating the operation of a
preferred embodiment of the present invention, and
[0029] FIGS. 10a-10d illustrate another example of how a view on
the display device can be formed and calculated according to the
viewing set up of FIG. 4.
DETAILED DESCRIPTION
[0030] FIG. 1 illustrates a simplified portable hand-held device
according to the present invention. The hand-held device is e.g. a
mobile phone or a Personal Digital Assistant (PDA). The display
device of the hand-held device displays information stored on a
memory of the hand-held device. The hand-held device is explained
more specifically in later examples. FIG. 1 represents the basic
browsing functionality. Information is browsed on the display
device by tilting (rotating) the hand-held device 40 towards
directions 2, 3, 4, and 5 around the axis 6 and 7. The memory of
the hand-held device 40 comprises a virtual data object comprising
characters, pictures, lines, links, video or pixels that can be
conveniently displayed on the display device at a time. A portion
of the virtual data object displayed on the display device is moved
at the same direction as the hand-held device is tilted. Moreover,
a certain orientation of the hand-held device 40 always displays
the same portion of the virtual data object on the display
device.
[0031] FIGS. 2a, 2b and 2c represent a more specific example of
tilting the hand-held device 40. It can be said that a typical
starting situation is that the hand-held device 40 is in a 20-30
degree angle with the horizontal plane 8. This plane is in one
embodiment set as a default xy-plane from which the rotation angles
of the hand-held device 40 are measured. It can also be said that
this starting point is the most appropriate one for viewing
information with the display device. So when the user tilts the
hand-held device 40, the viewing angle changes. The view on the
display device changes in real time to correspond to the new
viewing angle. A very important feature of the invention is that
the view on the display device depends on the viewing angle, and
the same viewing angle displays always the same view on the display
device. This feature is very natural and logical.
[0032] In FIG. 2a, angle a corresponds to the aforementioned 20-30
degrees. FIG. 2a is regarded as a starting position when the
browsing begins. In FIG. 2b, the hand-held device 40 has been
tilted to an angle .beta..sub.1, which is smaller than angle
.alpha.. The view on the display device changes based on the
tilting movements essentially in real time, and the movement of the
information on the display device is towards the same direction as
the hand-held device 40 is tilted. In FIG. 2c, the hand-held device
40 is tilted to an angle .beta..sub.2, which is bigger than angle
.alpha..
[0033] In one embodiment, the angle (.alpha.) is a predetermined
angle, and it is determined by the manufacturer of the hand-held
device 40. In the determination process it is defined that the
display view plane is based on axis x_VD and y_VD, which are
perpendicular to each other. The hand-held device is then set to a
certain position (.alpha.), and that position is set as a default
xy-plane. In FIGS. 2a, 2b and 2c, the default plane is determined
based on angle a. In another embodiment, the default plane can be
freely determined based on any x-axis, y-axis and/or z-axis.
[0034] From that moment on, the hand-held device 40 is tilted in
respective to this plane. When the default xy-plane is fixed, the
user of the hand-held device is always capable of returning to a
certain view by tilting the device back to the original orientation
when the sensors measuring the orientation of the hand-held device
do not cause any restrictions to the measured position. In another
embodiment, the angle .alpha. can be readjusted to a desired
value.
[0035] FIGS. 3 and 4 represent an exemplary embodiment of the setup
of a "mirroring system". It includes a viewpoint VP, a virtual
screen VS and a virtual display VD. The viewpoint VP represents the
location of a viewer of a hand-held device. The VD represents the
display device of the hand-held device. The virtual screen
represents the actual information browsed on the display
device.
[0036] For simplicity in the following the viewpoint VP is defined
to be at point [0 0 0]. Furthermore, the middle point of the
virtual display VD is defined to be at P_xyz wherein
P_xyz=[P_xyz.sub.1 P_xyz.sub.2 P_xyz.sub.3].sup.T, and the virtual
screen VS to be at plane x=kuva_shift.
[0037] The orientation of the virtual display VD is defined by
tilting angels .alpha..sub.x, .alpha..sub.y, .alpha..sub.z
indicating rotation angle over each coordinate axe. In FIG. 4, the
virtual display VD is a plane and has some size. Each coordinate in
this VD plane is defined using notation P=[P_xyz.sub.2+peili_y
P_xyz.sub.3+peili_z] when the orientation of the VD is defined to
be parallel with the x-plane.
[0038] It must be noted that FIGS. 3 and 4 represent only one
embodiment of the possible positions of the VS, VP and VD, and the
axes used.
[0039] In order to the determine the orientation of the VD, two
orthogonal vectors (in the x-plane) are defined as follows:
L=[0,1 ,-1].sup.T
M=[0,1 ,1].sup.T
[0040] Those vectors present the orthogonal direction vectors of
the VD. Next, the orientation of the virtual display VD is defined
using the rotation angles:
R x = [ 1 0 0 0 cos ( .alpha. x ) - sin ( .alpha. x ) 0 sin (
.alpha. x ) cos ( .alpha. x ) ] ##EQU00001## R y = [ cos ( .alpha.
y ) 0 sin ( .alpha. y ) 0 1 0 - sin ( .alpha. y ) 0 cos ( .alpha. y
) ] ##EQU00001.2## R z = [ cos ( .alpha. z ) - sin ( .alpha. z ) 0
sin ( .alpha. z ) cos ( .alpha. z ) 0 0 0 1 ] ##EQU00001.3##
[0041] Next the unit normal vector of the VD is calculated:
PT 1 = R x R y R z L ##EQU00002## PT 2 = R x R y R z M
##EQU00002.2## PNT = PT 1 .times. PT 2 ( cross product )
##EQU00002.3## PN = PNT PNT ##EQU00002.4##
[0042] where PN is the unit normal vector of the VD-plane. The PN
defines the applicable orientation of the VD to be used in the
projection calculation.
[0043] Next, the "image" on the virtual display VD is calculated.
Let's assume that there is a vector beginning from the VP and being
reflected via the VD. The point where the reflected vector hits on
the plane VS defines the projection of the point on the VS to the
point on the VD-plane. Hence, if all points on VD are processed as
described above, the image on the VD can be defined.
[0044] The idea of calculation is presented using vectors in FIG.
4. Using the vectors the algorithm works as follows:
[0045] 1. The points P and VP define a vector A.
[0046] 2. The projection proj of the vector A on the normal vector
PN is calculated.
[0047] 3. The sum of the vector A and proj*PN defines a point
Q.
[0048] 4. The points Q and VP define a vector B.
[0049] 5. The point defined as sum of the VP and 2*B defines a
point R.
[0050] 6. The direction vector that goes via P and R defines a
direction vector that hits the plane VS at point S.
[0051] 7. The result of this process is that the image of point P
in VD is the image of point S in VS.
[0052] By repeating phases 1-7 for all points in the VD-plane the
whole image of the virtual display VD is defined. Using vector
calculation the same can be presented as follows:
[0053] First the point P is defined:
P=P_xyz+R.sub.xR.sub.yR.sub.z[0peili_ypeili_z].sup.T
[0054] where P_xyz is the coordinate of the middle point of the VD,
peili_y is the y-coordinate on the VD plane-coordinate system and
peili_z is the z-coordinate on the VD plane-coordinate system
[0055] Next, the projection on the normal vector is defined:
A = P - VP ##EQU00003## proj = A PN PN ##EQU00003.2##
[0056] Hence the point Q can be defined:
Q=P-prof*PN
[0057] Further, the point R can be defined (the reason for the
factor 2 is that in mirror the arriving and departing light beam
have equal angles compared to the normal vector of the
surface).
B=Q-VP
R=VP+2*B
[0058] And finally the direction vector C is defined as
follows:
C=R-P.
[0059] Because the VS is located at plane x=kuva_shift, the vector
C hits that plane at the point
S = k * C + P ##EQU00004## where ##EQU00004.2## k = - P 1 +
kuva_shift C 1 ##EQU00004.3##
where P.sub.1 is the x-component of the point P and C.sub.1 is the
x-component of the vector C. Note that in this calculation the VP
was defined to the origin to simplify the presentation of the
algorithm. However, in practice the VP can locate freely in the
coordinate space. It must be noted that the image on the virtual
screen VS is horizontally inversed when the virtual screen VS is
viewed from the viewpoint VP direction.
[0060] The system of FIG. 4 has several characteristics:
[0061] 1. The view on the display device moves into the same
direction as it is tilted. In one embodiment, the movement of the
portion of the virtual data object displayed on the display device
is proportional to the change amount and/or rate of the rotational
movement.
[0062] 2. When the distance between the VP and VD increases, the
same tilting angle causes greater movements on the virtual screen
VS. In other words, the browsing speed of the information on the
display device increases as the distance between the VP and VD
increases. In one embodiment, this movement factor can be adjusted
by the user of the hand-held device.
[0063] 3. When rotating the display device, the view on the display
device remains unchanged in relative to the user.
[0064] 4. The view on the display device depends on the position
and orientation of the VS, VP and VD.
[0065] 5. A certain VS-VP-VD position/orientation combination
always constitute the same view on the display device.
[0066] 6. When the position of the VD alters, the viewing angle
between the VP and VD changes.
[0067] 7. Zooming can be implemented by changing the position of
the VS, VP and VD.
[0068] 8. Zooming can be implemented by enlarging the object on the
VS or altering the radius of curvature of the mirror (VD).
[0069] 9. If the figure on the VS is in the right way when viewed
from the VP, the view on the VD is mirrored (horizontally
inversed).
[0070] The present invention does not have to implement all the
aforementioned features, but the most appropriate ones can be
chosen. The ideal mirror-like functionality means that the
information on the display device changes when:
[0071] a) the location or orientation of the hand-held device in
proportion to the coordinates bound to the physical environment
changes,
[0072] b) the location of the user (VP) in proportion to the
coordinates bound to the hand-held device changes,
[0073] c) the virtual location of the data (virtual screen)
displayed on the display device in proportion to the coordinates
bound to the physical environment changes.
[0074] In order to simulate the operation of a mirror to the user,
the information on the display device is changed at least either
according to a) or b). If only a) or b) is taken into
consideration, the operation of the display is not so mirror-like
as if both a) and b) were implemented. In one embodiment, the
display device operates according to all a), b) and c).
[0075] FIGS. 10a-d illustrate another example of calculation which
is explained with reference to FIG. 4. FIG. 10c is a side view of
FIG. 10a and FIG. 10d is a side view of FIG. 10b. In this example
the virtual screen is referred to as the virtual surface 200.
[0076] In FIGS. 10a and 10c the default orientation of the display
201 is determined to be parallel with the yz-plane. The virtual
surface (VS) 200 is above the display plane and also parallel with
the yz-plane. A page having information to be browsed lies on the
virtual surface 200, and the size of the page is larger than the
size of the display 201.
[0077] The reference point VP is on the virtual surface 200. The
x-axis (not shown) runs through the reference point VP and the
middle point P of the display 201. After calculating the point S by
the method presented with reference to FIG. 4, the result is that
point S is equal to point VP. Of course, the relationship between
every single point in the area (2a*2b) of the display 201 and the
corresponding area (2a*2b) on the virtual surface 200 can be
calculated in a similar way. The portion of the page (2a*2b) that
is to be displayed then has a shape similar to the shape of the
display (2a*2b). In other words, on the point S on the virtual
surface 200 is the middle point of the determined rectangle 2a*2b
and all the other points residing around point S within the
rectangle relate to the corresponding the points residing around
point P on the display 201. That portion of the page surrounding
point S on the virtual surface is displayed on display 201.
[0078] In FIGS. 10b and 10d the display 201 has been tilted around
the y-axis, wherein the portion of the page shown on the display
201 changes in the following way:
[0079] Initially (i.e. when the virtual surface 200 and the display
surface 201 are parallel with respect to each other as shown in
FIGS. 10a and 10c) a reference line 203 drawn between point P and
point S meets the x-axis, i.e. it is parallel with the x-axis. The
normal of the display extending from point P is parallel with the
x-axis and the reference line 203. When display 201 is tilted by
angle a with respect to the virtual surface 200, the normal 204 of
the display is also tilted by angle a with respect to the x-axis.
After the display 201 is tilted as shown in FIGS. 10b and 10d, the
reference line 203 is mirrored with respect to the normal 204 of
the display wherein a mirror line 205 is generated. A hit point S'
is the point where the mirror line 205 hits the virtual surface
200. In the same manner as above, an area (shape) of the page
corresponding to the area (shape) of the display is determined. The
display 201 then shows the portion of the page around the hit point
S' and having a shape similar to the shape of the display 201.
[0080] FIG. 5 represents one example of a preferred hand-held
device 40. The hand-held device 40 is e.g. a mobile phone. The
hand-held device comprises a processor 30 and a display device 10
coupled to the processor 30. The data memory 60 and the program
memory 70 are also coupled to the processor 30. The program memory
70 contains e.g. the operation system. The sizes of the memories,
and the processing power of the processor 30 depend on the device
and application used. The program memory 60 can additionally
contain different kinds of software applications with which various
tasks can be executed. Application software comprise e.g. word
processing, graphical and spreadsheet software. The software
applications and data used by them are loaded into the data memory
60 in order to be able to use the software.
[0081] The display adapter 90 with the processor 30 controls the
display device 10. In order to not to use the data memory 60 for
storing display-related information, the display adapter 90
comprises a data buffer in which the information to be displayed on
the display device 10 is stored.
[0082] The hand-held device 40 comprises measuring means which in a
preferred embodiment of the invention refer to acceleration
sensor(s) 50. With the acceleration sensor(s) 50 it is possible to
measure tilting movements of the hand-held device 40. The processor
30 receives the measurement results and interprets them. The
acceleration sensor(s) 50 can be e.g. piezo-electric or capacitive
producing an analog voltage which is proportional to the
acceleration factor.
[0083] With the acceleration sensor(s) 50 it is possible to measure
one, two or three-dimensional accelerations. The measurement of
tilting movements is based on the fact that the highest
acceleration is parallel to the gravity of the earth. Therefore,
the orientation of the hand-held device 40 can be defined in
relation to the earth. It is also possible to use gyroscopes with
its various forms to measure the orientation of the hand-held
device 40. The quantities measured are e.g. tilting angle and
accelerations.
[0084] The relation information between the rotation degree of the
hand-held device and the memory address corresponding to the
displayed view is stored e.g. on the data memory 60. The processor
30 defines the orientation of the hand-held device 40 in relation
to the user or a reference position. The processor 30 may also
define the distance between the user and the hand-held device 40 or
the user orientation in relation to the hand-held device 40.
[0085] The most important point is not the way of how the
aforementioned definitions are made but the fact that the
orientation of the hand-held device 40 affects the information
displayed on the display device 10. The memory space can be
implemented logically, e.g. as a two-dimensional memory space. When
browsing starts, the processor 30 starts the definition process of
the new memory address from the current memory address so that
displacement in the memory space corresponds to the direction and
amount of change in orientation according to the relation
information.
[0086] The hand-held device 40 comprises also a browse lock 80 with
which it is signaled when the browsing is executed. The orientation
of the hand-held device 40 must remain in the same position in
order to keep the view on the display device unchanged. In a
preferred embodiment, the hand-held device 40 comprises a lock
feature, e.g. a push-button, with which the browsing can be locked.
The user can tilt the hand-held device back to an appropriate
viewing orientation in order to view the information on the display
device 10 properly. The browsing may then continue when the button
is released.
[0087] The hand-held device 40 in FIG. 6 is almost the same as the
hand-held device 40 in FIG. 5. In FIG. 5, the hand-held device
comprises also a locator 20. It is possible to control the view on
the display device 10 also by other means than acceleration
sensor(s) or equivalent means. The hand-held device 40 can comprise
e.g. a (video) camera measuring the orientation and location of the
hand-held device in relation to the user of the hand-held device 40
or to another reference point in the surroundings of the user. The
camera 20 may be set to recognize and measure distance to a certain
reference point, e.g. the eyes of the user. Therefore, when the
orientation and/or position of the hand-held device 40 changes, the
viewing angle measured by the camera also changes. Thus, it can be
concluded that the hand-held device 40 has been tilted and/or moved
towards some direction.
[0088] By analyzing the video image it is possible to define the
orientation of the hand-held device 40 in proportion to the
reference point and the distance of the hand-held device 40 to the
reference point tens of times within a second. The browsing
functionality can be implemented merely using the video camera, so
that additional acceleration sensor(s) are not necessarily needed.
The measuring of the distance can also be implemented with an
ultrasonic radar connected through an analog-digital converter to
the processor 30 of the hand-held device 40. In one embodiment,
from the user's perspective the information on the display device
10 is essentially browsed in the same manner as when looking in a
mirror. In other words, the view on the display 10 depends on the
viewing angle in relation to the display device plane as the view
in a mirror depends on the viewing angle to the mirror.
[0089] In one embodiment of FIG. 5, the locator 20 comprises a
video camera seeking the location of the head and eyes of the user.
Heuristic algorithms and neural network seeking the location of the
head and eyes can be used. Acceleration sensors are more
appropriate to use in hand-held devices than a video camera,
because they are cheaper. The acceleration sensors may also be a
more appropriate solution in devices which do not have a built-in
video camera for a default feature, e.g. in the (third generation)
mobile phones. The advantage of the use of the video camera is that
the use of the hand-held device is not restricted to the position
of the hand-held device, e.g. when being on one's back the
hand-held device can be used without problems. Also the selection
of starting point of browsing is more free, and choice (of the
starting point) can be given to the user of the hand-held device.
In one embodiment of FIG. 5, the display device surface level is
set as an xy-plane. A certain relation between the x-axial and/or
y-axial movement of the hand-held device and the amount of the
displacement of the portion of the virtual data object displayed on
the display device at a time has been determined. So, when the
hand-held device 40 is moved along x- and/or y-axis, the portion of
the virtual data object displayed on the display device moves in
the same direction as the hand-held device is moved in the xy-plane
according to the relation information.
[0090] In a preferred embodiment of FIGS. 5 and 6 the processor 30
comprises also means for filtering the x-axial, y-axial and/or
tilting movements before displaying the movements on the display
device. Therefore, minor unintentional movements can be filtered
out.
[0091] In one embodiment of FIGS. 5 and 6, the relation between the
tilting movements and the amount of the displacement of the portion
of the virtual data object displayed on the display device at a
time can be changed. Therefore, a user may define e.g. that from
now on a 10 degree tilting causes the same effect on the display as
a 15 degree tilting earlier. In one embodiment, the relation is
linear. In other words, the relation between the tilting movements
and the amount of the displacement of the portion of the virtual
data object displayed on the display device at a time does not
depend on the amount of the tilting. In another embodiment, the
relation is linear, but e.g. exponential. In other words, the
amount of the displacement of the portion of the virtual data
object displayed on the display device at a time depends on the
amount of the tilting. For example, the value of the relation
factor changes (e.g. exponentially) as the tilting amount
increases.
[0092] FIGS. 7a-7d represent the situation where the size of the
information on the display device depends on the zoom factor in
addition to the orientation of the hand-held device. The zoom
factor can be controlled in different ways. In one embodiment, the
zoom factor depends on the distance between the user and the
hand-held device. FIG. 7a represent the display device 10, on which
graphical FIGS. 21, 22 and 23 are seen. The view on the display
device 10 depends on the orientation of the hand-held device or the
viewing angle from which the user of the hand-held views the
display device. When the user of the hand-held device sets FIG. 21
in the middle of the display device, and the zoom factor is
increased, FIG. 21 grows as depicted in FIGS. 7b and 7c. In FIG.
7d, the zoom factor has decreased, and also the viewing angle
between the user and the hand-held device has changed.
[0093] The zoom factor can be modified with several different ways.
In one embodiment, the zoom factor depends on the distance between
the reference point (e.g. the eyes of the user) and the hand-held
device. When the distance decreases, FIG. 21 grows, and vice versa.
The display device 10 may have to be set to a zoom mode before the
zoom factor changes. If the zoom factor was all the time dependent
on the distance between the reference point and the hand-held
device, the browsing operation would not necessarily be practical
because the view on the display 10 would change whenever the
aforementioned distance changes.
[0094] In another embodiment, the zoom factor changes when rotating
the hand-held device around the axis being essentially
perpendicular to a predefined xy-plane. The xy-plane may be the
present plane of the display device 10 or some other predetermined
plane. Yet in another embodiment, the zoom factor is changed by
tilting the hand-held device. Before this the display device must
be set into a zoom mode. When the hand-held device is tilted, e.g.
to the right the zoom factor increases, and when the hand-held
device is tilted to the left, the zoom factor decreases. It is not
important which predefined tilting directions are used but that the
two directions can be separated sufficiently from each other. The
aforementioned zoom mode is set on and off e.g. with a
predetermined button of the hand-held device.
[0095] FIGS. 8a-8c represent different ways to implement the user
interface. In FIG. 8a, the display device 10 of the hand-held
device 40 contains information to be viewed by the user. In FIG.
8a, an A letter is on the display device 10, In one embodiment, the
information on the display device 10 remains in the same position
with respect to the user when the hand-held device 40 is rotated
around the axis being perpendicular to the display surface plane,
as depicted in FIG. 8b. In other words, the information on the
display device 10 remains in the same position because the
information is attached to the real physical coordinates.
[0096] In another embodiment, the information on the display device
10 remains in the same position with respect to the hand-held
device 40 when the hand-held device 40 is rotated around the axis
being perpendicular to the display surface plane, as depicted in
FIG. 8c. In other words, the orientation of the information on the
display device 10 changes with respect to the user of the hand-held
device 40 because the information is not attached to the real
physical coordinates but to the display device.
[0097] FIG. 9 represents a flow diagram describing the
functionality of a method of the present invention. FIG. 9
describes a hand-held device 40 comprising means for measuring
acceleration 50 and a processor 30. Means for measuring
acceleration refer e.g. to a multiaxial acceleration sensor suited
for measuring changes in the orientation of the hand-held device
40.
[0098] The hand-held device is switched on, and it is ready for
browsing information on the display device, as represented in phase
100. When the hand-held device is functional, the acceleration
sensor 50 measures constantly acceleration readings. The processor
30 receives the acceleration readings and defines the orientation
of the hand-held device and also the change in the orientation
compared to the prior measurement(s), as represented in phases 101
and 102. In phase 103, it is tested whether the browsing is on or
off. If the browsing is off, the processor 30 examines if a
predetermined browsing startup condition is fulfilled (phase 104).
If it is not fulfilled, the method returns back to phase 101. It
means that the orientation of the hand-held device has not changed
sufficiently, which would indicate that the user wishes to browse
information on the display device of the hand-held device.
[0099] If the predetermined browsing startup condition is
fulfilled, the processor 30 sets the browsing as started (phase
106) and determines the browsing speed based on the current
acceleration value (phase 108). The processor 30 also changes the
information presented on the display device according to a relation
between the rotation degree and the amount of the displacement of
the portion on the virtual data object stored on the data memory 60
and the determined browsing speed (phase 108). A certain
orientation of the hand-held device always causes the same view
(the same portion on the virtual data object stored on the memory)
on the display device. If it is observed in phase 103 that the
browsing is already on, and the browsing stopping condition is
fulfilled (phase 105), the processor 30 stops the browsing and sets
the browsing as stopped (phases 107 and 109). If it is observed
that the browsing stopping condition is not fulfilled (phase 105),
the processor 30 returns back to phase 101.
[0100] While the apparatus and method have been described in terms
of what are presently considered to be the most practical and
preferred embodiments, it is to be understood that the disclosure
need not be limited to the disclosed embodiments. It is intended to
cover various modifications and similar arrangements included
within the spirit and scope of the claims, the scope of which
should be accorded the broadest interpretation so as to encompass
all such modifications and similar structures. The present
disclosure includes any and all embodiments of the following
claims.
* * * * *