U.S. patent application number 14/842557 was filed with the patent office on 2015-12-24 for system and method to reduce bandwidth requirement for visibility event packet streaming using a predicted maximal view frustum and predicted maximal viewpoint extent, each computed at runtime.
This patent application is currently assigned to PRIMAL SPACE SYSTEMS, INC.. The applicant listed for this patent is PRIMAL SPACE SYSTEMS, INC.. Invention is credited to Barry L. Jenkins.
Application Number | 20150373153 14/842557 |
Document ID | / |
Family ID | 55022738 |
Filed Date | 2015-12-24 |
United States Patent
Application |
20150373153 |
Kind Code |
A1 |
Jenkins; Barry L. |
December 24, 2015 |
SYSTEM AND METHOD TO REDUCE BANDWIDTH REQUIREMENT FOR VISIBILITY
EVENT PACKET STREAMING USING A PREDICTED MAXIMAL VIEW FRUSTUM AND
PREDICTED MAXIMAL VIEWPOINT EXTENT, EACH COMPUTED AT RUNTIME
Abstract
There is provided a method of predictive prefetching and
transmitting from a server to a client device at least one partial
visibility event packet and/or deferred visibility event packet
including renderable graphics information occluded from a first
viewcell and not occluded from a second viewcell, including
otherwise renderable graphics information in a client view frustum
not previously transmitted to the client device; determining an
estimated maximal client view frustum; calculating a subset
comprising renderable graphics information that is included in the
estimated maximal client view frustum; determining whether the
calculated subset has previously been transmitted to the client
device by comparing the calculated subset to the stored renderable
graphics information previously transmitted; and transmitting the
at least one partial visibility event packet and/or deferred
visibility event packet to the client device if said packet has not
been previously transmitted to the client device.
Inventors: |
Jenkins; Barry L.; (Raleigh,
NC) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
PRIMAL SPACE SYSTEMS, INC. |
Raleigh |
NC |
US |
|
|
Assignee: |
PRIMAL SPACE SYSTEMS, INC.
Raleigh
NC
|
Family ID: |
55022738 |
Appl. No.: |
14/842557 |
Filed: |
September 1, 2015 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14754419 |
Jun 29, 2015 |
|
|
|
14842557 |
|
|
|
|
13445792 |
Apr 12, 2012 |
9171396 |
|
|
14754419 |
|
|
|
|
13420436 |
Mar 14, 2012 |
|
|
|
14754419 |
|
|
|
|
13420436 |
Mar 14, 2012 |
|
|
|
13445792 |
|
|
|
|
PCT/US2011/042309 |
Jun 9, 2011 |
|
|
|
13420436 |
|
|
|
|
PCT/US2011/051403 |
Sep 13, 2011 |
|
|
|
13420436 |
|
|
|
|
62044349 |
Sep 1, 2014 |
|
|
|
61360283 |
Jun 30, 2010 |
|
|
|
61382056 |
Sep 13, 2010 |
|
|
|
61384284 |
Sep 19, 2010 |
|
|
|
61452330 |
Mar 14, 2011 |
|
|
|
61474491 |
Apr 12, 2011 |
|
|
|
61476819 |
Apr 19, 2011 |
|
|
|
Current U.S.
Class: |
709/203 |
Current CPC
Class: |
G06T 2200/16 20130101;
H04L 45/08 20130101; G06T 15/20 20130101; A63F 13/355 20140902;
G09G 2350/00 20130101; G09G 2370/022 20130101; G09G 2360/122
20130101; G06T 2210/08 20130101; G06T 2210/36 20130101; G06F 3/1454
20130101; G06T 19/003 20130101; H04L 67/42 20130101; G06F 3/011
20130101; G06T 15/40 20130101; H04L 67/10 20130101 |
International
Class: |
H04L 29/06 20060101
H04L029/06; H04L 12/751 20060101 H04L012/751; H04L 29/08 20060101
H04L029/08 |
Claims
1. A method, conducted on a server, of predictive prefetching and
transmitting at least one partial visibility event packet from the
server to a client device, the at least one partial visibility
event packet being a subset of a complete visibility event packet,
the complete visibility event packet including renderable graphics
information occluded from a first viewcell and not occluded from a
second viewcell among a plurality of viewcells, the method
comprising: a) storing, using storage circuitry, information
representing a current client view frustum; b) storing, using the
storage circuitry, renderable graphics information previously
transmitted to the client device; c) determining, using a
processor, from the information representing the current client
view frustum, an estimated maximal client view frustum comprising a
volume of space intersected by a maximum possible movement of the
current client view frustum during a delay period of time at least
equal to a round-trip-time of data communication between the server
and the client device; d) calculating, using the processor, the
subset of the complete visibility event packet comprising
renderable graphics information that is included in the estimated
maximal client view frustum; e) determining, using the processor,
whether the calculated subset has previously been transmitted to
the client device by comparing the calculated subset to the stored
renderable graphics information previously transmitted to the
client device; and f) transmitting, to the client device, the at
least one partial visibility event packet comprising the calculated
subset of the complete visibility event packet, if the calculated
subset has not been previously transmitted to the client
device.
2. The method according to claim 1, wherein the information
representing the client view frustum includes a current client view
direction vector.
3. The method according to claim 2, wherein the information
representing the client view frustum includes a current client
viewpoint location.
4. The method according to claim 3, further comprising determining
the estimated maximal client view frustum by: c1) determining,
using the processor, a conservative maximal viewpoint extent
comprising a representation of a maximal possible subregion of the
second viewcell occupied by a client viewpoint during the delay
period; c2) determining, using the processor, a conservative
maximal view direction vector rotation comprising a representation
of a maximal rotational extent of the client view direction vector
during the delay period; and c3) calculating, using the processor,
the estimated maximal client view frustum from the conservative
maximal viewpoint extent and the conservative maximal view
direction vector rotation.
5. The method according to claim 4, wherein the information
representing the client view frustum includes a current velocity of
the current client viewpoint.
6. The method according to claim 4, wherein the information
representing the client view frustum includes a current
acceleration of the current client viewpoint.
7. The method according to claim 5, wherein the determining the
conservative maximal viewpoint extent is based on the current
client viewpoint location, the current velocity of the client
viewpoint, and a predetermined maximal value for viewpoint
velocity.
8. The method according to claim 4, wherein the information
representing the client view frustum includes a current angular
velocity of the current client view direction vector.
9. The method according to claim 4, wherein the information
representing the client view frustum includes a current angular
acceleration of the current view direction vector.
10. The method according to claim 9, wherein the conservative
maximal view direction vector rotation is based on the current
client view direction vector, the current angular velocity of the
current client view direction vector, and a predetermined maximal
value for the client view direction vector angular velocity.
11. The method according to claim 4, further comprising: g)
storing, using the storage circuitry, a complete data set for each
viewcell of the plurality of viewcells that is predictively
penetrated by the client viewpoint, as determined by the
conservative maximal viewpoint extent, the complete data set
representing a complete potentially visible set for said each
viewcell of the plurality of viewcells; h) storing, using the
storage circuitry, an incomplete data set for said each viewcell of
the plurality of viewcells that is predictively penetrated by the
client viewpoint, as determined by the conservative maximal
viewpoint extent, the incomplete data set representing, for said
each viewcell, a subset of the calculated subset actually
transmitted to the client device in step f); i) calculating, using
the processor, at least one deferred visibility event packet for
said each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, by comparing the
complete data set to the incomplete data set, the at least one
deferred visibility event packet including, for said each viewcell,
a subset of the complete potentially visible set not previously
transmitted to the client device; j) determining, using the
processor, for said each viewcell of the plurality of viewcells
that is predictively penetrated by the client viewpoint, whether
renderable graphics information of the at least one deferred
visibility event packet intersects the estimated maximal client
view frustum; and k) transmitting, to the client device, the at
least one deferred visibility event packet when the renderable
graphics information of the at least one deferred visibility event
packet intersects the estimated maximal client view frustum.
12. A method, conducted on a client device, of receiving at least
one partial visibility event packet from a server, the at least one
partial visibility event packet being a subset of a complete
visibility event packet, the complete visibility event packet
including renderable graphics information occluded from a first
viewcell and not occluded from a second viewcell among a plurality
of viewcells, the method comprising: a) transmitting to the server,
using a processor, client view information representing a current
client view frustum; b) transmitting to the server, using the
processor, data comprising at least one of a current client view
direction vector and a current client viewpoint location; and c)
receiving, from the server, the at least one partial visibility
event packet comprising renderable graphics information that
intersects a maximal client view frustum, wherein the maximal
client view frustum comprises a volume of space intersected by a
maximum possible movement of the current client view frustum, based
on said at least one of the current client view direction vector
and the current client viewpoint location, during a delay period of
time at least equal to a round-trip-time of data communication
between the server and the client device.
13. The method according to claim 12, further comprising: d)
receiving, from the server, at least one deferred visibility event
packet for said each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, the at least one
deferred visibility event packet including, for said each viewcell,
a subset of the complete potentially visible set not previously
received from the server, wherein the subset includes renderable
graphics information of the at least one deferred visibility event
packet that intersects the maximal client view frustum.
14. A method, conducted on a client device, of receiving at least
one partial visibility event packet from a server, the at least one
partial visibility event packet being a subset of a complete
visibility event packet, the complete visibility event packet
including renderable graphics information occluded from a first
viewcell and not occluded from a second viewcell among a plurality
of viewcells, the method comprising: a) determining, using a
processor, client view information representing a current client
view frustum and at least one of a current client view direction
vector and a current client viewpoint location; b) determining,
from the client view information, a maximal view frustum comprising
a volume of space intersected by a maximum possible movement of the
current client view frustum, based on said at least one of the
current client view direction vector and the current client
viewpoint location, during a delay period of time at least equal to
a round-trip-time of data communication between the server and the
client device; c) transmitting to the server, using the processor,
data representing the maximal view frustum; and d) receiving, from
the server, the at least one partial visibility event packet
comprising renderable graphics information that intersects the
maximal client view frustum.
15. A method, conducted on a server, of predictive prefetching and
transmitting at least one partial visibility event packet from the
server to a client device, the at least one partial visibility
event packet being a subset of a complete visibility event packet,
the complete visibility event packet including renderable graphics
information occluded from a first viewcell and not occluded from a
second viewcell among a plurality of viewcells, the method
comprising: a) receiving, from the client device, client view
information representing a current client view frustum and at least
one of a current client view direction vector and a current client
viewpoint location; b) determining, using a processor, from the
client view information, an estimated maximal client view frustum
comprising a volume of space intersected by a maximum possible
movement of the current client view frustum during a delay period
of time at least equal to a round-trip-time of data communication
between the server and the client device; c) calculating, using the
processor, the subset of the complete visibility event packet
comprising renderable graphics information that is included in the
estimated maximal client view frustum; and d) transmitting, at a
first time, to the client device, a reduced level-of-detail
representation of the at least one partial visibility event packet
comprising the calculated subset of the complete visibility event
packet that intersects the estimated maximal view frustum.
16. The method according to claim 15, wherein the first time is a
time or a time period when only the reduced level-of-detail
representation of the renderable graphics information can be
visually perceived.
17. The method according to claim 15, further comprising: e)
transmitting, at a second time later than the first time, to the
client device, the at least one partial visibility event packet
comprising the calculated subset as a high level-of-detail
representation of the subset of the renderable graphics
information.
18. The method according to claim 17, wherein the second time is
another time or another time period when the high level-of-detail
representation of the renderable graphics information can be
visually perceived.
19. The method according to claim 17, wherein a level-of-detail of
the high level-of-detail representation is proportional to human
visual acuity as a function of visual exposure time.
20. The method according to claim 17, wherein a level-of-detail of
the high level-of-detail representation is inversely proportional
to human visual acuity as a function of image-plane velocity of
renderable surfaces included in the at least one visibility event
packet.
21. The method according to claim 17, further comprising: f)
transmitting, at a third time, to the client device, at least one
deferred visibility event packet for said each viewcell of the
plurality of viewcells that is predictively penetrated by the
client viewpoint, the at least one deferred visibility event packet
including, for said each viewcell, a subset of the complete
visibility event packet that intersects the estimated maximal view
frustum and that has not been previously transmitted to the client
device, wherein the complete visibility event packet comprises an
additive combination of the at least one partial visibility event
packet and the at least one deferred visibility event packet.
22. The method according to claim 15, further comprising
determining the estimated maximal client view frustum by: b1)
determining, using the processor, a conservative maximal viewpoint
extent comprising a representation of a maximal possible subregion
of the second viewcell occupied by a client viewpoint during the
delay period; b2) determining, using the processor, a conservative
maximal view direction vector rotation comprising a representation
of a maximal rotational extent of a client view direction vector
during the delay period; and b3) calculating, using the processor,
the estimated maximal client view frustum from the conservative
maximal viewpoint extent and the conservative maximal view
direction vector rotation.
23. The method according to claim 22, further comprising: e)
storing, using storage circuitry, a complete data set for each
viewcell of the plurality of viewcells that is predictively
penetrated by the client viewpoint, as determined by the
conservative maximal viewpoint extent, the complete data set
representing a complete potentially visible set for said each
viewcell of the plurality of viewcells; f) storing, using the
storage circuitry, an incomplete data set for said each viewcell of
the plurality of viewcells that is predictively penetrated by the
client viewpoint, as determined by the conservative maximal
viewpoint extent, the incomplete data set representing, for said
each viewcell, a subset of the calculated subset actually
transmitted to the client device in step d); g) calculating, using
the processor, at least one deferred visibility event packet for
said each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, by comparing the
complete data set to the incomplete data set, the at least one
deferred visibility event packet including, for said each viewcell,
a subset of the complete potentially visible set not previously
transmitted to the client device; h) determining, using the
processor, for said each viewcell of the plurality of viewcells
that is predictively penetrated by the client viewpoint, whether
renderable graphics information of the at least one deferred
visibility event packet intersects the estimated maximal client
view frustum; and i) transmitting, to the client device, the at
least one deferred visibility event packet when the renderable
graphics information of the at least one deferred visibility event
packet intersects the estimated maximal client view frustum.
24. The method according to claim 23, further comprising: j)
transmitting, at a third time, to the client device, a reduced
level-of-detail representation of the at least one deferred
visibility event packet comprising the calculated subset of the
complete visibility event packet that intersects the estimated
maximal view frustum.
25. The method according to claim 24, wherein the third time is a
time or a time period when only the reduced level-of-detail
representation of the renderable graphics information can be
visually perceived.
26. The method according to claim 24, further comprising: k)
transmitting, at a fourth time later than the third time, to the
client device, the at least one deferred visibility event packet
comprising the calculated subset as a high level-of-detail
representation of the subset of the renderable graphics
information.
27. The method according to claim 26, wherein the fourth time is
another time or another time period when the high level-of-detail
representation of the renderable graphics information can be
visually perceived.
28. The method according to claim 26, wherein a level-of-detail of
the high level-of-detail representation is proportional to human
visual acuity as a function of visual exposure time.
29. The method according to claim 26, wherein a level-of-detail of
the high level-of-detail representation is inversely proportional
to human visual acuity as a function of image-plane velocity of
renderable surfaces included in the at least one visibility event
packet.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of priority under 35
U.S.C. .sctn.119(e) to U.S. Provisional Application No. 62/044,349,
filed on Sep. 1, 2014. This application is also a
continuation-in-part application of U.S. nonprovisional application
Ser. No. 14/754,419, filed on Jun. 29, 2015, and a
continuation-in-part application of U.S. nonprovisional application
Ser. No. 13/445,792, filed on Apr. 12, 2012, each of which is in
turn a continuation-in-part application of U.S. nonprovisional
application Ser. No. 13/420,436, filed on Mar. 14, 2012, which in
turn is a continuation-in-part of PCT application no.
PCT/US2011/042309, filed on Jun. 29, 2011, which itself claims the
benefit of priority to U.S. provisional application No. 61/360,283,
filed on Jun. 30, 2010. U.S. nonprovisional application Ser. No.
13/420,436 is also a continuation-in-part application of PCT
application no. PCT/US2011/051403, filed on Sep. 13, 2011, which
itself claims the benefit of priority to U.S. provisional
application No. 61/382,056, filed on Sep. 13, 2010, and to U.S.
provisional application No. 61/384,284, filed on Sep. 19, 2010.
U.S. nonprovisional application Ser. No. 13/420,436 also claims the
benefit of priority to U.S. provisional application No. 61/452,330,
filed on Mar. 14, 2011, U.S. provisional application No.
61/474,491, filed on Apr. 12, 2011, and U.S. provisional
application No. 61/476,819, filed on Apr. 19, 2011. The entirety of
each of the above-indicated applications is incorporated herein by
reference.
BACKGROUND
[0002] 1. Field
[0003] This application relates to a method and system for
predictive prefetching and transmitting at least one partial
visibility event packet and/or at least one deferred visibility
event packet from a server to a client device, the at least one
partial visibility event packet being a subset of a complete
visibility event packet, the complete visibility event packet
including renderable graphics information occluded from a first
viewcell and not occluded from a second viewcell among a plurality
of viewcells.
[0004] 2. Description of Background
[0005] In a 3D model of a densely occluded environment, most of the
renderable surfaces are completely occluded from any viewpoint
located in a small region that is deeply immersed in the
environment. This relationship is exploited by the referenced
copending U.S. application Ser. Nos. 14/754,419, 13/445,792, and
13/420,436, which describe a method of interactively delivering
renderable surfaces to a client unit by pre-computing surfaces that
become unoccluded if a viewpoint moves from one small navigable
region (viewcell) to an adjacent small navigable region of the
model. This precomputed information is stored as visibility event
(VE) packets, which can later be streamed interactively to a remote
client using scalable, adaptive, perception-based VE packet
streaming methods described in copending U.S. application Ser. Nos.
14/754,419, 13/445,792, and 13/420,436.
[0006] The copending U.S. application Ser. Nos. 13/420,436 and
13/445,792 specify how the payload of these visibility event
packets can be reduced if camera rotation is pre-limited by a
maximal view direction vector extent that is prescribed within each
viewcell. In this method, the maximum extent of the view frustum is
used to limit the precomputation of potentially-visible-set (PVS)
data within a viewcell, or delta-PVS data across viewcell boundary.
The method pre-computes the visibility event packet data, in one
embodiment, using a conservative model of visibility propagation.
This visibility propagation model can incorporate a predetermined,
prescribed maximal view frustum extent in order to restrict the
determination of from-viewcell visibility, and thus reduce VE
packet payloads when camera view direction vector is
pre-limited.
[0007] The method of using pre-limited view direction vectors to
compute smaller VE packets can be useful for streaming
in-game-engine cutscene material with a fixed or limited view
direction vector. Camera models in which the trajectory of the
viewpoint may be limited to a prescribed space curve, and for which
camera view direction vector rotation is limited, are common in
"rail shooter" games or episodes of games that provide relatively
restricted camera motion. In the general case of a walkthrough or
flythrough visualization, however (e.g., first-person, open-world
games), a view direction vector is not restricted. In the general
case, the view direction vector can be pointed in any direction for
any viewpoint within any viewcell.
SUMMARY
[0008] In exemplary embodiments, there is provided a method,
conducted on a server, of predictive prefetching and transmitting
at least one partial visibility event packet from the server to a
client device, the at least one partial visibility event packet
being a subset of a complete visibility event packet, the complete
visibility event packet including renderable graphics information
occluded from a first viewcell and not occluded from a second
viewcell among a plurality of viewcells, the method comprising: a)
storing, using storage circuitry, information representing a
current client view frustum; b) storing, using the storage
circuitry, renderable graphics information previously transmitted
to the client device; c) determining, using a processor, from the
information representing the current client view frustum, an
estimated maximal client view frustum comprising a volume of space
intersected by a maximum possible movement of the current client
view frustum during a delay period of time at least equal to a
round-trip-time of data communication between the server and the
client device; d) calculating, using the processor, the subset of
the complete visibility event packet comprising renderable graphics
information that is included in the estimated maximal client view
frustum; e) determining, using the processor, whether the
calculated subset has previously been transmitted to the client
device by comparing the calculated subset to the stored renderable
graphics information previously transmitted to the client device;
and f) transmitting, to the client device, the at least one partial
visibility event packet comprising the calculated subset of the
complete visibility event packet, if the calculated subset has not
been previously transmitted to the client device.
[0009] In exemplary embodiments, such a method may further comprise
determining the estimated maximal client view frustum by: c1)
determining, using the processor, a conservative maximal viewpoint
extent comprising a representation of the maximal possible
subregion of the second viewcell occupied by a client viewpoint
during the delay period; c2) determining, using the processor, a
conservative maximal view direction vector rotation comprising a
representation of a maximal rotational extent of the client view
direction vector during the delay period; and c3) calculating,
using the processor, the estimated maximal client view frustum from
the conservative maximal viewpoint extent and the conservative
maximal view direction vector rotation.
[0010] In exemplary embodiments, such a method may also further
comprise g) storing, using the storage circuitry, a complete data
set for each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, as determined by
the conservative maximal viewpoint extent, the complete data set
representing a complete potentially visible set for said each
viewcell of the plurality of viewcells; h) storing, using the
storage circuitry, an incomplete data set for said each viewcell of
the plurality of viewcells that is predictively penetrated by the
client viewpoint, as determined by the conservative maximal
viewpoint extent, the incomplete data set representing, for said
each viewcell, a subset of the calculated data set actually
transmitted to the client device in step f); i) calculating, using
the processor, at least one deferred visibility event packet for
said each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, by comparing the
complete data set to the incomplete data set, the at least one
deferred visibility event packet including, for said each viewcell,
a subset of the complete potentially visible set not previously
transmitted to the client device; j) determining, using the
processor, for said each viewcell of the plurality of viewcells
that is predictively penetrated by the client viewpoint, whether
renderable graphics information of the at least one deferred
visibility event packet intersects the estimated maximal client
view frustum; and k) transmitting, to the client device, the at
least one deferred visibility event packet when the renderable
graphics information of the at least one deferred visibility event
packet intersects the estimated maximal client view frustum.
[0011] In exemplary embodiments, there is also provided a method,
conducted on a client device, of receiving at least one partial
visibility event packet from a server, the at least one partial
visibility event packet being a subset of a complete visibility
event packet, the complete visibility event packet including
renderable graphics information occluded from a first viewcell and
not occluded from a second viewcell among a plurality of viewcells,
the method comprising: a) transmitting to the server, using a
processor, client view information representing a current client
view frustum; b) transmitting to the server, using the processor,
data comprising at least one of a current client view direction
vector and a current client viewpoint location; and c) receiving,
from the server, the at least one partial visibility event packet
comprising renderable graphics information that intersects a
maximal client view frustum, wherein the maximal client view
frustum comprises a volume of space intersected by a maximum
possible movement of the current client view frustum, based on said
at least one of the current client view direction vector and the
current client viewpoint location, during a delay period of time at
least equal to a round-trip-time of data communication between the
server and the client device.
[0012] In exemplary embodiments, such a method may further
comprise: d) receiving, from the server, at least one deferred
visibility event packet for said each viewcell of the plurality of
viewcells that is predictively penetrated by the client viewpoint,
the at least one deferred visibility event packet including, for
said each viewcell, a subset of the complete potentially visible
set not previously received from the server, wherein the subset
includes renderable graphics information of the at least one
deferred visibility event packet that intersects the maximal client
view frustum.
[0013] In exemplary embodiments, such a method may also further
comprise: g) storing, using the storage circuitry, a complete data
set for each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, as determined by
the conservative maximal viewpoint extent, the complete data set
representing a complete potentially visible set for said each
viewcell of the plurality of viewcells; h) storing, using the
storage circuitry, an incomplete data set for said each viewcell of
the plurality of viewcells that is predictively penetrated by the
client viewpoint, as determined by the conservative maximal
viewpoint extent, the incomplete data set representing, for said
each viewcell, a subset of the complete data set actually
transmitted to the client device in step f); i) calculating, using
the processor, at least one deferred visibility event packet for
said each viewcell of the plurality of viewcells that is
predictively penetrated by the client viewpoint, by comparing the
complete data set to the incomplete data set, the at least one
deferred visibility event packet including, for said each viewcell,
a subset of the complete potentially visible set not previously
transmitted to the client device; j) determining, using the
processor, for said each viewcell of the plurality of viewcells
that is predictively penetrated by the client viewpoint, whether
renderable graphics information of the at least one deferred
visibility event packet intersects the estimated maximal client
view frustum; and k) transmitting, to the client device, the at
least one deferred visibility event packet when the renderable
graphics information of the at least one deferred visibility event
packet intersects the estimated maximal client view frustum.
[0014] In exemplary embodiments, there is also provided a method,
conducted on a client device, of receiving at least one partial
visibility event packet from a server, the at least one partial
visibility event packet being a subset of a complete visibility
event packet, the complete visibility event packet including
renderable graphics information occluded from a first viewcell and
not occluded from a second viewcell among a plurality of viewcells,
the method comprising: a) transmitting to the server, using a
processor, client view information representing a current client
view frustum; b) transmitting to the server, using the processor,
data comprising at least one of a current client view direction
vector and a current client viewpoint location; and c) receiving,
from the server, the at least one partial visibility event packet
comprising renderable graphics information that intersects a
maximal client view frustum, wherein the maximal client view
frustum comprises a volume of space intersected by a maximum
possible movement of the current client view frustum, based on said
at least one of the current client view direction vector and the
current client viewpoint location, during a delay period of time at
least equal to a round-trip-time of data communication between the
server and the client device.
[0015] In exemplary embodiments, such a method may also further
comprise: d) receiving, from the server, at least one deferred
visibility event packet for said each viewcell of the plurality of
viewcells that is predictively penetrated by the client viewpoint,
the at least one deferred visibility event packet including, for
said each viewcell, a subset of the complete potentially visible
set not previously received from the server, wherein the subset
includes renderable graphics information of the at least one
deferred visibility event packet that intersects the maximal client
view frustum.
[0016] In exemplary embodiments, there is also provided a method,
conducted on a client device, of receiving at least one partial
visibility event packet from a server, the at least one partial
visibility event packet being a subset of a complete visibility
event packet, the complete visibility event packet including
renderable graphics information occluded from a first viewcell and
not occluded from a second viewcell among a plurality of viewcells,
the method comprising: a) determining, using a processor, client
view information representing a current client view frustum and at
least one of a current client view direction vector and a current
client viewpoint location; b) determining, from the client view
information, a maximal view frustum comprising a volume of space
intersected by a maximum possible movement of the current client
view frustum, based on said at least one of the current client view
direction vector and the current client viewpoint location, during
a delay period of time at least equal to a round-trip-time of data
communication between the server and the client device; c)
transmitting to the server, using the processor, data representing
the maximal view frustum; and d) receiving, from the server, the at
least one partial visibility event packet comprising renderable
graphics information that intersects the maximal client view
frustum.
[0017] In exemplary embodiments, there is also provided a method,
conducted on a server, of predictive prefetching and transmitting
at least one partial visibility event packet from the server to a
client device, the at least one partial visibility event packet
being a subset of a complete visibility event packet, the complete
visibility event packet including renderable graphics information
occluded from a first viewcell and not occluded from a second
viewcell among a plurality of viewcells, the method comprising: a)
receiving, from the client device, client view information
representing a current client view frustum and at least one of a
current client view direction vector and a current client viewpoint
location; b) determining, using a processor, from the client view
information, an estimated maximal client view frustum comprising a
volume of space intersected by a maximum possible movement of the
current client view frustum during a delay period of time at least
equal to a round-trip-time of data communication between the server
and the client device; c) calculating, using the processor, the
subset of the complete visibility event packet comprising
renderable graphics information that is included in the estimated
maximal client view frustum; and d) transmitting, at a first time,
to the client device, a reduced level-of-detail representation of
the at least one partial visibility event packet comprising the
calculated subset of the complete visibility event packet that
intersects the estimated maximal view frustum.
[0018] In exemplary embodiments, such a method may also further
comprise: transmitting, at a second time later than the first time,
to the client device, the at least one partial visibility event
packet comprising the calculated subset as a high level-of-detail
representation of the subset of the renderable graphics
information.
[0019] In exemplary embodiments, such a method may also further
comprise: transmitting, at a third time, to the client device, at
least one deferred visibility event packet for said each viewcell
of the plurality of viewcells that is predictively penetrated by
the client viewpoint, the at least one deferred visibility event
packet including, for said each viewcell, a subset of the complete
visibility event packet that intersects the estimated maximal view
frustum and that has not been previously transmitted to the client
device, wherein the complete visibility event packet comprises an
additive combination of the at least one partial visibility event
packet and the at least one deferred visibility event packet.
[0020] In exemplary embodiments, such a method may also further
comprise: transmitting, at a third time, to the client device, a
reduced level-of-detail representation of the at least one deferred
visibility event packet comprising the calculated subset of the
complete visibility event packet that intersects the estimated
maximal view frustum.
[0021] In exemplary embodiments, such a method may also further
comprise: transmitting, at a fourth time later than the third time,
to the client device, the at least one deferred visibility event
packet comprising the calculated subset as a high level-of-detail
representation of the subset of the renderable graphics
information.
BRIEF DESCRIPTION OF THE DRAWINGS
[0022] FIG. 1 is an exemplary diagram showing several viewcells
from which portions of surfaces may be occluded from visibility
when making a viewcell to viewcell transition with a maximal view
frustum facing the direction of movement, according to an
embodiment.
[0023] FIG. 2 is an exemplary diagram showing several viewcells
from which different portions of surfaces may be occluded from
visibility when making another viewcell to viewcell transition with
a maximal view frustum facing the direction of movement, according
to an embodiment.
[0024] FIG. 3 is an exemplary diagram showing several viewcells
from which different portions of surfaces may be occluded from
visibility when making yet another viewcell to viewcell transition,
this time with a maximal view frustum rotated 180 degrees compared
to the direction of movement shown in FIG. 2, according to an
embodiment.
[0025] FIG. 4 is an exemplary flowchart showing a method of
maintaining a shadow copy of a client's potentially-visible-set
(PVS) for a specific viewcell, and testing untransmitted (deferred)
surfaces for inclusion into a conservative estimate of the client's
maximal view frustum, in order to determine whether or not to
transmit the deferred surfaces to the client, according to an
embodiment.
[0026] FIG. 5A is a diagram showing an exemplary viewpoint and a
corresponding exemplary view frustum having a 90 degree horizontal
field of view applicable to the viewcells of FIGS. 1-3, according
to an embodiment.
[0027] FIG. 5B is a diagram showing a conservative current maximal
viewpoint extent (CCMVE) of penetration into a viewcell from a
known position after 166 ms of elapsed time using the exemplary
view frustum having a 90 degree horizontal field of view according
to FIG. 5A.
[0028] FIG. 6 is a diagram showing an exemplary method of
determining a conservative from-region frustum enveloping a
conservative current maximal viewpoint extent, for the general case
in which the view direction vector is rotating during traversal of
the viewpoint through the viewcell, and the viewpoint moves through
a specified penetration subregion of the viewcell.
[0029] FIG. 7 is a diagram showing that the resulting conservative
from-subregion frustum is larger than the corresponding from-point
frustum, even if it assumed that no view direction vector rotation
has occurred.
[0030] FIG. 8 is a diagram showing a resulting conservative
from-subregion frustum that results from viewpoint penetration into
the viewcell over 166 milliseconds for a CCMVE subregion, together
with rotation of the view direction vector 15 degrees to the right
or 15 degrees to the left from an initial view direction vector
orientation.
[0031] FIG. 9 is a diagram showing an additional angular region of
an extended frustum compared to that shown in FIG. 8, which spans
an additional 15 degrees on each side of an extended 120 degree
frustum shown in FIG. 8.
[0032] FIG. 10 is a diagram showing a top-down view of a view
frustum having a horizontal field of view of 90 degrees, and
undergoing rotation in the horizontal plane at a rate of 90 degrees
per second.
[0033] FIGS. 11A-11C show exemplary graphs of contrast sensitivity
(visual acuity) as a function of exposure time for a human visual
cortex.
[0034] FIG. 12 is a diagram showing a video frame of a simulation
of a state of rotation at an instant approximately 400 ms after
initiation of full speed 60 degree view frustum rotation from rest
to stop over a time period of 666 ms.
[0035] FIG. 13 is a diagram showing a later video frame, relative
to that shown in FIG. 12, of a simulation of a state of rotation at
an instant approximately 500 ms after initiation of full speed 60
degree view frustum rotation from rest to stop over a time period
of 666 ms.
[0036] FIG. 14A is an exemplary graph showing that the relative
spatial acuity of a human visual cortex is reduced by approximately
90% when a target surface is moving at 10 degrees per second on the
retina.
[0037] FIG. 14B is an exemplary graph showing the gain of the
oculomotor smooth pursuit system of a human visual cortex for
tracking targets at various angular velocities.
DETAILED DESCRIPTION OF THE EMBODIMENTS
[0038] In exemplary embodiments, there is described a method to
reduce bandwidth requirements for visibility event packet streaming
using a predicted maximal view frustum and a predicted maximal
viewpoint extent, each computed at runtime to determine a
conservative subset of visibility event packet data
transmitted.
[0039] For example, as described herein, a system includes a server
and a client computer device. The server delivers renderable
graphics information to the client. The renderable graphics
information comprises precomputed delta-PVS (potentially visible
set) information organized as visibility event packets that are
predictively prefetched from the server to the client based on
navigation prediction. The precomputed visibility event packet
information includes information comprising the renderable surfaces
that become unoccluded on crossing a specific boundary between
contiguous viewcells. This visibility event packet information is
used to incrementally maintain a collection of PVSs corresponding
to viewcells reachable by the client-user viewpoint.
[0040] In exemplary embodiments, the visibility event packet
information is precomputed assuming that the view direction vector
can assume any orientation within a viewcell. At runtime, the
server computes a conservative estimate of the client's maximal
view frustum during penetration of a viewcell using the linear and
angular velocities and accelerations of the client's viewpoint and
view direction vector. The conservative estimate of the client's
view frustum is used to determine the subset of the precomputed,
omnidirectional visibility event packet information that should be
sent to the client, and the subset for which the sending can be
deferred. The server maintains a copy of the client's PVS
information for each viewcell. At runtime, the conservative
estimate of the client's view frustum is tested against the
server's copy of the client's PVS to determine if any previously
deferred renderable surfaces now intersect the current conservative
estimate of the client's view frustum and should consequently be
sent to the client.
[0041] While the view direction vector has unrestricted range in
these cases, it takes a certain amount of time for the view frustum
to sweep out a full 360 degree squared directional extent, for
example, while the camera/viewpoint is within a specific viewcell.
The present disclosure exploits the finite angular velocity and
angular acceleration of the view frustum to reduce bandwidth
requirements by transmitting only a partial visibility event packet
at any given time, the partial visibility event packet being a
subset of a complete visibility event packet. The partial
visibility even packet may be a subset of the pre-computed VE
packet data at any given time.
[0042] This partial visibility event packet or subset is limited to
that portion of the renderable surfaces of the corresponding
visibility event packet that has not yet been transmitted and which
currently intersects a maximal view frustum, the maximal view
frustum computed to be a conservative representation of the largest
potential volume of 3-space intersected by the view frustum as the
user/client viewpoint moves within the corresponding viewcell, and
as the user/client view direction vector rotates within the
corresponding viewcell.
[0043] Also consistent with the disclosed embodiments, the present
disclosure exploits the finite angular velocity and angular
acceleration of the view frustum to reduce bandwidth requirements
by deferring transmission of visibility event packets. Such
deferred visibility event packets may include, for said each
viewcell, a subset of the complete potentially visible set not
previously transmitted.
[0044] Here, the deferred visibility event packets are also limited
to that portion of the renderable surfaces of the corresponding
visibility event packet that has not yet been transmitted and which
currently intersects a maximal view frustum. Further, the deferred
visibility event packets, if transmitted, would complete the
client's PVS representation for a currently occupied viewcell. For
example, the client's PVS representation may have become incomplete
due to previous transmission of only partial visibility event
packets. Consequently, the server may determine that the deferred
visibility event packets constitute the client's complete PVS minus
any previously untransmitted packets or partial packets.
[0045] Therefore, consistent with the disclosed embodiments,
partial visibility event packets may be understood as a subset of a
complete visibility event packet, the complete visibility event
packet including renderable graphics information occluded from a
first viewcell and not occluded from a second viewcell among a
plurality of viewcells. The partial visibility event packets may
thus be understood as those packets which omit some renderable
graphics information, however, when such information does not
intersect with the maximal view frustum when crossing or
predictively crossing a viewcell to viewcell boundary.
[0046] Likewise, consistent with the disclosed embodiments,
deferred visibility event packets may be understood as a subset of
a complete visibility event packet not previously transmitted. Such
packets may not have been previously transmitted because, while
they too may include renderable graphics information, such
information also did not intersect with the maximal view frustum
when crossing or predictively crossing a viewcell to viewcell
boundary at the time when the partial visibility event packets were
sent.
[0047] Thus, while the incorporated disclosure of copending U.S.
application Ser. Nos. 14/754,419, 13/445,792, and 13/420,436
describes transmitting precomputed complete visibility event
packets, which reflect the entire set of newly unoccluded surfaces
that could become visible upon crossing a viewcell to viewcell
boundary, the disclosed embodiments herein enable significant
improvements in computational efficiency and significantly reduced
bandwidth requirements for streaming transmission of visibility
event packets, by initially transmitting only partial visibility
event packets in which otherwise renderable surfaces of the
client's PVS have been culled because they are outside the maximal
view frustum. And, the disclosed embodiments herein enable
significant improvements in computational efficiency and
significantly reduced bandwidth requirements for streaming
transmission of visibility event packets, by deferring transmission
of visibility event packets or partial visibility event packets
until such time as the renderable surfaces of the client's PVS
therein would be within a conservative estimate of the client's
maximal view frustum due to changes concomitant with client
movement within a given viewcell.
[0048] Consistent with an embodiment, the maximal view frustum is
determined by the predicted extent of two separate components
during viewpoint motion within the corresponding viewcell. The
first component is the current maximal predicted sweep of the view
direction vector. This component is dependent upon the initial
angular kinetic state of the view direction vector on making the
corresponding transition between adjacent viewcells, on any known
changes in this state during movement in the viewcell, and also on
the maximum angular velocity, acceleration, and jerk (second
derivative of the angular velocity). The second component is the
current maximal predicted extent of the viewpoint within the
corresponding viewcell. This component is dependent upon the
location of penetration of the viewcell, the initial linear
velocity, on any known changes of these values during movement
within the viewcell, as well as the maximum linear velocity,
acceleration, and jerk (second derivative of the viewpoint linear
velocity).
[0049] Using these values in a Newtonian kinetic model, a
conservative current maximal view frustum can be computed at
various times during viewpoint and view direction movement within
the corresponding viewcell. Consistent with an embodiment, the
current conservative maximal view frustum is determined by the
server unit, as a conservative server-based representation of the
client user's actual maximal view frustum--considering both maximal
possible viewpoint penetration into the corresponding viewcell as
well as maximal view direction vector rotation during penetration
of the corresponding viewcell. The server also executes a process
in which in which the current maximal view frustum is used to
classify untransmitted surfaces of the corresponding VE packet in
order to determine which surfaces of the VE packet should be
prefetched at the current time. By using a conservatively predicted
current maximal view frusta, the subsets are selected and sent
before they intersect with the actual instantaneous client-side
view frustum. This prefetch prevents late VE packet arrival.
[0050] The server-side prediction of current maximal view direction
vector and current maximal viewpoint extent must, of course, take
into account ping latency between the visibility event server and
the visibility event client. At higher latencies, there are greater
angular and linear/translational regions of uncertainty, which is
reflected in larger predicted conservative regions calculated for
the current maximal view direction vector and current maximal
viewpoint extents. The method also accommodates embodiments in
which a Boolean difference constructive solid geometry calculation
is used to determine the difference between consecutive maximal
frustra for intersecting unsent surfaces. This difference volume is
smaller than the aggregate maximal frustum for the entire traversal
period of the viewcell, but it can be time consuming to compute.
The resulting difference volume can be highly irregular and more
difficult to use as a culling volume.
[0051] Turning now to FIG. 1, even if a surface (e.g., surface A in
FIG. 1) is part of the pre-computed visibility event packet for a
particular viewcell-viewcell transition (e.g., in FIG. 1, surface A
becomes potentially visible on moving from viewcell 1 to viewcell
2), and even if this surface is inside the conservative current
maximal view frustum, the surface may not actually be visible from
the current viewpoint within the frustum. That is, while surface A
may be unoccluded and within the frustum, it may be backfacing
(i.e., facing away) with respect to the current conservatively
predicted viewpoint extent. For example, FIG. 1 shows that at least
part of Surface A is within the maximal conservative predicted view
frustum (CPMVF), which is bounded in the horizontal plane by CPMVF
2a1 and CPMVF 2a2. However, in the example of FIG. 1, Surface A is
backfacing with every possible viewpoint located within the
conservative maximal viewpoint extent CCMVE 2a, and is therefore
invisible from the current predicted viewpoint in VIEWCELL 2.
[0052] Thus, in the example shown in FIG. 1, Surface A is an
element in the VE packet corresponding to the transition from
viewcell 1 to viewcell 2. It is a potentially newly-visible surface
on moving across the boundary shared by viewcell 1 and viewcell 2
in the direction from viewcell 1 to viewcell 2.
[0053] FIG. 1 shows two client viewpoint locations SVP1 and SVP2,
as determined by the server unit. The size of the viewpoints
represents a region, zone, or sphere of uncertainty that is related
to the round trip ping time between the server and the client, as
well as the kinetics and navigational degree of freedom of
viewpoint motion.
[0054] During a given period of time, the server's estimate of the
viewpoint location changes from SVP1 to SVP2, which corresponds to
a penetration of the boundary between VIEWCELL 1 and VIEWCELL 2.
The conservative current maximal viewpoint extent (CCMVE) predicted
by the server a short time after the current time corresponding to
SVP2 is shown as the small volume CCMVE 2a within VIEWCELL 2. This
conservative current maximal viewpoint extent (CCMVE) reflects not
only the intrinsic uncertainty due the ping latency, but
additionally reflects the current and maximum achievable velocity
and acceleration of the client-user controlled viewpoint. In some
implementations, the computation of the current maximal viewpoint
extent also considers the current and maximum achievable turning
velocity and acceleration (which can be identical to the view
direction vector velocity and acceleration, in some
embodiments).
[0055] The current predicted maximal conservative view frustum
(CPMVF) is determined from the current client-user view direction
vector, as well as the current angular velocity and acceleration of
this vector, and the maximal values for the angular velocity and
acceleration of this vector. The current predicted maximal
conservative view frustum corresponding to CCMVE 2a is, in the
horizontal plane, bounded by the conservative frustum boundaries
CPMVF 2a1 and CPMVF 2a2. Thus, as described earlier, while Surface
A is within the current conservative predicted maximal viewcell
frustum, Surface A is actually backfacing with respect to the
current conservative maximal viewpoint extent CCMVE 2a. Therefore,
Surface A is actually invisible from CCMVE 2a as a consequence of
this backfacing orientation.
[0056] In the example shown in FIG. 1, the actual horizontal view
angle is 90 degrees and the maximum rotational velocity of the view
direction vector is 90 degrees per second. Also in the example
shown in FIG. 1, the ping time between the server and the client is
100 ms. Consequently, the predicted frustum may be in error by as
much as 9 degrees in either direction in the horizontal plane. To
compensate for this intrinsic uncertainty, the horizontal angle of
the conservative predicted maximal frustum is increased by at least
18 degrees (9 degrees on each side) from 90 degrees to 108 degrees,
immediately upon penetration of viewcell 2. This angle can then be
further increased with time based on the initial directional state,
maximal rotational kinetic parameters, and hang time in the
viewcell.
[0057] Consistent with the diagram shown in FIG. 1, in order to
actually be visible from a current conservatively predicted
viewpoint, a surface must be: 1) unoccluded, 2) within the
predicted current maximal view frustum, and 3) have an orientation
that is not backfacing with respect to the current predicted
viewpoint extent. Exemplary embodiments of the present method
exploit condition number 3 by precomputing which surfaces become
unoccluded on moving across specific viewcell boundaries. This
information is stored as visibility event (VE) packets. Exemplary
embodiments of the present method employ the methods of the
copending applications, including copending U.S. application Ser.
Nos. 14/754,419, 13/445,792, and 13/420,436, to precompute this
information at a level of precision and granularity suitable for
real-time streaming of the packets using navigation-driven
predictive prefetch.
[0058] Exemplary embodiments of the present method also exploit
condition number 3 by testing that a surface in the VE packet is a)
within the current predicted maximal view frustum, and b) not
backfacing with respect to the current predicted maximal viewpoint
extent.
[0059] Note that while Surface A is not backfacing for all
viewpoints located in VIEWCELL 2, it is backfacing for a
substantial portion of VIEWCELL 2, including CCMVE 2a. The method
of computing a current conservative maximal viewpoint extent within
a viewcell can be effectively employed when the transit time
through the viewcell is substantially greater than the ping time
between the server and the client. In this case, there is generally
sufficient time to determine the subset of VE packet surfaces that
have not been sent and that are not backfacing with respect to the
evolving current conservative maximal viewpoint volume.
[0060] Likewise, the method of determining and sending, at runtime,
unsent surfaces of the VE packet that are within the evolving
conservative current maximal view frustum extent can be employed
when transit times in the viewcell are substantially larger than
the ping time, and where the maximal rotational velocity and
acceleration of the view direction vector are modest (e.g., at a
maximum of about 90-100 degrees per second).
[0061] Also, placing reasonable limits on the linear velocity of
the viewpoint and angular velocity of the view direction vector
actually improves human visual performance, which is limited at
high viewpoint velocity and high view direction vector velocities
by dynamic visual acuity constraints. Exemplary embodiments of the
present method incorporate specific techniques which exploit the
limited performance of the human visual system at high view
direction velocities. These techniques are described in a later
section of this disclosure.
[0062] For exemplary embodiments of the present method, placing
reasonable limits on the viewpoint and view direction vector
velocities enhances the predictability of the navigation and
decreases the size of both the conservative maximal viewpoint
volume and the conservative maximal predicted view frusta. For the
present method, these reasonable limits on locomotive performance
can significantly decrease the transmission bandwidth requirement
for VE packet streaming. In exemplary embodiments of the present
method, this reduced transmission bandwidth requirement is used to
transmit higher level-of-detail representations of the newly
visible surfaces, precisely under conditions when the client-side
human visual system or robotic visual system can actually perceive
the additional detail.
[0063] Conversely, during periods of relatively high linear and
angular velocities, the size of the predicted penetration regions
and predicted maximal frusta will increase. But, under these
conditions, exemplary embodiments of the present method will
transmit lower level-of-detail VE packets. Under such conditions,
only the lower level-of-detail can be perceived by the client-side
visual system. In this way, exemplary embodiments of the present
method exploit the similar spatio-temporal performance
characteristics of the visibility event protocol and human (and
robotic) vision systems to minimize the bandwidth required to
deliver a perceptually lossless visibility event packet stream.
[0064] In exemplary embodiments, there is also described a method
to reduce bandwidth requirements for visibility event packet
streaming using a predicted maximal view frustum and predicted
maximal viewpoint extent, each computed at runtime--in conjunction
with maintaining a complete version of an incrementally constructed
PVS on the server.
[0065] The runtime conservative frustum and conservative viewpoint
extent methods are used to prevent the transmission of invisible
surfaces of a VE packet during viewpoint motion with a single
viewcell. The disclosed method exploits the fact that, although the
surfaces of a VE packet are potentially visible from at least one
viewpoint located in the corresponding viewcell, the predicted
maximal extent of penetration into the viewcell as well as the
predicted maximal view frustum can actually make many of the
surfaces of the corresponding VE packet currently invisible to the
client-user viewpoint and view frustum.
[0066] The disclosed method can therefore be used to defer
transmission of significant portions of the visibility event
packet. Overall, however, the visibility event protocol defined by
a navigation-driven predictive prefetch of precomputed VE packets
is intrinsically an incremental and progressive method of streaming
the content. Consequently, the disclosed method delivers a series
of partial and/or deferred VE packets that reflect the (predicted)
penetration of a corresponding sequence of viewcell-to-viewcell
boundaries. When the runtime conservative culling methods are
employed, some parts of a VE packet corresponding to a first
viewcell boundary may go untransmitted, even as the viewpoint
(i.e., predicted viewpoint) penetrates later transited viewcell
boundaries.
[0067] For example, FIG. 1 shows Surface A and Surface B, both of
which are occluded from view from the view frustum perspective of
VIEWCELL 1 but are not necessarily occluded from the view frustum
perspective of VIEWCELL 2. Therefore, portions of both Surface A
and Surface B would be represented in the newly visible (i.e.,
newly unoccluded) surfaces component of the VE packet corresponding
to the viewcell boundary transition from VIEWCELL 1 to VIEWCELL 2.
That is, these surfaces would be in the newly-occluded surfaces
component of the VE packet corresponding to the viewcell boundary
transition from VIEWCELL 2 to VIEWCELL 1.
[0068] FIG. 2 shows the same viewcells and surfaces as represented
in FIG. 1, as well as the path taken by the viewpoint from the view
frustum perspective as it transits through VIEWCELL 2. The
viewpoint transit path P23 is shown in FIG. 2, as well as the
resulting location of the predicted viewpoint SVP3, for the
indicated conservative current maximal viewcell extent (CCMVE2) and
a corresponding conservative current maximal view frustum (CPMVF3)
for transit to VIEWCELL 3. During the viewpoint movement through
VIEWCELL 2 along path P23, there is depicted little view direction
vector rotation of the view frustum. Consequently, neither Surface
A nor Surface B are visible from the current predicted viewpoint
location and predicted view direction vector when the viewpoint
transits into VIEWCELL 3.
[0069] FIG. 3 shows a further evolution of a conservatively
predicted viewpoint and view direction vector motion within
VIEWCELL 3, relative to the depiction shown in FIG. 2. As shown in
FIG. 3, the view direction vector has rotated approximately 180
degrees during penetration of the client-user viewpoint into
VIEWCELL 3, relative to the view direction vector shown in FIG. 2.
However, the actual instantaneous view frustum shown in FIG. 3 is
not a predicted maximal view frustum. Rather, the predicted maximal
view frustum now essentially fills much of the volume of the
modeled space as a consequence of the approximately 180 degrees of
view direction vector rotation while the client-user viewpoint is
within VIEWCELL 3. Clearly, a portion of Surface A is visible from
viewpoint SVP3b, which is the predicted conservative maximal
viewpoint penetration into VIEWCELL 3. However, in this case,
Surface A has not been transmitted from the server to the client.
Surface A is instead represented as a newly exposed surface for the
VIEWCELL 1 to VIEWCELL 2 transition VE packet.
[0070] In other words, because it never passed the frustum test it
was never transmitted. Note that Surface A would eventually pass
the backfacing test along path P23c, but the position of the
surface is such that when it passes the backfacing test, it fails
the frustum test. Consequently, Surface A was not transmitted to
the client by the time the viewpoint entered VIEWCELL 3, that is,
Surface A is not a newly visible surface for the VIEWCELL 2 to
VIEWCELL 3 transition. Therefore, consistent with exemplary
embodiments of the disclosed method, as the view direction vector
undergoes rotation while in VIEWCELL 3, Surface A would pass the
frustum test, but as a consequence of the failed frustum test
during the entire transition through VIEWCELL 2, it has not yet
been sent to the client.
[0071] Thus, when employing the disclosed method of using runtime
predicted view frusta and/or predicted maximal viewpoint extents,
the incrementally constructed PVS for any viewcell maintained by
the client may be incomplete, since some deferred surfaces have not
been transmitted.
[0072] One solution to this problem is to test, on the server, all
deferred surfaces belonging to the PVS of the current (and/or
predicted) viewcell representative of unoccluded surfaces whose
transmission has been deferred by the backfacing or frustum tests
against the current predicted maximal frustum & maximal
viewcell extent, in order to determine if these previously deferred
surfaces should be predictively prefetched.
[0073] In the examples of FIGS. 1-3, Surface A became newly
unoccluded during the VIEWCELL 1 to VIEWCELL 2 transition. For
example, Surface A may be represented as a newly-unoccluded surface
in the VE packet for the VIEWCELL 1 to VIEWCELL 2 transition using
the methods described in copending U.S. application Ser. Nos.
14/754,419, 13/445,792, and 13/420,436, and remains potentially
visible (unoccluded) from VIEWCELL 3. Surface A is not included as
either newly-unoccluded or newly-occluded surface for the VE packet
corresponding the VIEWCELL 2 to VIEWCELL 3 transition.
[0074] In exemplary embodiments of the disclosed method, the
deferred surfaces that must be prefetched are identified by
maintaining a "shadow PVS" on the server for all penetrated
viewcells. This shadow PVS contains the omnidirectional PVS (that
is, the complete PVS assuming an omnidirectional view frustum) for
each actually penetrated viewcell, or viewcells for which the
navigation prediction algorithm has predicted penetration. In the
shadow PVS maintained by the server, each surface is tagged as
already prefetched or currently deferred. The maximal predicted
viewcell extent and maximal predicted frustum test are conducted on
the server against the deferred surfaces of the shadow PVS for the
current viewcell, and the deferred surfaces of the PVS that pass
these tests are prefetched to the client.
[0075] Note that because the PVS is typically larger than the
delta-PVS or dPVS (Visibility Event packets, per se), the frustum
testing incurs additional compute cycles on the server, but results
in lower bandwidth requirements for VE packet streaming. Of course,
only the deferred elements of the PVS need to be subjected to these
server-side tests, the deferred elements can be pre-labeled and
clustered for fast processing.
[0076] Moreover, a conservatively predicted maximal view frustum
cull test conducted on the server can be performed in Log N.sub.v
time with any basic hierarchical spatial subdivision or
hierarchical bounding box organization. The "V" subscript indicates
that the cull is much faster than the typical per-frame frustum
cull that occurs in the graphics pipeline for rendering a game
level, since in this case the occluded surfaces have already been
removed by the visibility event packet encoding process, as
described in the exemplary embodiments of copending U.S.
application Ser. Nos. 14/754,419, 13/445,792, and 13/420,436.
[0077] Turning back to the example of FIG. 3, the deferred Surface
A is transmitted when the frustum test is passed, but Surface B is
not transmitted because, although it is part of the PVS for
VIEWCELL 2, it becomes occluded in the VIEWCELL 2 to VIEWCELL 3
transition, and thus is not represented in the server-side shadow
PVS for VIEWCELL 3. In this case, Surface B, which is a newly
unoccluded surface for the delta-PVS VE packet corresponding to the
VIEWCELL 1 to VIEWCELL 2 transition, is never actually prefetched,
because it has been removed from the shadow-PVS corresponding to
VIEWCELL 3 by the time it would pass the predicted frustum
test.
[0078] Together, these techniques, implemented in an exemplary
embodiment of the disclosed method, exploit the fact that
visibility of a surface requires the surface to be: 1) unoccluded,
2) in the frustum, and 3) not backfacing. Of these three elements
of visibility, the most extensive to compute is occlusion. Existing
real-time graphics hardware typically relies on a z-buffer method
of resolving occlusion at the pixel level. This can be very
inefficient for rendering densely occluded environments.
Consequently, modern graphics systems incorporate several runtime
methods including: z-pass, early z rejection, occlusion queries,
and other runtime methods, to reduce the amount of occluded
surfaces submitted to the later stages of the rendering pipeline.
In contrast, exemplary embodiments of the disclosed method employ a
high-precision process to precompute occlusion, e.g., with
application of the methods of copending U.S. application Ser. Nos.
14/754,419, 13/445,792, and 13/420,436, which reduces the need for
expensive runtime occlusion culling methods. This approach also
accelerates the other two visibility tests of frustum inclusion and
front facing orientation, since when using the PVSs maintained by
the visibility event packets, the vast majority of occluded
surfaces are not included in the frustum or backface tests.
[0079] FIG. 4 is a flowchart showing the method of maintaining a
shadow copy of the client's PVS for a specific viewcell, and
testing untransmitted (deferred) surfaces for inclusion into the
conservative estimate of the client maximal view frustum, in order
to determine if these deferred surfaces should currently be
transmitted to the client.
[0080] In FIG. 4, data store 410 represents the complete PVS for a
particular viewcell. In exemplary disclosed embodiments, the
specific viewcell represents a viewcell for which penetration by
the client-user's viewpoint has already been predicted (using
navigation prediction), and for which the relevant visibility event
data (e.g., delta-PVS data) has already been transmitted to the
client. The data store 410 is a representation of the PVS that
includes all elements of the relevant delta-PVS and visibility
event packet data have been added to the PVS. That is, the data
store 410 is a representation of the viewcell PVS that assumes an
omnidirectional view frustum.
[0081] In contrast, data store 420 represents a copy of the client
PVS for the viewcell that includes only those renderable surfaces
of the PVS that have actually already been transmitted to the
client. The data store 420 reflects the fact that some of the
renderable surfaces of the corresponding delta-PVS (visibility
event packet data) may not have been transmitted at the time that
the viewpoint was predicted to cross the viewcell to viewcell
boundary corresponding to the visibility event packet. In exemplary
embodiments of the disclosed method, the transmission of some of
the renderable surfaces of a visibility event packet may be
deferred if these surfaces are not within the conservative estimate
of the client maximal view frustum.
[0082] Then, in step 430, the data stores 410 and 420 are used to
determine those surfaces of the complete PVS that have not yet been
transmitted to the client.
[0083] In step 440, the untransmitted surfaces are tested for
inclusion in the conservative estimate of the client maximal view
frustum. As described, in exemplary embodiments, this estimate of
the client maximal view frustum is constructed by the server unit
taking into account factors including the maximal extent of
viewpoint penetration into the viewcell (which is a function of the
viewpoint linear and angular velocity and acceleration) as well as
the angular velocity and acceleration of the view direction vector,
as well as other factors including the horizontal and vertical
field-of-view and the network latency between the client and the
server.
[0084] In decision step 450, it is determined if the deferred
surface of the PVS for the viewcell is within the conservative
estimate of the client maximal view frustum. If, in step 450, it is
determined that the deferred renderable surfaces of the PVS are not
within the conservative estimate of the client maximal view
frustum, then processing continues to step 460, where the
renderable surfaces not in the conservative estimate of the client
maximal view frustum are not transmitted to the client. If, in step
450, it is determined that the deferred renderable surfaces of the
PVS are within the conservative estimate of the client maximal view
frustum, then processing continues to step 470, where the
renderable surfaces not in the conservative estimate of the client
maximal view frustum are transmitted to the client.
[0085] In step 480, the transmitted renderable surfaces are added
to the list of the transmitted surfaces for the PVS as stored in
data store 420.
[0086] These runtime methods can be employed effectively in cases
where the predictability of navigation is high relative to the ping
time. The predictability of navigation is a function of the maximal
locomotive performance of the viewpoint and view direction vector,
as is also determined by navigational constraints in the
environment. As specified in copending U.S. application Ser. Nos.
14/754,419, 13/445,792, and 13/420,436, these determinants of
navigational predictability can be adaptively regulated by the
visibility event packet streaming system in order to prevent late
packet arrival, while insuring that the content is delivered to the
user at a rate which is approximately matched to the spatiotemporal
performance limits of the human or robotic client's vision
system.
[0087] In exemplary embodiments, there is also described a method
of using the extended view frustum to buffer against view direction
vector rotation.
[0088] For example, FIG. 5A shows a viewpoint, VP 5.10, and a
corresponding view frustum having a 90 degree horizontal field of
view. FIG. 5B shows a top-down view of a 90 horizontal field of
view frustum enveloping the CCMVE-5. FIG. 5B also shows a
conservative current maximal viewpoint extent, CCMVE-5, of
penetration into the viewcell from a known position after 166 ms of
elapsed time. As with CCMVE-2a, shown in FIG. 1, consistent with
disclosed embodiments, CCMVE-5 is determined from a last known
position and the maximal linear and angular velocity and
acceleration of the viewpoint.
[0089] For example, for a typical 90 degree field of view such as
the 90 degree from-point frustum shown in FIG. 5A, rotation rates
of the frustum approaching 130 to 140 degrees per second (such as
that of typical multiplayer action video games) can cause
difficulty in aiming, especially for people older than 30 years of
age, and can cause difficulty in viewing for a spectator. However,
a 90 degree yaw ability to scan the environment is more suitable
(such as that of more narrative-driven, cinematic campaign-type
video games), such that aiming can be improved, and viewing would
be more enjoyable for a spectator.
[0090] In exemplary embodiments, such a conservative from-region
frustum enveloping a conservative current maximal viewpoint extent,
such as CCMVE-5 shown in FIG. 5B, can be determined using the
method specified in copending U.S. application Ser. Nos.
14/754,419, 13/445,792, and 13/420,436. This is shown, for example,
in FIG. 6, for the general case in which the view direction vector
is rotating during traversal of the viewpoint through the viewcell,
and the viewpoint moves through a specified penetration subregion
of the viewcell. In the exemplary implementation shown in FIG. 5B,
the view direction vector is assumed to have undergone no rotation
during the 166 ms of viewcell penetration.
[0091] FIG. 7 shows that the resulting conservative from-subregion
frustum is larger than the corresponding from-point frustum at
viewpoint VP 4.10, even if it assumed that no view direction vector
rotation has occurred, for a CCMVE-5 representative of predicted
viewcell penetration at 166 ms.
[0092] FIG. 8 shows a resulting conservative from-subregion frustum
that results from a CCMVE-5 representative of viewpoint penetration
into the viewcell subregion over 166 milliseconds, together with
rotation of the view direction vector 15 degrees to the right or 15
degrees to the left from an initial view direction vector
orientation. In this exemplary case, assuming a maximum view
direction rotation rate of 90 degrees per second, if the ping
latency between the visibility event client and the server is 166
ms, the resulting 30 degree rotation would represent the
uncertainty of the client's view direction vector, as experienced
by the server. Accordingly, consistent with disclosed embodiments,
the server can employ the extended 120 degree frustum (i.e., 120
degree predicted maximum from-subregion frustum) to determine the
subset of the visibility event packet data to actually transmit to
the client. This determination is made by determining the set of
unsent surfaces of the corresponding visibility event packet that
intersect the extended frustum. Once again, in exemplary disclosed
embodiments, the visibility event packet data is precomputed using
the method of first-order from region visibility as specified in
copending U.S. application Ser. Nos. 14/754,419, 13/445,792, and
13/420,436. The set of surfaces belonging to the corresponding PVS,
incrementally maintained using the delta-PVS VE packets, that have
not already been sent is maintained using the technique of
maintaining the shadow PVS on the server.
[0093] In some embodiments, the visibility event packets are
precomputed assuming a full omnidirectional view frustum spanning
12.56 steradians of solid angle. Consistent with disclosed
embodiments, the methods described in copending U.S. application
Ser. Nos. 14/754,419, 13/445,792, and 13/420,436, may be employed
to precompute the omnidirectional visibility event packets. As
described, in exemplary disclosed embodiments, the visibility event
server can employ the extended view frustum to cull portions of the
precomputed visibility event packet that fall outside of the
maximum possible predicted extent of the client view frustum, as
determined from the ping latency and the maximal angular velocity
and acceleration of the view frustum, as well as the maximum
predicted extent of penetration of the viewpoint into the view
cell. This method ensures that all of the potentially visible
surfaces are transmitted, while minimizing bandwidth requirements,
by deferring the transmission of VE packet surfaces that are not
within the current conservative extended frustum, or which happen
to be backfacing with respect to the conservative current maximal
viewpoint extent of penetration into the viewcell.
[0094] In exemplary embodiments, there is also described a method
of using reduced level-of-detail models in periphery of extended
view frustum to reduce bandwidth requirements for buffering against
view direction vector rotation, i.e., level-of-detail vs. predicted
exposure durations.
[0095] The above-disclosed methods comprise determining a
conservative representation of the client's view frustum from the
temporal reference frame of the server, and using this extended
frustum to cull those surfaces of the corresponding visibility
event packet that could not possibly be in the client's view
frustum. Consistent with disclosed embodiments, all of the
transmitted surface information is represented at the highest
level-of-detail. As described in copending U.S. application Ser.
Nos. 14/754,419, 13/445,792, and 13/420,436, the visibility event
packets can be encoded using geometric and surface models at a
plurality of levels-of-detail, including a plurality of levels of
geometric, texture, and other surface detail.
[0096] In exemplary disclosed embodiments, however, the VE packets
can be transmitted at a lower level-of-detail during periods of low
bandwidth availability, and/or high bandwidth requirement, in order
to maximize the probability that the information encoding newly
exposed surfaces arrives on time (e.g., before the surface is
actually exposed in the client viewport). As described in copending
U.S. application Ser. Nos. 14/754,419, 13/445,792, and 13/420,436,
under some conditions, a VE packet containing relatively low
level-of-detail surface information can initially be transmitted
and later replaced by a VE packet containing higher level-of-detail
information. This exploits the fact that the human visual system
has lower visual acuity for newly exposed surfaces. As shown in
FIG. 11, for example, and as described in detail in copending U.S.
application Ser. Nos. 14/754,419, 13/445,792, and 13/420,436, human
visual acuity (expressed as contrast sensitivity on a logarithmic
scale) is very limited during the first 400-500 milliseconds of
exposure. That is, the human visual system's ability to perceive
detail on a surface is very limited for the first 400-500
milliseconds that the surface is exposed to the visual system.
[0097] In exemplary disclosed embodiments, this limitation of human
visual performance (as well as the limited spatiotemporal
performance of some robotic vision systems, including 3D
map-matching navigation systems) can be exploited by sending low
level-of-detail surface information if the surfaces fall outside
the region of the extended view frustum, as determined, in
exemplary embodiments, using one or more of the following pieces of
information: the ping latency, the maximum viewpoint translation
velocity and acceleration, the maximum angular velocity, and
acceleration of the view direction vector. For example, FIG. 9
shows an additional angular region of extended view frustum that
spans an additional 15 degrees on each side of the extended 120
degree frustum shown in FIG. 8. In exemplary embodiments, the
visibility event server transmits surfaces that fall in the
subfrustum between 120 degrees and the maximally extended frustum
of 150 degrees at a lower level-of-detail than the other visibility
event surface data that fall within the 120 degree extended
frustum.
[0098] The disclosed method thus provides an additional buffer
against view direction vector rotation, which may be useful if the
directional visibility gradient (e.g., the rate of exposure of
surfaces per degree of view direction vector rotation) is high, or
if the available bandwidth has a high degree of variability (e.g.,
network jitter). In such exemplary embodiments, the low
level-of-detail surface information can potentially be replaced by
a higher level-of-detail representation. If the replacement occurs
at a rate which exceeds the dynamic visual acuity performance
curves of FIGS. 11A-11C, for example, then the switch can be
difficult or impossible for viewers to perceive.
[0099] FIG. 10 shows a top-down view of a view frustum having a
horizontal field of view of 90 degrees, and undergoing rotation in
the horizontal plane at a rate of 90 degrees per second in a
direction from region A toward region D. In this exemplary case,
surfaces to the right-hand side of the view frustum will undergo
incursion into the rotating frustum at region D, whereas surfaces
near the left-hand extreme of the view frustum at region A will
exit the frustum during frustum rotation. In the exemplary case
shown in FIG. 10, those surfaces in region A have been in the
frustum for between 750 ms and 1000 ms as a consequence of exposure
via regions D, C, and B during the rotation. In region B, for
example, the surfaces have been in the frustum for between 500 ms
and 750 ms; in region C, the surfaces have been in the frustum for
between 250 ms and 500 ms; and in region D, the surfaces have been
in the frustum for between 0 ms and 250 ms.
[0100] Surfaces that have been in the frustum for only a brief
period of time have also been exposed to the client-user's visual
system for a concomitantly brief period of time. For example, FIG.
12 shows a similar case of rotation of a 90 degree horizontal
field-of-view frustum from rest (with instantaneous angular
acceleration) to a view direction vector rotational velocity of 90
degrees per second in a direction from Region 6 toward Region 1,
which is sustained for 666 ms, giving a rotation of 60 degrees. In
this exemplary case, at the end of the rotation, those surfaces in
Region 1 were in the frustum at the time the rotation began, as a
consequence of exposure via Regions 6, 5, 4, 3, and 2 during the
rotation. Also in this exemplary case, at the end of the rotation,
those surfaces in Region 2 have been in the frustum from 500 ms to
666 ms, as a consequence of exposure via Regions 6, 5, 4, and 3
during the rotation. Also in this exemplary case, at the end of the
rotation, those surfaces in Region 3 have been in the frustum from
375 ms to 500 ms, as a consequence of exposure via Regions 6, 5,
and 4 during the rotation. Also in this exemplary case, at the end
of the rotation, those surfaces in Region 4 have been in the
frustum from 250 ms to 375 ms, as a consequence of exposure via
Regions 6 and 5 during the rotation. Also in this exemplary case,
at the end of the rotation, those surfaces in Region 5 have been in
the frustum from 125 ms to 250 ms, as a consequence of exposure via
Region 6 during the rotation. Also in this exemplary case, at the
end of the rotation, those surfaces in Region 6 have been in the
frustum from 0 ms to 125 ms.
[0101] FIG. 12, for example, thus shows the state of the rotation
visible in the view frustum at an instant approximately 400 ms
after the initiation of the rotation. In the example of FIG. 12,
the depicted low level-of-detail graphical model of a statue was
just outside of the view frustum immediately prior to the start of
the view direction vector rotation. At 400 ms into the rotation,
the depicted graphical model enters into visibility in Region 3 of
the view frustum. In this case, no part of the depicted graphical
model has been exposed to the visual system for more than 400 ms.
Consequently, any client-user cannot perceive high frequency
geometric or other surface detail of the model. In exemplary
embodiments of the disclosed method, therefore, the server sends a
low level-of-detail representation of the graphical model during
initial view frustum incursion, since the client-user cannot
initially perceive higher levels of detail.
[0102] In exemplary disclosed embodiments, as the exposure time of
the graphical model increases, the low level-of-detail model
initially streamed by the visibility event server can be replaced
by a higher level-of-detail model. For example, FIG. 13 shows a
later time window of the same view frustum rotation shown in FIG.
12. In FIG. 13, however, the initial low level-of-detail graphical
model of the statue has been replaced by a higher level-of-detail
model. In this case, the higher level-of-detail model has 20 times
the number of polygons as the lower level-of-detail initially
streamed by the visibility event server and displayed by the
visibility event client shown in FIG. 12. The graphical model of
the statue in FIG. 13 also has texture maps that are 20 times the
resolution of the texture maps belonging to the lower
level-of-detail model shown in FIG. 12.
[0103] Thus, according to exemplary embodiments of the disclosed
method, the visibility event server initially streams the lower
level-of-detail models as they initially penetrate the view
frustum. In exemplary disclosed embodiments, these streamed lower
level-of-detail models may be low level-of-detail versions of
graphical models that exist just outside of the maximal
instantaneous conservative view frustum, for example, in the
regions labeled 166 ms Buffer Zone in FIG. 9.
[0104] In exemplary disclosed embodiments, this pre-transmission of
the low level-of-detail graphical model as shown in FIG. 12
prevents missing surfaces in transmitted VE packets, and allows for
additional time to transmit the higher level-of-detail
representation for presentation later in the view frustum
incursion, that is, when the surfaces of the graphical model have
been exposed to the client-user's visual system for a longer
period, and are therefore able to be resolved with higher visual
acuity. If this replacement occurs faster than the contrast
sensitivity (visual acuity) vs. exposure time curves of FIGS.
11A-11C, then it is difficult or impossible for human client-user's
visual system to perceive the switch from the displayed low
level-of-detail model to the displayed higher level-of-detail
model. In exemplary disclosed embodiments, the transition between
displaying the low level-of-detail representation and the high
level-of-detail representation is made to occur over several frame
display intervals, with blending during the transition. This
technique further masks the transition.
[0105] As shown in FIGS. 12 and 13, the depicted graphical model of
a statue corresponds to frames from a video showing a simulation in
which the 666 ms view frustum rotates at 60 frames per second. In
this case, the rotation requires 40 frame intervals (all image
frames being computed on the visibility event client system). For
the initial 500 ms of the rotation, only the low level-of-detail
graphical model is displayed in the video. In the final 166 ms of
the rotation, the higher level-of-detail model, which has arrived
at the visibility event client, is then displayed. In the video, it
is very difficult for human viewers to see that a very low
level-of-detail model (e.g., having only 5% the detail of the final
model) was initially displayed. The inability of human viewers to
perceive the transition is, in part, caused by one of the specific
limitations of human dynamic visual acuity: limited acuity at low
surface exposure times.
[0106] Exemplary embodiments of the disclosed method also send
and/or display different component surfaces of a single graphical
model at different levels-of-detail, depending on the exposure
duration of the component surfaces. For example, in the case of the
low level-of-detail and high level-of-detail representations of the
graphical statue model of FIGS. 12 and 13, respectively, the
component surfaces of the model's right shoulder can be transmitted
and displayed at a higher level-of-detail than the component
surfaces of the model's left shoulder, since the right shoulder
surfaces are exposed to the client-user's visual system for a
longer period of time before the left shoulder surfaces would
become exposed to the client-user's visual system for the same
longer period of time.
[0107] In exemplary embodiments, there is also described a method
of using reduced level-of-detail models in the periphery of an
extended view frustum to reduce bandwidth requirements for
buffering against view direction vector rotation, i.e.,
level-of-detail vs. estimated image-space velocity and retinal
slip.
[0108] As described above, visual acuity (i.e., the ability to
resolve spatial detail) is markedly limited at low exposure times.
Visual acuity is also limited for surfaces that are moving at a
substantial angular velocity in the image plane. This limitation
results because the oculomotor system has a finite gain in smoothly
pursuing rapidly moving targets in the image plane/visual field.
For any moving visual targets that are not perfectly pursued by the
oculomotor system, the projection of the object will actually be
moving on the retina. This motion of image elements relative to the
retina is called retinal slip. The visual system has limited visual
acuity for elements that are moving on the retina. Since these
moving image elements are not stabilized in space-time on the rods
and cones of the retina, or on subsequent neural visual processing
systems (e.g., the lateral geniculate nucleus of the thalamus, area
17 of the visual cortex, etc.), they are incompletely processed and
incompletely resolved.
[0109] FIG. 14A shows that the relative spatial acuity is reduced
by approximately 90% when the target surface is moving 10 degrees
per second on the retina. FIG. 14B shows the gain of the oculomotor
smooth pursuit system for tracking such targets at various angular
velocities. Two gain curves are shown, one for young subjects
(labeled Young) and another for older subjects (labeled Elderly).
For young subjects, target velocities above 30 degrees per second
are imperfectly tracked, which causes retinal slip. For example, at
a target velocity of approximately 50 degrees per second, the gain
of the smooth pursuit oculomotor tracking system is approximately
80%, resulting in a retinal slip of approximately 10 degrees per
second and a decrease in acuity of approximately 90%. For the older
subjects, a much slower target velocity of only 15 degrees per
second causes the same degree of retinal slip and loss of
acuity.
[0110] In exemplary embodiments of the disclosed method, the
technique of initially sending/display low level-of-detail models
during frustum incursion also exploits the fact that graphical
surfaces have an image plane velocity that is generally
proportional to the angular velocity of the view direction vector.
Consequently, even as the exposure time of the graphical surfaces
increase with increasing incursion into the view frustum, the
ability of the human visual system to perceive detail can still be
significantly limited if the frustum continues to rotate. If the
horizontal rotation of the view frustum exceeds the horizontal
field-of-view of the frustum, then graphical surfaces that enter
the view frustum will still be moving on the image plane as they
exit the other side of the view frustum. In some cases, this
results in graphical surfaces that may be poorly resolved during
their entire lifespan in the view frustum. In these cases,
exemplary embodiments of the present method send only low
level-of-detail representations of the graphics surfaces, without
replacement.
* * * * *