U.S. patent application number 14/138045 was filed with the patent office on 2014-06-26 for system and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file fo.
This patent application is currently assigned to G. ANTHONY REINA. The applicant listed for this patent is JAMES OMER L'ESPERANCE, G. ANTHONY REINA, JAMES PAUL SMURRO. Invention is credited to JAMES OMER L'ESPERANCE, G. ANTHONY REINA, JAMES PAUL SMURRO.
Application Number | 20140176661 14/138045 |
Document ID | / |
Family ID | 50974170 |
Filed Date | 2014-06-26 |
United States Patent
Application |
20140176661 |
Kind Code |
A1 |
SMURRO; JAMES PAUL ; et
al. |
June 26, 2014 |
SYSTEM AND METHOD FOR SURGICAL TELEMENTORING AND TRAINING WITH
VIRTUALIZED TELESTRATION AND HAPTIC HOLOGRAMS, INCLUDING METADATA
TAGGING, ENCAPSULATION AND SAVING MULTI-MODAL STREAMING MEDICAL
IMAGERY TOGETHER WITH MULTI-DIMENSIONAL [4-D] VIRTUAL MESH AND
MULTI-SENSORY ANNOTATION IN STANDARD FILE FORMATS USED FOR DIGITAL
IMAGING AND COMMUNICATIONS IN MEDICINE (DICOM)
Abstract
The invention relates generally to a medical apparatus and
method of using the same for receiving and transmitting streaming
medical imagery and audio signals in real time, and allowing remote
operators to annotate and telestrate with same. The invention
acquires streaming medical imagery and audio signals through a
telestreamer input device, allowing users to electronically
collaborate, generally by telestrating, annotating, and sketching
image overlays on streaming medical imagery. Video images of
streaming imagery data displayed on a monitor are superimposed onto
a virtual mesh projected via computer graphics. The vertices of the
mesh move according to equations of motion based on a computational
physics engine. Virtual tools are projected above the mesh via
computer graphics. These virtual tools interact with the virtual
mesh according to the physics engine. The superposition of the
video images onto the virtual mesh makes it appear that points
within the video image are moving in a realistic manner and
reacting to the virtual tools with a realistic response. The
invention allows for recursive superposition of mesh layers, also
known as `surgi-skins`, and creation of a multi-layered virtual
mesh. Multi-layered surgi-skins synthesized from multi-modal
streaming medical imagery and saved together with multi-dimensional
[4-D] virtual mesh and multi-sensory annotation in single file
format as DICOM files are also known as `DICOM mesh` or `haptic
holograms`.
Inventors: |
SMURRO; JAMES PAUL; (SAN
CLEMENTE, CA) ; REINA; G. ANTHONY; (CORONADO, CA)
; L'ESPERANCE; JAMES OMER; (LA MESA, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
SMURRO; JAMES PAUL
REINA; G. ANTHONY
L'ESPERANCE; JAMES OMER |
SAN CLEMENTE
CORONADO
LA MESA |
CA
CA
CA |
US
US
US |
|
|
Assignee: |
REINA; G. ANTHONY
CORONADO
CA
L'ESPERANCE; JAMES OMER
LA MESA
CA
SMURRO; JAMES PAUL
SAN CLEMENTE
CA
MARIOTTI; MARK
WESTFORD
MA
MOLINARI, JR.; ANTHONY R.
GRAFTON
MA
HALEBLIAN; GEORGE E.
BELMONT
MA
|
Family ID: |
50974170 |
Appl. No.: |
14/138045 |
Filed: |
December 21, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61745383 |
Dec 21, 2012 |
|
|
|
Current U.S.
Class: |
348/14.06 ;
348/14.07 |
Current CPC
Class: |
G16H 20/40 20180101;
G06T 11/00 20130101; G16H 40/67 20180101; G16H 30/40 20180101; G06T
7/30 20170101; G16H 50/50 20180101; G06T 2207/10016 20130101; G06T
9/001 20130101; H04N 19/25 20141101; H04N 7/15 20130101; G16H 30/20
20180101; G06T 11/60 20130101; H04N 7/155 20130101 |
Class at
Publication: |
348/14.06 ;
348/14.07 |
International
Class: |
H04N 7/15 20060101
H04N007/15; G06Q 50/24 20060101 G06Q050/24 |
Claims
1. A method to perform video annotation using an augmented reality
telestrator.
2. The method of claim 1, wherein the video images are projected
onto a virtual mesh which moves based on a physically-realistic
computational model of the actual object being displayed in the
video images. In the current embodiment, the virtual mesh is
constructed in computer graphics as a rectangle made from
equilateral triangles whose vertices are interconnected. Movement
between vertices of the virtual mesh is calculated via
physics-based calculations, including but not limited to Hooke's
spring law and Newton's laws of motion. A UV-map is constructed to
superimpose the video images onto the virtual mesh. As the vertices
of the virtual mesh move based on the physics-based calculations,
the superimposed video images are transformed in the corresponding
positions. Physical parameters of the mesh and tools, including
multi-sensory (e.g. haptic) feedback may be saved using
standardized file formats (e.g. COLLADA) to be used in conjunction
with the DICOM medical imagery.
3. The method of claim 1, wherein computer-generated, virtual tools
are overlaid on the video images and can manipulate the images in a
realistic manner based on the physically-realistic model mesh of
claim 2. In the current embodiment, these virtual tools are
three-dimensional rendering of scissors, sutures, and forceps which
can be used to cut, stitch, and push/pull/twist points within the
video images. In effect, these video images appear to react in a
realistic manner to the virtual tools based on the physics-based
calculations of the virtual mesh of claim 2.
4. A network system apparatus allowing users to capture, retrieve
and view both real time and archived medical images for synchronous
or asynchronous communication, collaboration and consultation by
one or more users using illustrations over the medical images,
comprising: a telenetwork server including at least one associated
database having the capability to communicate with a local area
network; and at least one telestreamer in network communication
with a telenetwork server via a local area network wherein the
telestreamers capture one or more medical images and provide the
medical images via the network communication to the telenetwork
server via the local area network as it receives medical images
from at least one source
5. A network system apparatus system allowing users to capture,
retrieve and view both real time and archived medical imagery
streams for synchronous or asynchronous communication,
collaboration and consultation by one or more users using
illustrations over the medical images as in claim 4 wherein; one or
more users retrieve and view the medical images, creating
illustrations over the medical images such as, but not limited to
drawing, annotating, telestrating; and storing the medical images
with the annotations, and saving all the medical images together
with annotations and metadata on the telenetwork server, on a
picture archiving and communications system server, in a known
digital imaging and communications in medicine format.
6. A method for allowing one or more users to capture, retrieve and
view both real time and archived medical images for synchronous or
asynchronous communication, collaboration and consultation by one
or more users using illustrations over the medical images
comprising; running a computer program, storing the program on each
of user's computers, displaying the graphical user interface output
of that program on a computer display; linking each user's computer
to a telenetwork server using a local area network, each user
communicating with the telenetwork server, the telemedicine image
management system server providing permission to each user wherein
linking the user to a digital imaging and communications in
medicine modality worklist utility, a medical image archive server,
and telestreamers for capturing, retrieving and viewing medical
images, user's illustrating over the medical images, telenetwork
server managing all illustration file sharing wherein new user's
illustrations are appending periodically to the telenetwork server,
maintaining the file on the user's computer, the telenetwork server
linking to the internet and other users, wherein the images remain
on the telenetwork server and the user illustrations are appended
to that telenetwork server; streaming images into a local area
network wherein the telenetwork server having associated database
in communication with telestreamers streamers connected directly to
medical imaging modalities for acquiring one or more medical
images, streaming those medical images to the local area
network;
7. A method for allowing users to capture, retrieve and view both
real time and archived medical imagery streams for synchronous or
asynchronous communication, collaboration and consultation by one
or more users using illustrations over the medical images as in
claim 6 wherein; one or more users retrieve and view the medical
images, creating illustrations over the medical images such as, but
not limited to drawing, annotating, telestrating and storing the
medical images with the illustrations, and viewing all of the
user's illustrations as they happen and can save all the
illustrations from all participant clients on their local
respective computer storage devices, on the telenetwork server, on
a picture archiving and communications system, in a known digital
imaging and communications in medicine format.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of:
[0002] U.S. Provisional Application No. 61/745,383 filed Dec. 21,
2012 entitled "SYSTEM AND METHOD FOR SURGICAL TELEMENTORING USING
VIRTUALIZED TELESTRATION," naming as inventors, G. Anthony Reina
and James Omer L'Esperance, which is incorporated herein by
reference in its entirety.
[0003] This application may be related to the following commonly
assigned and commonly filed U.S. patent applications, each of which
is incorporated herein by reference in its entirety: [0004] 1. U.S.
patent application Ser. No. US 2011/0282141 A1 entitled "METHOD AND
SYSTEM OF SEE-THROUGH CONSOLE OVERLAY", naming as inventors
Itkowitz et al., filed on 17 Nov. 2011. [0005] 2. U.S. patent
application Ser. No. US 2011/0282140 A1 entitled "METHOD AND SYSTEM
OF HAND SEGMENTATION AND OVERLAY USING DEPTH DATA", naming as
inventors Itkowitz et al., filed on 17 Nov. 2011. [0006] 3. U.S.
patent application Ser. No. US 2010/0164950 A1 entitled "EFFICIENT
3-D TELESTRATION FOR LOCAL ROBOTIC PROCTORING", naming as inventors
Zhao et al., filed on 1 Jul. 2010. [0007] 4. U.S. patent
application Ser. No. 8,169,468 B2 entitled "AUGMENTED STEREOSCOPIC
VISUALIZATION FOR SURGICAL ROBOT", naming as inventors Scott et
al., filed on 1 May 2012. [0008] 5. U.S. patent application Ser.
No. US 2009/0036902 A1 entitled "INTERACTIVE USER INTERFACE FOR
ROBOTIC MINIMALLY INVASIVE SURGICAL SYSTEMS", naming as inventors
DiMaio et al., filed on 5 Feb. 2009. [0009] 6. U.S. patent
application Ser. No. US 2011/0107238 A1 entitled "NETWORK-BASED
COLLABORATED TELESTRATION ON VIDEO, IMAGES, OR OTHER SHARED VISUAL
CONTENT", naming as inventors Liu and Zhou, filed on 5 May 2011.
[0010] 7. U.S. patent application Ser. No. 7,492,363 B2 entitled
"TELESTRATION SYSTEM", naming as inventors Meier et al., filed on
17 Feb. 2009. [0011] 8. Patent application Ser. No. CA2545508 C
entitled "CAMERA FOR COMMUNICATION OF STREAMING MEDIA TO A REMOTE
CLIENT", naming as inventors Kavanagh et al., filed on Oct. 7,
2003. [0012] 9. U.S. patent application Ser. No. US 20090210801 A1
entitled "N-way multimedia collaboration systems", naming as
inventors Bakir et al., filed on 19 Feb. 2008. [0013] 10. U.S.
patent application Ser. No. US 20060122482 A1 entitled "Medical
image acquisition system for receiving and transmitting medical
images instantaneously and method of using the same", naming as
inventors Mariotti et al., filed on 22 Nov. 2004. [0014] 11. U.S.
patent application Ser. No. US20110126127 A1 entitled "System and
method for collaboratively communicating on images and saving those
communications and images in a standard known format", naming as
inventors Mariotti et al., filed on 23 Nov. 2009.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
[0015] Not applicable.
REFERENCE TO SEQUENCE LISTING, A TABLE, OR A COMPUTER PROGRAM
LISTING COMPACT DISC APPENDIX
[0016] Not applicable
BACKGROUND OF THE INVENTION
[0017] 1. Field of Invention
[0018] Aspects of this invention are related to telestration for
remote video collaborating with streaming medical imagery and are,
more specifically, related to enhancing a remote telementor's
ability to annotate and interact with the images in a more
realistic, yet virtualized manner through simulating the movement
and reaction of the displayed images according to a computational
physics model.
[0019] 2. Description of Related Art
[0020] Industries that develop, manufacturer, and maintain complex
products often find an insufficient number of employees with
extensive training and experience to meet demand. This is
particularly relevant as businesses become more geographically
diverse. It is inefficient (and sometime physically impossible) to
deploy an expert "into the field" on every occasion at a moment's
notice. Rather, companies typically deploy technicians with
relative degrees of experience who collaborate with the expert
remotely. For example, a multi-national aerospace company might
have local technicians in an Italian production plant conferring
with senior designers in the United States regarding the
fabrication concerns for a specialized airframe. Similarly,
technicians on an ocean oil rig may consult with shore side experts
to address problems with specialized drilling machinery.
Traditionally, video monitoring, as described in previous art, has
been instrumental in achieving this collaboration.
[0021] Conventional tele-monitoring (aka teleconferencing) allows
real-time audio and video tele-collaboration to improve education,
training, and performance in many fields. Current collaboration
methods include telestration, which can be performed either locally
or remotely to identify regions of interest within the video
images. For example, television personalities routinely annotate
video of live or replayed video broadcasts to highlight their
commentary. Similarly, flight engineers can remotely inspect
possible damage to space vehicles using telestrated,
high-definition images of the equipment while it is still in orbit.
In short, expert know-how can be maintained at a centralized
location while being mobilized anywhere at a moment's notice.
[0022] Current telestration techniques, as defined in prior art,
primarily display freehand and other two-dimensional drawings over
a video image or series of images. However, true collaboration is
better achieved if the remote expert can demonstrate information
through movement and manipulation of the images. In this invention,
a computer simulation of the objects within the video images is
constructed so that they can be manipulated in a more realistic
manner.
Medical Imaging and Medical Information Technology
[0023] The promotion of electronic medical records has spurred the
expansion of healthcare information technology (HIT) infrastructure
and led to the growth of medical information technologies, such as
networked medical imaging and virtual reality (VR).
[0024] Traditionally, a medical image is produced when an operator
or technician conducts a scan of the patient with a medical imaging
apparatus. Medical imaging modalities include X-ray, CT, MRI, and
ultrasound scanners. The operator uses the imaging apparatus to
save the image (in still or motion video format) onto a hard copy
(e.g. film), into the memory, or into an image storage database or
repository, such as, a Picture Archiving and Communications System
(PACS). PACS is a storage and management system for multiple
medical imaging modalities. These images, such as X-rays, MRI and
CAT scans, generally require a greater amount of storage than
images in non-medical industries. An operator, or user, such as a
surgeon, can use PACS to retrieve the saved images either locally
or remotely and conceivably use them for navigational or
interventional guidance during a surgical procedure.
[0025] Digital Imaging and Communications in Medicine (DICOM) is a
standard for managing medical data, including medical imaging.
DICOM has many roles in healthcare information technology: It is a
standard for exchanging digital information which ensures
interoperability between medical imaging equipment (such as
radiological imaging) and other systems. It is a protocol for
medical device communication over a network, defining syntax and
semantics for commands and associated information that can be
exchanged. It is a file format and medical directory structure to
facilitate access to images and related information stored on media
that shares information. It is a printing and display standard to
ensure that medical imagery is uniformly presented independent of
the device.
Application of Virtual Reality (VR) Technologies in Healthcare
[0026] Virtual reality applications in the healthcare industry are
associated with many areas of medical technology innovation
including robot-assisted surgery, augmented reality (AR) surgery,
computer-assisted surgery (CAS), image-guided surgery (IGS),
surgical navigation, pre-operative surgical planning, virtual
colonoscopy, virtual surgical simulation, and virtual reality
exposure therapy (VRET). In addition to intraoperative surgical
navigation and guidance, VR tools are often used for medical data
visualization, including multi-modality image fusion and advanced
2D/3D/4D image reconstruction. Education and training applications
include virtual surgical and procedural simulators. Patient use of
VR tools find application in rehabilitation and therapy, including
immersive VR systems for pain management, behavioral therapy,
psychological therapy, physical rehabilitation, and motor skills
training Clinical benefits of healthcare VR technology include
improved patient outcomes, reduced medical errors, improved
minimally-invasive surgical (MIS) technique, improved physician
collaboration in diagnosis, and improved psychological and motor
rehabilitation.
BRIEF SUMMARY OF THE INVENTION
[0027] The invention relates generally to a multimedia
collaborative teleconferencing system and method of using the same
for generating telestrations and annotations on streaming medical
imagery and saving same for tele-consultation, tele-collaboration,
tele-monitoring, tele-proctoring, and tele-mentoring with others
users.
[0028] The apparatus includes a medical image acquisition system
adapted for receiving and transmitting medical images, constructed
from a computer having communications capability adapted for
acquisition and transmission of a plurality of medical imaging and
video signals. Wherein the medical image and video signals are
acquired at the medical device's native resolutions, the apparatus
transmits the signals at their native resolutions and native frame
rates to a receiving device, receiving the medical imaging video
signals in analog or digital form, and if required, compressing
and/or scaling the signal, converting the signal to digital form
for transmission, and transmitting the digital signals to a display
device.
[0029] A computer can be defined as typically made of several
components such as a main circuit board assembly having a central
processing unit, memory storage to store programs and files, other
storage devices such as hard drives, and portable memory storage, a
power supply, a sound and video circuit board assembly, a display,
and an input device such as a keyboard, mouse, stylus pen and the
like allowing control of the computer graphics user interface
display, where any two or more of such components may be physically
integrated or may be separate. Any user on the network can store
files on the server and a network server is a computer that manages
network traffic.
[0030] The medical image acquisition system is capable of acquiring
signals from a plurality of medical imaging systems including but
not limited to, ultrasound, computer tomography (CT) scan,
fluoroscopy, endoscopy, magnetic resonance imaging, nuclear
medicine, echocardiogram ultrasound and microscopy. The medical
receiving device acquires the video image signal from a plurality
of video sources, including but not limited to, S-video, composite
color and monochrome, component red blue green video (RGB, three
additive primary colors), Digital Visual Interface (DVI), any video
transport protocol including digital and analog protocols, high
definition multimedia interface (HDMI, compact audio video
interface uncompressed digital data), serial digital interface
(SDI), and DICOM video in their native, enhanced or reduced
resolutions or their native, enhanced or reduced frame rates.
[0031] The apparatus includes a storage device adapted for
archiving the video signal in a predetermined digital format,
including Digital Imaging and Communications for Medicine (DICOM).
Data is transmitted using secure encryption protocols and video
signal resolution is transmitted at the same resolution as the
received signal. In one illustration, a remote location
communicates with the networked computer, for the purpose of
collaborating and conferencing.
[0032] The present invention improves on existing telestration
techniques via the addition of virtual telestration tools that can
physically manipulate the video images in a natural way based on a
physics model of the object(s) being displayed. Telestration
techniques described in prior art rely on freehand drawing of lines
or shapes which are then displayed as overlays onto the video
images. In the current embodiment, the user controls virtual tools
which are able to cut, push, pull, twist, and suture the video
images as if they were actually manipulating human tissue.
[0033] While the current embodiment is a natural fit for
telestrating/telementoring over real-time or stored medical images,
such as with surgical telemedicine, the method can be applicable to
any telestration requiring one user to demonstrate the use of a
tool to an operator who is actually using the tool at that time.
Although this technique is naturally suited to such remote
student-mentor scenarios, it can also be applied to single-user
interfaces. Most notably, with the application of the computational
physics model included in the current invention, the user can
practice a technique in a virtualized manner on live video images
prior to actually performing the maneuver.
[0034] This flexibility makes the technique adaptable for the use
in remote fieldwork. For example, a telecommunications technician
working in a remote location can receive realtime guidance from an
expert located elsewhere. Through virtual tool telestration, the
expert can annotate which segments to push, pull, twist, and cut in
a realistic, but still virtualized manner. The local technician can
also use the same annotation tools to practice the task under the
guidance of the expert before actually performing the task. By
adjusting parameters of the virtual video mesh and computational
physics model described below, these annotation techniques can be
applied to approximate any objects displayed within the video.
[0035] The present invention is accomplished using a combination of
both hardware and software. The software used for the present
invention is stored on one or more processor readable storage media
including hard disk drives, RAM, ROM, optical drives, and other
suitable storage devices. In alternative embodiments, some or all
of the software may be replaced with dedicated hardware, including
custom integrated circuits and electronic processors.
[0036] The novelty of this invention is: [0037] (1) the ability to
create and modify `synthetic` DICOM information objects. These
objects, referred to as `surgi-skins`, are multi-modal, multi-layer
virtual meshes synthesized from streaming medical imagery. [0038]
(2) the ability to encapsulate `surgi-skins` with user metadata,
synchronized audio annotations, and haptic annotation, plus save
that data together in a single file format structure based on DICOM
and referred to as the `DICOM Mesh`.
[0039] The advantages and novelty of the present invention will
appear more clearly from the following description and figures in
which the preferred embodiment of the invention is described in
detail.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING
Description List
[0040] Within the figures, the following reference characters are
used to refer to the following elements of the exemplary system
illustrated in the drawings. [0041] 10 is an exemplary video
stream. [0042] 12 is a 3D mesh object virtual tool exemplification.
[0043] 14 is a tele-video mesh overlay. [0044] 16 is an exemplary
mesh deformation. [0045] 18 is an exemplary mesh tear.
Figures
[0046] FIG. 1 is a detailed view of the virtual mesh telestration.
In this example, a rectangular 12-column grid (14) of equilateral
triangles (aka virtual mesh) is constructed via computer graphics.
Each vertex (black circle) is connected to another via a
computational physic model (spring) which calculates the vertex's
three-dimensional position using pre-programmed parameters,
including a spring constant, gravitational acceleration, and a
damping factor. The border vertices (black squares) remain in fixed
positions. The video image of an outstretched left arm (10) is
superimposed onto the virtual mesh. A virtual scalpel (12) is
superimposed over both 10 and 14.
[0047] FIG. 2 is a detailed view based on FIG. 1 after the virtual
scalpel has been moved to the left which simulates a cut to the
virtual mesh (12'.fwdarw.12). The vertices of the virtual mesh (14)
move according to the computational physics engine and create new
sub-triangles within the mesh (16). This movement creates a void
(18) in the mesh. The superimposed video image of the outstretched
left arm (10) moves according to the displacement of the associated
vertices of the virtual mesh and gives the appearance that the
virtual scalpel (12) has in fact "cut" the arm in a realistic
manner. Nevertheless, although the original video image is
displayed in a distorted manner, the data (and the actual arm)
remain unchanged.
[0048] FIG. 3 is a detailed view of the virtual mesh telestration
using a forceps tool (12). As with FIG. 1, the virtual mesh is
constructed with a 12-column rectangular arrangement of equilateral
triangles (14) whose vertices move according to a computational
physics model (spring). A video image of an outstretched left arm
(10) is superimposed onto the virtual mesh.
[0049] FIG. 4 is a detailed view based on FIG. 4 after the virtual
forceps have moved a vertex up and to the left (12'.fwdarw.12).
With this tool movement, no vertices are created nor destroyed, but
instead move according to the computational physics model
(stretched and squeezed springs). The superimposed video image of
the outstretched left arm (10) moves according to the displacement
of the associated vertices of the virtual mesh and gives the
appearance that the virtual forceps has pulled a section of the arm
up and to the left. Nevertheless, although the original video image
is displayed in a distorted manner, the data (and the arm) remain
unchanged.
[0050] FIG. 5 is a workflow diagram of the application and method.
An imaging device (e.g. video camera) (#4) is captured by a
Surgicom Telenetwork server (#3) and sent to the Surgicom
Telestreamer (#2) which digitizes its content and transmits it over
telecommunication lines in realtime. A 3D virtual tool telestrator
(#1) receives the video telestream and allows the client to
annotate the video images as described in FIGS. 1 and 2 using
virtual telestration tools. These annotations are streamed back to
the Surgicom Telestreamer (#2) which updates the original video
source device stream (#1) with the annotated version. Note that
multiple 3D virtual tool telestrators (#1) may act as clients to
the Surgicom Telestreamer (#2). All clients view the same video
images and can annotate them independently. Also, note that the
Surgicom Telenetwork server (#3) is able to save, store, and
transmit data to the Surgicom Telestreamer (#2) from recorded
sources.
[0051] FIG. 6 is a network diagram of the "Surgicom Telementoring
Network" consisting of a Surgicom Telenetwork server, Surgicom
Streamer Stack, Local (O.R.) clients, LAN, WAN, and web-based
tele-proctors and clients.
[0052] FIG. 7 is a workflow diagram of the Surgicom P.A.C.S. and
Holographic systems connected to the Surgicom Streamer Stack and
the Surgicom Telenetwork Server.
[0053] FIG. 8 is an exemplar of the haptic holographic display
using a virtual mesh and virtual 3D tools.
[0054] FIG. 9 is a workflow diagram of a Surgicom Surgical
Telementoring Session.
[0055] FIG. 10 is an exemplar of the Surgicom Console session. FIG.
10A ("See One") shows how pre-operative surgical planning, review,
road map, and holographic `priors` can be accessed from the system.
FIG. 10B ("Do One") shows how a surgeon can simulate alternative
approaches with virtual `cuts` and synthesize into haptic
holograms. FIG. 10C ("Teach One") shows how to annotate and save
haptic holograms as teaching files with mulit-modal medical imagery
and send to PACS.
DETAILED DESCRIPTION OF THE INVENTION
[0056] In the following description, a preferred embodiment of the
invention is described with regard to process and design elements.
However, those skilled in the art would recognize, after reading
this application, that alternate embodiments of the invention may
be implemented with regard to hardware or software without
requiring undue invention.
General Features of the Method and System
[0057] There are 3 main components to this method: [0058] (1) the
virtual mesh [0059] (2) the UV texture map [0060] (3) the virtual
tools.
Virtual Mesh
[0061] The virtual mesh is a computer graphics representation of a
video display where each vertex of the mesh corresponds to a
position within the video image. In a static display, the virtual
mesh is analogous to a pixel map of the video image. In this
invention, however, the vertices of the virtual mesh are not
necessarily aligned with the pixels of the video image. More
importantly, the locations of the vertices are not fixed in space,
but rather can move with respect to one another as if each vertex
were a physical object (or a part of a physical object) in the real
world.
[0062] In the current instantiation, the virtual mesh is
constructed using equilateral triangles arranged in a 12-column
grid (FIG. 1). Equilateral triangles were chosen because they are
computationally easier to sub-divide than other geometric shapes.
Nevertheless, any shape (2D or 3D) can be used to create the mesh.
In addition, multiple meshes of varying configurations can be
produced to represent features and objects within the streamed
imaging modality. Further, the overall mesh is rectangular in shape
because video images are usually displayed in this manner; but, the
shape of the mesh can changed to conform to the needs of the
telestration.
[0063] Machine vision techniques may be applied to sub-divide the
mesh according to objects within the video image. For example, a
mesh displaying a video of an automobile could be sub-divided into
body, wheels, and background--with each sub-segment of the mesh
being programmed to mimic the physical characteristics of the
objects they represent. This would compensate for any relative
movement among the camera, objects, or field of view.
[0064] In the current embodiment, a surgeon could identify regions
of interest within the image (e.g. major organs, nerves, or blood
vessels) by encircling them with conventional freehand drawing
telestration. An optical flow algorithm, such as the Lucas Kinade
method, could be used to track each region of interest within the
realtime video. The virtual mesh would be continually updated to
change the parameters of the sub-meshes based on the regions of
interest. This would ensure, for example, that a cut in the mesh
which was made to overlay the prostate would keep the same relative
position and orientation with respect to the prostate regardless of
movement.
[0065] The vertices of the virtual mesh are interconnected in
movement using a computational physics model of the object being
represented. In the current instantiation, the physics model
assumes that vertices are connected via springs which obey the
physical constraints of Hooke's Law and gravitational acceleration.
By changing the parameters, such as spring constant, gravitational
acceleration, and damping factor, the behavior of the virtual mesh
can be adjusted between various levels of fluidity. For example,
the current embodiment can be made to approximate human skin, but
different types of human tissue could also be represented in the
same telestrated video. The properties for both the virtual mesh
and physics model can be stored in a standard known format, such as
the Collaborative Design Activity (COLLADA).
[0066] It should be noted that although the computational physics
model is currently formulated to simulate movement in typical
environments, it could be equally used to simulate movement of
objects in exotic environments, such as in space or underwater by
computationally changing the nature of the virtual mesh.
UV Mapping
[0067] UV mapping is a three-dimensional (3D) modeling process
which maps a two-dimensional (2D) image onto the three-dimensional
surface. Other patents and techniques sometimes refer to this
technique as "texture mapping". Every 3D object in computer
graphics is made up of a series of connected polygons. UV mapping
allows these polygons to be painted with a color from a 2D image
(or texture). Although in its current instantiation the virtual
mesh is a 2D object, it can be texture mapped with a 2D video image
in the same manner. Further, using the UV mapping, the same
technique can be applied to true 3D virtual meshes of any
configuration.
[0068] By superimposing the video image onto the virtual mesh using
a UV map, the video image will be distorted whenever the virtual
mesh is distorted. In effect, the process allows points and
segments of the video image to move and react to the telestration.
In fact, if polygons within the virtual mesh are deleted (e.g.
cutting the mesh as in FIG. 2), the projected video image will not
display the area which is mapped to those polygons. Similarly, if
the polygon changes shape (e.g. pulling the mesh as in FIG. 4), the
projected video image will display the area mapped to that polygon
with precisely the same geometric distortion.
Virtual Tools
[0069] Virtual tools are computer-generated objects which are
programmed to interact with the virtual mesh according to a
computational physics engine. In the current instantiation, the
invention uses three virtual tools: a virtual scalpel, a virtual
forceps, and a virtual suture. All three tools are programmed to
push, pull, and twist the virtual mesh according to the physics
engine using standard ray-casting techniques and colliders.
[0070] The virtual scalpel separates the connections between the
triangles that are in contact with the scalpel tip. This results in
a void between those triangles and makes the video image appear to
have been cut in the mapped area. Further, if an entire section of
the virtual mesh is "cut" from the existing mesh, the UV mapped
area of the video image will appear to be physically removed from
the remainder of the video image. The edge of the cut mesh then
acts as an edge of tissue; so the edge of the cut surface will
deform when manipulated, independent of the other side of the cut
mesh.
[0071] The virtual forceps attaches to the triangle closest to the
forceps tip when activated. It creates an external force on the
attached triangles within the computation physics model of the
virtual mesh. The forceps can be used to drag the attached
triangles (FIG. 4) and gives the illusion that the video image is
being grabbed by the forceps in a realistic manner. After the
forceps is deactivated, the external force is removed from the
computational physics model. The affected triangles will continue
to react to internal (reaction) forces until they eventually return
to a steady-state position.
[0072] The virtual suture allows the telestrator to add connections
between triangles. The suture is modeled by a spring. When
activated, the suture tool adds a spring to the computational
physics engine between any two points specified. This tool can be
used to join previously cut sections of the virtual mesh.
[0073] Although in its current instantiation the virtual tools are
limited to these three, the flexibility of the computational
physics engine allows the technique to be readily expanded to
include the use of any tool or object which can be modeled,
including drills, retractors, stents, and suction devices. It also
allows for multisensory annotations, including haptic and audio
feedback from tool use, to be realistically modeled and stored.
[0074] In addition, the parameters for the virtual tools, mesh, and
physics engine can be saved along with multi-sensory (e.g. haptic)
data in standard known 3D file formats, such as the Collaborative
Design Activity (COLLADA) and Immersion Force Resource (IVS/IFR)
specification, and Haptic Multimedia Broadcasting formats, such as
MPEG-4 Binary Format for Scene (BIFS) and the University of Iowa's
3D Holovideo format.
Application
[0075] In order to illustrate the method proposed in this
invention, consider the field of surgery. Adequate surgical
collaboration requires one practitioner demonstrating a technique
to another practitioner. Current telestration techniques are unable
to demonstrate surgical techniques, such as dissection, clamping,
and suturing. It is not sufficient to know simply where or when to
cut; the surgeon must be able to also demonstrate how to cut--how
to hold the instrument, how hard to push, and how quickly to move.
These limitations of conventional telestration as described in
prior art are exacerbated in situations where the practitioners may
be in different locations. These telestration techniques are
insufficient for true surgical telementoring or any video
annotation requiring a procedure to be demonstrated especially when
complex techniques are being demonstrated to new students.
[0076] Virtual tool telestration, as described herein and which
makes up at least a part of the present disclosure, may allow the
mentoring surgeon to interact with a virtual video-overlay mesh of
the operative field and mimic the technique needed to perform the
operation. The surgeon mentor can demonstrate suturing and
dissecting techniques while they are virtually overlaid on a video
of the actual operative field. Notably, the mentoring surgeon can
demonstrate the surgical technique effectively without actually
changing the operative field.
[0077] Current telestration methods have limited conventional
telemedicine to non-surgical fields of medicine. However, with the
system and method of the present disclosure, it may be possible
that telemedicine/telementoring will become crucial to surgical
practice and, indeed, any field where collaboration requires
demonstrating rather than merely describing an idea.
[0078] In fact, there is growing concern that the advance of
minimally invasive surgery (MIS) is grossly outpacing the evolution
of surgical training This application will assist in bridging the
learning curves for surgeons performing the MIS procedures. In
addition, as live video and other imaging modalities become more
prevalent in clinical practice, the telestration described herein
will become inherent to all forms of medicine. A virtual tool
telestrator is the critical element to enable adequate surgical
telestration. Such a telestrator may be adapted to work in a 2-D or
a 3-D video environment with applications not just with visible
light images, but with other modalities, including (but not limited
to) fluoroscopy, tomography, and magnetic resonance imagery.
[0079] Additionally, telestration is currently used in a number of
non-medicine fields. The most common application is with
professional sports broadcasting whereby sports commentators can
"draw" on the televideo and emphasize certain elements of the
video, such as the movement of the players. Adding 3D virtual
telestration tools, as described herein, to these existing
telestration devices and tools could be invaluable to such
modalities. For example, bomb disposal experts could use virtual
tools to interact with the remote video signal transmitted by
ordinance disposal robots to signal the robot to push or pull
certain areas of the field of view. Sculptors could use virtual
hands to indicate to their student the proper finger position on a
piece of unformed clay--and demonstrate how the clay should move
without actually affecting the real world object. Any real world
object that can be imaged can be transmitted and manipulated in a
collaborative, yet virtualized manner. Such a method and device
would be a natural fit for wearable computers or head-mounted
displays, such as Google Glass and the Occulus VR Rift, to provide
better augmented reality solutions.
[0080] Virtual tool telestration may be equally effective in a 2-D
or a 3-D environment or representation and differs from what
currently exists in the field of telestration. It is typically
constructed from three components (FIG. 5):
[0081] 1. a 3D virtual tool telestrator
[0082] 2. a Surgicom Telestreamer
[0083] 3. a Surgicom Telenetwork server
[0084] These elements (as demonstrated in the drawings) may be
related to each other in the following exemplary and non-limiting
fashion.
[0085] The Surgicom Telestreamer (#2) may be a computer networking
device which allows for audio and video signals to be sent in
realtime to remote viewers. In one embodiment, the Surgicom
Telestreamer captures streaming medical imagery and transmits it
over the internet using a real-time streaming protocol (RTSP) in a
H.264 video compression/decompression (codec) format at 1080p
resolution of 60 frames per second.
[0086] The 3D virtual tool telestrator (#1) may be a computer
program which displays the Surgicom telestream (#2) as a 3D mesh
object on a video monitor, allows for a remote users to overlay
virtual 3D tools (e.g. forceps, scalpels) which can be moved by the
remote user and which can interact with the video mesh. For
example, the remote user may virtually grab a section of the video
mesh with the forceps and that part of the mesh will move in a
manner similar to that of the actual object being displayed in the
video (e.g. a section of the bladder neck during prostate
removal).
[0087] The 3D Virtual Tool Telestrator (#1) will transmit the
virtualized surgical telestration of the remote user back to the
source Surgicom Telestreamer (#2) for display. To conserve
transmission bandwidth, the 3D Virtual Tool Telestrator (#1) only
sends the position and orientation of the virtual tools and the
virtual mesh to the Surgicom Telestreamer (#2) along with the
timestamp of the current video frame. In this manner, bandwidth
requirements and latency are minimized.
[0088] The 3D virtual tool telestrator (#1) may be comprised of
computer software written, by way of an exemplary and non-limiting
example, with mostly open-sourced software development packages,
such as by using a programming environment like but not limited to
C++, C#, Mono, Silverlight, and Unity3D. The telestrator may
include 3D graphics rendering engine, such as but not limited to
Unity3D, which may be used to display the 3D virtual tools and a
virtual mesh with triangular vertices. The telestrator may also
include a physics simulator, such as but not limited to PhysX, to
handle the virtual simulation and interaction between the
virtualized 3D tools and the video mesh. The telestrator may also
include a multimedia player, such as but not limited to AVPro
LiveCapture, which may be used to overlay a video input stream from
#2 onto the virtual mesh to create a virtual operative field. The
telestrator will use human input devices, such as the Razer Hydra
joystick or the Geomagic Touch to control movement of the virtual
tools in a natural way.
[0089] A similar computer program exists on the Surgicom
Telestreamer (#2). However, unlike the 3D virtual tool telestrator
(#1), this program renders the graphics without the computational
physics engine. Instead, the position and orientation of the
virtual tools and virtual mesh that were passed back from the
virtual tool telestrator (#1) are used to create an exact rendering
of the virtual tool telestration at that timestamp. In this way,
the Surgicom Telestreamer (#2) can display an exact rendering of
the 3D virtual tool telestration to all clients simultaneously.
[0090] While the invention has been described with reference to
preferred embodiments, it is to be understood that the invention is
not intended to be limited to the specific embodiments set forth
above. Thus, it is recognized that those skilled in the art will
appreciate that certain substitutions, alterations, modifications,
and omissions may be made without departing from the spirit or
intent of the invention. Accordingly, the foregoing description is
meant to be exemplary only, the invention is to be taken as
including all reasonable equivalents to the subject matter of the
invention, and should not limit the scope of the invention set
forth in the following claims.
[0091] The Surgicom Telenetwork server (#3) can save and store the
medical images having the overlaid drawn annotated and telestrated
images in a PACS using the DICOM format, and saving the session
information that includes the collaboration session ID, client
information, image information including associated metadata, and
date and times of the session
* * * * *