U.S. patent application number 16/599414 was filed with the patent office on 2020-04-16 for robotically-assisted surgical device, robotically-assisted surgery method, and system.
The applicant listed for this patent is Medicaroid Corporation Ziosoft, Inc.. Invention is credited to Shusuke CHINO, Jota IDA, Yutaka KARASAWA, Yukihiko KITANO, Tsuyoshi NAGATA, Shinichiro SEO.
Application Number | 20200113637 16/599414 |
Document ID | / |
Family ID | 70161964 |
Filed Date | 2020-04-16 |
![](/patent/app/20200113637/US20200113637A1-20200416-D00000.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00001.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00002.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00003.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00004.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00005.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00006.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00007.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00008.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00009.png)
![](/patent/app/20200113637/US20200113637A1-20200416-D00010.png)
View All Diagrams
United States Patent
Application |
20200113637 |
Kind Code |
A1 |
IDA; Jota ; et al. |
April 16, 2020 |
ROBOTICALLY-ASSISTED SURGICAL DEVICE, ROBOTICALLY-ASSISTED SURGERY
METHOD, AND SYSTEM
Abstract
A robotically-assisted surgical device assists robotic surgery
with a surgical robot that includes at least one robot arm holding
a surgical instrument. The robotically-assisted surgical device
includes a processing unit and a display unit. The processing unit
is configured to: acquire 3D data of a subject; acquire kinematic
information regard to the robot arm; acquire information of an
surgical procedure for operating the subject; acquire information
regarding a position of at least one port which is to be pierced on
a body surface of the subject; derive a 2D range on the body
surface where errors are allowed for the piercing of the port based
on the 3D data, the kinematic information, the information of the
surgical procedure, and the position of the port; and cause the
display unit to display the information regarding the position of
the port and information indicating the 2D range.
Inventors: |
IDA; Jota; (Kobe-shi,
JP) ; KITANO; Yukihiko; (Kobe-shi, JP) ;
CHINO; Shusuke; (Minato-ku, JP) ; NAGATA;
Tsuyoshi; (Minato-ku, JP) ; KARASAWA; Yutaka;
(Minato-ku, JP) ; SEO; Shinichiro; (Minato-ku,
JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Medicaroid Corporation
Ziosoft, Inc. |
Kobe-shi
Minato-ku |
|
JP
JP |
|
|
Family ID: |
70161964 |
Appl. No.: |
16/599414 |
Filed: |
October 11, 2019 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A61B 34/70 20160201;
A61B 34/30 20160201; A61B 6/032 20130101; A61B 2034/305
20160201 |
International
Class: |
A61B 34/30 20060101
A61B034/30; A61B 34/00 20060101 A61B034/00; A61B 6/03 20060101
A61B006/03 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 11, 2018 |
JP |
2018-192931 |
Claims
1. A robotically-assisted surgical device that assists minimally
invasive robotic surgery with a surgical robot that includes at
least one robot arm holding a surgical instrument, the
robotically-assisted surgical device comprising a processing unit
and a display unit, wherein the processing unit is configured to:
acquire 3D data of a subject; acquire kinematic information regard
to the robot arm; acquire information of a surgical procedure for
operating the subject; acquire information regarding a position of
at least one port which is to be pierced on a body surface of the
subject; derive a 2D range on the body surface where errors are
allowed for the piercing of the port based on the 3D data, the
kinematic information, the information of the surgical procedure,
and the position of the port; and cause the display unit to display
the information regarding the position of the port and information
indicating the 2D range.
2. The robotically-assisted surgical device according to claim 1,
wherein there are at least two different distances between the
position of the port and respective positions on a contour of the
2D range.
3. The robotically-assisted surgical device according to claim 1,
wherein the processing unit is configured to perform a
pneumoperitoneum simulation on volume data of the subject to
generate the 3D data of a virtual pneumoperitoneum state.
4. The robotically-assisted surgical device according to claim 3,
wherein the processing unit is configured to: perform a plurality
of pneumoperitoneum simulations on volume data of the subject with
different amounts of pneumoperitoneum to generate the 3D data of a
plurality of virtual pneumoperitoneum states; derive a plurality of
2D ranges based on the 3D data of the plurality of virtual
pneumoperitoneum states; and display the information regarding the
position of the port and information regarding the plurality of 2D
ranges.
5. The robotically-assisted surgical device according to claim 3,
wherein the processing unit is configured to: perform a plurality
of pneumoperitoneum simulations on volume data of the subject with
different amounts of pneumoperitoneum to generate the 3D data of a
plurality of virtual pneumoperitoneum states; derive a plurality of
2D ranges based on the 3D data of the plurality of virtual
pneumoperitoneum states; derive a minimum allowable range which is
a range on the body surface commonly included in the plurality of
2D ranges; and display the information regarding the position of
the port and information regarding the minimum allowable range.
6. The robotically-assisted surgical device according to claim 1,
wherein a shape of the 2D range includes a primitive shape.
7. The robotically-assisted surgical device according to claim 1,
wherein the processing unit is configured to: cause the display
unit to visualize the 3D data with an annotation of the information
regarding the position of the port and information regarding a
position of the 2D range.
8. The robotically-assisted surgical device according to claim 1,
wherein the processing unit is configured to cause a projection
unit to project visible light representing the information
regarding the position of the port and information regarding a
position of the 2D range to the body surface.
9. A robotically-assisted surgery method of a robotically-assisted
surgical device that assists robotic surgery with a surgical robot
that includes a robot arm holding a surgical instrument, the
robotically-assisted surgery method comprising: acquiring 3D data
of a subject; acquiring kinematic information regard to the robot
arm of the surgical robot; acquiring information of a surgical
procedure for operating the subject; acquiring information
regarding a position of a port that is to be pierced on a body
surface of the subject; deriving a 2D range on the body surface of
the subject where errors are allowed for the piercing of the port
based on the 3D data, the kinematic information of the surgical
robot, the surgical procedure, and the position of the port, and
displaying the information regarding the position of the port and
information indicating the 2D range.
10. The robotically-assisted surgery method according to claim 9,
wherein there are at least two different distances between the
position of the port and respective positions on a contour of the
2D range.
11. The robotically-assisted surgery method according to claim 9,
further comprising: performing a pneumoperitoneum simulation on
volume data of the subject to generate the 3D data of a virtual
pneumoperitoneum state.
12. The robotically-assisted surgery method according to claim 11,
further comprising: performing a plurality of pneumoperitoneum
simulations on volume data of the subject with different amounts of
pneumoperitoneum to generate the 3D data of a plurality of virtual
pneumoperitoneum states; deriving a plurality of 2D ranges based on
the 3D data of the plurality of virtual pneumoperitoneum states;
and displaying the information regarding the position of the port
and information regarding the plurality of 2D ranges.
13. The robotically-assisted surgery method according to claim 11,
further comprising: performing a plurality of pneumoperitoneum
simulations on volume data of the subject with different amounts of
pneumoperitoneum to generate the 3D data of a plurality of virtual
pneumoperitoneum states; deriving a plurality of 2D ranges based on
the 3D data of the plurality of virtual pneumoperitoneum states;
deriving a minimum allowable range which is a range on the body
surface commonly included in the plurality of 2D ranges; and
displaying the information regarding the position of the port and
information regarding the minimum allowable range.
14. The robotically-assisted surgery method according to claim 9,
wherein a shape of the 2D range includes a primitive shape.
15. The robotically-assisted surgery method according to claim 9,
further comprising: causing the display unit to visualize the 3D
data with an annotation of the information regarding the position
of the port and information regarding a position of the 2D
range.
16. The robotically-assisted surgery method according to claim 9,
further comprising: causing a projection unit to project visible
light representing the information regarding the position of the
port and information regarding a position of the 2D range to the
body surface.
17. A robotically-assisted surgery system of a robotically-assisted
surgical device that assists robotic surgery with a surgical robot
that includes at least one robot arm holding a surgical instrument,
the robotically-assisted surgery system comprising: acquiring 3D
data of a subject; acquiring kinematic information regard to the
robot arm of the surgical robot; acquiring information of an
surgical procedure for operating the subject; acquiring information
regarding a position of a port that is to be pierced on a body
surface of the subject; deriving a 2D range on the body surface of
the subject where errors are allowed for the piercing of the port
based on the 3D data, the kinematic information of the surgical
robot, the surgical procedure, and the position of the port, and
displaying the information regarding the position of the port and
information indicating the 2D range.
18. The robotically-assisted surgery system according to claim 17,
wherein there are at least two different distances between the
position of the port and respective positions on a contour of the
2D range.
19. The robotically-assisted surgery system according to claim 17,
further comprising: performing a pneumoperitoneum simulation on
volume data of the subject to generate the 3D data of a virtual
pneumoperitoneum state.
20. The robotically-assisted surgery system according to claim 19,
further comprising: performing a plurality of pneumoperitoneum
simulations on volume data of the subject with different amounts of
pneumoperitoneum to generate the 3D data of a plurality of virtual
pneumoperitoneum states; deriving a plurality of 2D ranges based on
the 3D data of the plurality of virtual pneumoperitoneum states;
and displaying the information regarding the position of the port
and information regarding the plurality of 2D ranges.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is based on and claims priority under 35
USC 119 from Japanese Patent Application No. 2018-192931 filed on
Oct. 11, 2018, the contents of which are incorporated herein by
reference.
TECHNICAL FIELD
[0002] The present disclosure relates to a robotically-assisted
surgical device that assists robotic surgery with a surgical robot,
a robotically-assisted surgery method, and a system.
BACKGROUND ART
[0003] In the related art, when robotic surgery is operated using a
surgical robot, a port is pierced to insert forceps into the body
of a patient being operated. The position of the port is
approximately determined depending on a surgical procedure, but the
optimal position thereof has yet to be established. US2014/0148816A
discloses port placement planning. Specifically, a surgical port
placement system disclosed in US2014/0148816A generates a surgical
port placement model based on a plurality of parameter sets
associated with a plurality of past surgical procedures, receives a
given parameter set for a given surgical procedure including
physical characteristics of a given patient, and plans at least one
port position for the given patient for the given surgical
procedure based on live given parameter set and the surgical port
placement model.
SUMMARY OF INVENTION
[0004] The present disclosure provides a robotically-assisted
surgical device capable of recognizing piercing accuracy required
for piercing a port, a robotically-assisted surgery method, and a
system.
[0005] According to one aspect of the disclosure, a
robotically-assisted surgical device assists minimally invasive
robotic surgery with a surgical robot that includes at least one
robot arm holding a surgical instrument. The robotically-assisted
surgical device includes a processing unit and a display unit. The
processing unit is configured to: acquire 3D data of a subject;
acquire kinematic information regard to the robot arm; acquire
information of surgical procedure for operating the subject;
acquire information regarding a position of at least one port which
is to be pierced on a body surface of the subject; derive a 2D
range on the body surface where errors are allowed for the piercing
of the port based on the 3D data, the kinematic information, the
information of the surgical procedure, and the position of the
port; and cause the display unit to display the information
regarding the position of the port and information indicating the
2D range.
[0006] According to another aspect of the disclosure, a
robotically-assisted surgery method is a method of a
robotically-assisted surgical device that assists robotic surgery
with a surgical robot that includes a robot arm holding a surgical
instrument. The robotically-assisted surgery method includes:
acquiring 3D data of a subject; acquiring kinematic information
regard to the robot arm of the surgical robot; acquiring
information of a surgical procedure for operating the subject;
acquiring information regarding a position of a port that is to be
pierced on a body surface of the subject; deriving a 2D range on
the body surface of the subject where errors are allowed for the
piercing of the port based on the 3D data, the kinematic
information of the surgical robot, the surgical procedure, and the
position of the port, and displaying the information regarding the
position of the port and information indicating the 2D range.
[0007] According to further another aspect of the disclosure, a
robotically-assisted surgery system is a system of a
robotically-assisted surgical device that assists robotic surgery
with a surgical robot that includes at least one robot arm holding
a surgical instrument. The robotically-assisted surgery system
includes: acquiring 3D data of a subject; acquiring kinematic
information regard to the robot arm of the surgical robot;
acquiring information of a surgical procedure for operating the
subject; acquiring information regarding a position of a port that
is to be pierced on a body surface of the subject; deriving a 2D
range on the body surface of the subject where errors are allowed
for the piercing of the port based on the 3D data, the kinematic
information of the surgical robot, the surgical procedure, and the
position of the port, and displaying the information regarding the
position of the port and information indicating the 2D range.
[0008] According to the present disclosure, the piercing accuracy
required for piercing the port can be recognized.
BRIEF DESCRIPTION OF DRAWINGS
[0009] FIG. 1 is a block diagram illustrating a hardware
configuration example of a robotically-assisted surgical device
according to a first embodiment;
[0010] FIG. 2 is a block diagram illustrating a functional
configuration example of the robotically-assisted surgical
device;
[0011] FIG. 3 is a view illustrating examples of MPR images of an
abdomen before and after performing a pneumoperitoneum
simulation;
[0012] FIG. 4 is a view illustrating a measurement example of a
port position of a pre-pierced port;
[0013] FIG. 5A is a view illustrating a first placement example of
port positions placed on a body surface of a subject;
[0014] FIG. 5B is a view illustrating a second placement example of
port positions placed on the body surface of the subject;
[0015] FIG. 5C is a view illustrating a third placement example of
port positions placed on the body surface of the subject;
[0016] FIG. 6 is a view illustrating an example of a positional
relationship between the subject, ports, trocars, and robot arms
during robotic surgery;
[0017] FIG. 7 is a flowchart illustrating an example of a procedure
of a port position simulation by the robotically-assisted surgical
device;
[0018] FIG. 8 is a flowchart illustrating an operation example when
a port position score is calculated by the robotically-assisted
surgical device;
[0019] FIG. 9 is a view illustrating an example of working areas
determined based on port positions;
[0020] FIG. 10 is a flowchart illustrating a derivation procedure
of allowable error information by the robotically-assisted surgical
device;
[0021] FIG. 11 is a view illustrating a first example of an error
for port positions in a thickness direction of the subject;
[0022] FIG. 12 is a view illustrating a second example of the error
for port positions in the thickness direction of the subject;
[0023] FIG. 13 is a view illustrating a measurement example of a
straight line distance;
[0024] FIG. 14 is a view illustrating a measurement example of a
curved distance;
[0025] FIG. 15 is a view illustrating an adjustment example of the
allowable error depending on the amount of pneumoperitoneum;
[0026] FIG. 16 is a view illustrating a first display example of
guide information including information regarding an allowable
region of a port to be pierced;
[0027] FIG. 17 is a view illustrating a second display example of
guide information including information regarding an allowable
region of a port to be pierced;
[0028] FIG. 18 is a view illustrating a display example of port
positions according to Comparative Example:
[0029] FIG. 19 is a view illustrating a display example of port
positions and allowable error information according to the
embodiment; and
[0030] FIG. 20 is a view illustrating designation of a piercing
position on a 2D plane and misplacement of the piercing position in
a 3D space.
DESCRIPTION OF EMBODIMENTS
[0031] Hereinafter, embodiments of the present disclosure will be
described using the drawings.
[0032] In the present disclosure, a robotically-assisted surgical
device assists minimally invasive robotic surgery with a surgical
robot that includes at least one robot arm holding a surgical
instrument. The robotically-assisted surgical device includes a
processing unit and a display unit. The processing unit is
configured to: acquire 3D data of a subject, acquire kinematic
information regard to the robot arm; acquire information of a
surgical procedure for operating the subject; acquire information
regarding a position of at least one port which is to be pierced on
a body surface of the subject; derive a 2D range on the body
surface where errors are allowed for the piercing of the port based
on the acquired 3D data, the acquired kinematic information, the
acquired information of the surgical procedure, and the acquired
position of the port; and cause the display unit to display the
information regarding the position of the port and information
indicating the derived 2D range.
[0033] According to the present disclosure, the
robotically-assisted surgical device displays the port position and
the information indicating the 2D range, and thus, a user can
recognize the degree to which the error is allowed during the
piercing of the port PT. That is, the piercing accuracy required
for piercing the port PT can be recognized.
[0034] (Circumstances for Achievement of Aspect of Present
Disclosure)
[0035] In some cases, an assistant pierces a port according to
preoperative planning. However, it is difficult to accurately
pierce a port at a planned port position. For example, as
illustrated in FIG. 20, when a port position to be pierced is
preoperatively planned on a 2D plane, for example, a port is
planned to be pierced at a position at a distance L1 from a navel,
even with the same distance on the 2D plane, the position largely
changes in a forward-backward direction of a subject as it moves
toward a lateral part of the subject (refer to a range .alpha.).
Therefore, it is difficult to accurately perform the measurement
and to uniquely determine a port position to be pierced during port
piercing.
[0036] In addition, in robotic surgery, pneumoperitoneum is
performed in many cases. During pneumoperitoneum, carbon dioxide
gas is injected into the abdominal cavity to secure a working space
in the abdominal cavity. Since the degree of elevation of abdominal
wall and condition in an abdominal cavity varies depending on the
pneumoperitoneum state, a 3D position planned on a body surface of
a patient is also variable.
[0037] In addition, in robotic surgery, arms with an end effector
(forceps) that are included in a surgical robot may come into
contact with each other such that a movable range of the arms is
limited. In addition, during minimally invasive surgery using a
surgical robot, application of stress to a port is limited. It is
generally understood that, due to the above-described reasons,
higher piercing accuracy during port piercing is required as
compared to minimally invasive surgery by a human. As a result, it
takes a long period of time for high-accuracy measurement.
[0038] In addition, various end effectors (forceps) are inserted
into a subject through ports, and a work (treatment) is performed
according to a surgical procedure. As the end effectors, effectors
for various uses corresponding to works are present. The piercing
accuracy for the ports varies depending on the contents of works or
the uses of end effectors. Accordingly, like the ports, ports that
require high piercing accuracy and ports that do not require high
piercing accuracy may be present together. Accordingly, if a
piercer (for example, an assistant) who performs piercing can
recognize whether or not a port requires high piercing accuracy
during piercing, the piercer can easily prepare for piercing the
port.
[0039] In the following embodiment, a robotically-assisted surgical
device capable of recognizing piercing accuracy required for
piercing a port, a robotically-assisted surgery method, and a
program will be described.
First Embodiment
[0040] FIG. 1 is a block diagram illustrating a configuration
example of a robotically-assisted surgical device 100 according to
a first embodiment. The robotically-assisted surgical device 100
assists robotic surgery with a surgical robot 300 and performs, for
example, a preoperative simulation, an intraoperative simulation,
and intraoperative navigation.
[0041] The surgical robot 300 includes a robot operation terminal,
a robot main body, and an image display terminal.
[0042] The robot operation terminal includes a hand controller or a
foot switch manipulated by an operator. The robot operation
terminal operates a plurality of robot arms AR provided in the
robot main body according to a manipulation of the hand controller
or the footswitch by the operator. In addition, the robot operation
terminal includes a viewer. The viewer may be a stereo viewer and
may merge images input through an endoscope to display a 3D image.
A plurality of robot operation terminals may be present such that a
plurality of operators operate the plurality of robot operation
terminals to perform robotic surgery.
[0043] The robot main body includes: a plurality of robot arms for
performing robotic surgery; and an end effector EF (forceps, an
instrument) as a surgical instrument that is mounted on the robot
arm AR.
[0044] The robot main body of the surgical robot 300 includes four
robot arms AR including: a camera arm on which an endoscope camera
is mounted; a first end effector arm on which an end effector EF
operated by a right-hand controller of the robot operation terminal
is mounted; a second end effector arm on which an end effector EF
operated by a left-hand controller of the robot operation terminal
is mounted; and a third end effector arm on which an end effector
EF for replacement is mounted. Each robot arm AR includes a
plurality of joints and includes a motor and an encoder
corresponding to each joint. Each robot arm AR has at least 6
degrees of freedom and preferably 7 or 8 degrees of freedom,
operates in a 3D space, and may be movable in each direction in the
3D space. The end effector EF is an instrument that actually comes
into contact with a treatment target in a subject PS during robotic
surgery, and can perform various treatments (for example, gripping,
dissection, exfoliation, or suture).
[0045] Examples of the end effector EF may include gripping
forceps, exfoliating forceps, an electric knife, and the like. A
plurality of different end effectors EF may be prepared for
respective functions. For example, in robotic surgery, a treatment
of dissecting a tissue with one end effector EF while holding or
pulling the tissue with two end effectors EF may be performed. The
robot arm AR and the end effector EF may operate based on an
instruction from the robot operation terminal.
[0046] The image display terminal includes a monitor, a controller
for processing an image captured by a camera of an endoscope to
display the image on a viewer or a monitor, and the like. The
monitor is checked by, for example, an assistant of robotic surgery
or a nurse.
[0047] The surgical robot 300 receives a manipulation of the hand
controller or the footswitch of the robot operation terminal by the
operator, controls the operation of the robot arm AR or the end
effector EF of the robot main body, and performs robotic surgery in
which various treatments are performed on the subject PS. In
robotic surgery, laparoscopic surgery is performed in the subject
PS.
[0048] In robotic surgery, a port PT is pierced on the body surface
of the subject PS, and pneumoperitoneum is performed through the
port PT. In pneumoperitoneum, carbon dioxide may be injected to
inflate the abdominal cavity of the subject PS. In the port PT, a
trocar TC may be provided. The trocar TC includes a valve and
maintains the inside of the subject PS to be airtight. In addition,
in order to maintain the airtight state, air (for example, carbon
dioxide) is intermittently introduced into the subject PS.
[0049] The end effector EF (shaft of the end effector EF) is
inserted into the trocar TC. The valve of the trocar TC is opened
during insertion of the end effector EF and is closed during the
separation of the end effector EF. The end effector EF is inserted
from the port PT through the trocar TC such that various treatments
are performed according to the surgical procedure. Robotic surgery
may be applied to not only laparoscopic surgery in which the
surgery target is the abdomen but also arthroscopic surgery in
which the surgery target includes a region other than the
abdomen.
[0050] As illustrated in FIG. 1, the robotically-assisted surgical
device 100 includes a communication unit 110, a user interface (UI)
120, a display 130, a processor 140, and a memory 150. The UI 120,
the display 130, and the memory 150 may be included in the
robotically-assisted surgical device 100 or may be provided
separately from the robotically-assisted surgical device 100.
[0051] A CT (Computed Tomography) apparatus 200 is connected to the
robotically-assisted surgical device 100 through the communication
unit 110. The robotically-assisted surgical device 100 acquires
volume data from the CT apparatus 200 and processes the acquired
volume data. The robotically-assisted surgical device 100 may be
configured by a PC (Personal Computer) and software installed on
the PC. The robotically-assisted surgical device 100 may be
configured as a part of the surgical robot 300.
[0052] The surgical robot 300 is connected to the
robotically-assisted surgical device 100 through the communication
unit 110. The robotically-assisted surgical device 100 may provide
various data, information, or images from, for example, the
surgical robot 300 to assist robotic surgery. The
robotically-assisted surgical device 100 may acquire, from, for
example, the surgical robot 300, information regarding a mechanism
or the operation of the surgical robot 300 or data obtained before,
during, or after robotic surgery such that various kinds of
analysis or interpretation can be performed based on the acquired
information or data. The analysis result or the interpretation
result may be visualized.
[0053] A measuring instrument 400 is connected to the
robotically-assisted surgical device 100 through the communication
unit 110. The measuring instrument 400 may measure information (for
example, a body surface position of the subject PS) regarding the
subject PS (for example, a patient) to be operated by the surgical
robot 300. The measuring instrument 400 may measure a position of
the port PT provided on the body surface of the subject PS. The
measuring instrument 400 may be, for example, a depth sensor 410.
The depth sensor 410 may be included in the surgical robot 300 (for
example, the robot main body) or may be provided in the ceiling or
the like of an operating room where robotic surgery is performed.
In addition, the measuring instrument 400 may receive an input of
the result of manual measurement of an operation unit of the
measuring instrument 400. In the manual measurement, for example,
information regarding a patient or a port position on the body
surface may be measured by a ruler or a tape measure.
[0054] In addition, the CT apparatus 200 is connected to the
robotically-assisted surgical device 100. Alternatively, instead of
the CT apparatus 200, a device capable of capturing various images
may be connected to the robotically-assisted surgical device 100.
This device may be, for example, an angiographic device or an
ultrasound device. This device may be used to check the internal
state of the subject PS before and during robotic surgery.
[0055] The CT apparatus 200 irradiates an organism with X-rays and
acquires images (CT images) using a difference in X-ray absorption
depending on tissues. The subject PS may be for example, a human
body or an organism. The subject PS may not be a human body nor an
organism. For example, the subject PS may be an animal or a phantom
for surgical training.
[0056] A plurality of CT images may be acquired in a time series.
The CT apparatus 200 generates volume data including information
regarding any portion inside the organism. Here, any portion inside
the organism may include various organs (for example, brain, heart,
kidney, colon, intestine, lung, chest, lacteal gland, and prostate
gland). By acquiring the CT image, it is possible to obtain a pixel
value (CT value, voxel value) of each pixel (voxel) of the CT
image. The CT apparatus 200 transmits the volume data as the CT
image to the robotically-assisted surgical device 100 via a wired
circuit or a wireless circuit.
[0057] Specifically, the CT apparatus 200 includes a gantry (not
illustrated) and a console (not illustrated). The gantry includes
an X-ray generator (not illustrated) and an X-ray detector (not
illustrated) and acquires images at a predetermined timing
instructed by the console to detect an X-ray transmitted through
the subject PS and to obtain X-ray detection data. The X-ray
generator includes an X-ray tube (not illustrated). The console is
connected to the robotically-assisted surgical device 100. The
console acquires a plurality of X-ray detection data from the
gantry and generates volume data based on the X-ray detection data.
The console transmits the generated volume data to the
robotically-assisted surgical device 100. The console may include
an operation unit (not illustrated) for inputting patient
information, scanning conditions regarding CT scanning, contrast
enhancement conditions regarding contrast medium administration,
and other information. This operation unit may include an input
device such as a keyboard or a mouse.
[0058] The CT apparatus 200 continuously captures images to acquire
a plurality of 3D volume data such that a moving image can also be
generated. Data of the moving image generated the plurality of 3D
volume data will also be referred to as 4D (four-dimensional)
data.
[0059] The CT apparatus 200 may capture CT images at each of a
plurality of timings. The CT apparatus 200 may capture a CT image
in a state where the subject PS is contrast-enhanced. The CT
apparatus 200 may capture a CT image in a state where the subject
PS is not contrast-enhanced.
[0060] In the robotically-assisted surgical device 100, the
communication unit 110 performs communication of various data or
information with other devices. The communication unit 110 may
perform communication of various data with the CT apparatus 200,
the surgical robot 300, and the measuring instrument 400. The
communication unit 110 performs wired communication or wireless
communication. The communication unit 110 may be connected to the
CT apparatus 200, the surgical robot 300, and the measuring
instrument 400 in a wired or wireless manner.
[0061] The communication unit 110 may acquire various information
for robotic surgery from the surgical robot 300. The various
information may include, for example, kinematic information of the
surgical robot 300. The communication unit 110 may transmit various
information for robotic surgery to the surgical robot 300. The
various information may include, for example, information (for
example, an image or data) generated by a processing unit 160.
[0062] The communication unit 110 may acquire various information
for robotic surgery from the measuring instrument 400. The various
information may include, for example, position information of the
body surface of the subject PS or information of a port position
pierced on the body surface of the subject PS that is measured by
the measuring instrument 400.
[0063] The communication unit 110 may acquire volume data from the
CT apparatus 200. The acquired volume data may be transmitted
immediately to the processor 140 for various processes, or may be
stored in the memory 150 first and then transmitted to the
processor 140 for various processes as necessary. In addition, the
volume data may be acquired via a recording medium.
[0064] The volume data acquired by the CT apparatus 200 may be
transmitted from the CT apparatus 200 to an image data server such
as (PACS: Picture Archiving and Communication Systems; not
illustrated) and stored therein. Instead of acquiring from the CT
apparatus 200, the communication unit 110 may acquire volume data
from the image data server. This way, the communication unit 110
functions as an acquisition unit that acquires various data such as
volume data.
[0065] The UI 120 may include a touch panel, a pointing device, a
keyboard, or a microphone. The UI 120 receives an input operation
from a user of the robotically-assisted surgical device 100. The
user may include a doctor, a radiographer, or other paramedic
staffs. The doctor may include an operator that manipulates a
surgeon console to operate robotic surgery or an assistant that
assists robotic surgery near the subject PS.
[0066] The UI 120 receives an operation such as a designation of a
region of interest (ROI), a setting of luminance conditions, and
the like in the volume data. The region of interest may include
various tissues (such as blood vessels, bronchial tubes, organs,
bones, brain, heart, feet, neck, and blood flow). The tissues may
broadly include tissues of the subject PS such as diseased tissue,
normal tissue, organs, and parts. In addition, the UI 120 may
receive an operation such as a designation of the region of
interest or a setting of luminance conditions in the volume data
with respect to an image (for example, a 3D image or a 2D image
described below) based on the volume data.
[0067] The display 130 may include a Liquid Crystal Display (LCD)
and displays various information. The various information may
include a 3D image or a 2D image obtained from the volume data. The
3D image may include, for example, a volume rendering image, a
surface rendering image, a virtual endoscope image (VE image), a
virtual ultrasound image, or a Curved Planar Reconstruction (CPR)
image. The volume rendering image may include a RaySum image (also
simply referred to as "SUM image"), a Maximum Intensity Projection
(MIP) image, a Minimum Intensity Projection (MinIP) image, an
average image, or a Raycast image. The 2D image may include an
axial image, sagittal image, a coronal image, a Multi Planar
Reconstruction (MPR) image, or the like. The 3D image and the 2D
image may include a color fusion image.
[0068] The memory 150 includes a primary storage device such as
various Read Only Memories (ROMs) or Random Access Memories (RAMs).
The memory 150 may include a secondary storage device such as a
Hard Disk Drive (HDD) or a Solid State Drive (SSD). The memory 150
may include a third storage device such as a USB memory or an SD
card. The memory 150 stores various information. The various
information includes information acquired via the communication
unit 110, information and an image generated from the processor
140, setting information set by the processor 140, and various
programs. The information acquired via the communication unit 110
may include, for example, information from the CT apparatus 200
(for example, volume data), information from the surgical robot
300, information from the measuring instrument 400, or information
from an external server. The memory 150 is an example of a
non-transitory recording medium in which a program is recorded.
[0069] A projection unit 170 projects visible light (for example,
laser light) to the subject. The projection unit 170 projects the
visible light to display various information (for example, the
information of the port position) on the body surface of the
subject PS (for example, the body surface of the abdomen). The
visible light, that is, the information displayed on the body
surface of the subject PS is recognized by the users (for example,
an assistant).
[0070] The processor 140 may include a Central Processing Unit
(CPU), a Digital Signal Processor (DSP), or a Graphical Processing
Unit (GPU). The processor 140 executes the program stored in the
memory 150 to function as the processing unit 160 controlling
various processes and controls.
[0071] FIG. 2 is a block diagram illustrating a functional
configuration example of the processing unit 160.
[0072] The processing unit 160 includes a region segmentation unit
161, an image generator 162, a deformation simulator 163, a port
position processing unit 164, a display controller 166, and a
projection controller 167.
[0073] The processing unit 160 integrates the respective units of
the robotically-assisted surgical device 100. The respective
sections included in the processing unit 160 may be implemented as
different functions by one piece of hardware or may be implemented
as different functions by a plurality of pieces of hardware. In
addition, the respective sections included in the processing unit
160 may be implemented by a dedicated hardware component.
[0074] The region segmentation unit 161 may perform segmentation
processing in the volume data. In this case, the UI 120 receives an
instruction from a user and transmits information of the
instruction to the region segmentation unit 161. The region
segmentation unit 161 may perform segmentation processing from the
volume data based on the information of the instruction using a
well-known method to segment the region of interest. In addition,
the region of interest may be set manually in accordance with the
specific instruction from the user. In addition, when an
observation target is predetermined, the region segmentation unit
161 may perform segmentation processing from the volume data to
segment the region of interest including the observation target
without the user instruction. The segmented region may include
regions of various tissues (for example, blood vessels, bronchial
tubes, organs, bones, brain, heart, feet, neck, blood flow, lacteal
gland, chest, and tumor). The observation target may be a target to
be treated by robotic surgery.
[0075] The image generator 162 may generate a 3D image or a 2D
image based on the volume data acquired from the communication unit
110. The image generator 162 may generate a 3D image or a 2D image
from the volume data acquired from the communication unit 110 based
on a designated region or the region segmented by the region
segmentation unit 161.
[0076] The deformation simulator 163 may perform a process relating
to deformation in the subject PS as a surgery target. For example,
the deformation simulator 163 may perform a pneumoperitoneum
simulation of virtually performing pneumoperitoneum on the subject
PS. A specific method of the pneumoperitoneum simulation may be a
well-known method, for example, a method described in Takayuki
Kitasaka, Kensaku Mori, Yuichiro Hayashi, Yasuhito Suenaga, Makoto
Hashizume, and Junichiro Toriwaki, "Virtual Pneumoperitoneum for
Generating Virtual Laparoscopic Views Based on Volumetric
Deformation", MICCAI (Medical Image Computing and Computer-Assisted
Intervention), 2004, P559-P567 which is incorporated herein by
reference. That is, the deformation simulator 163 may perform the
pneumoperitoneum simulation based on the volume data (volume data
before pneumoperitoneum (non-pneumoperitoneum state)) acquired from
the communication unit 110 or the region segmentation unit 161 to
generate volume data after pneumoperitoneum (volume data in the
pneumoperitoneum state). Through the pneumoperitoneum simulation,
the user can simulate a state where pneumoperitoneum is performed
on the subject PS without actually performing pneumoperitoneum on
the subject PS to observe a state where pneumoperitoneum is
virtually performed. Among pneumoperitoneum states, a state of
pneumoperitoneum estimated by the pneumoperitoneum simulation will
be referred to as "a virtual pneumoperitoneum state", and a state
where pneumoperitoneum is actually performed will also be referred
to as "an actual pneumoperitoneum state".
[0077] FIG. 3 is a view illustrating examples of MPR images of the
abdomen before and after performing the pneumoperitoneum
simulation. An image G11 illustrate the state before performing the
pneumoperitoneum simulation, which is a state (non-pneumoperitoneum
state) where the abdomen of the subject PS is not inflated. An
image G12 illustrate the state after performing the
pneumoperitoneum simulation, which is a state (virtual
pneumoperitoneum state) where the abdomen of the subject PS is
inflated and includes a pneumoperitoneum space KS. In robotic
surgery, the subject PS is operated in the pneumoperitoneum state.
Therefore, the pneumoperitoneum simulation is performed on the
volume data acquired in the non-pneumoperitoneum state by the
deformation simulator 163 and the volume data in the virtual
pneumoperitoneum state is derived.
[0078] The deformation simulator 163 may virtually deform the
observation target such as an organ or a disease in the subject PS.
The observation target may be a surgery target to be operated by
the operator. The deformation simulator 163 may simulate a state
where an organ is pulled, pressed, or dissected by forceps of the
end effector EF. In addition, the deformation simulator 163 may
simulate, for example, movement of an organ by a postural
change.
[0079] The port position processing unit 164 acquires information
of a plurality of ports PT provided on the body surface of the
subject PS. The information of the port PT may include, for
example, identification information of the port PT, information
regarding a position (port position) on the body surface of the
subject PS where the port PT is pierced, information regarding the
size of the port PT, or the like. The information of a plurality of
ports may be stored in the memory 150 or the external server as a
template. The information of the plurality of ports may be
determined according to the surgical procedure. The information of
the plurality of ports may be used for preoperative planning.
[0080] The port position processing unit 164 may acquire the
information of the plurality of ports positions from the memory
150. The port position processing unit 164 may acquire the
information of the plurality of port positions from the external
server via the communication unit 110. The port position processing
unit 164 may receive a designation of port positions of the
plurality of ports PT via the UI 120 to acquire the information of
the plurality of port positions. The information of the plurality
of ports may be the information of a combination of the plurality
of port positions.
[0081] The port position processing unit 164 acquires kinematic
information of the surgical robot 300. The kinematic information
may be stored in the memory 150. The port position processing unit
164 may acquire the kinematic information from the memory 150. The
port position processing unit 164 may acquire the kinematic
information from the surgical robot 300 or the external server via
the communication unit 110. The kinematic information may vary
depending on the surgical robot 300.
[0082] The kinematic information may include, for example, shape
information regarding the shape of an instrument (for example, the
robot arm AR or the end effector EF) for robotic surgery included
in the surgical robot 300 or operation information regarding the
operation thereof. This shape information may include information
of at least a part, for example, the length or weight of each
portion of the robot arm AR or the end effector EF, the angle of
the robot arm AR with respect to a reference direction (for
example, a horizontal plane), or the inclination angle of the end
effector EF with respect to the robot arm AR. This operation
information may include information of at least a part, for
example, the movable range of the robot arm AR or the end effector
EF in the 3D space, the position, velocity, or acceleration of the
arm during the operation of the robot arm AR, or the position,
velocity, or acceleration of the end effector EF relative to the
robot arm AR during the operation of the end effector EF.
[0083] In kinematics, not only the movable range of one arm but
also the movable range of another arm are considered and regulated
such that the robot arms AR do not interfere with each other.
Accordingly, the surgical robot 300 operates based on the
kinematics of each robot arm AR of the surgical robot 300, and
therefore, interference between the plurality of robot arms AR
during operation can be avoided.
[0084] The port position processing unit 164 acquires information
of the surgical procedure. The surgical procedure refers to the
procedure of surgery on the subject PS. The surgical procedure may
be designated via the UI 120. Each treatment in robotic surgery may
be determined depending on the surgical procedure. Depending on the
treatment, the end effector EF required for the treatment may be
determined. Accordingly, the end effector EF mounted on the robot
arm AR may be determined depending on the surgical procedure, and
the type of the end effector EF mounted on the robot arm AR may be
determined depending on the surgical procedure. In addition, a
minimum region that is required for the treatment or a recommended
region that is recommended to be secured for the treatment may be
determined depending on the treatment.
[0085] The port position processing unit 164 acquires information
of a target region. The target region may be a region including
targets (for example, tissues (such as blood vessels, bronchial
tubes, organs, bones, brain, heart, feet, and neck) to be treated
by robotic surgery. The tissues may broadly include tissues of the
subject PS such as diseased tissues, normal tissues, organs, and
parts.
[0086] The port position processing unit 164 may acquire
information regarding the position of the target region from the
memory 150. The port position processing unit 164 may acquire the
information of the position of the target region from the external
server via the communication unit 110. The port position processing
unit 164 may receive a designation of the position of the target
region via the UI 120 to acquire the information regarding the
position of the target region.
[0087] The port position processing unit 164 may execute a port
position simulation. The port position simulation may be a
simulation in which the user operates the UI 120 to determine
whether or not desired robotic surgery can be performed on the
subject PS. In the port position simulation, while simulating
surgery, the user may operate the end effector EF inserted into
each of the port positions in a virtual space to determine whether
or not the target region as a surgery target is accessible. That
is, in the port position simulation, while receiving the manual
operation of the surgical robot 300, the user may determine whether
or not a moving part (for example, the robot arm AR and the end
effector EF) of the surgical robot 300 relating to robotic surgery
is accessible to the target region as a surgery target without a
problem. The port position processing unit 164 may obtain port
position planning information through the port position
simulation.
[0088] In the port position simulation, whether or not the target
region is accessible may be determined based on the volume data of
the subject PS, the acquired combination of the plurality of port
positions, the kinematics of the surgical robot 300, the surgical
procedure, the volume data of the virtual pneumoperitoneum state,
and the like. While changing the plurality of port positions on the
body surface of the subject PS, the port position processing unit
164 may determine whether or not the target region is accessible at
each port position or may sequentially perform the port position
simulation. The port position processing unit 164 may designate
information regarding a finally preferable (for example, optimal)
combination of port positions according to the user input via the
UI 120. As a result, the port position processing unit 164 may plan
the plurality of port positions to be pierced. The details of the
port position simulation will be described below.
[0089] Using the plurality of port positions provided on the body
surface of the subject PS, the port position processing unit 164
may derive (for example, calculate) a port position score
representing the appropriateness for robotic surgery. That is, the
port position score based on the combination of the plurality of
port positions indicates the value of the combination of the
plurality of port positions for robotic surgery. The port position
score may be calculated based on the combination of the plurality
of port positions, the kinematics of the surgical robot 300, the
surgical procedure, the volume data of the virtual pneumoperitoneum
state, and the like. The port position score is derived for each
port position. The details of the port position score will be
described below.
[0090] The port position processing unit 164 may adjust the port
position based on the port position score. In this case, the port
position processing unit 164 may adjust the port position based on
the variation of the port position score according to the movement
of the port position. The details of the port position adjustment
will be described below.
[0091] As described above, the port position processing unit 164
may derive the plurality of port positions to be pierced according
to the port position simulation. In addition, the port position
processing unit 164 may derive the plurality of port positions to
be pierced based on the port position score.
[0092] The display controller 166 causes the display 130 to display
various data, information, or images. The display controller 166
may display the 3D image or the 2D image generated by the image
generator 162. The display controller 166 may display an image
showing the information of the plurality of ports PT (for example,
the information of the port positions) generated by the image
generator 162.
[0093] The display controller 166 may display an image which is
generated by the image generator 162 and indicates information
(allowable error information) indicating the error that is allowed
for each port when piercing each port PT. In this case, the display
controller 166 may display the allowable error information to
superimpose the 3D image or the 2D image.
[0094] The projection controller 167 controls the projection of the
visible light from the projection unit 170. The projection
controller 167 may control, for example, a frequency or an
intensity of the visible light, a position to which the visible
light is projected, or a timing at which the visible light is
projected.
[0095] The projection controller 167 causes the projection unit 170
to project the visible light to the subject PS and displays various
information on the body surface of the subject PS (for example, the
body surface of the abdomen). The projection controller 167 may
project laser light to the body surface of the subject PS to mark a
specific position on the body surface. The specific position may
be, for example, the port position to be pierced or a position on
the volume data where the observation target (for example, the
affected part) is present when shifted from the specific position
on the body surface in the normal direction. That is, the
projection controller 167 may be a laser pointer indicating the
port position. In this case, the port position may be displayed in
a state where the port position is extended to a range indicating
the allowable error.
[0096] In addition, the projection controller 167 may cause the
projection unit 170 to project the visible light to the body
surface of the subject PS to superimpose and display information
assistant robotic surgery (for example, the information regarding
the port position or the allowable error information) on the body
surface of the subject PS. The superimposing information may be,
for example, character information or graphic information. That is,
the projection controller 167 may assist the user in robotic
surgery using an augmented reality (AR) technique.
[0097] FIG. 4 is a view illustrating a measurement example of a
port position of the pre-pierced port PT1. The measurement of the
port position may be the 3D measurement. In FIG. 4, the subject PS
(for example, a patient) is horizontally placed on a bed BD.
[0098] The depth sensor 410 may include: a light-emitting portion
that emits infrared light; a light-receiving portion that receives
infrared light; and a camera that captures an image. The depth
sensor 410 may detect the distance front the depth sensor 410 to
the subject PS based on the infrared light that is emitted from the
light-emitting portion to the subject PS and reflected light that
is reflected from the subject PS and received by the
light-receiving portion. The depth sensor 410 may detect the upper,
lower, left, and right sides of an object using the image captured
by the camera. As a result, the depth sensor 410 may acquire
information of a 3D position (3D coordinates) of each position (for
example, the port position of the pre-pierced port PT1) on the body
surface of the subject PS.
[0099] The depth sensor 410 may include a processor and an internal
memory. The internal memory may store information regarding the
shape of the trocar TC. Referring to the shape information of the
trocar TC stored in the internal memory, the depth sensor 410 may
detect (recognize) the trocar TC provided in the port FT pierced on
the body surface of the subject PS to detect (measure) a 3D
position of the trocar TC.
[0100] In addition, a predetermined mark may be formed on a surface
of the trocar TC. The depth sensor 410 may capture an image using
the predetermined mark on the trocar TC as a feature point to
detect (recognize) the trocar TC by image recognition. As a result,
the depth sensor 410 can improve the recognition accuracy of the
trocar TC and can improve the measurement accuracy of the 3D
position of the trocar TC.
[0101] In addition, the depth sensor 410 may include a stereo
camera instead of the infrared sensor (the light-emitting portion
and the light-receiving portion) such that the 3D position of the
trocar TC can be measured by image processing. In this case, the
depth sensor 410 may measure the 3D position of the trocar TC by
recognizing the trocar TC by object recognition in an image
captured by a stereo camera, detecting (recognizing) the position
of the trocar TC on the body surface of the subject, and
calculating the distance to the trocar TC.
[0102] The depth sensor 410 may measure each position or the
position of the trocar TC on the body surface of the subject PS in
a range that can be reached by the infrared light emitted from the
infrared sensor or in a range where an image can be captured by the
camera (refer to a range A1 in FIG. 4).
[0103] The deformation simulator 163 of the robotically-assisted
surgical device 100 may acquire information regarding each position
on the body surface of the subject PS in the actual
pneumoperitoneum state, that is, information regarding the shape of
the body surface of the subject PS in the actual pneumoperitoneum
state from the depth sensor 410. In addition, the deformation
simulator 163 may extract the contour (corresponding to the body
surface) of the subject PS based on the volume data of the subject
PS in the non-pneumoperitoneum state to acquire information
regarding each position on the body surface of the subject PS in
the non-pneumoperitoneum state, that is, information regarding the
shape of the body surface of the subject PS in the
non-pneumoperitoneum state.
[0104] The deformation simulator 163 may calculate a difference
between each position on the body surface of the subject PS in the
actual pneumoperitoneum state and each position on the body surface
of the subject PS in the non-pneumoperitoneum state, that is, a
difference between the shape of the body surface of the subject PS
in the actual pneumoperitoneum state and the shape of the body
surface of the subject PS in the non-pneumoperitoneum state. As a
result, the deformation simulator 163 can recognize the amount of
pneumoperitoneum for allowing the actual pneumoperitoneum state of
the subject PS.
[0105] In addition, the deformation simulator 163 may correct a
simulation method or a simulation result of the pneumoperitoneum
simulation based on the difference between the actual
pneumoperitoneum state and the virtual pneumoperitoneum state in
the pneumoperitoneum simulation. That is, the deformation simulator
163 may correct a simulation method or a simulation result of the
pneumoperitoneum simulation based on the actual amount of
pneumoperitoneum. The deformation simulator 163 may store the
correction information in the memory 150. In addition, the
deformation simulator 163 may receive the amount of scavenging air
from a pneumoperitoneum device via the communication unit 110 to
correct a simulation method or a simulation result of the
pneumoperitoneum simulation. As a result, the robotically-assisted
surgical device 100 can improve the accuracy of the
pneumoperitoneum simulation.
[0106] Next, an example of displaying a port position will be
described.
[0107] The deformation simulator 163 performs the pneumoperitoneum
simulation on the volume data obtained in the non-pneumoperitoneum
state (for example, by preoperative CT scanning) to generate the
volume data of the virtual pneumoperitoneum state. The image
generator 162 may perform volume rendering on the volume data of
the virtual pneumoperitoneum state to generate a volume rendering
image. The image generator 162 may perform surface rendering on the
volume data of the virtual pneumoperitoneum state to generate a
surface rendering image.
[0108] The deformation simulator 163 may perform the
pneumoperitoneum simulation on the volume data obtained in the
non-pneumoperitoneum state (for example, by preoperative CT
scanning) to generate deformation information regarding deformation
from the non-pneumoperitoneum state to the virtual pneumoperitoneum
state. The image generator 162 may generate a surface from the
volume data acquired in the non-pneumoperitoneum state (for
example, by preoperative CT scanning) to generate a surface
rendering image. The image generator 162 may apply the shape
information to the surface generated from the volume data acquired
in the non-pneumoperitoneum state (for example, by preoperative CT
scanning) to generate a surface rendering image of the virtual
pneumoperitoneum state.
[0109] The display controller 166 may cause the display 130 to
visualize the 3D data (the volume rendering image or the surface
rendering image of the virtual pneumoperitoneum state) with an
annotation of the port position derived from the port position
processing unit 164. The display controller 166 causes the display
130 to display the allowable error information.
[0110] The projection controller 167 may project visible light to
the port position on the body surface of the subject PS (for
example, a patient) derived by the port position processing unit
164 to indicate the port position using the visible light and to
visualize the port position. As a result, the user can perform a
treatment such as piercing on the port position while checking the
port position on the body surface of the subject PS. In addition,
the projection controller 167 may project visible light to display
the allowable error information on the body surface of the subject
PS.
[0111] The projection controller 167 may project visible light to
the subject PS to display information regarding the port position
on the body surface of the subject PS (for example, a patient)
derived by the port position processing unit 164. In this case, the
projection controller 167 may display the information regarding the
port position (for example, the identification information of the
port or an arrow indicating the port position) to superimpose the
subject PS using an AR technique. In addition, the projection
controller 167 may display the allowable error information to be
superimposed on the subject PS using an AR technique. As a result,
referring to guide information by the visible light, the user can
perform a treatment such as piercing on the port position while
checking the information regarding the port position on the body
surface of the subject PS.
[0112] Here, the deformation information will be described in
detail.
[0113] The deformation simulator 163 detects movement (deformation)
of each of the portions included in the volume data to generate the
deformation information based on the plurality of volume data (CT
images) obtained before and after pneumoperitoneum. In this case,
the deformation simulator 163 performs movement analysis
(deformation analysis) on the deformation of the plurality of
volume data based on the plurality of volume data regarding the
amount of pneumoperitoneum to acquire the deformation information
in the volume data. A specific method of the deformation analysis
is described in, for example, U.S. Pat. No. 8,311,300 and Japanese
Patent No. 5408493 which is incorporated herein by reference. These
methods are examples of non-rigid registration but may be rigid
registration.
[0114] The deformation simulator 163 may acquire, as the
deformation information, information regarding the amount of
movement or information regarding the velocity at a given point of
the volume data. When the method described in US2014/0148816A which
is incorporated herein by reference is applied, the deformation
simulator 163 separates the volume data into a 2D lattice node (k,
l), and 2D coordinates (x, y) in a phase node (k, l, t) of a phase
t of the 2D lattice is obtained. In this case, based on a
difference between a plurality of nodes (k, l, t) obtained by
changing the value of the phase t, the information regarding the
amount of movement at the lattice point of the node (k, l) may be
calculated. In addition, the deformation simulator 163 may
differentiate the information regarding the amount of movement with
time to calculate the information regarding the velocity. The
information regarding the amount of movement or the velocity may be
expressed by a vector.
[0115] When the deformation simulator 163 interpolates the
deformation information of the 2D lattice at each point of the
entire volume data, the deformation information of each point of
the volume data can be obtained. When the deformation information
of a predetermined point is applied to each point of a region
including an observation site, the deformation information of each
point of the region including the observation site can be
obtained.
[0116] In addition, when the method described in Japanese Patent
No. 5408493 is applied, the deformation simulator 163 may generate
the deformation information based on volume data tk-1 and time
information tk-1 thereof and volume data tk and time information tk
thereof among the volume data (before and after pneumoperitoneum)
aligned in time series. The deformation information may indicate
information regarding a corresponding position on the plurality of
volume data or correspondence of a corresponding object or
information regarding the process of a change in the movement of
the position and the object. A pixel of each volume data is an
index indicating a position at any time between time k-1 and time
k.
[0117] The deformation simulator 163 is not limited to the method
of US2014/0148816A and may perform deformation analysis using
another well-known registration method. The robotically-assisted
surgical device 100 performs deformation analysis on each point or
the observation site using the deformation information, and thus
the movement of any position in the subject before and after
pneumoperitoneum can be grasped.
[0118] Next a specific example of a standard port position will be
described.
[0119] FIG. 5A is a view illustrating a first placement example of
port positions placed on the body surface of the subject PS. FIG.
5B is a view illustrating a second placement example of port
positions placed on the body surface of the subject PS. FIG. 5C is
a view illustrating a third placement example of port positions
placed on the body surface of the subject PS. The placement of a
plurality of port positions may be determined, for example,
according to the surgical procedure. In FIGS. 5A to 5C, the
physical size of the subject PS or the position or size of a
disease or the like of the observation target is not
considered.
[0120] A plurality of port positions illustrated in FIGS. 5A to 5C
are port positions that are planned to be pierced. There may be
some errors between the port positions that are planned to be
pierced and the port positions that are actually pierced. For
example, there may be an error of about 25 mm.
[0121] The ports PT provided on the body surface of the subject PS
may include a camera port PTC into which a camera CA is inserted,
an end effector port PTE into which the end effector EF is
inserted, and an auxiliary port PTA into which forceps held by an
assistant are inserted. A plurality of ports PT may be present for
each of the types (for example, for each of the camera port PTC,
the end effector port PTE, and the auxiliary port PTA), or the
sizes of the different types of ports PT may be the same as or
different from each other. For example, the end effector port PTE
into which the end effector EF for holding an organ or the end
effector EF of which the movement in the subject PS is complex is
inserted may be larger than the end effector port PTE into which
the end effector EF as an electric knife is inserted. The placement
position of the auxiliary port PTA may be planned relatively
freely.
[0122] In FIG. 5A, large numbers of the end effector ports PTE and
the auxiliary ports PTA are linearly arranged in the right
direction of the subject PS and in the left direction of the
subject PS, respectively, with respect to the port position of the
camera port PTC as a reference (the vertex).
[0123] In FIG. 5B, large numbers of end effector ports PTE and the
auxiliary ports PTA are linearly aligned with a position of a navel
hs interposed therebetween. In addition, the camera port PTC is
also placed near the navel hs.
[0124] In FIG. 5C, large numbers of end effector ports PTE and the
auxiliary ports PTA are linearly aligned. The position of the navel
hs is slightly shifted from the position on the straight line. In
addition, the camera port PTC is also placed near the navel hs.
[0125] The reason why a large amount of ports PT are linearly
placed is presumed to be that the user can easily recognize the
port positions and feels safe. Among the plurality of ports PT, the
camera port PTC may be placed at the center of the body surface of
the subject PS.
[0126] FIG. 6 is a view illustrating an example of a positional
relationship between the subject PS, the ports PT, the trocars TC,
and the robot arms AR during robotic surgery.
[0127] In the subject PS, one or more ports PT are provided. In
each of the ports PT, the trocar TC is placed. The end effector EF
is connected (for example, is inserted) to the trocar TC and a work
(treatment) can be performed using the end effector EF in the
subject. The port position is disposed to be fixed and does not
move during operation. Accordingly, the position of the trocar TC
disposed at the port position does not also move. On the other
hand, according to the treatment during operation, the robot arms
AR and the end effectors are controlled based on the manipulation
of the surgeon console, and the robot arms AR move. Accordingly,
the positional relationship between the robot arms AR and the
trocars TC changes and the angles of the trocars TC with respect to
the body surface of the subject PS or the angles of the end
effectors EF attached to the trocars TC change. In FIG. 6, a
monitor held by an assistant is also illustrated as an end
effector.
[0128] Next, the operation of the robotically-assisted surgical
device 100 will be described.
[0129] First, the procedure of the port position simulation will be
described. FIG. 7 is a flowchart illustrating an example of the
procedure of the port position simulation.
[0130] First, the port position processing unit 164 acquires the
volume data including the subject PS, for example, via the
communication unit 110 (S11). The port position processing unit 164
acquires the kinematic information from the surgical robot 300, for
example, via the communication unit 110 (S12). The deformation
simulator 163 performs the pneumoperitoneum simulation (S13) to
generate the volume data of the virtual pneumoperitoneum state of
the subject PS.
[0131] The port position processing unit 164 acquires the
information of the surgical procedure (S14). The port position
processing unit 164 acquires and sets the positions (initial
positions) of the plurality of ports PT according to the acquired
surgical procedure (S14). In this case, the port position
processing unit 164 may set the positions of the plurality of ports
PT in terms of 3D coordinates.
[0132] The port position processing unit 164 acquires the
information of the target region (S15).
[0133] The port position processing unit 164 determines whether or
not each of the end effectors EF inserted from each of the ports PT
is accessible to the target region based on the positions of the
plurality of ports acquired in S14 and the position of the target
region (S16). Whether or not each of the end effectors EP is
accessible to the target region may correspond to whether or not
each of the end effectors EF can reach all the positions in the
target region. That is, whether or not each of the end effectors EF
is accessible to the target region shows that whether or not
robotic surgery can be performed by the end effector EF
(optionally, the plurality of end effectors EF) according to the
acquired surgical procedure, and when each of the end effectors EF
is accessible to the target region, robotic surgery can be
performed.
[0134] When at least one of the end effectors ET is not accessible
to at least a part of the target region, the port position
processing unit 164 moves, a port position of at least one port PT
included in the plurality of ports PT to be pierced along the body
surface of the subject PS (S17). In this case, the port position
processing unit 164 may move the port position based on the user
input via the UI 120. The port PT to be moved includes at least a
port PT into which the end effector EF that is not accessible to at
least a part of the target region is inserted.
[0135] When each of the end effectors EF is accessible to the
target region, the processing unit 160 ends the process of the port
position simulation of FIG. 7.
[0136] As described above, the robotically-assisted surgical device
100 performs the port position simulation such that whether or not
each of the end effectors EF is accessible to the target region
using the acquired plurality of port positions can be determined
and thus whether or not robotic surgery can be performed by the
surgical robot 300 using the acquired plurality of port positions
can be determined. When the target region is not accessible using
the plurality of port positions, the robotically-assisted surgical
device 100 may change at least a part of the port positions via the
UI 120 so as to determine again whether or not the target region is
accessible using the changed plurality of port positions. The
robotically-assisted surgical device 100 can plan a combination of
a plurality of port positions that are accessible to the target
region as the plurality of port positions to be pierced. This way,
the robotically-assisted surgical device 100 can plan the port
position by the user manually adjusting the port position.
[0137] Next, an example of calculating the port position score will
be described.
[0138] The plurality of port positions are determined, for example,
according to the surgical procedure, and it may be assumed that
each port position is disposed at any positions on the body surface
of the subject PS. Accordingly, as the combination of the plurality
of port positions, various combinations of port positions may be
assumed. One end effector EF mounted on the robot arm AR can be
inserted from one port PT into the subject PS. Accordingly, a
plurality of end effectors EF mounted on a plurality of robot arms
AR can be inserted from a plurality of ports PT into the subject
PS.
[0139] A range where one end effector EF can reach the subject PS
through the port PT is a working area (individual working area WA1)
where a work (treatment in robotic surgery) can be performed by one
end effector EF. Accordingly, an area where the individual working
areas WA1 of the plurality of end effectors EF superimpose each
other is a working area (entire working area WA2) where the
plurality of end effectors EF can simultaneously reach the inside
of the subject PS through the plurality of ports PT. In a treatment
according to the surgical procedure, a predetermined number (for
example, three) of end effectors EF needs to be operated at the
same time. Therefore, the entire working area WA2 where the
predetermined number of end effectors EF can simultaneously reach
the inside of the subject PS is considered.
[0140] In addition, the position where the end effector EF can
reach the subject PS varies depending on the kinematics of the
surgical robot 300, and thus is added to the derivation of a port
position as a position where the end effector EF is inserted into
the subject PS. In addition, the position of the entire working
area WA2 in the subject PS that is required to be secured varies
depending on the surgical procedure, and thus is added to the
derivation of a port position corresponding to the position of the
entire working area WA2.
[0141] The port position processing unit 164 may calculate the port
position score for each of the acquired (assumed) combinations of
the plurality of port positions. The port position processing unit
164 may plan a combination of port positions having a port position
score (for example, a maximum port score) that satisfies
predetermined conditions among the assumed combinations of the
plurality of port positions. That is, the plurality of port
positions included in the planned combination of the port positions
may be planned as the plurality of port positions to be
pierced.
[0142] A relationship between the port position and the operation
of the moving part of the surgical robot 300 may satisfy a
relationship described in, for example, Mitsuhiro Hayashibe, Naoki
Suzuki, Makoto Hashizume, Kozo Konishi, Asaki Hattori, "Robotic
surgery setup simulation with the integration of inverse-kinematics
computation and medical imaging", computer methods and programs in
biomedicine, 2006, P63-P72 and Pal Johan From, "On the Kinematics
of Robotic-assisted Minimally Invasive Surgery", Modeling
Identification and Control, Vol. 34, No. 2, 2013, P69-P82, which
are incorporated herein by reference.
[0143] FIG. 8 is a flowchart illustrating an operation example when
the port position score is calculated by the robotically-assisted
surgical device 100.
[0144] Before the process of FIG. 8, the acquisition of the volume
data of the subject PS, the acquisition of the kinematic
information of the surgical robot 300, the execution of the
pneumoperitoneum simulation, and the acquisition of the information
of the surgical procedure are performed as in S11 to S14 of the
port position simulation illustrated in FIG. 8. In addition, the
kinematic information may include the information of each of the
end effectors EF mounted on each robot arm according to the
surgical procedure. The initial value of the port position score is
0. The port position score is an evaluation function (evaluation
value) indicating the value of the combination of the port
positions. A variable i is an example of identification information
of a work, and a variable j is an example of identification
information of a port.
[0145] The port position processing unit 164 generates a work list
works, which is a list of works work_i in which each end effector
EF is used, according to the surgical procedure (S21). The work
work_i includes information for allowing each end effector EF to
perform the work in the surgical procedure according to the
surgical procedure. The work work_i may include, for example,
gripping, dissection, or suture. The work may include a solo work
that is performed by a single end effector EF or a cooperative work
that is performed by a plurality of end effectors EF.
[0146] Based on the surgical procedure and the volume data of the
virtual pneumoperitoneum state, the port position processing unit
164 determines a minimum region least_region_i, which is a region
necessary for performing the works work_i included in the work list
works (S22). The minimum region may be specified as a 3D region in
the subject PS. The port position processing unit 164 generates a
minimum region list least_regions, which is a list of the minimum
regions least_region_i (S22).
[0147] Based on the surgical procedure, the kinematics of the
surgical robot 300, and the volume data of the virtual
pneumoperitoneum state, the port position processing unit 164
determines an effective region effective_region_i that is
recommended for performing the work work_i included in the work
list works (S23). The port position processing unit 164 generates
an effective region list effective_regions, which is a list of the
effective regions effective_region_i (S23). The effective region
may include not only the minimum space (minimum region) for
performing the work but also a space that is effective, for
example, the end effector EF to operate.
[0148] The port position processing unit 164 acquires information
of a port position list ports, which is a list of a plurality of
port positions port_j (S24). The port position may be specified by
3D coordinates (x, y, z). The port position processing unit 164 may
receive, for example, a user input through the UI 120 to acquire
the port position list ports including one or more port positions
designated by the user. The port position processing unit 164 may
acquire the port position list ports that are stored in the memory
150 as a template.
[0149] Based on the surgical procedure, the kinematics of the
surgical robot 300, the volume data of the virtual pneumoperitoneum
state, and the acquired plurality of port positions, the port
position processing unit 164 determines a port working region
region_i, which is a region where each of the end effectors EF can
perform each of the works work_i through each of the port positions
port_j (S25). The port working region may be specified as a 3D
region. The port position processing unit 164 generates a port
working region list regions, which is a list of the port working
regions region_i (S25).
[0150] The port position processing unit 164 subtracts the port
working region region_i from the minimum region least_region_i for
each of the works work_i to calculate a subtracted region
(subtracted value) (S26). The port position processing unit 164
determines whether or not the subtracted region is an empty region
(the subtracted value is negative) (S26). Whether or not the
subtracted region is an empty region shows that whether or not a
region that is not covered with the port working region region_i (a
region that cannot be reached by the end effector EF through the
port PT) is present in at least a part of the minimum region
least_region_i.
[0151] When the subtracted region is an empty region, the port
position processing unit 164 calculates a volume value volume_i,
which is the product of the recommended region effective_region_i
and the port working region region_i (S27). The port position
processing unit 164 sums the volume values volume_i calculated for
each of the works work_i to calculate a sum value volume_sum. The
port position processing unit 164 sets the sum value volume sum as
the port position score (S27).
[0152] That is, when the subtracted region is an empty region, it
is preferable that the region that is not covered with the port
working region is not present in the minimum region and this port
position list ports (the combination of the port positions port_j)
is selected. Therefore, in order to promote the selection of the
port position list, the value for each of the works work_i is added
to the port position score. In addition, by determining the port
position score based on the volume value volume_i, as the minimum
region or the port working region increases, the port position
score increases, and this port position list ports is more likely
to be selected. Accordingly, the port position processing unit 164
is more likely to select a combination of port positions in which
the minimum region or the port working region is large and each
treatment is easy in surgery.
[0153] On the other hand, when the subtracted region is not an
empty region, the port position processing unit 164 sets the port
position score of the port position list ports to a value of 0
(S28). That is, since the region that is not covered with the port
working region is present in at least a part of the minimum region
and the work of the target work work_i may not be completed, it is
not preferable to select this port position list ports. Thus, in
order to make the selection of the port position list ports
difficult, the port position processing unit 164 sets the port
position score to a value of 0 and excludes the port position list
from candidates of the selection. In this case, when the subtracted
region is an empty region in a case where another work work_i is
performed using the same port position list ports, the port
position processing unit 164 sets the port position score to a
value of 0 as a whole.
[0154] The port position processing unit 164 may calculate a port
position score for all the works work_i by repeating the respective
steps of FIG. 8 for all the works work_i.
[0155] as described above, the robotically-assisted surgical device
100 derives the port position score, and when the robotic surgery
is performed using the plurality of port positions provided on the
body surface of the subject PS, the appropriateness of the
combination of the port positions to be pierced can be grasped. The
individual working area WA1 and the entire working area WA2 depend
on the placement positions of the plurality of pons to be pierced.
Even in this case, by using a score (port position score) for each
combination of a plurality of port positions, the surgical robot
300 can derive a combination of a plurality of port positions in
which, for example, the port position score is a threshold th1 or
higher (for example, maximum), and the port positions with which
robotic surgery can be easily performed can be set.
[0156] In addition, by appropriately securing the working area
based on the port position score, the user can secure a wide visual
field in the subject PS that cannot be directly visually observed
in robotic surgery, a wide port working region can be secured, and
unexpected events can be easily handled.
[0157] In addition, in robotic surgery, the port positions to be
pierced are not variable. However, the robot arms AR on which the
end effectors inserted into the port positions are mounted are
movable in a predetermined range. Therefore, in robotic surgery,
depending on the planned port positions, the robot arms AR may
interfere with each other. Therefore, port position planning is
important. In addition, the positional relationship between the
surgical robot 300 and the subject PS cannot be changed during
operation in principle. Therefore, port position planning is
important.
[0158] FIG. 9 is a view illustrating an example of working areas
determined based on the port positions. The individual working area
WA1 is an individual working area corresponding to each of the port
positions port_j. The individual working area WA1 may be a region
in the subject PS that can be reached by each of the end effectors
EF. An area where the respective individual working areas WA1
superimpose each other is the entire working area WA2. The entire
working area WA2 may correspond to the port working region
region_i. The robotically-assisted surgical device 100 can optimize
each of the port positions using the port position score, and the
suitable individual working areas WA1 and the suitable entire
working area WA2 can be derived.
[0159] Next, the details of the port position adjustment will be
described.
[0160] The port position processing unit 164 acquires information
of the plurality of port positions (candidate positions), for
example, based on the template stored in the memory 150 or the user
instruction via UI 120. The port position processing unit 164
calculates the port position score for the case using the plurality
of port positions based on the acquired combination of the
plurality of port positions.
[0161] The port position processing unit 164 may adjust the
position of the port PT based on the port position score. In this
case, the port position processing unit 164 may adjust the position
of the port PT based on the port position score for the acquired
plurality of port positions and the port position score obtained
when at least one port position among the plurality of port
positions is changed. In this case, the port position processing
unit 164 may also consider a small movement or a differential of
the port position in each of the directions (x direction, y
direction, and z direction) in a 3D space.
[0162] The x direction may be a direction along a left-right
direction with respect to the subject PS. The y direction may be a
forward-backward direction (thickness direction of the subject PS)
with respect to the subject PS. The z direction may be an up-down
direction (body axis direction of the subject PS) with respect to
the subject PS. The x direction, the y direction, and the z
direction may be three directions defined by Digital Imaging and
Communications in Medicine (DICOM). The x direction, the y
direction, and the z direction may be directions other than the
above-described directions and are not necessarily the directions
with respect to the subject PS.
[0163] For example, the port position processing unit 164 may
calculate a port position score F (ports) for the plurality of port
positions according to (Expression 1) to calculate a differential
value F' of F.
F (port_j(x+.DELTA.x, y, z))-F (port_j(x, y, z))
F (port_j(x, y+.DELTA.y, z))-F (port_j(x, y, z)) (Expression 1)
F (port_j(x, y, z+.DELTA.z))-F (port_j(x, y, z))
[0164] That is, the port position processing unit 164 calculates
the port position score F for the port position F
(port_j(x+.DELTA.x, y, z)), calculates the port position score F
for the port position F (port_j(x, y, z)), and calculates a
difference therebetween. This difference value indicates a change
in the port position score with respect to a small change of the
port position F (port_j(x, y, z)) in the x direction, that is, the
differential value F' of F in the x direction.
[0165] In addition, the port position processing unit 164
calculates the port position score F for the port position F
(port_j(x, y+.DELTA.y, z)), calculates the port position score F
for the port position F (port-j(x, y, z)), and calculates a
difference therebetween. This difference value indicates a change
in the port position score with respect to a small change of the
port position F (port_j(x, y, z)) in the y direction, that is, the
differential value F' of F in the y direction.
[0166] In addition, the port position processing unit 164
calculates the port position score F for the port position F
(port_j(x, y, z+.DELTA.z)), calculates the port position score F
for the port position F (port_j(x, y, z)), and calculates a
difference therebetween. This difference value indicates a change
in the port position score with respect to a small change of the
port position F (port_j(x, y, z)) in the z direction, that is, the
differential value F' of F in the z direction.
[0167] The port position processing unit 164 calculates a maximum
value of the port position score based on the differential value F
of each of the directions. In this case, the port position
processing unit 164 may calculate a port position having the
maximum port position score according to the steepest descent
method based on the differential value F'. The port position
processing unit 164 may adjust the port position to optimize the
port position such that the calculated port position is a position
to be pierced. Instead of the port position in which the port
position score is the maximum, the port position may be, for
example, a position in which the port position score is the
threshold th1 or higher as long as the port position score is
improved (increases).
[0168] The port position processing unit 164 may apply this port
position adjustment to the adjustment of another port position
included in the combination of the plurality of port positions or
to the adjustment of port positions of another combination of a
plurality of port positions. As a result, the port position
processing unit 164 can plan the plurality of ports PT of which the
respective port positions are adjusted (for example, optimized) as
the port positions to be pierced.
[0169] Regarding the plurality of port positions (coordinates of
the port positions), there may be an error of about a predetermined
length (for example, 25 mm) between a piercing-planned position and
an actual piercing position, and it is presumed that a port
position planning accuracy of 3 mm at most is sufficient.
Therefore, the port position processing unit 164 may set a
plurality of port positions included in the combination of port
positions as piercing-planned positions per predetermined length of
the body surface of the subject PS, and the port position score may
be calculated for each of the plurality of port positions. That is,
the piercing-planned positions may be placed in a lattice shape
(grid) of the predetermined length (for example, 3 mm) on the body
surface of the subject PS. In addition, when it is assumed that the
number of ports (for example, the number of intersections in a
lattice shape) on the body surface is n and the number of ports
included in the combination of port positions is m, the port
position processing unit 164 may combine by sequentially selecting
m port positions from n port positions and may calculate the port
position score for each of the combinations. This way, when the
grid is not excessively small as in a lattice shape having an
interval of 3 mm, the calculation load of the port position
processing unit 164 can be inhibited front being excessive, and the
port position scores of all the combinations can be calculated.
[0170] The port position processing unit 164 may adjust the
plurality of port positions using a well-known method. The port
position processing unit 164 may plan the port positions to be
pierced as the plurality of port positions included in the adjusted
combination of port positions. The well-known method of the port
position adjustment may include techniques described in the
followings. Shaun Selha, Pierre Dupont, Robert Howe, David
Torchiana, "Dexterity optimization by port placement in
robot-assisted minimally invasive surgery", SPIE International
Symposium on Intelligent Systems and Advanced Manufacturing,
Newton, Mass., 28-31, 2001; Zhi Li, DejanMilutinovic, Jacob Rosen,
"Design of a Multi-Arm Surgical Robotic System for Dexterous
Manipulation", Journal of Mechanisms and Robotics, 2016; and
US2007/0249911 A, which is incorporated herein by reference.
[0171] Next, the allowable error of a port position will be
described.
[0172] The port position processing unit 164 derives (for examples,
calculates) information (allowable error information) indicating
errors that are allowed for each of ports. The port position
processing unit 164 may calculate the allowable error information
based on the port position score. The port position processing unit
164 may calculate the allowable error information based on the
variation of the port position score according to the movement of
the port position. The allowable error may be, for example, a value
that is more than a threshold th2 (for example, error: 3 mm)
representing the highest level of the piercing accuracy of the port
position.
[0173] The allowable error information may be displayed on the body
surface of the subject PS. In this case, the projection controller
167 may project visible light representing the allowable error
information to the body surface of the subject PS. In addition, the
display controller 166 may display the allowable error information
to superimpose a rendering image obtained by rendering the volume
data of the subject PS.
[0174] The allowable error information may be displayed as graphic
information or character information. The graphic information may
be displayed in a range including the allowable error that includes
a port position to be pierced. This range may be a 2D range on the
body surface of the subject PS. The 2D range may be a range having
a circular shape (for example, an ellipse, a true circle, or other
circles), a polygonal shape (for example, a rectangle, a square, a
triangle, or other polygons), or other shapes. The circle or the
polygon will also be referred to as "primitive shape". The
allowable error information may be displayed as other information
(for example, information regarding a display manner (a display
color, a display size, a display pattern, or a flashing pattern)).
For example, when the allowable error of the port PT is large, the
port PT may be displayed by a first color, and when the allowable
error of the port PT is small, the port PT may be displayed by a
second color.
[0175] The robotically-assisted surgical device 100 displays the
allowable error information and the user can recognize the
allowable error information and can rapidly recognize an allowable
range of the piercing of a port position to be pierced.
Accordingly, for example, when a spatial (for example, planar)
range represented by the allowable error information is large, the
user can recognize that a port can be carelessly pierced at a port
position to be pierced. In addition, for example, when a spatial
(for example, planar) range represented by the allowable error
information is small, the user can recognize that a port is
required to be accurately pierced at a port position to be pierced.
Accordingly, when the robotically-assisted surgical device 100
displays, for example, allowable error information having a large
allowable error, the robotically-assisted surgical device 100 can
reduce a mental burden of the user who pierces the port PT. In
addition, when the robotically-assisted surgical device 100
displays, for example, allowable error information having a large
allowable error, the robotically-assisted surgical device 100 can
reduce the number of processes required for the user who pierces
the port PT to place the port, and can reduce the operative
duration. In addition, when the robotically-assisted surgical
device 100 displays, for example, allowable error information
having a small allowable error, the robotically-assisted surgical
device 100 can notify the user that high accuracy is required for
the piercing of the port PT.
[0176] FIG. 10 is a flowchart illustrating a derivation procedure
of allowable error information by the robotically-assisted surgical
device 100. In FIG. 10, the acquisition of the volume data of the
subject PS, the acquisition of the kinematic information of the
surgical robot 300, the execution of the pneumoperitoneum
simulation, and the acquisition of the information of the surgical
procedure are performed in advance as illustrated in FIG. 8.
[0177] The port position processing unit 164 acquires information
of a plurality of port positions (positions of piercing candidates)
(S31). The port position processing unit 164 performs the port
position simulation to calculate the port position score based on
the acquired plurality of port positions (S32). In this case, the
port position processing unit 164 may calculate the port position
score based on the surgical procedure, the kinematics of the
surgical robot 300, the volume data of the virtual pneumoperitoneum
state, and the acquired plurality of port positions. That is, here,
the port position processing unit 164 may calculate the port
position score for the acquired port positions.
[0178] The port position processing unit 164 acquires allowable
decrease information (total allowable decrease information)
regarding the degree of decreases in port position scores that are
allowed for the acquired plurality of port positions (S33). The
allowable decrease information may include information regarding
the amount or ratio of the decreases in port position scores that
are allowed for the port positions. The port position processing
unit 164 may receive a user input via the UI 120 to acquire the
total allowable decrease information. The port position processing
unit 164 may acquire the total allowable decrease information from
the memory 150. The port position processing unit 164 may acquire
the total allowable decrease information from the external server
is the communication unit 110.
[0179] The port position processing unit 164 acquires allowable
decrease information (individual allowable decrease information)
regarding the degree of a decrease in port position score that is
allowed for each of the port positions (S34). The individual
allowable decrease information of the respective ports PT may be
the same or different. The port position processing unit 164 may
derive (for example, calculate) the individual allowable decrease
information based on the total allowable decrease information. In
this case, the port position processing unit 164 may divide the
amount of allowable decreases represented by the total allowable
decrease information by the number of the ports PT to calculate the
amount of an allowable decrease for each of the ports represented
by the individual allowable decrease information.
[0180] In addition, the port position processing unit 164 may
acquire the individual allowable decrease information without
acquiring the total allowable decrease information in S33. In this
case, the port position processing unit 164 may receive a user
input via the UI 120 to acquire the individual allowable decrease
information. The port position processing unit 164 may acquire the
individual allowable decrease information from the memory 150. The
port position processing unit 164 may acquire the individual
allowable decrease information from the external server via the
communication unit 110.
[0181] When at least one port position is moved, the port position
processing unit 164 may derive (for example, calculate) decrease
information indicating the degree of a decrease caused by the
movement of the port position based on a port position score at the
port position (a combination of port positions) before the movement
and a port position score at the port position (a combination of
port positions) after the movement (S35). The decrease information
may include the amount or ratio of a decrease caused by the
movement of the port position. In this case, the port position
processing unit 164 may subtract the port position score at the
port position after the movement from the port position score at
the port position before the movement to calculate the amount of
the decrease in port position score.
[0182] When the port position is moved, the port position
processing unit 164 may calculate the amount of a change (amount of
a decrease) in the port position score according to the
above-described Expression 1. In this case, the amount of a
decrease in the port position score before and after the movement
of a port position by a predetermined distance (for example, a
small distance) may correspond to the differential value F' of the
port position score F.
[0183] In addition, the movement of the port position may be
performed in any direction along the body surface of the subject
PS. In this case, when a body surface is a plane, the port
positions before and after the movement are positioned in a 2D
plane along the body surface. In addition, when the body surface
includes a curved surface, the port positions before and after the
movement are positioned in a 3D space.
[0184] For each of the port positions of the subject PS, the port
position processing unit 164 derives a region (allowable region PR)
where the decrease information satisfies the allowable decrease
information (S36). In this case, the port position processing unit
164 may calculate the allowable region PR where the amount of a
decrease represented by the decrease information is less than or
equal to the amount of an allowable decrease represented by the
allowable decrease information. The allowable region PR may be a
region in a 3D space. Accordingly, a contour of the allowable
region PR is a position in a 3D space that matches the amount of an
allowable decrease represented by the allowable decrease
information with respect to the port position. The allowable region
PR is an example of the allowable error information.
[0185] The port position processing unit 164 causes the display
controller 166 or the projection controller 167 to display a port
position and the allowable region PR for the port position for each
of the ports PT (S37). In this case, the display controller 166
causes the display 130 to display the port position and the
allowable region PR for the port position to superimpose the
rendering image of the subject PS for each of the ports PT. In
addition, the projection controller 167 may cause the projection
unit 170 to project visible light representing the port position
and the allowable region PR for the port position to the body
surface of the subject PS for each port PT to display the port
position and the allowable region PR.
[0186] The port position processing unit 164 may extract the
contour of the volume data of the virtual pneumoperitoneum state to
acquire information regarding the body surface of the subject PS on
which pneumoperitoneum is performed. The port position processing
unit 164 may derive (for example, calculate) a superimposing range
where the body surface of the subject PS on which pneumoperitoneum
is performed and the allowable ration PR superimpose each other.
The image generator 162 may perform surface rendering on the volume
data of the virtual pneumoperitoneum state to generate a surface
rendering image of the derived superimposing range. The display
controller 166 may cause the display 130 to display the surface
rendering image of the superimposing range.
[0187] In addition, the allowable region PR in the rendering image
may be displayed by other information instead of being directly
displayed in the region. For example, the display controller 166
may display the allowable region PR by displaying the radius of a
range obtained by projecting the allowable region PR to the surface
of the rendering image. In addition, the projection controller 167
may display the allowable region PR by projecting visible light
representing information indicating the radius of a range obtained
by projecting the allowable region PR to the body surface of the
subject PS.
[0188] In addition, instead of directly displaying the allowable
region PR, the display controller 166 or the projection controller
167 may change the respective port positions using different
display manners (for example, a display color, a display pattern,
or a flashing pattern) corresponding to the sizes of the allowable
regions PR. As a result, by checking the display manner, the user
can recognize the size of the allowable region PR and can check
whether the allowable error is large or small.
[0189] As described above, by deriving the allowable error
information, the robotically-assisted surgical device 100 can
derive the piercing accuracy required for the piercing of the port
PT. In addition, the robotically-assisted surgical device 100
displays (visualizes) the allowable error information such that the
user can visually recognize the allowable error information.
Accordingly, the user can check, for example, whether or not the
port to be pierced can be carelessly pierced or whether or not it
is necessary to carefully pierce the port, and the preparation for
the piercing becomes easy.
[0190] Next, errors for port positions in the thickness direction
(y direction) of the subject PS will be described.
[0191] FIG. 11 is a view illustrating a first example of errors for
port positions in the thickness direction (y direction) of the
subject PS. FIG. 12 is a view illustrating a second example of
errors for port positions in the thickness direction (y direction)
of the subject PS. FIGS. 11 and 12 illustrate errors based on the
straight line distances.
[0192] In many cases, the port positions are illustrated in a plan
view when the subject PS horizontally placed on the bed BD is seen
from the top (the negative side in the y direction, a so-called
front view). The error for port positions illustrated in a plan
view is the error in a direction along the bed BD and is the error
in the xz direction. In practice, positions, which are not
illustrated in a plan view, in the y direction perpendicular to the
xz direction vary depending on the positions on the body
surface.
[0193] In FIG. 11, pneumoperitoneum is performed on the subject PS,
and a pneumoperitoneum space ks is present. In FIG. 11, both a port
A1 and a port B1 have an error range of 10 mm in the x direction in
a plan view. However, when the y direction other than the xz
direction indicated in a plan view is taken into consideration, an
end portion of the subject in the x direction, that is, the
position of the port B1 has a larger error range on the body
surface than a center portion of the subject in the x direction.
Accordingly, the port B1 has a high influence of errors on the
working area in the y direction. It is because the port position
score in consideration of the working area changes depending on the
position of the port that is variable within the error range.
[0194] In addition, even in FIG. 12, pneumoperitoneum is performed
on the subject PS and a pneumoperitoneum space ks is present. As
the information of the port position, FIG. 12 illustrates not only
information regarding the error range in the xz direction in a plan
view but also information regarding the error range in the y
direction. Specifically, FIG. 12 illustrates that the distance of
the error range of the port A2 from the position corresponding to
the navel hs on the body surface in the x direction is in a range
of 15 to 20 mm. In addition, in FIG. 12, the distance of the error
range of the port B2 from the bed BD is in a range of 150 to 160
mm. The display controller 166 may determine and display at least
one information (information regarding the error range along the x
direction and the error range along the y direction) as a display
target. The error range illustrated in FIGS. 11 and 12 may be
within or outside the tolerance.
[0195] Next, a method of measuring the distance for determining the
port position will be described.
[0196] For example, when the position of a port is measured from a
reference position (for example, the navel hs or a port position
adjacent thereto) in order to pierce the port, a straight line
distance or a curved distance may be measured. That is, the
distance between the position of the port and the reference
position may be represented by a straight line distance or a curved
distance (for example, the distance along the body surface of the
subject PS). As a method of measuring these distances, a method of
virtually measuring the distance when the port PT is not actually
pierced may be used, or a method of actually measuring the distance
when the port PT is actually pierced may be used.
[0197] FIG. 13 is a view illustrating a measurement example of a
straight line distance. The straight line distance may be a
distance L11 along an xz plane (x direction) or a distance L12
(distance from the bed BD) along the y direction. In addition, for
example, when measuring the port position using a ruler, the ruler
is put on the body surface of the subject PS for the measurement.
Therefore, even when a straight line distance is measured, the
straight line distance may be a distance that is not parallel to
the xyz direction or a distance on a straight line that is parallel
to a part of the body surface.
[0198] FIG. 14 is a view illustrating a measurement example of a
curved distance. The curved distance may be a surface distance L13
or L14 from the reference position (for example, the navel hs, the
port position adjacent thereto, or the bed BD) along the body
surface of the subject PS. The curved distance may be a surface
distance in a convex hull generated based on the reference
position. In addition, for example, when measuring the port
position using a tape measure, since the measurement is performed
along the body surface of the subject PS for the measurement,
measuring the curved distance is convenient. The measured distance
may be input to the measuring instrument 400 as distance
information via the operation unit of the measuring instrument 400
such that the distance information is transmitted to the
robotically-assisted surgical device 100. In addition, the measured
distance may be input by the user via the UI 120 of the
robotically-assisted surgical device 100.
[0199] Next, the adjustment of the allowable error depending on the
amount of pneumoperitoneum will be described.
[0200] FIG. 15 is a view illustrating an adjustment example of the
allowable error depending on the amount of pneumoperitoneum.
[0201] The deformation simulator 163 may perform a pneumoperitoneum
simulation kn1 using an amount of pneumoperitoneum kr1 to generate
volume data v1. A contour of the volume data v1 is a body surface
bs1 of the subject PS of the virtual pneumoperitoneum state that is
estimated by the pneumoperitoneum simulation kn1. The deformation
simulator 163 may perform a pneumoperitoneum simulation kn2 using a
larger amount of pneumoperitoneum kr2 (that is kr2>kr1) than the
amount of pneumoperitoneum kr1 to generate volume data v2. A
contour of the volume data v2 is a body surface bs2 of the subject
PS of the virtual pneumoperitoneum state that is estimated by the
pneumoperitoneum simulation kn2. The deformation simulator 163 may
perform a pneumoperitoneum simulation kn3 using a larger amount of
pneumoperitoneum kr3 (that is kr3>kr2) than the amount of
pneumoperitoneum kr2 to generate volume data v3. A contour of the
volume data v3 is a body surface bs3 of the subject PS of the
virtual pneumoperitoneum state that is estimated by the
pneumoperitoneum simulation kn3.
[0202] The port position processing unit 164 derives a 3D allow
able region PR based on the volume data v2 of the virtual
pneumoperitoneum state obtained based on the amount of
pneumoperitoneum kr2. In this case, the port position processing
unit 164 may derive (for example, calculate) an allowable range PA2
based on an intersection between the allowable region PR and the
body surface bs2. In addition, the port position processing unit
164 may derive (for example, calculate) the allowable ranges PA1
and PA3 based on intersections between the allowable region PR and
the body surfaces bs1 and bs3 represented by the contours of the
volume data v1 and v3 other than the volume data v2 for deriving
the allowable region PR. As the amount of pneumoperitoneum
increases, the allowable range may increase. As the amount of
pneumoperitoneum decreases, the allowable range may decrease.
[0203] The port position processing unit 164 may project the
allowable ranges PA1 and PA3 respectively corresponding to the body
surfaces bs1 and bs3 to the body surface bs2 to obtain allowable
ranges PA1' and PA3' such that a superimposing range of the
allowable ranges PA1', PA2, and PA3' is a minimum allowable range
MPA. Even when pneumoperitoneum is performed at any of the amounts
of pneumoperitoneum kr1 to kr3, the minimum allowable range MPA is
included in the allowable region PR. Accordingly, by piercing the
port PT in the minimum allowable range MPA, the port position is
within the tolerance irrespective of the amount of
pneumoperitoneum. Information indicating the minimum allowable
range MPA may be displayed on the display 130 or the body surface
of the subject PS. The minimum allowable range MPA may be
determined using minimum values of diameters of the allowable
ranges PA1', PA2, and PA3'.
[0204] This way, the robotically-assisted surgical device 100 may
use a plurality of pneumoperitoneum states to acquire, for example,
a result of a standard pneumoperitoneum simulation, a result of a
pneumoperitoneum simulation in which the abdomen is largely
inflated, and a result of a pneumoperitoneum simulation in which
the abdomen is not largely inflated. The robotically-assisted
surgical device 100 may acquire superimposing ranges (the allowable
ranges PA1 to PA3) of the body surfaces bs1 to bs3 of the results
of the respective pneumoperitoneum simulations and the allowable
region PR to display the allowable ranges PA1 to PA3. In addition,
the robotically-assisted surgical device 100 may derive the minimum
allowable range MPA including the same region of the subject PS for
the allowable ranges PA1 to PA3 to display the minimum allowable
range MPA.
[0205] That is, the port position processing unit 164 may adjust
the allowable error based on a variation in the degree of
pneumoperitoneum (that is, the amount of pneumoperitoneum), and
thus may adjust the allowable region PR based on the port position.
The adjustment of the allowable error may include the projection of
the allowable region PR to the respective body surfaces bs1 to bs3.
In addition, the port position processing unit 164 may derive (for
example, calculate) the minimum allowable range MPA as the
allowable region PR that does not depend on the variation in the
degree of pneumoperitoneum (that is, the amount of
pneumoperitoneum). The port position processing unit 164 may cause
the display controller 166 or the projection controller 167 to
display information indicating the derived minimum allowable range
MPA (for example, display a figure representing an outer edge of
the minimum allowable range MPA or a display manner that can
identify the minimum allowable range MPA). In addition, by using
the minimum allowable range MPA as one index, the
robotically-assisted surgical device 100 can solve a problem that
the position to be pierced deviates from the tolerance depending on
the degree of pneumoperitoneum.
[0206] In addition, the port position processing unit 164 may
project a 3D allowable region PR of the port obtained from spatial
coordinates to the body surfaces bs1 to bs3 of the subject PS based
on the movement of the subject PS. The movement of the subject PS
is, for example, the movement caused by breathing or heartbeat.
[0207] In addition, even when gas is injected into the subject PS
under the same conditions, the pneumoperitoneum state may vary
depending on the subject PS. For example, there are a person whose
body surface portion is likely to extend and a person whose body
surface portion is not likely to extend. In addition, depending on
a positional relationship with an organ, a part of the body surface
of the subject PS is likely to extend, and another part of the body
surface is not likely to extend. Even in this case, the
robotically-assisted surgical device 100 can derive the allowable
error or the allowable region PR in consideration of the plurality
of pneumoperitoneum states. Therefore, unexpected allowable error
or an unexpected allowable region PR generated due to the actual
amount of pneumoperitoneum can be suppressed.
[0208] Next, a variation of a process relating to the allowable
error of a port position will be described.
[0209] The port position processing unit 164 may perform the port
position simulation or the port position adjustment in
consideration of the allowable error. For example, the port
position processing unit 164 may newly plan, as a port position to
be pierced, any position in the allowable region PR based on the
originally acquired or planned port position. Accordingly, the user
can pierce the port PT in a wide range such as any position in the
allowable region PR without being limited to a narrow range
(position) such as the originally acquired or planned port
position. Accordingly, the robotically-assisted surgical device 100
can reduce the mental burden of the user during piercing.
[0210] The port position processing unit 164 may separately process
the allowable error of the port PT according to the directions in a
3D space. For example, the port position processing unit 164
separately process the allowable error in the xz direction and the
allowable error in the y direction. For example, the port position
processing unit 164 may distinguish between a port PT for which it
is not necessary to pay attention to the thickness direction (y
direction) of the subject PS and a port PT for which it is
necessary to pay attention to the thickness direction (y direction)
of the subject PS.
[0211] In addition, the port position processing unit 164 may set
whether or not to consider the allowable error of the port
according to the directions in the 3D space. For example, regarding
a port position placed at any position in the 3D space, the
allowable error in the xy direction is considered, and the
allowable error in the y direction is not considered. In this case,
for example, when a position in the thickness direction of the
subject PS is pierced, the user does not need to pay attention, and
a mental burden during piercing can be reduced.
[0212] In addition, the allowable errors in the respective
directions in the 3D space may be the same or different. For
example, when the allowable errors in the respective directions in
the 3D space are the same, an outer edge of the allowable error is
a spherical surface. For example, when the allowable errors in the
respective directions in the 3D space are different, an outer edge
of the allowable error is, for example, a surface of an ellipsoid
or other shapes. In addition, the allowable error may be visualized
using any coordinate system that is different from the
above-described coordinate system including the x direction, the y
direction, and the z direction.
[0213] The port position processing unit 164 may cause the display
controller 166 or the projection controller 167 to display that a
port PT whose allowable error is a threshold th3 or more (a port
whose allowable error at be large) can be marked before
pneumoperitoneum. In this case, the port position processing unit
164 may cause the display controller 166 or the projection
controller 167 to display a port position estimated before
pneumoperitoneum. For example, the port position processing unit
164 may perform the port position simulation or the port position
adjustment to plan the port position based on volume data of the
non-pneumoperitoneum state instead of the volume data of the
virtual pneumoperitoneum state. The port position processing unit
164 may cause the display controller 166 or the projection
controller 167 to display the planned port position.
[0214] The port position processing unit 164 may cause the display
controller 166 or the projection controller 167 to display that a
port PT of which allowable error is less than the threshold th3 (a
port of which allowable error is small) cannot be marked before
pneumoperitoneum.
[0215] In order to display the allowable region PR, the port
position processing unit 164 may use a landmark in the subject PS
such as a rib or the navel hs. For example, in a predetermined
region including a predetermined port position, the allowable
region PR may be a region excluding a rib or a region at a
predetermined distance from the navel hs.
[0216] This way, as the port PT, there are a port PT in which the
piercing accuracy has a high influence on the individual working
area WA1 and a port PT in which the piercing accuracy has a low
influence on the individual working area WA1. In addition, the
influence of the piercing accuracy on the working area (the
individual working area WA1 or the entire working area WA2) varies
depending on the surgical procedure.
[0217] On the other hand, by determining the allowable error based
on the degree of influence of the piercing accuracy on the working
area, the robotically-assisted surgical device 100 can adjust the
port positions with respect to the port position of the port PT
having a small allowable error. Therefore, the robotically-assisted
surgical device 100 can improve the usability of the port position
simulation or the port position adjustment.
[0218] In addition, the robotically-assisted surgical device 100
can distinguishably display information regarding the port PT that
is required to be carefully pierced during robotic surgery and
information regarding the port PT that can be carelessly pierced.
Accordingly, by checking the display to pierce the port PT, the
user can reduce the operative duration. In addition, for the port
that can be carelessly pierced, a cooperative work can be easily
performed by a plurality of users, which leads to a reduction in
the operative duration.
[0219] In addition, a region (difficult-to-use region) where it is
difficult to provide a port, for example, due to medical history
and adhesion or the like may be present on the body surface of the
subject PS. In addition, in the allowable region PR, the
difficult-to-use region and a region (usable region) that is not
the difficult-to-use region may be present together. The port
position processing unit 164 may receive a user input via the UI
120 to designate the difficult-to-use region. Setting information
of the difficult-to-use region may be stored in the memory 150 to
be appropriately referred to. In this case, even when a port
position to be pierced is included in the difficult-to-use region,
the port position processing unit 164 may plan, as a new port
position, any position in the usable region of the allowable region
PR for the port position to be pierced. The port position
processing unit 164 may cause the display controller 166 or the
projection controller 167 to distinguishably display the newly
planned port position or the usable region of the allowable region
PR. As a result, even when adhesion or the like is present in the
body of the subject PS, the robotically-assisted surgical device
100 can assist the piercing of another position in the tolerance of
the port position to be pierced as a new port position.
[0220] FIG. 16 is a view illustrating a first display example of
guide information including information indicating the allowable
region PR of the port PT to be pierced. FIG. 17 is a view
illustrating a second display example of guide information
including information indicating the allowable region PR of the
port PT to be pierced. FIG. 16 illustrates a coronal section of the
subject PS. FIG. 17 illustrates a sagittal section of the subject
PS.
[0221] The images of FIGS. 16 and 17 are the examples displayed by
the display 130. However, the guide information may be displayed on
the body surface using the visible light by projecting visible
light from the projection unit 170 to the body surface of the
subject PS. Information regarding lengths included in the guide
information is merely exemplary and may be other lengths.
[0222] In FIG. 16, the volume rendering image is displayed, and the
guide information is also displayed to superimpose the volume
rendering image. The guide information may include identification
information (for example, a port A, B, C) of the port PT displayed
at the port position to be pierced. The identification information
of the port PT is not necessarily displayed.
[0223] In addition, FIG. 16 illustrates that an allowable region
PRA where the error is allowed for the piercing of the port A has a
radius of 15 mm (r15) from the position of the port A. That is, the
allowable region PRA of the port A is in a true circle having a
radius of 15 mm whose center is the position of the port A. In
addition an allowable region PRB where the error is allowed for the
piercing of the port B is within a range of 120 mm.+-.10 mm from
the navel hs in the body axis direction (x direction) at a position
moved from the navel hs by 20 mm in a direction (z direction) along
the body axis direction. The allowable region PRB of the port B may
be within a range of 0 mm.+-.20 mm from the navel its in the z
direction and within a range of 120 mm.+-.10 mm from the navel hs
in the x direction. In addition, an allowable region PRC where the
error is allowed for the piercing of the port C is within a range
of a 10 mm.times.20 rectangle centering on the position of the port
C along the body surface at a position moved from a side end
portion (end portion in the x direction) of the subject PS by 100
mm in a direction perpendicular to the body axis direction from the
navel hs.
[0224] In FIG. 17, the volume rendering image is displayed, and the
guide information is also displayed to superimpose the volume
rendering image. The guide information may include identification
information (for example, a port D or E) of the port PT displayed
at the port position to be pierced. The identification information
of the port PT is not necessarily displayed.
[0225] In addition, FIG. 17 illustrates that an allowable region
PRD where the error is allowed for the piercing of the port D is
within a range of an ellipse having a major axis of 20 mm and a
minor axis of 10 mm and centering on a position moved from the
navel hs along the body surface in a direction (y direction)
perpendicular to the body axis. In addition, in FIG. 17, an
allowable region PRE where the error is allowed for the piercing of
the port E is within a range of a half of an ellipse having a major
axis of 30 mm and a minor axis of 10 mm and centering on a position
moved from the navel hs along the body surface in a direction alone
the y direction and moved from the bed BD by a distance of 100
mm.
[0226] In addition, in the guide information, an arrow shows a
distance between head positions of two direction arrows. In FIGS.
16 and 17, the distance is expressed in mm units. In addition, the
position of the navel hs is also shown by an arrow. This distance
may be a distance having a certain amount of width (for example,
120 mm.+-.10 mm).
[0227] It can be said that the 2D allowable region PDs (PDA is PDE)
illustrated in FIGS. 16 and 17 are the above-described allowable
ranges projected to the surface represented by the images of FIGS.
16 and 17.
[0228] Next, display examples of port positions according to
Comparative Example and the embodiment will be described.
[0229] FIG. 18 is a view illustrating a display example of port
positions according to Comparative Example. In FIG. 18, A, B, C, D,
and E are examples of the identification information of the ports,
and the lengths thereof are expressed in mm units. The respective
values illustrated in FIG. 18 are merely exemplary and may be other
values.
[0230] In FIG. 18, a port position of a port C is instructed and
displayed at a position moved from the navel hs to the head side in
the body axis direction such that the distance between the navel hs
and the port C is 20 mm. In addition, the ports A, B, C, D and E
are instructed and displayed such that the ports A to E are
linearly placed in a direction perpendicular to the body axis
direction. That is, the ports A, B, C, D and E are instructed and
displayed such that the distance between the port A and the port B
is 50 mm, the distance between the port B and the port C is 50 mm,
the distance between the port C and the port D is 60 mm, and the
distance between the port D and the port E is 40 mm. This way, in
Comparative Example, the allowable error of each of the ports A to
E is not considered. Therefore, the user needs to accurately pierce
each of the ports A to E at the designated position.
[0231] FIG. 19 is a view illustrating a display example of port
positions and allowable error information according to the
embodiment.
[0232] In FIG. 19, the distance between ports adjacent to each
other is instructed and displayed to have a certain amount of width
(allowable error). That is, the ports A to E are instructed and
displayed such that the distance between the navel hs and the port
B is 20 mm to 25 mm, the distance between the port A and the port B
is 50 mm to 60 mm, the distance between the port B and the port C
is 50 mm to 60 mm, the distance between the port C and the port D
is 60 mm to 70 mm, and the distance between the port C and the port
E is 100 mm to 105 mm. This way, in the embodiment, the allowable
error of each of the ports A to E is considered. Therefore, the
user can pierce each of the ports A to E with a certain amount of
allowance with respect to the designated position, and a mental
burden for piercing the port PT can be reduced.
[0233] Hereinbefore, various embodiments have been described with
reference to the drawings. However, it is needless to say that the
present disclosure is not limited to these examples. It is obvious
to those skilled in the art that various changes or modifications
can be conceived within the scope of the claims. Of course, it can
be understood that these changes or modifications belong to the
technical scope of the present disclosure
[0234] In the first embodiment, the volume data as the captured CT
images are transmitted from the CT apparatus 200 to the
robotically-assisted surgical device 100. Instead, the volume data
may be transmitted to a network server to temporarily accumulate
the data and then stored in a server or the like. In this case, as
necessary, the communication unit 110 of the robotically-assisted
surgical device 100 may acquire the volume data from the server or
the like via a wired circuit or a wireless circuit, or may acquire
the volume data via any storage medium (not illustrated).
[0235] In the first embodiment, the volume data as the captured CT
images are transmitted from the CT apparatus 200 to the
robotically-assisted surgical device 100 via the communication unit
110. This example also includes a case where the CT apparatus 200
and the robotically-assisted surgical device 100 are substantially
integrated into one product. In addition, the example may also
include a case where the robotically-assisted surgical device 100
is considered as a console of the CT apparatus 200.
[0236] In the first embodiment, the CT apparatus 200 captures
images to generate volume data including information regarding the
inside of an organism. However, another device may capture images
to generate volume data. Examples of the other device include a
Magnetic Resonance Imaging (MRI) device, a Positron Emission
Tomography (PET) device, an angiographic device, and other modality
devices. In addition, the PET device may be used in combination
with other modality devices.
[0237] In the first embodiment, the surgical robot 300 is connected
to the robotically-assisted surgical device 100. However, the
surgical robot 300 is not necessarily connected to the
robotically-assisted surgical device 100. The reason is for this is
that this connection is not particularly limited as long as the
kinematic information of the surgical robot 300 is acquired in
advance. In addition, the surgical robot 300 may be connected after
the end of the piercing of the ports. In addition, only a device
that is a part of devices constituting the surgical robot 300 may
be connected to the robotically-assisted surgical device 100. In
addition, the robotically-assisted surgical device 100 itself may
be a part of the surgical robot 300.
[0238] In the first embodiment, the surgical robot 300 is a
surgical robot for minimal invasion. However, the surgical robot
300 for minimal invasion may be a surgical robot that assists
laparoscopic surgery. In addition, the surgical robot 300 may be a
surgical robot that assists endoscopic surgery.
[0239] In the first embodiment, the robotically-assisted surgical
device 100 plans the port positions based on the volume data of the
virtual pneumoperitoneum state of the subject, but the present
disclosure is not limited thereto. For example, when the
observation target is a respiratory organ, or a cervical part,
robotic surgery may be performed without pneumoperitoneum. That is,
the robotically-assisted surgical device 100 may plan the port
positions based on the volume data of the non-pneumoperitoneum
state.
[0240] In the first embodiment, the subject PS is a human body but
may be an animal body.
[0241] The present disclosure is also applicable to a program that
implements the function of the robotically-assisted surgical device
according to the first embodiment, in which the program is supplied
to the robotically-assisted surgical device via a network or
various storage media and is read and executed by a computer in the
robotically-assisted surgical device.
[0242] As described above, the robotically-assisted surgical device
100 according to the embodiment assists minimally invasive robotic
surgery by the surgical robot 300. The processing unit 160 may
acquire 3D data of the subject PS (for example, the volume data of
the non-pneumoperitoneum state or the volume data of the virtual
pneumoperitoneum state). The processing unit 160 may acquire
operation information (for example, kinematic information) regard
to a moving part (for example, the robot arm AR or the end effector
EF) of the surgical robot 300 for performing the robotic surgery.
The processing unit 160 may acquire information of a surgical
procedure for operating the subject PS. The processing unit 160 may
acquire information regarding the position of a port that is to be
pierced on a body surface of the subject PS. The processing unit
160 may derive a 2D range on the body surface of the subject PS
where the error is allowed for the piercing of the port PT based on
the 3D data, the operation information (the kinematic information)
of the surgical robot 300, the surgical procedure, and the position
of the port PT. This 2D range is an example of the allowable region
PD (allowable range) displayed on the display 130 or the body
surface of the subject PS. The processing unit 160 may cause the
display unit (for example, the display 130) to display the
information regarding the position of the port PT and information
indicating the 2D range.
[0243] As a result, the robotically-assisted surgical device 100
displays the port position and the information indicating the 2D
range, and thus, the user can recognize the degree to which the
error is allowed during the piercing of the port PT. That is, the
piercing accuracy required for piercing the port PT can be
recognized. For example, the user can carelessly perform a piercing
work on the port PT that does not require high piercing accuracy
and thus, the operative duration can be reduced. For example, the
user can carefully perform a piercing work on the port PT that
requires high piercing accuracy, and thus, the piercing accuracy
can be secured. Since the robotically-assisted surgical device 100
visually provides the user with the information regarding the
allowable error (required piercing accuracy), a mental burden of
the user during piercing can be reduced.
[0244] In addition, distances between the position of the port PT
and respective positions on a contour of the 2D range where the
error is allowed for the port PT may include a plurality of
different distances.
[0245] As a result, the robotically-assisted surgical device 100
can also allow a state where the allowable error of the port
position in a 3D space as a real space is large in a first
direction and is small in a second direction. That is, the
robotically-assisted surgical device 100 can provide the
information regarding the allowable error that is more suitable for
an actual condition such that the allowable error of the port
position in a 3D space as a real space is not necessarily uniform
and may have directivity. For example, the user can check a
piercing position without being excessively careful in a direction
in which the allowable error is large and can carefully check a
piercing position in a direction in which the allowable error is
small. For example, the user can perform measurement only in the
direction in which the allowable error is small, and can visually
determine the position in the direction in which the allowable
error is large.
[0246] In addition, the processing unit 160 may perform a
pneumoperitoneum simulation on volume data of the subject PS to
generate the 3D data of a virtual pneumoperitoneum state.
[0247] As a result, the user can check the allowable error in
consideration of the 3D data of the virtual pneumoperitoneum
state.
[0248] In addition, the processing unit 160 may perform a plurality
of pneumoperitoneum simulations on volume data of the subject PS
with different amounts of pneumoperitoneum to generate the 3D data
of a plurality of virtual pneumoperitoneum states. The processing
unit 160 may derive a plurality of 2D ranges based on the 3D data
of the plurality of virtual pneumoperitoneum states. The processing
unit 160 may display the information regarding the position of the
port and information indicating the plurality of 2D ranges.
[0249] As a result, the robotically-assisted surgical device 100
can display the information regarding the plurality of allowable
errors corresponding to the pneumoperitoneum state. Since the way
of extension of the body surface during pneumoperitoneum or the
placement of organs in the subject PS varies, even when gas is
injected into the respective subjects PS under the same conditions,
the pneumoperitoneum states may vary, respectively. In addition,
since the amount of gas during pneumoperitoneum varies depending on
patients in consideration of the ventilation capacities of the
patients or the risk of complications, the pneumoperitoneum states
may vary. Therefore, it is difficult to accurately estimate the
pneumoperitoneum state before pneumoperitoneum. On the other hand,
the robotically-assisted surgical device 100 can display the
respective information regarding the allowable errors corresponding
to various pneumoperitoneum states, and the user can easily check
the allowable errors corresponding to various pneumoperitoneum
states.
[0250] In addition, the processing unit 160 may perform a plurality
of pneumoperitoneum simulations on volume data of the subject PS
with different amounts of pneumoperitoneum to generate the 3D data
of a plurality of virtual pneumoperitoneum states. The processing
unit 160 may derive a plurality of 2D ranges based on the 3D data
of the plurality of virtual pneumoperitoneum states. The processing
unit 160 may derives a minimum allowable range that is a range on
the body surface of the subject commonly included in the plurality
of 2D ranges. The processing unit 160 may display the information
regarding the position of the port and information regarding the
minimum allowable range MPA.
[0251] As a result, the robotically-assisted surgical device 100
can display the information regarding the plurality of allowable
errors corresponding to the pneumoperitoneum state. Since the way
of extension of the body surface during pneumoperitoneum or the
placement of organs in the subject PS varies, even when gas is
injected into the respective subjects PS under the same conditions,
the pneumoperitoneum states may vary, respectively. Therefore, it
is difficult to accurately estimate the pneumoperitoneum state
before pneumoperitoneum. On the other hand, the
robotically-assisted surgical device 100 can display the
information regarding the allowable errors in consideration of
various pneumoperitoneum states, and the user can easily check the
allowable errors corresponding to various pneumoperitoneum
states.
[0252] In addition, a shape of the 2D range may include a primitive
shape.
[0253] As a result, the robotically-assisted surgical device 100
can visually provide the user with information regarding the
allowable error in a primitive shape that can be easily understood
by the user.
[0254] In addition, the processing unit 160 may render the 3D data
to generate a rendering image. The processing unit 160 may cause
the display unit (for example, the display 130) to visualize the 3D
data with an annotation of the information regarding the position
of the port PT and information regarding a position of the 2D
range.
[0255] As a result, the user can check the port position of the
port to be pierced on the subject PS and the allowable error on the
display 130. In addition, the robotically-assisted surgical device
100 uses the 3D data such that the user can estimate the influence
of the piercing of the port on the inside of the subject PS in
consideration of the allowable error.
[0256] In addition, the processing unit 160 may causes the
projection unit 170 to project visible light representing the
information regarding the position of the port PT and information
regarding a position of the 2D range to the body surface of the
subject such that the information regarding the position of the
port PT and information indicating the plurality of 2D ranges are
displayed on the body surface of the subject PS.
[0257] As a result, the robotically-assisted surgical device 100
can project the information regarding the port position of the port
PT to be pierced and the 2D range directly to the subject PS on
which robotic surgery is performed. Therefore, the user can check
the information regarding the port position and the 2D range
projected to the subject PS to be pierced. Accordingly, the user
can recognize the allowable error using the visible light on the
subject PS as a mark, and can pierce the port PT with a reduced
mental burden.
[0258] The present disclosure is useful for, for example, a
robotically-assisted surgical device capable of recognizing a
piercing accuracy required for piercing a port, a
robotically-assisted surgery method, and a program.
* * * * *