U.S. patent application number 12/958985 was filed with the patent office on 2011-08-04 for system for remote viewing and display of a slide.
This patent application is currently assigned to Carl Zeiss MicroImaging GmbH. Invention is credited to Rui-Tao Dong, Usman Rashid, Jack A. Zeineh.
Application Number | 20110187846 12/958985 |
Document ID | / |
Family ID | 34079727 |
Filed Date | 2011-08-04 |
United States Patent
Application |
20110187846 |
Kind Code |
A1 |
Zeineh; Jack A. ; et
al. |
August 4, 2011 |
SYSTEM FOR REMOTE VIEWING AND DISPLAY OF A SLIDE
Abstract
A method for creating a virtual slide is provided. A virtual
slide is a digital representation of an area of interest of a
microscope slide. One method is to use a motorized microscope that
can move a specimen with respect to a microscope objective. With
such a system, one can capture one or more images through a
microscope objective, such that a region of interest is imaged.
Each image is then joined together to form a composite or "virtual
image." In one embodiment, after a virtual slide is created, a user
may fully utilize the full capabilities of the remote microscope.
Among these capabilities is a set of "optical objectives" and
"virtual objectives." Optical objectives are images created by
digitizing an image through a microscope objective in real time.
Virtual objectives are digitally created magnifications created by
utilizing the existing virtual slide data to digitally create a
field of view.
Inventors: |
Zeineh; Jack A.; (Newport
Beach, CA) ; Rashid; Usman; (Newport Beach, CA)
; Dong; Rui-Tao; (Newport Beach, CA) |
Assignee: |
Carl Zeiss MicroImaging
GmbH
Jena
DE
|
Family ID: |
34079727 |
Appl. No.: |
12/958985 |
Filed: |
December 2, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11502816 |
Aug 11, 2006 |
|
|
|
12958985 |
|
|
|
|
10620016 |
Jul 14, 2003 |
|
|
|
11502816 |
|
|
|
|
Current U.S.
Class: |
348/79 ;
348/E7.085 |
Current CPC
Class: |
G02B 21/367 20130101;
G06T 9/007 20130101 |
Class at
Publication: |
348/79 ;
348/E07.085 |
International
Class: |
H04N 7/18 20060101
H04N007/18 |
Claims
1. (canceled)
2. A system for remote displaying of a slide, comprising: a
microscopy system comprising a microscope having a stage to receive
a specimen slide and a camera to couple with the microscope; a
display monitor showing a first image of a portion of the specimen
slide and showing a plurality of graphical control features, the
plurality of control features comprising: one or more first
selections to command the microscopy system to capture a real time
version of a second image of at least part of the specimen slide
and to transmit the captured second image to the display monitor;
and one or more second selections to command a previously-captured
version of the second image to be transmitted to the display
monitor, wherein the previously-captured version of the second
image includes an image of a portion of a virtual slide, the
virtual slide comprising a plurality of previously-captured images
stitched together to form a composite virtual image.
3. The system of claim 2, wherein the display monitor is further
showing a thumbnail view of the specimen slide.
4. The system of claim 3, wherein the thumbnail view of the slide
comprises multiple images joined together.
5. The system of claim 2, wherein the one or more first selections
are further to specify a microscope objective of the microscope
through which the microscope system is to capture the real time
version of the second image.
6. The system of claim 2, wherein the one or more second selections
are further to specify a magnification of the previously-captured
version of the second image to be transmitted.
7. The system of claim 2, wherein the one or more second selections
are further to specify a region of interest of the slide, the
region of interest to be shown by the previously-captured version
of the second image.
8. The system of claim 7, wherein if the first image of the portion
of the specimen slide to be shown on the display monitor comprises
part of the specified region of interest, then the
previously-captured version of the second image to be transmitted
to the display monitor is to not include the part of the specified
region of interest.
9. The system of claim 2, wherein the plurality of graphical
control features further comprises one or more third selections to
specify qualifying information for at least one of the real time
version of the second image and the previously-captured version of
the second image.
10. The system of claim 9, wherein the qualifying information
comprises an x, y microscope position.
11. The system of claim 10, wherein the qualifying information
comprises an x, y, z microscope position.
12. The system of claim 10, wherein the qualifying information
comprises image dimensions.
13. The system of claim 10, wherein the qualifying information
comprises an exposure setting.
14. The system of claim 10, wherein the qualifying information
comprises a compression level.
15. The system of claim 10, wherein the qualifying information
comprises a compression type.
16. A method for remote displaying of a slide, comprising:
capturing a plurality of images through a microscope objective,
each of the plurality of images corresponding to a portion of a
specimen slide; joining the plurality of images together to form a
composite, virtual slide; providing a display monitor showing a
first image of a portion of the specimen slide and showing a
plurality of graphical control features; receiving, by receipt of a
selection of one or more of the graphical control features, a
command to obtain a second image of at least part of the specimen
slide on the display monitor, the command specifying the second
image as either a previously-captured second image or a for-capture
second image; and in response to the command: if the command
specified the second image as a previously-captured second image,
retrieving and transmitting a portion of the virtual slide to the
display monitor; and if the command specified the second image as
for capture, capturing the second image in real time through the
microscope objective and transmitting the second image to the
display monitor.
17. The method of claim 16, wherein the display monitor further
shows a thumbnail view of the slide.
18. The method of claim 16, wherein the command specifying the
second image as a previously-captured second image further
specifies a magnification of the second image, and wherein if the
command specifies the other image as a previously-captured second
image, the previously-captured second image that is retrieved is at
the specified magnification.
19. The method of claim 16, wherein the command further specifies
the second image with qualifying information.
20. The method of claim 19, wherein the qualifying information
comprises an x, y microscope position.
21. The method of claim 19, wherein the qualifying information
comprises an x, y, z microscope position.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 11/502,816, filed Aug. 11, 2006, which is a
divisional application filed from application Ser. No. 10/620,016,
filed Jul. 14, 2003, which claims benefit of U.S. Provisional
Application No. 60/087,523, filed Jun. 1, 1998, U.S. application
Ser. No. 09/323,371, filed Jun. 1, 1999, and U.S. application Ser.
No. 10/448,913, filed May 30, 2003. U.S. patent application Ser.
No. 10/620,016 is incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0002] This invention relates to a method and apparatus for viewing
remote microscope images.
[0003] Currently there is increasing demand for pathologist review
of samples at remote locations. There exist multiple systems to
address this need. They typically fall within one of two
categories: live remote microscopy and virtual slide imaging.
[0004] In live remote control microscopy, a user receives images
that are taken from a slide on a microscope. In virtual slide
imaging, a user receives images previously captured. Virtual slide
systems take one or more images of an area of interest and assemble
them together (if there is more than one image) to form a virtual
slide. Each of these techniques has its advantages. Live remote
imaging provides users with the closest approximation to manual
manipulation. Virtual slides allow faster image viewing, since
images are already captured.
[0005] Virtual slide systems take one or more images and assemble
them to form a "virtual slide."
[0006] However, users in the past were limited in their ability to
integrate these technologies. One could only view and manipulate
live and virtual images independently of one another. A user would
have to clumsily go back and forth between these two modes of
operation to separately look at the virtual slides and live
microscope slides. We present a new method that integrates these
ideas into one seamless operating environment.
SUMMARY OF THE INVENTION
[0007] In accordance with the present invention, a method and
apparatus are provided for the analysis of remote slides in a
hybrid live and virtual medium. Users obtain benefits of each
technique in a unified environment.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
[0008] A virtual slide is a digital representation of an area of
interest of a microscopic slide. A virtual slide can be created
multiple ways.
[0009] One method is to use a motorized microscope that can move a
specimen with respect to a microscope objective (e.g., a microscope
with a motorized stage). With such system, one can capture one or
more images through a microscope objective, such that a region of
interest (all or part of the microscopic slide) is imaged. Each
image is then joined together to form a composite or "virtual
image." Multiple methods of joining images together are known in
the art. One example is when images are simply abutted one next to
another. However, this method does not generally produce virtual
slides without seams, because errors such as camera rotation
relative to the axis of motion are difficult to correct. Even with
submicron accuracy stages it is, in practice, difficult to obtain
consistent positioning. Another method is to utilize overlap
between adjacent images to edge align images for maximum
seamlessness. This can be done by sequentially shifting overlapping
regions in the x and/or y axis, for example, by a stepping motor,
and calculating a correlation value (or measure of goodness of
overlap). The shift which results in the best correlation value is
then used to join the images together (FIG. 1). While this method
can be computationally expensive, it reduces reliance on
difficult-to-attain mechanical positioning requirements and
ultimately produces the best images in the sense of
seamlessness.
[0010] In another method, the virtual slide is made simply by
utilizing an imaging device with optics suitable to take a an image
of the area of interest on the slide in one snapshot. This method
is embodied in the form of a conventional digital or analog camera
with a macro lens.
[0011] This virtual slide can then be used to create a thumbnail
view of the slide. To create the thumbnail view, the virtual slide
is shrunk in resolution from its original, base resolution to a
target resolution. If the target resolution is the same as the base
resolution, then the image is unchanged. However, typically the
resolution of the thumbnail desired is several times smaller than
the base resolution.
[0012] With the virtual slide created, the user may fully utilize
the full capabilities of the remote microscope. The user is
presented with an image window and a set of control features (FIG.
2). Among these control features is a set of "optical objectives"
and "virtual objectives."
[0013] Optical objectives are images that are created by a camera
digitizing an image through a microscope objective (e.g.,
10.times., 20.times., or 40.times.) in real time (i.e., an image is
captured at the time the user requests the image). Virtual
objectives are digitally created magnifications created not by
digitizing in real time, but rather by utilizing the existing
virtual slide data to digitally create a field of view.
[0014] When a user selects one of the optical objectives, a "change
objective" command is sent to the microscope. This change objective
command can also, specify additional qualifying information, such
as microscope x, y, z positions, exposure setting, compression type
and level, and image dimensions. If additional qualifying
information is not sent, then the implicit qualifying information
is the current state of the microscope or the last specified state.
When the microscope receives the command, actions are taken to
change the objective lens and to change the state of the microscope
commensurate with the command (e.g., change the relative position
of the objective lens relative to the microscopic slide, change
exposure, etc.). An image is then digitized, compressed if so
specified, and then transmitted to the user for display.
[0015] When a user selects a virtual objective, a virtual objective
command is sent to the microscope. Similar to an optical objective
command, this virtual objective command can also specify additional
qualifying information, such as microscope x, y, and/or z position.
If additional qualifying information is not sent, then the implicit
qualifying information is current microscope state or the last
specified state. A region of interest is defined by the virtual
request--it is the area on the microscope slide included in the
field specified by the coordinates of the stage x, y and
magnification of the command. This region of interest may
optionally be trimmed such that image information already residing
at the requesting user's view is not retransmitted to the user.
[0016] An image of the region of interest can be created from the
virtual slide in multiple ways. If the virtual slide is not
compressed, the retrieval of image information corresponding to the
region of interest can be done by simply copying data from the
virtual slide. If the virtual slide is compressed, a region
corresponding to at least the region of interest can be
decompressed to a raw bitmap from the main compressed image. If the
virtual slide was stored as multiple compressed images rather than
one large image, additional efficiencies are possible. For example,
only those images that contain the desired region of interest need
be accessed for decompression, rather than the entire area of the
virtual slide. This enhances performance.
[0017] The decompression itself can also be performed in various
ways. Strategies such as scaled decoding, as in the case of jpeg
type compression, can be employed to improve the speed of the
decompression by coupling a resolution reduction process with the
decompression to speed up decompression when resolution reduction
is required. Once the region of interest is decompressed, it can
then be recompressed using a variety of strategies known to those
skilled in the imaging field, which need not be the same as the
method by which the virtual slide was compressed.
[0018] An alternative type of decompression/recompression step can
also be used involving partial decompression. Partial
decompression, such as decoding of Huffman-encoded data, as in the
case of jpeg, can be performed to produce raw coefficients, rather
than full decompression, which produces a raw bitmap. The raw
coefficients corresponding to the area of the region of interest
can then be selected. These coefficients which correspond to the
region of interest are then re-encoded. In the case of jpeg
compression, this would involve re Huffman encoding of the
coefficients, rather than in normal full compression, where a dct
must be performed followed by quantization and then Huffman
coding.
[0019] Whichever the technique, the result is a compressed region
of interest. The compressed region of interest is then transmitted
to the user for viewing. The described method is more advantageous
than sending the entire virtual slide, as one efficiently sends
only that information required by the user.
[0020] However, direct transfer of the compressed image without
decompression is feasible when the virtual slide is stored as
multiple compressed images. The compressed images that include the
area specified by the region of interest can be directly
transferred to the user, rather than going through a
decompression/recompression step. The disadvantage is that one may
transfer more information than is needed if, for example, the
compressed images are at a higher resolution that the requested
resolution. This can be partially solved by creation of multiple
resolution versions of the virtual slide. There are also
compression strategies available that allow only portions of the
compressed images to be sent, such that a given resolution can be
attained depending on which portions of the compressed image one
chooses to send (e.g., progressive encoding). However, there is
still the issue that the region of interest only partially covers
the area of the compressed image. In this case, direct transfer of
the image results in inefficiently sending data including both the
region of interest and data outside the region of interest to the
user.
[0021] With this invention, users are afforded a streamlined method
of utilizing the features of virtual and live microscopy
techniques.
BRIEF DESCRIPTION OF THE DRAWINGS
[0022] FIG. 1 is a drawing showing the overlap between adjacent
images during optimization of overlap.
[0023] FIG. 2 is a photograph of a user's view of the remote
microscope, showing a thumbnail view, microscope imaging window,
and a set of microscope controls.
[0024] While the apparatus and methods of the present invention
have been illustrated in terms of certain embodiments, the
invention claimed herein is not limited to embodiments disclosed in
this application. Rather, the scope of the invention is defined by
the claims attached hereto.
[0025] While the invention has been illustrated and explained
herein in terms of certain embodiments the invention is not limited
to the specific embodiments disclosed. Rather, the invention is
defined by the scope of the claims appended hereto.
[0026] A method for simultaneously viewing remote microscope images
comprising virtual and live images and producing a seamless view of
live and virtual images comprises: providing a virtual slide;
providing a live microscope slide; and automatically and
sequentially shifting overlapping regions of images from the slides
to obtain the optimal image is disclosed.
* * * * *