U.S. patent application number 12/016539 was filed with the patent office on 2008-06-26 for method and system for training a visual prosthesis.
Invention is credited to Wolfgang Fink, Mark A. Tarbell.
Application Number | 20080154338 12/016539 |
Document ID | / |
Family ID | 33457049 |
Filed Date | 2008-06-26 |
United States Patent
Application |
20080154338 |
Kind Code |
A1 |
Fink; Wolfgang ; et
al. |
June 26, 2008 |
METHOD AND SYSTEM FOR TRAINING A VISUAL PROSTHESIS
Abstract
A method for training a visual prosthesis includes presenting a
non-visual reference stimulus corresponding to a reference image to
a visual prosthesis patient. The visual prosthesis including a
plurality of electrodes. Training data sets are generated by
presenting a series of stimulation patterns to the patient through
the visual prosthesis. Each stimulation pattern in the series,
after the first, is determined at least in part on a previous
subjective patient selection of a preferred stimulation pattern
among stimulation patterns previously presented in the series and a
fitness function optimization algorithm. The presented stimulation
patterns and the selections of the patient are stored and presented
to a neural network off-line to determine a vision solution.
Inventors: |
Fink; Wolfgang; (Montrose,
CA) ; Tarbell; Mark A.; (Walnut, CA) |
Correspondence
Address: |
ROPES & GRAY LLP
PATENT DOCKETING 39/41, ONE INTERNATIONAL PLACE
BOSTON
MA
02110-2624
US
|
Family ID: |
33457049 |
Appl. No.: |
12/016539 |
Filed: |
January 18, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10837163 |
Apr 30, 2004 |
7321796 |
|
|
12016539 |
|
|
|
|
60467037 |
May 1, 2003 |
|
|
|
Current U.S.
Class: |
607/54 |
Current CPC
Class: |
A61N 1/36046 20130101;
A61F 9/08 20130101 |
Class at
Publication: |
607/54 |
International
Class: |
A61N 1/36 20060101
A61N001/36 |
Claims
1. A non-visual stimulus tool for use in training a visual
prosthesis comprising: a array of components whose relative
positions are movable with respect to one another to form a
physical relief suitable for use as a reference image to present to
a vision impaired patient having the visual prosthesis; and a
training processor coupled to the array of components for
controlling the relative positions of the components to generate
the training image.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. Application
10/837,163, filed Apr. 30, 2004, which claims priority to U.S.
Provisional Patent Application No. 60/467,037, entitled "Blind
Patient in the Loop Optimization Algorithm for Electrical
Stimulation Patterns for Retinal Implants Electrode Arrays" filed
on May 1, 2003, the entirety of which is hereby incorporated by
reference.
FIELD OF INVENTION
[0002] The invention relates generally to visual prosthesis
technology, and in one embodiment, to methods for training retinal
implants.
BACKGROUND OF THE INVENTION
[0003] A healthy individual's visual perception process begins with
the individual's retina(s) receiving stimulation in the form of
light. The individual's nerve cells within the retina communicate a
signal corresponding to the stimulation to the optic nerve. The
optic nerve subsequently transmits a corresponding signal to the
visual cortex through the lateral geniculate nucleus. For a vision
impaired patient, visual perception may be induced by providing
electrical stimulation at one or more of these locations, depending
on the nature patient's given impairment.
[0004] Previous studies have shown that controlled electrical
stimulation of the retina induces visual perception in blind
patients. A healthy retina has over 100 million photoreceptors. Far
fewer, however, are required to restore at least low resolution
vision in blind individuals. For example, to enable a blind person
to attain unaided mobility and large print reading, two important
quality of life indicators, tests have shown that such abilities
can be provided with orders of magnitude fewer photoreceptors being
active. Implants, such as those developed by Second Sight, LLP of
Sylmar, Calif., or described, for example, in U.S. Pat. No.
5,935,155 by Humayan et al. and U.S. Pat. No. 5,109,844 to De Juan,
Jr. et al., which include arrays of electrodes coupled to nerve
cells of a patient's retina, have been shown to be able to restore
low resolution visual perception to blind patients.
[0005] While current implant technology has been demonstrated to
stimulate some amount of visual perception, each implant needs to
be trained for its individual patient in order to effectively
elicit the appropriate visual perception. Prime candidates for the
retinal implants are previously sighted individuals who have had
their normal retinal nerve activity damaged by various conditions,
for example macular degeneration or retinitis pigmentosa. However,
due to the retinal damage in such candidates, predicting in advance
the perception induced by a retinal implant in a particular
candidate has proven difficult.
SUMMARY OF THE INVENTION
[0006] The invention addresses the deficiencies in the art by, in
one aspect, providing a method for training a visual prosthesis
(e.g., a retinal implant, a cortical implant, a lateral geniculate
nucleus implant, or an optical nerve implant) to adapt to the
patient in which it is implanted. According to this aspect, the
method includes providing a non-visual reference stimulus to a
patient having a visual prosthesis based on a reference image. The
non-visual reference stimulus is intended to provide the patient an
expectation of the visual image the visual prosthesis will induce.
Non-visual reference stimuli include, without limitation, a
pinboard, Braille text, or a verbal communication. The visual
prosthesis stimulates the patient's nerve cells with a series of
stimulus patterns attempting to induce a visual perception that
matches the patient's expected perception derived from the
non-visual reference stimulus. The patient provides feedback to
indicate which of the series of stimulus patterns induces a
perception that most closely resembles the expected perception. The
invention employs the patient feedback as a fitness function (also
referred to as a cost function or an energy function). Subsequent
stimuli provided to the patient through the visual prosthesis are
based, at least in part, on the previous feedback of the patient as
to which stimulus pattern(s) induce the perception that best
matches the expected perception. According to one embodiment, the
subsequent stimulus patterns are also based, at least in part, on a
fitness function optimization algorithm. In one embodiment, the
fitness function optimization algorithm is a simulated annealing
algorithm. In another implementation, the fitness function
optimization algorithm is a genetic algorithm.
[0007] According to one feature, the invention stores the reference
image and the series of stimulus patterns presented to the patient,
along with the choices indicated by the patient, as a first
training set. The invention may generate additional training sets
using additional reference images and series of corresponding
stimulus patterns. In one embodiment, the fitness function
optimization is modified between the stimulation of each series of
stimulation patterns. In other embodiments, the fitness function
optimization algorithm is static. According to one feature, a
neural network analyzes the training set offline to determine a
vision solution for the visual prosthesis.
[0008] In another aspect, the invention provides a system for
training a visual prosthesis such as a retinal implant, a cortical
implant, a lateral geniculate nucleus implant, or an optical nerve
implant. The system includes a training processor configured to
generate and present series of stimulation patterns corresponding
to a reference image. As in the above-described method, subsequent
stimulation patterns are based, at least in part, on patient input
and/or a fitness function optimization algorithm. The system
includes a tool for identifying to a patient the reference image
that the patient should expect to perceive using a non-visual
reference stimulus. The system includes a user input for the
patient to identify which of a set of stimulation patterns in each
series induces a perception most closely resembling the expected
perception derived from the non-visual reference stimulus.
BRIEF DESCRIPTION OF THE FIGURES
[0009] The foregoing discussion will be understood more readily
from the following detailed description of the invention with
reference to the following drawings.
[0010] FIG. 1 is a schematic depiction of a retinal implant
according to an illustrative embodiment of the invention.
[0011] FIG. 2 is a block diagram of a system for retinal implant
training according to an illustrative embodiment of the
invention.
[0012] FIG. 3 is a flowchart of a method of training a retinal
implant according to an illustrative embodiment of the
invention.
[0013] FIG. 4 is an illustrative non-visual reference
identification tool according to an illustrative embodiment of the
invention.
[0014] FIG. 5 is a flowchart of a method for generating training
data sets for training a retinal implant according to an
illustrative embodiment of the invention.
ILLUSTRATIVE DESCRIPTION
[0015] For illustrative purposes, the methods and systems below are
described with specific reference to retinal implants. The methods
also apply to the training of optical nerve implants, lateral
geniculate nucleus implants, and cortical implants.
[0016] FIG. 1 is a schematic depiction of a retinal implant system
100 according to one embodiment of the invention. The retinal
implant system 100 includes elements implanted within a patient's
eye 102 and portions exterior to the patient's eye 102. For
example, in the illustrative example, the retinal implant system
100 includes an electrode array 104, a receiver 106, and an
electrode array processor 108 implanted within the patient's eye
102. An image processor 110 and video camera 112 remain outside of
the patient's eye. In other embodiments, the image processor 110
can be implanted within the patient's eye 110 or elsewhere within
the patient. In other embodiments, the video camera 112 can be
replaced with other image capture devices such as CCD arrays.
[0017] The electrode array processor 108 is electrically connected
to the electrode array 104 and generates an electrical stimulus for
output through each electrode in the electrode array 104. Electrode
stimulation does not necessarily have a one-to-one correspondence
to a patient perceiving a pixel of light. For example, in some
patients, in order to induce perception of one particular pixel of
vision, two or more electrodes may need to be energized. The
electrode array processor 108 receives its drive instructions from
the receiver 106 which in turn wirelessly receives instructions
from the image processor 110.
[0018] In the illustrative embodiment, the patient wears the image
processor 110, for example, on his or her belt. In addition to the
image processor 110, the retinal implant system 100 includes a
camera 112 for providing vision information to the image processor
110. The image processor 110 then processes the vision information
to determine the appropriate electrical stimuli patterns for the
retinal implant 100. The image processor 110 can be implemented in
either hardware, such as, and without limitation, an ASIC, Digital
Signal Processor, or other integrated circuit, or in software
operating on a computing device, or in a combination of the
two.
[0019] In one embodiment, the image processor 110 receives a
digital image from the video camera 112. The image processor 110
converts the received image to a gray scale and reduces/downscales
the resolution of the image to a resolution matching the number of
electrodes in the electrode array 104. In other embodiments, the
image processor 110 retains the image in color. The image processor
110 then determines the appropriate electrical stimulus for each
electrode in the electrode array 104 based on a vision solution.
The vision solution determines the values of a number of parameters
that can be manipulated for each electrode, including, without
limitation, for example, the amplitude of the voltage applied to
each electrode (if any), the timing of the onset of the electrical
signal applied to each electrode relative to the other electrodes,
the shape of the signal applied to each electrode, the width of a
pulse applied to the electrode, the frequency of the signal
applied, and the duration of the voltage applied to the electrode.
In contrast to methods of training optical implants that tune
temporal-spatial filters to model a healthy retina, the
illustrative training method determines direct electrode outputs in
response to received images.
[0020] The image processor 110, after generating a stimulus pattern
for the electrode array 104, transmits the stimulus pattern
wirelessly to the receiver 106 within the patient's eye 102.
[0021] FIG. 2 is a conceptual high-level block diagram of a retinal
implant training system 200. The training system 200 includes a
training processor 202, a user input 204, and a reference image
identification tool 206. The elements of the training system 200
will be best understood with reference to the method described in
FIG. 3.
[0022] FIG. 3 is a flowchart of a method 300 for training a retinal
implant 100, such as the implant described in FIG. 1, to determine
a vision solution for an individual patient's retinal implant 100.
A patient participates in training his or her individual implant
100 with the aid of the training system 200. In general, the
training method 300 can be divided into two portions, generating
training data (step 302) and processing training data (step 304) to
produce the vision solution.
[0023] Generating training data (step 302) can be further divided
into three processes: i) selecting a training reference image (step
306), ii) communicating the reference image to the patient (step
308), and iii) training the patient's implant to successfully
produce the reference image (step 310).
[0024] First, the training processor 202 selects a first training
reference image (step 306) to attempt to stimulate perception of in
the patient. In one embodiment the training reference image is
selected from a predefined group, such as letters, numbers, or
basic shapes. For example, the first training reference image may
be a solid white rectangle. Subsequent training reference images
include individual straight vertical and horizontal lines at the
extremities of the patient's inducible field-of-view. Other
preliminary training reference images include individual bright
spots at various locations in a patient's field of view. The
training set then identifies the first training reference image to
the patient using a non-visual reference stimulus (step 308), for
example by employing the reference image identification tool 206.
The non-visual reference stimulus provides the patient an
expectation of the perception the implant 100 is attempting to
induce.
[0025] FIG. 4 is an example 400 of a reference image identification
tool 206. The tool 400 includes an array 402 of pins (e.g., 404)
whose positions relative to one another can be rearranged to form a
physical relief corresponding to the reference image. The array 402
generally has one pin 404 per pixel of resolution of the reference
image. For example, the configuration of the pins of the tool 400
corresponds to a reference image having a simple single bright spot
in the center of the patient's field-of-view. The training
processor 202 controls the arrangement of the pins 404 in the array
402. Other deformable tools can be similarly employed to provide a
tactile stimulus representing the first training reference image.
Alternatively, if the patient previously had been sighted, the
patient can be informed of the first training reference image
verbally. In such embodiments, the tool for non-visually
communicating the reference image can be a person, for example a
doctor aiding in the training, or an audio speaker connected to a
speech synthesizer. In another embodiment, the training system 200
can identify the reference image using Braille text.
[0026] Referring back to FIGS. 2 and 3, the training system 200
then trains the implant 100 to correctly induce perception of the
reference image (step 310).
[0027] FIG. 5 is a flow chart of a method 500 for training the
retinal implant to induce perception of a reference image,
according to an illustrative embodiment of the invention. The
training processor 202 communicates a first stimulation pattern to
the retinal implant (step 502) by transmitting instructions to the
electrode array processor 108. Subsequently, the training processor
202 generates a second stimulation pattern (step 504). The
electrode processor 108 energizes the electrodes 104 according to
the stimulation patterns. The patient, perceiving the results
induced by the stimulation patterns, and having an expectation of
the induced perception based on the non-visual reference stimulus,
selects, using the input 204, the stimulation pattern that induces
a visual perception closest to the expected perception (step 506).
Based on the selection by the patient and preferably a fitness
function optimization algorithm, the training system 200 generates
yet another stimulation pattern (step 504). The patient chooses
between the previously selected stimulation pattern and the most
recently generated stimulation pattern, once again, based on which
stimulation pattern induces the visual perception most closely
matching the expected perception (step 506). This process continues
until the generated stimulation pattern induces a visual perception
that sufficiently matches the expectations of the patient (step
512).
[0028] In practice, no stimulation pattern may generate a
perception that identically matches the patient's expected
perception of the training reference image. According to one
feature, the training processor 202 generates new stimulation
patterns until the patient recognizes diminishing returns in each
new stimulation pattern. At that point, the patient chooses the
best available fit.
[0029] As indicated above, the generation of each stimulation
pattern after the first is based at least in part on a fitness
function optimization algorithm. In one embodiment, the fitness
function optimization algorithm is a simulated-annealing algorithm.
Simulated annealing is a widely used and well-established
optimization technique, particularly suited for high-dimensional
configuration spaces. The goal of the algorithm is to minimize an
energy function, in this case, the subjective evaluation by the
blind patient of the difference between the perceived image and the
expected perception.
[0030] The minimization is performed by randomly changing the value
of one or more of the stimulation parameters within physiologically
tolerable limits and reevaluating the energy function. Two cases
can occur: 1) the change in the variable values results in a new,
lower energy function value, i.e. the patient prefers the changed
stimulation pattern; or 2) the energy function value is higher or
unchanged, i.e. the patient prefers the prior stimulation pattern
or has no opinion on which pattern is better. In the first
scenario, the new set of parameter values is stored and the change
accepted. In the second scenario, the new set of variable values is
only stored with a certain likelihood (Boltzmann probability,
including an annealing temperature). This ensures that the overall
optimization algorithm does not get stuck in local minima of the
energy function too easily. The annealing temperature directly
influences the Boltzmann probability by making it less likely to
accept an energetically unfavorable step, the longer the
optimization lasts (cooling schedule). Then, the overall procedure
is repeated until the annealing temperature has reached its end
value or a preset number of iterations has been exceeded.
[0031] In one embodiment, the training system 200 applies a
derivative of the canonical simulated annealing algorithm, by
replacing the Boltzmann probability acceptance with an energy
threshold acceptance: each configuration with an energy
E<Emin+Ethreshold will be automatically accepted, with
Ethreshold oscillating between to preset boundaries similar to
"simulated reheating and cooling".
[0032] In another embodiment, the fitness function optimization
algorithm is a genetic algorithm. In general, genetic algorithms
try to mimic evolutionary principles used by nature in applying
genetic operations such as point mutation, cross-over, and
inversion to parameter strings, called "genes" or "chromosomes", in
order to "evolve" a set of parameters that achieves a high fitness
as determined by a fitness function (similar to the energy function
in simulated annealing).
[0033] In one such embodiment, the training system 200 uses the
following genetic algorithm to generate stimulation patterns. The
training system 200 begins with generating a stimulation pattern
having a set of parameters having random values within
physiologically tolerable limits. The training system 200 replaces
patient rejected stimulation patterns with stimulation patterns
generated by executing genetic operations on a randomly selected
set of the stimulation pattern parameters.
[0034] In another embodiment, the algorithm operates in the
following fashion. The training system 200, after receiving input
from a patient compares the most recently selected stimulation
pattern with the most recently rejected stimulation pattern to
identify differences in electrode output parameters in the
stimulation patterns. In the subsequently generated stimulation
pattern, at least one identified difference between the stimulation
patterns is further emphasized. For example, if the voltage applied
to a particular electrode in the electrode array in the selected
stimulation pattern is 5% greater than the voltage applied to that
electrode in the rejected stimulation pattern, a subsequent
stimulation pattern would apply a voltage to that electrode that is
8% greater than that applied in the rejected stimulation pattern.
Such increases in voltage to that electrode in subsequently
generated stimulation patterns continue until the patient rejects
the newest stimulation pattern or the voltage reaches
physiologically tolerable limits. Then, a different stimulation
parameter is altered in subsequently generated stimulation
patterns.
[0035] After the training system 200 successfully generates a
stimulation pattern that induces the visual perception expected by
the patient based on the non-visual reference stimulus, the
training system 200 repeats (step 514) the training process 500
with additional training reference images. The training system 200
stores training data sets including the reference image, records of
all the stimulation patterns that were generated, which stimulation
patterns were compared to each other, which of those patterns were
selected, and the final successful stimulation patterns.
[0036] In one embodiment, the training system 200 analyzes training
sets between training sessions that use different reference images
in order to further optimize (step 312) the fitness function
optimization algorithm. An ordered choice of reference images can
facilitate this optimization (step 312).
[0037] After successful stimulation patterns for one or more
training reference images are achieved, the training data set(s) is
analyzed (step 304) by, for example, an artificial
feed-forward-type multi-layer neural network configured, e.g.,
using the Error-Backpropagation algorithm; a Hopfield-type
Attractor neural network configured with, e.g., the Projection Rule
or the Pseudoinverse algorithm; and/or a recurrent neural network
trained, e.g., with algorithms such as Real-Time Recurrent Learning
algorithm and the Time-Dependent Recurrent Backpropagation to
determine a vision solution. The neural network analysis yields the
vision solution, which, in one embodiment, is itself a neural
network. The vision solution provides the capability to generate
stimulation patterns for correctly inducting perception of the
reference images and of images not presented as part of the
training by applying the "knowledge" extracted from the analysis of
the training data set(s). The resulting vision solution is
transferred to the image processor 110.
[0038] The fitness function optimization algorithm employed by the
training system 200 can further be optimized by testing the
optimization algorithm on sighted subjects. Instead of
communicating a reference stimulus to the sighted subject
non-visually, the sighted subject is presented with the reference
image along with two potential images that correspond at least in
part to the reference image. As with the training of the retinal
implant, the sighted subject selects the image most closely
resembling the reference image. A next set of images is selected
for presentation using the fitness function optimization algorithm,
however, the electrical stimulus parameters are substituted with
direct visual parameters such as brightness and contrast.
Optimization algorithms are objectively compared by comparing the
average number of iteration steps before a presented image suitably
resembles the reference image.
[0039] The invention may be embodied in other specific forms
without departing form the spirit or essential characteristics
thereof. The forgoing embodiments are therefore to be considered in
all respects illustrative, rather than limiting of the
invention.
* * * * *