U.S. patent application number 15/859126 was filed with the patent office on 2018-07-05 for deep learning ingredient and nutrient identification systems and methods.
The applicant listed for this patent is Suggestic, Inc.. Invention is credited to Victor Chapela, Ricardo Corral Corral.
Application Number | 20180189636 15/859126 |
Document ID | / |
Family ID | 62709046 |
Filed Date | 2018-07-05 |
United States Patent
Application |
20180189636 |
Kind Code |
A1 |
Chapela; Victor ; et
al. |
July 5, 2018 |
Deep Learning Ingredient and Nutrient Identification Systems and
Methods
Abstract
Provided herein are exemplary methods and systems that include a
deep learning ingredient and nutrient identification system that
can learn from websites, databases, ontologies, recipes, food lists
and restaurant menu items to identify ingredients, ingredient
quantity, nutrient composition and diet adherence of different
products, recipes or restaurant menu items, a propagation algorithm
that can enhance or correct ingredient lists, ingredient quantity,
nutrient composition and diet adherence for similar items in the
food or activity databases, an adaptive ontology that can be
learned from the internet and can be encoded to aid the machine
learning or artificial intelligence algorithms to better learn the
relationships between different food elements and groupings; and a
contextual filtering and adherence scoring system that identifies
and selects recipes, food products, supplements, medications and
restaurant menu items according to a personalized plan.
Inventors: |
Chapela; Victor; (Palo Alto,
CA) ; Corral Corral; Ricardo; (Mexico City,
MX) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Suggestic, Inc. |
San Francisco |
CA |
US |
|
|
Family ID: |
62709046 |
Appl. No.: |
15/859126 |
Filed: |
December 29, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62440924 |
Dec 30, 2016 |
|
|
|
62440982 |
Dec 30, 2016 |
|
|
|
62440689 |
Dec 30, 2016 |
|
|
|
62440801 |
Dec 30, 2016 |
|
|
|
62441043 |
Dec 30, 2016 |
|
|
|
62441014 |
Dec 30, 2016 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A23V 2002/00 20130101;
G06N 3/0427 20130101; G06N 3/02 20130101; G16H 20/60 20180101; A61M
2005/14208 20130101; A61M 5/1723 20130101; A61B 5/021 20130101;
G06N 5/025 20130101; G06F 16/9024 20190101; G09B 5/02 20130101;
A61B 5/681 20130101; A61M 5/14244 20130101; A61B 5/02438 20130101;
A61B 5/14532 20130101; G06T 11/60 20130101; A23L 33/40 20160801;
G06F 16/27 20190101; G06T 19/006 20130101; A61B 5/14551 20130101;
A61B 5/486 20130101; G06N 5/045 20130101; G06T 11/206 20130101;
A61B 5/4806 20130101; G16H 50/20 20180101; A61B 5/08 20130101; A61B
5/4866 20130101; G06F 1/163 20130101; G06N 3/0454 20130101; G06N
5/022 20130101; G06N 20/00 20190101; A61B 5/6803 20130101; G06N
7/005 20130101; G09B 19/0092 20130101; G16H 50/30 20180101; G09B
19/00 20130101; A61B 5/0205 20130101; G06N 3/08 20130101 |
International
Class: |
G06N 3/04 20060101
G06N003/04; G06N 3/08 20060101 G06N003/08; G06F 17/30 20060101
G06F017/30 |
Claims
1. A system, comprising: a deep learning nutrient and ingredient
identification system that can learn from websites, databases,
ontologies, recipes, food lists and restaurant menu items to
identify ingredients, ingredient quantity, nutrient composition and
diet adherence of different food or nutrition products, recipes or
restaurant menu items; a propagation algorithm that can enhance or
correct ingredient lists, ingredient quantity, nutrient composition
and diet adherence for similar items in the food or activity
databases; and an adaptive ontology that can be learned from the
internet and can be encoded to aid the machine learning or
artificial intelligence algorithms to better learn the
relationships between different food elements and groupings.
Description
CROSS-REFERENCE TO RELATED TO APPLICATIONS
[0001] The present patent application claims the benefit of U.S.
Provisional Patent Application No. 62/440,924, filed Dec. 30, 2016,
and titled "Personalized Program Generation System with Plan and
Ruleset Stacking", U.S. Provisional Patent Application No.
62/440,689, filed Dec. 30, 2016, and titled "Dynamic and
Feedback-Based Ecosystem", U.S. Provisional Patent Application No.
62/440,982, filed Dec. 30, 2016, and titled "Personalized Program
Generation System with Adaptive Program Engine", U.S. Provisional
Patent Application No. 62/440,801, filed Dec. 30, 2016, and titled
"Contextual Filtering and Adherence Scoring Systems and Methods",
U.S. Provisional Patent Application No. 62/441,014, filed Dec. 30,
2016, and titled "Deep Learning and Ingredient Identification
Systems and Methods", and U.S. Provisional Patent Application No.
62/441,043, filed Dec. 30, 2016, and titled "Multivariate Causation
Systems and Methods". The present patent application is related to
Non-Provisional U.S. patent application Ser. No. 15/858,713, filed
Dec. 29, 2017, and entitled "Augmented Reality Systems Based on a
Dynamic feedback-based Ecosystem and Multivariate Causation System"
(Attorney Docket No. PA9009US). The present patent application is
also related to Non-Provisional U.S. patent application Ser. No.
15/859,062, filed Dec. 29, 2017, and entitled "Augmented Reality
and Blockchain Technology for Decision Augmentation Systems and
Methods Using Contextual Filtering and Personalized Program
Generation" (Attorney Docket No. PA9010US). All of the
aforementioned disclosures are hereby incorporated by reference
herein in their entireties including all references and appendices
cited therein.
FIELD OF THE PRESENT TECHNOLOGY
[0002] The present technology relates generally to a dynamic and
feedback-based ecosystem that provides tailored solutions for
users. The systems and methods provided herein comprise a plurality
of individual feedback loops that provide users with adaptive
health, wellness, productivity, activity and/or longevity programs
that are being constantly adapted based on coded rulesets generated
from empirical studies, personal biomarkers, genome, microbiome,
blood test analysis, preferences, restrictions, beliefs and goals,
as well as, sensor feedback, user feedback, external sources and
input from multivariate causation analysis.
SUMMARY OF THE PRESENT TECHNOLOGY
[0003] Provided herein are exemplary methods and systems that
include a deep learning ingredient and nutrient identification
system that can learn from websites, databases, ontologies,
recipes, food lists and restaurant menu items to identify
ingredients, ingredient quantity, nutrient composition and diet
adherence of different products, recipes or restaurant menu items,
a propagation algorithm that can enhance or correct ingredient
lists, ingredient quantity, nutrient composition and diet adherence
for similar items in the food or activity databases, an adaptive
ontology that can be learned from the internet and can be encoded
to aid the machine learning or artificial intelligence algorithms
to better learn the relationships between different food elements
and groupings; and a contextual filtering and adherence scoring
system that identifies and selects recipes, food products,
supplements, medications and restaurant menu items according to a
personalized plan.
[0004] Further exemplary embodiments include the contextual
filtering and adherence scoring system providing augmented reality
selections of food products, supplements, medications, recipes and
menu items using images obtained from a restaurant menu, the
contextual filtering and adherence scoring system using deep
learning, machine learning or artificial intelligence to correctly
identify and characterize, food products, supplements, medications,
recipes and menu items, the contextual filtering and adherence
scoring system also using feedback from a user to fine tune image
recognition, as well as increase content accuracy and data
quality.
BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The accompanying drawings, where like reference numerals
refer to identical or functionally similar elements throughout the
separate views, together with the detailed description below, are
incorporated in and form part of the specification, and serve to
further illustrate embodiments of concepts that include the claimed
disclosure, and explain various principles and advantages of those
embodiments.
[0006] The methods and systems disclosed herein have been
represented where appropriate by conventional symbols in the
drawings, showing only those specific details that are pertinent to
understanding the embodiments of the present disclosure so as not
to obscure the disclosure with details that will be readily
apparent to those of ordinary skill in the art having the benefit
of the description herein.
[0007] FIG. 1 illustrates an example ecosystem of the present
disclosure.
[0008] FIG. 2 illustrates an example causation sequence performed
by the multivariate causation system.
[0009] FIG. 3 shows an exemplary architecture for a multi-model,
multi-ontology, multi-label deep neural network, otherwise known as
"mLOM."
[0010] FIG. 4 is a table illustrating ROC curves for ingredient
prediction. Each ROC curve represents the output for each meal in a
validation set.
[0011] FIG. 5 is a diagrammatic representation of an example
machine in the form of a computer system.
DESCRIPTION OF EXEMPLARY EMBODIMENTS
[0012] FIG. 1 illustrates an example ecosystem 100 of the present
disclosure. The ecosystem generally comprises a ruleset generator
system 102, a personalized program generation system 104, a
contextual filtering and adherence scoring system 106, a passive
and active feedback system 108, and a multivariate causation system
110. These various systems can be executed using, for example, a
server or within a cloud-based computing environment. In some
embodiments, each of the various systems of the ecosystem 100 can
be consolidated into a single system.
[0013] In one embodiment, the ruleset generator system 102 obtains
input from two separate sub-systems. In one embodiment, the ruleset
generator system 102 obtains lifestyle, dietary, nutrition and
empirical evidence-based diets and programs from various sources
112. For example, this could include peer-reviewed or other similar
publications or data regarding diets and exercise such as
ketogenic, paleo, vegan, low carbohydrate, low-fat, or even
specific dietary plans such as whole 30, Daniel Diet, and so forth.
These programs or diets can also be submitted directly by
researchers, healthcare professionals, user groups or individual
users. These data are received by the ruleset generator system 102
and converted into rulesets that can be applied to a personal
program for a user. For example, the ruleset can include rules for
those persons with specific biomarkers or characteristics, with
specific ratios of macronutrients that would be found in a
particular dietary plan, as well as restricted or promoted food
items.
[0014] The ruleset generator system 102 can also obtain input from
the multivariate causation system 110, as will be described in
greater detail below. In general, the multivariate causation system
110 can generate both updated lifestyle, dietary, nutrition and
empirical evidence-based data for the various sources 112, as well
as ruleset updates that are each based on output of the passive and
active feedback system 108. In general the passive feedback from
sensors and external sources and active feedback from user input in
system 108 is measuring empirical feedback from biometric resources
or other applications that are tracking exactly how the user is
behaving or acting based on the information given and plan
established by the ecosystem 100. For example, if the user is
following a prescribed plan that is designed to reduce weight but
the user is not achieving results based on scale feedback received
by the passive and active feedback system 108, the multivariate
causation system 110 can assess the lack of progress and determine
a change to the ruleset(s) that might positively correlate with an
improvement towards the goal. For example, if a user desires to
lose weight and has not done so, the multivariate causation system
110 might suggest that the user reduce caloric intake or
potentially remove a food item from their diet or substitute one
item for another in their diet to improve their outcome.
[0015] To be sure, the multivariate causation system 110 can
utilize artificial intelligence techniques such as machine
learning, deep learning, or big data that include information from
other users having similar genetics, biomarkers, profile,
activities, background, clinical data or other demographic or
personal information. Thus, the user is not only analyzed in
context of their own personal goals and personal information, but
the multivariate causation system 110 can also derive or infer new
rules based on what has worked or not worked for other similarly
situated individuals. In other embodiments of the multivariate
causation engine it also includes any type of data streams or log
data to derive or infer new rules based on the sequences and
patterns found. The sequence or log data can include, but is not
limited to sensors, test results, biomarkers, activities, symptoms,
supplements, medicine intake, food, beverages or locations. The
multivariate causation engine can also determine the likelihood
that each pattern or sequence of events will have a predicted
outcome.
[0016] The personalized program generation system 104 can utilize
ruleset stacking to create a converging solution for a set of
dietary considerations or limitations for a user. For example, a
user may have the dietary considerations of being a vegetarian, as
well as being on a reduced-salt diet. The user also does not like
certain gluten products. The personalized program generation system
104 can overlap these diets for a single user. In other embodiments
the personalized program generation system 104 can overlap the
dietary restrictions of multiple users to create a single
converging solution for multiple parties.
[0017] The personalized program generation system 104 can also
implement an adaptive program algorithm and create a personalized
program for a user. The personalized program generation system 104
receives one or more rulesets that are applicable to the user based
on information known about the user. For example, the user prefers
paleo diet. Thus, the personalized program generation system 104
will obtain rulesets for paleo adherents and will further
personalize each of the rules based on the user's information. The
rulesets obtained from the ruleset generator system 102 can be
selectively adjusted based on other information such as a user's
genetic information, their microbiome, their biomarkers, their
clinical, medical or health data, activities, their age, weight,
height, ethnic background, other demographic information, and so
forth.
[0018] In some embodiments, the personalized program generation
system 104 can implement a data gathering process whereby a user is
questioned using a chatbot or other means to obtain information
that is used to select any appropriate ruleset and goal for the
user. To be sure, this can be augmented with more detailed
information about the user such as specific objective and
subjective demographic information, genetic test information,
microbiome testing, and so forth. This information can also be
obtained from medical records, including electronic medical
records. An example method for collecting user information
comprises the use of a chatbot that is programmed to interact with
a user to request diet preferences and health conditions, as well
as a target goal(s). Another example is to obtain the user's
permission to connect to their health application or service that
will further enhance the personalization of their program.
[0019] The contextual filtering and adherence scoring system 106 is
configured to execute dynamic adherence scoring algorithms to
determine the adherence level of any meal or activity against the
personalized program, diet or plan. It obtains information from
external and internal sources such as restaurant and recipe
sub-systems or databases (e.g. nutrient identification system 114).
In some embodiments, the contextual filtering and adherence scoring
system 106 obtains recipe, supplement, grocery, and restaurant menu
information using deep learning and artificial intelligence
information gathering and processing techniques. The contextual
filtering and adherence scoring system 106 can also obtain feedback
on these types of information from user interaction with the
ecosystem 100. This user feedback assists in resolving errors or
inconsistencies with data.
[0020] According to some embodiments, the contextual filtering and
adherence scoring system 106 can use specific techniques to examine
menus, recipes, and ingredient lists from a wide variety of sources
and correlate and/or harmonize what is known about individual
meals, activities or places. In this way, the contextual filtering
and adherence scoring system 106 can select appropriate meals or
activities for the user based on their goals and personalized
program.
[0021] The contextual filtering and adherence scoring system 106
provides personalized programs to the user through their computing
device 116. The contextual filtering and adherence scoring system
106 provides the user with a personalized program that is tailored
based on selection of lifestyle, dietary, nutrition and empirical
evidence-based programs that are converted to rulesets and applied
to a user's goals, preferences, and demographics. Contextual
filtering is applied in some embodiments to selectively tailor the
recipe or menu suggestions provided to the user in accordance with
their personalized plan generated by the personalized program
generation system 104.
[0022] The computing device 116 executes a client side application
that provides personalized plans and receives both passive and
active feedback, in some embodiments.
[0023] In some embodiments, the passive and active feedback system
108 receives data from the user through the computing device 116.
For example, the user can create a food log or record their
exercise. The user can also take pictures of food, menus,
ingredient lists, and so forth. This information can be fed back
into the restaurant and recipe sub-systems or databases (e.g.
nutrient identification system 114). This gathered information can
also be redirected back to the passive and active feedback system
108 for further analysis by the multivariate causation system
110.
[0024] In some embodiments, the passive and active feedback system
108 collects information from external sensors 118 from sensors
such as wearables (e.g., smart glasses, watches, etc.), sleep
sensors, blood pressure monitors, glucose monitors and insulin
pumps, blood pressure sensors, respiration monitors, pulse
oximeters, heart rate meters, and so forth--just to name a few.
[0025] The multivariate causation system 110 is configured to
receive empirical feedback about the user and their behavior from
the computing device 116 and the external sensors 118. The
multivariate causation system 110 uses the specific information
known about the user and those users who are similar in one way or
another (goals, biometrics, biomarkers, genetics, demographics,
lifestyle, and so forth), as well as feedback from the external
sensors 118 to selectively modify how a user's diet is prioritized
and potentially if rulesets are adjusted for the user. For example,
as different users progress towards a goal, their passive and
active feedback is analyzed by the multivariate causation system
110 that determines what has worked. It then modifies and
reprioritizes the program rulesets so that the patterns and
activity sequences that work best are suggested, and those patterns
or sequences that do not work are reduced or avoided. The
multivariate causation system 110 can adjust priority and/or rules
for the diets and programs to more closely align with the goals of
the successful users. The multivariate causation system 110
receives streams of data from user passive and active feedback, as
well as the programs, goals and personal data and adjusts the
rulesets on the fly or periodically.
[0026] The multivariate causation system 110 can also act as a data
producing system that reports back information for use in the
lifestyle, dietary, nutrition and empirical evidence-based diets
and programs from various sources 112. The multivariate causation
system 110 can deeply analyze user feedback and determine specific
variations on behaviors and determine how they affect the desired
outcomes. For example, the multivariate causation system 110 may
determine that the user moves closer to achieving a goal when they
restrict carbohydrate consumption in the morning and evening, or if
they eat vegetables as snacks as opposed to combining them with
protein sources.
[0027] FIG. 2 illustrates an example causation sequence performed
by the multivariate causation system 110. The causal event sequence
discovery is a process performed from events gathered from user
behaviors and the external sensors 118 and the computing device
116. Other example methodologies that can be implemented by the
multivariate causation system 110 are described in Attachments A
(Human Activity Language) and B (Low entropy approaches for causal
inference), which are incorporated by reference herein in its
entirety, including all references cited therein. The multivariate
causation system 110 can also implement a deep multi-label,
multi-ontology a multi-model network architecture as described in
Attachment C, which is incorporated by reference herein in its
entirety, including all references cited therein.
[0028] These systems of the ecosystem 100 work together in a
collective and synergistic manner to provide a user with empirical
and evidence-based plans for improving their health. The ecosystem
100 uses various feedback loops (represented by the individual
systems of the ecosystem 100) to create an environment that learns
based on empirical feedback and fine tunes a plan for a user based
on this information.
[0029] In some embodiments, the ecosystem 100 can comprise a
comprehensive knowledgebase of recipes and menu items from the
restaurant and recipe sub-systems (e.g. nutrient identification
system 114).
[0030] In some embodiments, the knowledgebase is created and
updated by searching the Internet for recipes, grocery products,
meal plans, ingredient lists, and restaurant menus--just to name a
few. In addition to obtaining recipes and menu items, the ecosystem
100 generates nutritional and ingredient information for a recipe,
grocery product, or restaurant menu items based on several
different algorithms. The algorithms include but are not limited
to: deep learning algorithms, dynamic food ontologies and
propagation algorithms. For example, when a new restaurant menu is
incorporated to the system it will automatically populate the
expected ingredients and nutrients for each of the menu items by
reading the menu, comparing it to recipes, other restaurants and
the ontology it has learned. And with that accumulated knowledge
the system would infer that a Caesar Salad most probably contains
romaine lettuce and croutons dressed with Parmesan cheese, lemon
juice, olive oil, egg, Worcestershire sauce, garlic, and black
pepper. It will also infer the expected amount of each ingredient
as well as the nutrients. If a user were to send feedback stating
that this particular Caesar Salad also contains chicken. That
feedback would not only modify the ingredient list for that
specific restaurant's Caesar Salad as containing Chicken, but it
would also increase the probability of all other similar Caesar
Salads, at similar restaurants to also contain chicken.
[0031] Prior to searching and recommending items to a user, the
knowledge base is built using deep learning or other machine
learning or artificial intelligence methodologies. For example,
deep learning algorithms are used to make inferences about a menu
at a restaurant by learning from menu items from other restaurants
or recipes that match the menu items at the restaurant in question.
The inferences can be affected by user input as well. For example,
user feedback or input can be used that provides corrections or
context for a restaurant menu, a menu item, or recipe--just to name
a few. The user can specify, for example, if a restaurant is a
vegan restaurant. Thus, even if the menu items are purposefully or
unintentionally mis-descriptive, the ecosystem 100 can correctly
identify a menu item based on ingredients or contextual
information. If the restaurant serves only vegan dishes (as listed
on the menu), this knowledge is used to specify that a menu item of
cheeseburger is likely to have vegan ingredients. The contextual
filtering and adherence scoring system 106 can determine the actual
ingredients from those listed on the menu, a restaurant website, or
other similar recipes for vegan cheeseburgers.
[0032] In some embodiments the deep learning algorithms learn from
different food and recipe datasets which ingredients or nutrients
are common for a specific type of meal and then infer these
ingredients and nutrients for restaurant menu items. An example
methodology that can be implemented by the deep learning ingredient
and nutrient identification system 114 is described in Attachment
C, which is incorporated by reference herein in its entirety,
including all references cited therein.
[0033] In some embodiments, food and activity ontologies are built
based on a continuous relation of descriptions and data such as
those contained in recipes, menu items, dictionaries, databases and
the Web pages, among others. These food and activity ontologies are
then used to generate a complex network (graph) representation of
the different activities, places, ingredients, menu items, recipes
and food products. With this representation the deep learning
algorithms learn to distinguish the relations of different
activity, place, food, nutrition, ingredient groups and
classifications. An example methodology that can be implemented to
create activity, place, food, nutrition and ingredient
classifications used by the deep learning ingredient and nutrient
identification system 114 is described in Attachment A, which is
incorporated by reference herein in its entirety, including all
references cited therein. These examples involve what is referred
to as human activity analysis methods.
[0034] In some embodiments the propagation algorithm allows for
corrections or enhancements to be propagated to all similar menu
items, recipes or products. The corrections or enhancements can be
automatically generated by other algorithms, can be imported from
external databases or can be fed back into the system through user
feedback.
[0035] FIG. 3 shows an exemplary architecture for a multi-model,
multi-ontology, multi-label deep neural network, otherwise known as
"mLOM."
[0036] To be able to search restaurant menus, one needs to know the
ingredients and nutrients every menu item has, out of thousands of
possible options. Restaurants do not publish the ingredients they
use. And only around 800 chain restaurants in the US are required
to make available a reduced set of 11 nutrients out of the 200 that
they may use.
[0037] To query over half a million restaurant menus, mLOM
represents artificial intelligence ("AI") technology that learns to
predict accurately every food ingredient, nutrient, flavor,
portion, course, cuisine and quantity. mLOM has achieved over 98.5%
of median AUC (area under the curve) in a composite ROC curve (an
indicator of accuracy), and when compared to humans, mLOM is almost
twice as effective in knowing the ingredients in a menu item.
[0038] FIG. 4 shows a table illustrating ROC curves for ingredient
prediction. Each ROC curve represents the output for each meal in a
validation set:
[0039] One of the problems solved herein is inferring food related
labels from meal names and their descriptions. Such relevant labels
can belong to different ontological domains, like courses and
cuisine type, additionally to ingredients, nutrients, substances
and other characteristics. Thus, mLOM handles multi-label learning,
which comprises learning label subsets of particular objects from a
base label set. This task is different from multi-class learning,
where an output is expected to be only one of many mutually
exclusive labels. Note that multi-class learning is a particular
case of multi-label learning, thus, mLOM is also able to handle any
multi-class setup if needed.
[0040] Multi-ontology label learning capabilities implemented in
mLOM solve two apparently opposite challenges in contrast with
simple multi-label learning. First, knowledge from different
ontological domains is transferred to others, and labeling under a
particular domain is fine-tuned to concentrate in specific domain
characteristics.
[0041] In some embodiments, mLOM can synergistically combine
different heterogeneous approaches. Each approach is referred as a
base model, thus, mLOM can be composed of an ensemble of any number
of base models, depending of the combination that offers the
overall superior predictive performance under any chosen
performance criteria.
[0042] Accordingly, the mLOM architecture incorporates knowledge
from m base models, each performing multi-label learning over
ontological domains. mLOM's principal objective transfers
multi-ontology knowledge inside each model and across different
models. Base models might be any neural network able to perform
multi-domain multi-label learning, this means that output neurons
should represent label scores for the entire label set.
[0043] To extrapolate knowledge from inferences made inside a
domain to any other domain, a multi-ontology merging block is
employed. A multi-ontology merging block takes label scores from
different domains from a base model as input and outputs the same
labels after some transformation layers to optimize with respect to
expected output.
[0044] Particular architecture of the multi-ontology merging block
is may vary depending of the particular dataset to be analyzed.
[0045] To combine knowledge from different models and fine tune
single domain label estimates, all layers from different models for
each domain are merged. Then m+1 additional layers are added to a
fine tune layer stack. At each fine tune layer, a highway to each
model is merged in such a way that raw model output can be rescued
before additional transformations. This fine tune step obtains
consensus from the different base models considered.
[0046] As an additional input for the fine tune, the output from a
Viable Auto Encoder ("VAE") is used. Here, Viable Auto Encoder
refers to any distribution estimation technique. The skilled in the
art will recognize such distribution estimation techniques as any
auto regressive auto encoders, masked autoencoder distribution
estimators, variational autoencoders, generative adversarial
networks among others.
[0047] The VAE is trained with observed label sets for each
ontology. In this way, any given label set is more consistent with
observed ones. For example, if lettuce, tomato, onion and chocolate
are combined as a set, the chocolate might be removed from this
label set, as there are not examples of meals containing chocolate
and lettuce as ingredients.
[0048] An example mLOM architecture for two base models and two
ontologies are shown in FIG. 3.
[0049] The aforementioned mLOM architecture requires that an
initial target object is transformed with an appropriate input
encoder for each base model.
[0050] Such initial object can be any digital piece holding a meal
representation like text, audio or image file among others. An
input encoder is used to transform this raw initial object into an
adequate representation for each particular chosen base model.
[0051] Raw output from each base model among different domains is
taken as input for a Multi Ontology Merging Block (MOMB), which
comprises additional layers that can refine independent domain
predictions taking into account information from all domains being
considered. A fine tuning step consists of a stack of layers for
each domain. Corresponding domain outputs from different base
models are connected by gated layers, such gated layers are
dynamically adjusted to learn how much of previous layers
information should be remembered for upcoming layers. The skilled
on the art can recognize particular implementations of such
strategy as highways networks or even more restricted setups like
residual networks. mLOM has the ability to understand portion sizes
by restaurant depending on the price, type of restaurant, menu
name, etc. This will allow the ability to add specific nutrient
quantities that are required in some diets and nutrition plans.
Other ontologies and labels may be added to create better and more
accurate experiences for users.
[0052] Because of the multiple model nature of mLOM, new "points of
view" may be added to analyze food. Therefore, the ability to
recognize food plate images through a model, when combined with the
knowledge mLOM already has of food ontologies and food components,
allows the ability to predict the ingredients and nutrients within
images of food plates.
[0053] FIG. 5 is a diagrammatic representation of an example
machine in the form of a computer system 1, within which a set of
instructions for causing the machine to perform any one or more of
the methodologies discussed herein may be executed. In various
example embodiments, the machine operates as a standalone device or
may be connected (e.g., networked) to other machines. In a
networked deployment, the machine may operate in the capacity of a
server or a client machine in a server-client network environment,
or as a peer machine in a peer-to-peer (or distributed) network
environment. The machine may be a personal computer (PC), a tablet
PC, a set-top box (STB), a personal digital assistant (PDA), a
cellular telephone, a portable music player (e.g., a portable hard
drive audio device such as an Moving Picture Experts Group Audio
Layer 3 (MP3) player), a web appliance, a network router, switch or
bridge, or any machine capable of executing a set of instructions
(sequential or otherwise) that specify actions to be taken by that
machine. Further, while only a single machine is illustrated, the
term "machine" shall also be taken to include any collection of
machines that individually or jointly execute a set (or multiple
sets) of instructions to perform any one or more of the
methodologies discussed herein.
[0054] The example computer system 1 includes a processor or
multiple processor(s) 5 (e.g., a central processing unit (CPU), a
graphics processing unit (GPU), or both), and a main memory 10 and
static memory 15, which communicate with each other via a bus 20.
The computer system 1 may further include a video display 35 (e.g.,
a liquid crystal display (LCD)). The computer system 1 may also
include an alpha-numeric input device(s) 30 (e.g., a keyboard), a
cursor control device (e.g., a mouse), a voice recognition or
biometric verification unit (not shown), a drive unit 37 (also
referred to as disk drive unit), a signal generation device 40
(e.g., a speaker), and a network interface device 45. The computer
system 1 may further include a data encryption module (not shown)
to encrypt data.
[0055] The disk drive unit 37 includes a computer or
machine-readable medium 50 on which is stored one or more sets of
instructions and data structures (e.g., instructions 55) embodying
or utilizing any one or more of the methodologies or functions
described herein. The instructions 55 may also reside, completely
or at least partially, within the main memory 10 and/or within the
processor(s) 5 during execution thereof by the computer system 1.
The main memory 10 and the processor(s) 5 may also constitute
machine-readable media.
[0056] The instructions 55 may further be transmitted or received
over a network via the network interface device 45 utilizing any
one of a number of well-known transfer protocols (e.g., Hyper Text
Transfer Protocol (HTTP)). While the machine-readable medium 50 is
shown in an example embodiment to be a single medium, the term
"computer-readable medium" should be taken to include a single
medium or multiple media (e.g., a centralized or distributed
database and/or associated caches and servers) that store the one
or more sets of instructions. The term "computer-readable medium"
shall also be taken to include any medium that is capable of
storing, encoding, or carrying a set of instructions for execution
by the machine and that causes the machine to perform any one or
more of the methodologies of the present application, or that is
capable of storing, encoding, or carrying data structures utilized
by or associated with such a set of instructions. The term
"computer-readable medium" shall accordingly be taken to include,
but not be limited to, solid-state memories, optical and magnetic
media, and carrier wave signals. Such media may also include,
without limitation, hard disks, floppy disks, flash memory cards,
digital video disks, random access memory (RAM), read only memory
(ROM), and the like. The example embodiments described herein may
be implemented in an operating environment comprising software
installed on a computer, in hardware, or in a combination of
software and hardware.
[0057] One skilled in the art will recognize that the Internet
service may be configured to provide Internet access to one or more
computing devices that are coupled to the Internet service, and
that the computing devices may include one or more processors,
buses, memory devices, display devices, input/output devices, and
the like. Furthermore, those skilled in the art may appreciate that
the Internet service may be coupled to one or more databases,
repositories, servers, and the like, which may be utilized in order
to implement any of the embodiments of the disclosure as described
herein.
[0058] The corresponding structures, materials, acts, and
equivalents of all means or step plus function elements in the
claims below are intended to include any structure, material, or
act for performing the function in combination with other claimed
elements as specifically claimed. The description of the present
technology has been presented for purposes of illustration and
description, but is not intended to be exhaustive or limited to the
present technology in the form disclosed. Many modifications and
variations will be apparent to those of ordinary skill in the art
without departing from the scope and spirit of the present
technology. Exemplary embodiments were chosen and described in
order to best explain the principles of the present technology and
its practical application, and to enable others of ordinary skill
in the art to understand the present technology for various
embodiments with various modifications as are suited to the
particular use contemplated.
[0059] Aspects of the present technology are described above with
reference to flowchart illustrations and/or block diagrams of
methods, apparatus (systems) and computer program products
according to embodiments of the present technology. It will be
understood that each block of the flowchart illustrations and/or
block diagrams, and combinations of blocks in the flowchart
illustrations and/or block diagrams, can be implemented by computer
program instructions. These computer program instructions may be
provided to a processor of a general purpose computer, special
purpose computer, or other programmable data processing apparatus
to produce a machine, such that the instructions, which execute via
the processor of the computer or other programmable data processing
apparatus, create means for implementing the functions/acts
specified in the flowchart and/or block diagram block or
blocks.
[0060] These computer program instructions may also be stored in a
computer readable medium that can direct a computer, other
programmable data processing apparatus, or other devices to
function in a particular manner, such that the instructions stored
in the computer readable medium produce an article of manufacture
including instructions which implement the function/act specified
in the flowchart and/or block diagram block or blocks.
[0061] The computer program instructions may also be loaded onto a
computer, other programmable data processing apparatus, or other
devices to cause a series of operational steps to be performed on
the computer, other programmable apparatus or other devices to
produce a computer implemented process such that the instructions
which execute on the computer or other programmable apparatus
provide processes for implementing the functions/acts specified in
the flowchart and/or block diagram block or blocks.
[0062] While this technology is susceptible of embodiment in many
different forms, there is shown in the drawings and will herein be
described in detail several specific embodiments with the
understanding that the present disclosure is to be considered as an
exemplification of the principles of the technology and is not
intended to limit the technology to the embodiments
illustrated.
[0063] The terminology used herein is for the purpose of describing
particular embodiments only and is not intended to be limiting of
the technology. As used herein, the singular forms "a", "an" and
"the" are intended to include the plural forms as well, unless the
context clearly indicates otherwise. It will be further understood
that the terms "comprises" and/or "comprising," when used in this
specification, specify the presence of stated features, integers,
steps, operations, elements, and/or components, but do not preclude
the presence or addition of one or more other features, integers,
steps, operations, elements, components, and/or groups thereof.
[0064] It will be understood that like or analogous elements and/or
components, referred to herein, may be identified throughout the
drawings with like reference characters. It will be further
understood that several of the figures are merely schematic
representations of the present technology. As such, some of the
components may have been distorted from their actual scale for
pictorial clarity.
[0065] The flowchart and block diagrams in the Figures illustrate
the architecture, functionality, and operation of possible
implementations of systems, methods and computer program products
according to various embodiments of the present technology. In this
regard, each block in the flowchart or block diagrams may represent
a module, segment, or portion of code, which comprises one or more
executable instructions for implementing the specified logical
function(s). It should also be noted that, in some alternative
implementations, the functions noted in the block may occur out of
the order noted in the figures. For example, two blocks shown in
succession may, in fact, be executed substantially concurrently, or
the blocks may sometimes be executed in the reverse order,
depending upon the functionality involved. It will also be noted
that each block of the block diagrams and/or flowchart
illustration, and combinations of blocks in the block diagrams
and/or flowchart illustration, can be implemented by special
purpose hardware-based systems that perform the specified functions
or acts, or combinations of special purpose hardware and computer
instructions.
[0066] In the following description, for purposes of explanation
and not limitation, specific details are set forth, such as
particular embodiments, procedures, techniques, etc. in order to
provide a thorough understanding of the present invention. However,
it will be apparent to one skilled in the art that the present
invention may be practiced in other embodiments that depart from
these specific details.
[0067] Reference throughout this specification to "one embodiment"
or "an embodiment" means that a particular feature, structure, or
characteristic described in connection with the embodiment is
included in at least one embodiment of the present invention. Thus,
the appearances of the phrases "in one embodiment" or "in an
embodiment" or "according to one embodiment" (or other phrases
having similar import) at various places throughout this
specification are not necessarily all referring to the same
embodiment. Furthermore, the particular features, structures, or
characteristics may be combined in any suitable manner in one or
more embodiments. Furthermore, depending on the context of
discussion herein, a singular term may include its plural forms and
a plural term may include its singular form. Similarly, a
hyphenated term (e.g., "on-demand") may be occasionally
interchangeably used with its non-hyphenated version (e.g., "on
demand"), a capitalized entry (e.g., "Software") may be
interchangeably used with its non-capitalized version (e.g.,
"software"), a plural term may be indicated with or without an
apostrophe (e.g., PE's or PEs), and an italicized term (e.g.,
"N+1") may be interchangeably used with its non-italicized version
(e.g., "N+1"). Such occasional interchangeable uses shall not be
considered inconsistent with each other.
[0068] Also, some embodiments may be described in terms of "means
for" performing a task or set of tasks. It will be understood that
a "means for" may be expressed herein in terms of a structure, such
as a processor, a memory, an I/O device such as a camera, or
combinations thereof. Alternatively, the "means for" may include an
algorithm that is descriptive of a function or method step, while
in yet other embodiments the "means for" is expressed in terms of a
mathematical formula, prose, or as a flow chart or signal
diagram.
[0069] The terminology used herein is for the purpose of describing
particular embodiments only and is not intended to be limiting of
the invention. As used herein, the singular forms "a", "an" and
"the" are intended to include the plural forms as well, unless the
context clearly indicates otherwise. It will be further understood
that the terms "comprises" and/or "comprising," when used in this
specification, specify the presence of stated features, integers,
steps, operations, elements, and/or components, but do not preclude
the presence or addition of one or more other features, integers,
steps, operations, elements, components, and/or groups thereof.
[0070] It is noted at the outset that the terms "coupled,"
"connected", "connecting," "electrically connected," etc., are used
interchangeably herein to generally refer to the condition of being
electrically/electronically connected. Similarly, a first entity is
considered to be in "communication" with a second entity (or
entities) when the first entity electrically sends and/or receives
(whether through wireline or wireless means) information signals
(whether containing data information or non-data/control
information) to the second entity regardless of the type (analog or
digital) of those signals. It is further noted that various figures
(including component diagrams) shown and discussed herein are for
illustrative purpose only, and are not drawn to scale.
[0071] While specific embodiments of, and examples for, the system
are described above for illustrative purposes, various equivalent
modifications are possible within the scope of the system, as those
skilled in the relevant art will recognize. For example, while
processes or steps are presented in a given order, alternative
embodiments may perform routines having steps in a different order,
and some processes or steps may be deleted, moved, added,
subdivided, combined, and/or modified to provide alternative or
sub-combinations. Each of these processes or steps may be
implemented in a variety of different ways. Also, while processes
or steps are at times shown as being performed in series, these
processes or steps may instead be performed in parallel, or may be
performed at different times.
[0072] While various embodiments have been described above, it
should be understood that they have been presented by way of
example only, and not limitation. The descriptions are not intended
to limit the scope of the invention to the particular forms set
forth herein. To the contrary, the present descriptions are
intended to cover such alternatives, modifications, and equivalents
as may be included within the spirit and scope of the invention as
defined by the appended claims and otherwise appreciated by one of
ordinary skill in the art. Thus, the breadth and scope of a
preferred embodiment should not be limited by any of the
above-described exemplary embodiments.
* * * * *