U.S. patent application number 14/480670 was filed with the patent office on 2014-12-25 for system and method for a human machine interface.
The applicant listed for this patent is Ford Global Technologies, LLC. Invention is credited to Pietro Buttolo, Yifan Chen, Dimitar Petrov Filev, Jeffrey Allen Greenberg, Krishnaswamy Venkatesh Prasad, James Stewart Rankin, Gary Steven Strumolo, Basavaraj Tonshal.
Application Number | 20140380240 14/480670 |
Document ID | / |
Family ID | 51226433 |
Filed Date | 2014-12-25 |
United States Patent
Application |
20140380240 |
Kind Code |
A1 |
Rankin; James Stewart ; et
al. |
December 25, 2014 |
System and Method for a Human Machine Interface
Abstract
A vehicle computer system comprising a wireless transceiver
configured to send a nomadic device human machine interface to a
nomadic device in a web browser format. The vehicle computer system
further comprises a vehicle server utilizing a contextual data
aggregator that utilizes vehicle data and off-board data to
generate a dynamic human machine interface, the server further
configured to generate an in-vehicle human machine interface for
output on a vehicle display and generate the nomadic device human
machine interface for the nomadic device to display.
Inventors: |
Rankin; James Stewart;
(Novi, MI) ; Tonshal; Basavaraj; (Northville,
MI) ; Buttolo; Pietro; (Dearborn Heights, MI)
; Chen; Yifan; (Ann Arbor, MI) ; Strumolo; Gary
Steven; (Canton, MI) ; Greenberg; Jeffrey Allen;
(Ann Arbor, MI) ; Filev; Dimitar Petrov; (Novi,
MI) ; Prasad; Krishnaswamy Venkatesh; (Ann Arbor,
MI) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Ford Global Technologies, LLC |
Dearborn |
MI |
US |
|
|
Family ID: |
51226433 |
Appl. No.: |
14/480670 |
Filed: |
September 9, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13767303 |
Feb 14, 2013 |
8866604 |
|
|
14480670 |
|
|
|
|
Current U.S.
Class: |
715/810 |
Current CPC
Class: |
B60K 2370/589 20190501;
G06F 3/04842 20130101; B60K 2370/573 20190501; B60K 2370/5899
20190501; B60K 37/06 20130101; B60K 2370/566 20190501; B60K
2370/592 20190501; B60K 35/00 20130101; B60K 2370/569 20190501;
G06F 3/048 20130101; G06F 3/0482 20130101; B60K 2370/563
20190501 |
Class at
Publication: |
715/810 |
International
Class: |
G06F 3/0482 20060101
G06F003/0482; G06F 3/0484 20060101 G06F003/0484 |
Claims
1-20. (canceled)
21. A vehicle computer system comprising: a wireless transceiver
configured to establish a connection with a nomadic device, wherein
the wireless transceiver is further configured to send a nomadic
device human machine interface (HMI) utilizing a web browser
format; a vehicle display configured to output an in-vehicle HMI
for controlling various vehicle functions, wherein the in-vehicle
HMI utilizing the web browser format; and a vehicle server
configured to generate the in-vehicle HMI for the vehicle display
and generate the nomadic device HMI for the nomadic device.
22. The vehicle computer system of claim 21, wherein the wireless
transceiver is further configured to communicate with an off-board
server, wherein the off-board server is configured to utilize or
provide the vehicle with off-board data for generating the nomadic
device HMI or the in-vehicle HMI.
23. The vehicle computer system of claim 22, wherein the wireless
transceiver is further configured to communicate with the off-board
server via a mobile device.
24. The vehicle computer system of claim 23, wherein the mobile
device is the nomadic device.
25. The vehicle computer system of claim 22, wherein the wireless
transceiver is further configured to communicate with the off-board
server via an embedded cellular connection.
26. The vehicle computer system of claim 21, wherein the in-vehicle
HMI includes a different configuration than the nomadic device
HMI.
27. The vehicle computer system of claim 21, wherein the web
browser format includes HTML5.
28. The vehicle computer system of claim 21, wherein the in-vehicle
HMI is based on a driver of the vehicle.
29. The vehicle computer system of claim 21, wherein the web
browser format is configured to be chrome-less.
30. An nomadic device comprising: a wireless transceiver
communicating with a vehicle; and a processor configured to:
receive data indicative of a human machine interface (HMI) via the
wireless transceiver; and output on a display of the nomadic device
a contextual HMI in a web browser format, wherein the contextual
HMI is configured to control vehicle functions.
31. The nomadic device of claim 30, wherein the wireless
transceiver is further capable of communicating with a server to
receive off-board data and send the off-board data to the
vehicle.
32. The nomadic device of claim 30, wherein the nomadic device
further comprises a vehicle HMI application to facilitate the
output of the contextual HMI utilizing the vehicle data.
33. The nomadic device of claim 32, wherein vehicle HMI application
is configured to generate a web page to display the contextual
HMI.
34. The nomadic device of claim 32, wherein the processor is
further configured to receive the contextual HMI from a vehicle
server.
35. The nomadic device of claim 32, wherein the off-board data
includes media data received from a media server.
36. The nomadic device of claim 32, wherein the wireless
transceiver is further configured to communicate with a query based
system that utilizes spoken dialogue.
37. The nomadic device of claim 30, wherein the contextual HMI
utilizes vehicle data and off-board data.
38. A vehicle computer system comprising: a vehicle server
configured to generate an in-vehicle human machine interface (HMI)
for output on a vehicle display and generate a nomadic device (ND)
HMI in a web browser format for a ND to display; and a wireless
transceiver configured to send the ND HMI to a ND in a web browser
format.
39. The vehicle computer system of claim 38, wherein the vehicle
server includes a data manager in communication with vehicle
modules to retrieve controller area network data from a vehicle
bus.
40. The vehicle computer system of claim 38, wherein the vehicle
server is further configured to generate the in-vehicle HMI in a
web browser format.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. application Ser.
No. 13/767,303 filed Feb. 14, 2013 the disclosure of which is
hereby incorporated in its entirety by reference herein.
TECHNICAL FIELD
[0002] The illustrative embodiments generally relate to an
apparatus and method for generating a user interface in a vehicle
multimedia system.
BACKGROUND
[0003] Below, a few prior art entertainment systems are described.
Additional device usage of these systems may be obtained and
described from the identified references below.
[0004] United States Patent Publication 2011/0234427 describes a
telemetric device for a vehicle that includes a location
determining device arranged to determine the vehicle's location and
a transmitting device arranged to continuously or nearly
continuously transmit location data concerning the vehicle in
real-time or near real-time. The transmission of the location data
is not based on a condition of the vehicle and is without prompting
by a user.
[0005] U.S. Pat. No. 8,204,734 describes an exemplary system that
includes a development subsystem configured to facilitate
development of a software application and a simulation subsystem
selectively and communicatively coupled to the development
subsystem. The simulation subsystem is configured to emulate a
plurality of processing device platforms, receive data
representative of a selection of at least one of the plurality of
processing device platforms, and simulate an execution of the
software application by one or more processing devices associated
with the at least one selected processing device platform.
[0006] United States Patent Publication 2012/0179325 describes an
information system located within a vehicle that includes a data
storage device configured to store programmed instructions to
implement a web browser, a data communication module, and a
controller operatively coupled to the data storage device, and the
data communication module, the controller configured to execute the
web browser to receive a plurality of data elements, identify a
respective content type for each of the plurality of data elements,
assign a relevance level to each of the plurality of data elements
based on the respective identified content type, compare the
assigned relevance to a predetermined relevance threshold; and
generate a user interface using at least one of the plurality of
data elements based upon the comparison.
SUMMARY
[0007] A first illustrative embodiment discloses a vehicle computer
system comprising a wireless transceiver configured to establish a
connection with a nomadic device, wherein the wireless transceiver
is further configured to send a nomadic device human machine
interface configured for output on the nomadic device using a first
web browser format. The vehicle computer system also includes a
vehicle display configured to output an in-vehicle human machine
interface using a second web browser format for controlling various
vehicle functions and a vehicle server configured to generate the
in-vehicle human machine interface for the vehicle display and
generate the nomadic device human machine interface for the nomadic
device.
[0008] A second illustrative embodiment discloses an off-board
computer system comprising a wireless transceiver communicating
with a vehicle and a server that includes a contextual data
aggregator which utilizes vehicle data retrieved from the wireless
transceiver and off-board data to generate a dynamic contextual
human machine interface based on the vehicle data and off-board
data in a web browser data format to be sent to the vehicle
utilizing the wireless transceiver.
[0009] A third illustrative embodiment discloses a vehicle computer
system comprising a wireless transceiver configured to send a
nomadic device human machine interface to a nomadic device in a web
browser format. The vehicle computer system further comprises a
vehicle server utilizing a contextual data aggregator that utilizes
vehicle data and off-board data to generate a dynamic human machine
interface, the server further configured to generate an in-vehicle
human machine interface for output on a vehicle display and
generate the nomadic device human machine interface for the nomadic
device to display.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] FIG. 1 shows an example block topology for a vehicle based
computing system for a vehicle.
[0011] FIG. 2 shows an example topology of the vehicle based
computing system communicating with a cloud based server configured
with contextual data aggregation.
[0012] FIG. 3 is an example of the vehicle cloud human machine
interface utilizing the context detector.
[0013] FIG. 4 is an example of a flow chart indicating the
interaction of the vehicle based computing system.
[0014] FIG. 5A is an illustrative embodiment of a network
architecture utilized to demonstrate a remote control employed to
control a seat using a mobile application.
[0015] FIG. 5B is an illustrative embodiment of a network
architecture utilizing various displays in the vehicle.
[0016] FIG. 6 is an illustrative embodiment of a human machine
interface displayed on different types of nomadic devices.
[0017] FIG. 7 illustrates an example of a query based HMI that may
utilize a spoken dialogue system.
DETAILED DESCRIPTION
[0018] As required, detailed embodiments of the present invention
are disclosed herein; however, it is to be understood that the
disclosed embodiments are merely exemplary of the invention that
may be embodied in various and alternative forms. The figures are
not necessarily to scale; some features may be exaggerated or
minimized to show details of particular components. Therefore,
specific structural and functional details disclosed herein are not
to be interpreted as limiting, but merely as a representative basis
for teaching one skilled in the art to variously employ the present
invention.
[0019] The invention now will be described more fully hereinafter
with reference to the accompanying drawings, in which embodiments
of the invention are shown. This invention, may however, be
embodied in many different forms and should not be construed as
limited to the embodiments set forth herein. Like numbers refer to
elements throughout. As used herein the term "and/or" includes any
and all combinations of one or more of the associated listed
items.
[0020] Many new features and functionality are packed into our
phones, tablets, MP3 players, etc. However, many of these features
are not developed with the vehicle environment in mind. Increasing
interoperability of mobile devices with a vehicle's computing
system allows customers to have a seamless experience whether or
not they are in a vehicle environment. One illustrative example of
a seamless customer experience is allowing a user to operate a user
interface related to the vehicle on a user's nomadic device. This
may allow a user to control a vehicle computer system from multiple
devices and in different locations, both within the vehicle and
outside of the vehicle.
[0021] FIG. 1 illustrates an example block topology for a vehicle
based computing system 1 (VCS) for a vehicle 31. An example of such
a vehicle-based computing system 1 is the SYNC system manufactured
by FORD MOTOR COMPANY. A vehicle enabled with a vehicle-based
computing system may contain a visual front end interface 4 located
in the vehicle. The user may also be able to interact with the
interface if it is provided, for example, with a touch sensitive
resistive/capacitive buttons that may also be utilized on a display
screen. In another illustrative embodiment, the interaction occurs
through, button presses, spoken dialog system with automatic speech
recognition and speech synthesis.
[0022] In the illustrative embodiment 1 shown in FIG. 1, a
processor 3 controls at least some portion of the operation of the
vehicle-based computing system. Provided within the vehicle, the
processor allows onboard processing of commands and routines.
Further, the processor is connected to both non-persistent 5 and
persistent storage 7. In this illustrative embodiment, the non-
persistent storage is random access memory (RAM) and the persistent
storage is a hard disk drive (HDD) or flash memory.
[0023] The processor is also provided with a number of different
inputs allowing the user to interface with the processor. In this
illustrative embodiment, a microphone 29, an auxiliary input 25
(for input 33), a USB input 23, a GPS input 24 and a BLUETOOTH
input 15 are all provided. An input selector 51 is also provided,
to allow a user to select between various inputs. Input to both the
microphone and the auxiliary connector is converted from analog to
digital by a converter 27 before being passed to the processor.
Although not shown, these and other components may be in
communication with the VCS over a vehicle multiplex network (such
as, but not limited to, a CAN bus) to pass data to and from the VCS
(or components thereof).
[0024] Outputs to the system can include, but are not limited to, a
visual display 4 and a speaker 13 or stereo system output. The
speaker is connected to an amplifier 11 and receives its signal
from the processor 3 through a digital-to-analog converter 9.
Output can also be made to a remote BLUETOOTH device such as PND 54
or a USB device such as vehicle navigation device 60 along the
bi-directional data streams shown at 19 and 21 respectively.
[0025] In one illustrative embodiment, the system 1 uses the
BLUETOOTH transceiver 15 to communicate 17 with a user's nomadic
device 53 (e.g., cell phone, smart phone, PDA, or any other device
having wireless remote network connectivity). The nomadic device
can then be used to communicate 59 with a network 61 outside the
vehicle 31 through, for example, communication 55 with a cellular
tower 57. In some embodiments, tower 57 may be a Wi-Fi access
point.
[0026] Exemplary communication between the nomadic device and the
BLUETOOTH transceiver is represented by signal 14.
[0027] Pairing a nomadic device 53 and the BLUETOOTH transceiver 15
can be instructed through a button 52 or similar input.
Accordingly, the CPU is instructed that the onboard BLUETOOTH
transceiver will be paired with a BLUETOOTH transceiver in a
nomadic device.
[0028] Data may be communicated between CPU 3 and network 61
utilizing, for example, a data-plan, data over voice, or DTMF tones
associated with nomadic device 53. Alternatively, it may be
desirable to include an onboard modem 63 having antenna 18 in order
to communicate 16 data between CPU 3 and network 61 over the voice
band. The nomadic device 53 can then be used to communicate 59 with
a network 61 outside the vehicle 31 through, for example,
communication 55 with a cellular tower 57. In some embodiments, the
modem 63 may establish communication 20 with the tower 57 for
communicating with network 61. As a non-limiting example, modem 63
may be a USB cellular modem and communication 20 may be cellular
communication.
[0029] In one illustrative embodiment, the processor is provided
with an operating system including an API to communicate with modem
application software. The modem application software may access an
embedded module or firmware on the BLUETOOTH transceiver to
complete wireless communication with a remote BLUETOOTH transceiver
(such as that found in a nomadic device). Bluetooth is a subset of
the IEEE 802 PAN (personal area network) protocols. IEEE 802 LAN
(local area network) protocols include Wi-Fi and have considerable
cross-functionality with IEEE 802 PAN. Both are suitable for
wireless communication within a vehicle. Another communication
means that can be used in this realm is free-space optical
communication (such as IrDA) and non-standardized consumer IR
protocols.
[0030] In another embodiment, nomadic device 53 includes a modem
for voice band or broadband data communication. In the
data-over-voice embodiment, a technique known as frequency division
multiplexing may be implemented when the owner of the nomadic
device can talk over the device while data is being transferred. At
other times, when the owner is not using the device, the data
transfer can use the whole bandwidth (300 Hz to 3.4 kHz in one
example). While frequency division multiplexing may be common for
analog cellular communication between the vehicle and the internet,
and is still used, it has been largely replaced by hybrids of Code
Domain Multiple Access (CDMA), Time Domain Multiple Access (TDMA),
Space-Domain Multiple Access (SDMA) for digital cellular
communication. These are all ITU IMT-2000 (3G) compliant standards
and offer data rates up to 2 mbs for stationary or walking users
and 385 kbs for users in a moving vehicle. 3G standards are now
being replaced by IMT-Advanced (4G) which offers 100 mbs for users
in a vehicle and 1 gbs for stationary users. If the user has a
data-plan associated with the nomadic device, it is possible that
the data- plan allows for broad-band transmission and the system
could use a much wider bandwidth (speeding up data transfer). In
still another embodiment, nomadic device 53 is replaced with a
cellular communication device (not shown) that is installed to
vehicle 31. In yet another embodiment, the ND 53 may be a wireless
local area network (LAN) device capable of communication over, for
example (and without limitation), an 802.11g network (i.e., Wi-Fi)
or a WiMax network.
[0031] In one embodiment, incoming data can be passed through the
nomadic device via a data-over-voice or data-plan, through the
onboard BLUETOOTH transceiver and into the vehicle's internal
processor 3. In the case of certain temporary data, for example,
the data can be stored on the HDD or other storage media 7 until
such time as the data is no longer needed.
[0032] Additional sources that may interface with the vehicle
include a personal navigation device 54, having, for example, a USB
connection 56 and/or an antenna 58, a vehicle navigation device 60
having a USB 62 or other connection, an onboard GPS device 24, or
remote navigation system (not shown) having connectivity to network
61. USB is one of a class of serial networking protocols. IEEE 1394
(FireWire.TM. (Apple), i.LINK.TM. (Sony), and Lynx.TM. (Texas
Instruments)), EIA (Electronics Industry Association) serial
protocols, IEEE 1284 (Centronics Port), S/PDIF (Sony/Philips
Digital Interconnect Format) and USB-IF (USB Implementers Forum)
form the backbone of the device-device serial standards. Most of
the protocols can be implemented for either electrical or optical
communication.
[0033] Further, the CPU could be in communication with a variety of
other auxiliary devices 65. These devices can be connected through
a wireless 67 or wired 69 connection. Auxiliary device 65 may
include, but are not limited to, personal media players, wireless
health devices, portable computers, nomadic device, key fob and the
like.
[0034] Also, or alternatively, the CPU could be connected to a
vehicle based wireless router 73, using for example a Wi-Fi (IEEE
803.11) 71 transceiver. This could allow the CPU to connect to
remote networks in range of the local router 73.
[0035] In addition to having exemplary processes executed by a
vehicle computing system located in a vehicle, in certain
embodiments, the exemplary processes may be executed by a computing
system in communication with a vehicle computing system. Such a
system may include, but is not limited to, a wireless device (e.g.,
and without limitation, a mobile phone) or a remote computing
system (e.g., and without limitation, a server) connected through
the wireless device. Collectively, such systems may be referred to
as vehicle associated computing systems (VACS). In certain
embodiments particular components of the VACS may perform
particular portions of a process depending on the particular
implementation of the system. By way of example and not limitation,
if a process has a step of sending or receiving information with a
paired wireless device, then it is likely that the wireless device
is not performing the process, since the wireless device would not
"send and receive" information with itself. One of ordinary skill
in the art will understand when it is inappropriate to apply a
particular VACS to a given solution. In all solutions, it is
contemplated that at least the vehicle computing system (VCS)
located within the vehicle itself is capable of performing the
exemplary processes.
[0036] FIG. 2 shows an example topology of the vehicle
communicating with the cloud to communicate different data from
various sources. The vehicle server in the cloud may allow for the
management and utilization of the data. The server may also be
located outside of the vehicle, or in the cloud. The server may be
located in the vehicle and may utilize hardware and/or software to
deliver web content to be accessed through the internet or via a
TCP/IP connection. The server may deliver requests to various
clients using HTTP or other means of delivering HTML documents and
additional content such as images, scripts, etc. The server's data
may be updated and aggregated. The data may come from a large
community in the cloud. The server may also provide data synthesis,
such as utilizing contextual information for extraction.
Additionally, the vehicle server in the cloud may allow for
intelligence and decision making by utilizing an agent in the
cloud. The vehicle server may be aware of a destination point, thus
the server may utilize the agent to proactively engage the driver
in delivering weather at a user's set destination point.
[0037] Real-time, by-directional data streaming between a vehicle
and the vehicle server, and between a mobile device and the vehicle
server, may be accomplished through the illustrative embodiment
disclosed. Relevant vehicle CAN and vehicle to smartphone (V2SP)
interaction data may be uploaded to the server. The data may be
utilized to create a contextual HMI that can be generated in real
time at the server, using the contextual data aggregator. The
contextual HMI may be served to one or more clients, such as a
vehicle, a device in the vehicle, or another module/display.
Websockets may be utilized to provide the communication channel.
The websocket specification, developed as part of the HTML5
initiative, introduced the WebSocket JavaScript interface. The
websocket interface allows a full-duplex single socket connection
in which messages can be sent between a client and server. The
websocket standard simplifies much of the complexity around
bi-directional web communication and connection management. The use
of web sockets makes it possible for real time updating of the
context-based HMI for different user scenarios. One example of a
particular user scenario may include displaying climate for various
situations on the vehicle display interface. For example, the car
area network (CAN) may have access to climate information that is
available via a climate data source that is off-board from the
vehicle and the vehicle server. Thus, the vehicle computer system
may be configured to display the destination climate by utilizing
the climate data from the off-board server, various data from the
vehicle, and the contextual data aggregator. Furthermore, it may be
capable of providing weather information at the driver's residence
or a way-point. The climate data may be fetched from a vehicle
cloud server via a persistent websocket connection with the vehicle
cloud server at the time of visual interface rendering. This may
allow for different HMI interfaces to be utilized on one's mobile
nomadic device versus an on-board vehicle system, such as MYFORD
TOUCH. Furthermore, it enables for direct interaction between
devices and a server. The user may be allowed to interact directly
with either the mobile device or the relayed-HMI on the vehicle's
on-board system.
[0038] The vehicle based computing system 201 of FIG. 2 may be
capable of communicating with a vehicle cloud server 203. The
vehicle based computing system may connect to the vehicle cloud
server 203 via a Bluetooth cell phone paired with the vehicle
system, an embedded cellular connection, and/or both a
long-range/short-range wireless connection. The vehicle cloud
server 203 may also be accessible by other nomadic devices or
computer systems. Additionally, accessibility may require various
security restrictions in order to allow for access to the vehicle
cloud based server. The vehicle may send different types of
relevant data to the server utilizing web sockets 217. This allows
for full-duplex communication channels via a single TCP connection.
Thus a web browser or web server of a client or server application
may utilize the websocket API to facilitate live real-time updating
of content. Although the websocket API and protocol are utilized in
this example, other similar alternative embodiments may be
utilized.
[0039] The vehicle cloud server 205 may store a variety of vehicle
and user related data on a database 205 of the server. In the
illustrative embodiment disclosed, the database 205 may include
vehicle CAN data 207, smart phone vehicle data 209, and vehicle HMI
data 211. The vehicle CAN data 207 may be utilized for utilizing
data associated with the vehicle system and performance. Thus it
could provide driver history, vehicle diagnostic, and maintenance
data. The relevant vehicle CAN data may be uploaded into the
server. Furthermore, data related to the vehicle and smart phone
interaction 209 may also be stored on the same server. The smart
phone vehicle data 209 may be utilized for identifying contacts,
the user device, etc. The vehicle computer system may interface
with more than one device brought into the vehicle. Thus the
vehicle server may help prepare the device to work with the vehicle
system. The vehicle HMI data 211 may be utilized for storing user
settings, providing voice recognition data, and data related to
interaction carried while driving, such as but not limited to
advertisement bookmarks, location bookmarks, and entertainment
preferences. Although the illustrative embodiments shows the
database including vehicle CAN data, smart phone vehicle data, and
vehicle HMI data, other data may be stored. For example, the
database 205 may store the vehicle manufacturer data and more.
[0040] Content may also be provided by the external cloud servers
219. Some examples of off-board data that the external cloud
servers may have access to is traffic data,
navigation/direction/routing data, music content,
dealership/vehicle maintenance data, weather content, social
interaction data gathered from social networks like Google Plus,
Facebook, Path, Twitter, etc. The external cloud server may provide
additional data to help enhance the contextual HMI generation. The
external cloud server may help provide location specific
information, such as Point of Interest (POI) information (hours,
location, menu, ratings, etc.), traffic, weather, etc.
[0041] The vehicle cloud server 203 may utilize different computer
processing to provide contextual data aggregation 213. The vehicle
may leverage the additional resources that the vehicle cloud server
203 provides in order to deliver enhanced user experience for the
driver. For example, in one illustrative embodiment, the vehicle
server may utilize the user's cell phone number, make, and model to
identify the capability of the device and prepare the HMI for the
device. The vehicle based computing system may obtain this data
from the user's cell phone once the Bluetooth phone has been paired
with the vehicle. Furthermore, by utilizing additional data, such
as the vehicle CAN data and the vehicle HMI data, the server may
generate a unique HMI for the current user's device. The HMI
generation 215 may be accomplished through algorithms and formulas
utilized by the contextual data aggregator 213. Both a unique
visual HMI and spoken dialogue system may be generated on the cloud
server to be presented to the user. Numerous combinations of
different data may be utilized to provide the output of the
contextual HMI. The different combinations of the data allows for
updates of applications to be available. Additionally, it allows
for very fast integration of new applications and use cases in the
vehicle based computing system. The HMI data may be generated in
real time on the server based on the contextual data aggregator.
The HMI data may also be served to the client, which in one
embodiment, may be the vehicle. In an alternative embodiment, the
client maybe a nomadic device that is either located within the
vehicle or located remote from the vehicle. Once generated by the
contextual aggregator of the server, the server may send the
necessary data to the vehicle via web sockets 217. The web sockets
help facilitate the real time updating of a context-based HMI
data.
[0042] FIG. 3 illustrates another illustrative embodiment of the
vehicle server utilizing an entirely off-board solution. The
vehicle's cloud HMI may utilize different web standards to create a
vehicle server. The vehicle server 401 may be located at the
vehicle, off-board, or utilize a hybrid approach that has certain
software features located at the vehicle and others located
off-board. An off-board or cloud based solution may allow for
servers with high processing power to aggregate and transform data
in an efficient manner, however, connections off-board may not
always be available in certain circumstances (i.e. no signal or
service for a long-range connection). Thus an on-board solution may
also be utilized for the vehicle server. The on-board vehicle
server may also include a context detector 403 and software to
generate context aware visual and audio metadata 405. The vehicle
server may not only utilize vehicle sensor data, but also utilize
other relevant information from off-board servers and other devices
or sensors not embedded in the vehicle.
[0043] The vehicle server 401 may also utilize the context detector
403 to understand the environment in which the vehicle or other
client is operating in. Furthermore, the context detector may
determine the environment in view of the different clients or
devices which are in communication with the vehicle server. The
vehicle cloud server may be in communication with an embedded
vehicle server or with a nomadic device, thus the context detector
has to determine the context for suitable HMI rendering. The
context detection may be done entirely in the cloud based on
vehicle sensor data and other relevant data. For example, if a
driver is approaching home and the vehicle server knows this
information and generates an HMI, it may send it to the vehicle.
The visual HMI display may have a screen for operation of the
home's garage door. Although context detection exists in the cloud
for the present embodiment, both on-board vehicle solutions and
hybrid solutions are alternative embodiments.
[0044] The vehicle server 401 may operate similar to a web server
with web socket support. The vehicle HMI renderer may simply be a
modern browser that, for example, supports HTML 5 and web sockets.
The vehicle may send data located within the vehicle data bus or on
vehicle sensors, such as vehicle speed, location, etc, to the
vehicle server. The vehicle cloud server may use this data for
context detection and drive pattern recognition. The vehicle
embedded browser may utilize the data to render the HMI for the
user. In an alternative embodiment, a graphics rendering
application may be utilized to render the HMI, rather than a
browser. The application may be utilized to mitigate or eliminate
rendering of a page on a browser.
[0045] The context detector 403 may be in communication with a home
server 411. The home server may provide data located in a user's
residence. Such data that may be utilized are home automation data,
e.g. home HVAC, security, lighting and lighting status, gas status,
etc. Furthermore, the context detector 403 may be in communication
with an external media server 412. The external media server may
utilize data such as photos, music, and video files.
[0046] Upon the context detector 403 receiving data from multiple
servers and sources, the vehicle server may generate context aware
visual and audio metadata 405. The metadata may then be sent to the
client 407, which may be the vehicle, nomadic device, or browser on
a personal computing device. The vehicle HMI rendering engine 409
may be used to generate a specific type of HMI based on the data,
the client, and other factors. The HMI rendering engine may be
utilized in conjunction with a web browser or an application.
[0047] In another embodiment, the hybrid approach may be utilized
to render the HMI utilizing the vehicle embedded server. This may
solve a potential problem if the vehicle is unable to maintain a
connection with the vehicle cloud server. The vehicle HMI rendering
engine maybe configured to work both with and without the vehicle
cloud server, as to render the HMI when no connection to the cloud
is present. This hybrid architecture may be ideally suitable for
vehicle HMI rendering when a connection to the server is not always
available.
[0048] FIG. 4 is an example of a flow chart indicating the
interaction of the vehicle based computing system. The vehicle
based computing system may be equipped with either an on-board,
off-board, or hybrid (combination of on-board and off-board)
server. Thus any combination or all of the steps may be performed
either within the vehicle, outside of the vehicle in a cloud, or a
hybrid combination that utilizes processing of both on-board and
off-board solutions. The server may retrieve vehicle data 501 from
various vehicle modules using a wired connection (e.g. CAN, USB,
Serial, etc.) or a wireless connection (e.g. Bluetooth, Wi-Fi,
Wi-Fi direct, cellular connection, etc.). In addition, the vehicle
data may include media data from a user's media collection,
navigation data (e.g. POI information, speed limit info, and other
mapping data), and cellular phone data. The vehicle data may be
utilized to output specific information on an interface (e.g. speed
of the vehicle or location of the vehicle), or it could be used to
generate a specific feature or interface for the HMI.
[0049] The vehicle computer system may be configured to determine
if a connection is present with an off-board cloud server or data
server 502. If the connection is present, the computer system may
receive off-board data 503 from various data sources that are not
located on or within the vehicle. These data sources may include
weather, traffic, sports scores, off-board navigation data, etc.
Similar to the vehicle data, the off-board data may be utilized to
output specific information on the user interface (e.g. a weather
map) or it could be used to generate a specific feature or
interface for the HMI. The server may generate the interface to be
used with a browser application.
[0050] Upon receiving the off-board data or determining that no
connection is present, the vehicle computer system may utilize a
contextual data aggregator. In some embodiments, the vehicle
computer system may be in communication with a context aggregator
that generates dynamic features and context based on the vehicle
data and/or the off-board data. The context aggregator may analyze
the data 505 to determine if any dynamic features may be utilized
by the interface. If the context aggregator does not recognize any
features that may be dynamic, it may use the static HMI features of
the vehicle computer system. In an alternate scenario, the context
aggregator may recognize that either the vehicle data or off-board
data can include the possibility of the vehicle computer system to
generate a dynamic feature or HMI 507. In one scenario, the context
aggregator may receive data indicating that the vehicle is in
motion and is located on a road with traffic. Thus, the aggregator
may utilize the data to display on the user interface a screen
asking a user to reroute around the traffic. In another scenario,
the context aggregator may recognize the mobile phone as a
secondary driver and display a user interface tailored to that
driver's need. For example, a certain skin or graphic may be used
and presets may be stored based on the driver. If no contextual
use-case or data is present, the system may simply generate a
static HMI to be rendered on a vehicle system. Additionally, the
HMI may be customized for features that the particular user uses
most frequently. The contextual aggregator may be located off-board
from the vehicle, on-board, or utilize a hybrid solution.
[0051] The vehicle computing system may determine whether a nomadic
device that is connected to the vehicle computing system is
utilizing an application 509. Utilization of an application by a
nomadic device may eliminate the need for the server to generate
the web page. Instead, if the nomadic device is determined to be
using a use interface application for the vehicle, the server may
simply send the relevant data for the nomadic device's HMI 511. The
data may be in the form of any data type, such as a JSON
(JavaScript Object Notation). Thus, the page will be pre-rendered
prior to the relevant vehicle data being sent from the nomadic
device. The nomadic device will utilize any relevant data to output
the necessary information to be displayed on the user interface.
Additionally, the application may be tailored for a specific
user-case scenario of driver or passenger, such as an application
to adjust the seating controls of the vehicle.
[0052] If the vehicle computing system determines that the device
is not using an application for the user interface, the device may
request the vehicle embedded server for the rendered HMI that may
be in the form of a web page. The vehicle computer system may
render the web page of the HMI 513 utilizing an on-board,
off-board, or hybrid solution. Upon rendering the HMI, the vehicle
computer system may send the HMI to the nomadic device 515. The
device may use a web browser conforming to the protocols of the
server in order to request or load the HMI, which may be in the
form of a web page. The web page may be chrome-less, meaning that a
browser chrome does not have to be shown. The browser may utilize a
rendering engine, Java Script, and browser plug-ins to enhance the
user experience. Furthermore, any of the data that is used by the
vehicle computer system may be utilized to generate the HMI.
[0053] FIG. 5A is an illustrative embodiment of a network
architecture utilized to demonstrate a smart phone employed to
control a seat. A nomadic device 601 may be brought into the
vehicle. The nomadic device may or may not have a vehicle interface
application 603 running in order to utilize a human machine
interface (HMI) for various vehicle controls. The smart phone may
allow the browser to request vehicle HMI web pages from the vehicle
embedded server. Further, the nomadic device may be utilized to
control various parts, components, or modules of the radio. In the
embodiment described below, the application is utilized to operate
the functionality of a seat. However, any and all modules,
components, or computers of the vehicle may be utilized. Some
examples include, but are not limited to, the radio, navigation,
system, sun roof, moon roof, trunk, lights, etc. This embodiment
demonstrates a wireless transceiver 605 communicating with the
nomadic device. Various wireless transceivers may include a Wi-Fi
transceiver, Bluetooth transceiver, near field communication (NFC),
infrared, etc. Although a wireless transceiver is shown in the
embodiment for the vehicle to communicate with the nomadic device,
a wired solution (e.g. USB, Serial, CAN, Firewire, etc.) may be
possible. Data communication between the nomadic device 601 and the
wireless transceiver 605 may be established when security has been
authorized. Devices may utilize a pairing sequence of generating a
random number and matching with the vehicle computer system, or
vice versa. Additionally, the vehicle or device may require a
button to be pressed at a certain moment for a certain amount of
time in order to establish a connection. Additionally, NFC may
utilize RFD tags to establish communication.
[0054] The wireless transceiver may be in direct communication with
the vehicle network 606. A vehicle computer may include a vehicle
embedded server 611 and a CAN/USB manager to communicate data
throughout the vehicle network infrastructure 602. Although the
vehicle embedded server 611 may include a CAN Manager and a USB
Seat Controller, they each may be separate individual modules in
other embodiments. The infrastructure 602 may utilize wireless
signals to communicate, or it may utilize a wired network 606, such
as a CAN, USB, DVI, etc. The vehicle embedded server 611 may be in
communication with one or more USB or CAN transceivers 609 to
communicate with other modules. In turn, the transceiver 609 may be
in communication with a seat & position control module 607. The
seat & position control module may communicate data back and
forth to the vehicle network infrastructure 602. The data may be
utilized to allow a user to control seat positions on a nomadic
device 601 like a mobile phone.
[0055] Further, vehicle embedded server 611 may generate or output
HMI to the various displays of the vehicle, such as on the display
of nomadic device 601. The vehicle embedded server may additionally
be connected to the off-board vehicle cloud servers to communicate
various data, such as off-board data, a standard user interface, or
a contextual user interface. The server 611 maybe output sound or
music through the vehicle speakers 627 utilizing DSP, 5.1, THX, or
other audio compression technology.
[0056] The steering wheel controls 613 may also be able to
communicate data to the server 611 and other devices within the
VCS. The steering wheel may allow for flexibility in controlling
the user interface while allowing a driver to safely operate the
vehicle. The steering wheel controls 613 may control only some of
the display's user interfaces or every different user interfaces
within the vehicle.
[0057] Another vehicle computer 629 may include an application
acting as a data manager, HUD module, and a voice recognition
engine. The data manager may be utilized to facilitate in the
transmission of relevant CAN data to the various vehicle modules.
The voice recognition engine can provide the ability of user
commands to be spoken and input via MIC 631. The voice recognition
engine may use a natural language voice recognition engine and
eliminate the need to speak only specific commands, but utilize
contextual awareness to interpret the user's spoken commands. Thus
after identifying the user's command, the voice engine will send
the relevant CAN message or data to the relevant module to act on
the command. The embedded vehicle server may be able to communicate
with all relevant HMI interfaces in the vehicle, such as but not
limited to the heads up display (HUD), Cluster, Center stack, and
Voice Recognition/Spoken Dialogue systems. The voice engine 629 may
output the dialogue utilizing various speakers 627 in the
vehicle.
[0058] A HUD 631 may be utilized to render a specific type of user
interface or to display specific data. The HUD may allow for a
projection to be displayed on a vehicle's windshield in order to
display relevant information within a driver's peripheral vision.
The HUD can act as one of type of display in the vehicle utilizing
the vehicle server's HMI.
[0059] Another type of display that may utilize the vehicle
server's HMI is the vehicle cluster. The vehicle cluster may be
utilized to display various vehicle information, such as speed,
RPM's, fuel economy, fuel level, etc. The cluster display 635 may
be in communication with a cluster manager 633. The cluster manager
633 may receive the interface data from the vehicle server to
output on the cluster display 635. Furthermore, the cluster manager
633 may be capable of rendering its own HMI or facilitating data
communication with other device within the vehicle network.
[0060] Another type of display that may utilize the vehicle
server's HMI is the vehicle center stack. The vehicle center stack
may be utilized to display various vehicle information, such as
audio information, navigation information, phone information,
vehicle settings, etc. The center stack display 639 may be in
communication with a center stack manager 637. The center stack
manager 637 may receive the interface data from the vehicle server
to output on the cluster display 639. Furthermore, the cluster
manager 637 may be capable of rendering its own HMI or facilitating
data communication with other devices within the vehicle
network.
[0061] FIG. 5B is an illustrative embodiment of a simplified block
diagram depicting the network architecture utilizing various
displays in the vehicle. The vehicle computer system 657 may host a
vehicle embedded server to communicate data and web pages to
various devices utilizing a browser or application. Some of the
displays may be hardwired, such as the vehicle cluster 655, center
stack 659, HUD 651, or other display for outputting an HMI
interface 653. Additionally, the web server may utilize a
wired/wireless connection 661, to bring in a nomadic device 663.
The nomadic device 663 may output various vehicle HMI's 665 for
utilization on the nomadic device. The various devices 663 may be
in communication with the vehicle web server 657 through a wired or
wireless connection 661. The various displays and devices may use
an application to output the relevant HMI data to output an
interface 665. In another embodiment, the interface 665 may be
output utilizing a nomadic device browser that renders its own
page.
[0062] FIG. 6 is an illustrative embodiment of a human machine
interface displayed on different types of nomadic devices. 701 is
an example of a tablet that may connect with the vehicle server in
order to operate the vehicle computer system. The interface of the
nomadic device may allow for a user to control various features,
such as the radio navigation, climate control, cell phone, and
other features and settings. The tablet's interface 702 may mimic
the interface of an in-vehicle HMI, or be tailored to specific
features for the tablet. The interface may run on different
operating systems, such as iOS and Android. The vehicle server may
send the HMI to the tablet 701 formatted for the screen size and
resolution of the tablet.
[0063] 703 is an example of a cell phone that may connect with the
vehicle server in order to operate the vehicle computer system. The
tablet 701 and cell phone 703 may connect via a wired, short-range
wireless (e.g. Bluetooth, Wi-Fi, Wi-Fi Direct), or long-range
wireless (CDMA, GSM, LTE, etc). The interface 705 may be tailored
to fit and display optimally on the screen of a cell phone.
Although FIG. 6 illustrates an example of a cell phone and tablet
being utilized to display the interface, other mobile devices may
be used. This may include an MP3 player, a laptop computer, PDA, or
portable gaming device.
[0064] FIG. 7 illustrates an example of a query based HMI that may
utilize a spoken dialogue system. The query/search based system 801
may allow for a voice recognition engine to be utilized in
conjunction with the cloud server for a human machine interface
environment. The query based system 801 may be in the vehicle,
located on an off-board server, or utilize a hybrid combination
that encompasses certain aspects of processing both on the vehicle
and off-board. In he query/search based automotive HMI, a user can
query for functionality or search for functionality or information
utilizing the spoken dialogue system that works along with the
cloud based vehicle server. The vehicle server may provide
different options for the user. For example, the user can query for
a multi-contour functionality and the user is offered with related
visual HMI display ad an audible description of the multi-contour
feature.
[0065] A spoken dialogue system 803 may output audible dialogue for
a user. This may allow the user to hear a computer synthesized
voice (text to speech) through the vehicle speakers and provide
commands through a MIC. The dialogue system 803 may utilize various
speech recognition engines to recognize the commands. Additionally,
the spoken dialogue system may be off-board on another server for
additional computing power. Furthermore, the spoken dialogue system
803 may be utilized in conjunction with the visual HMI elements.
For example, any menu or text that is graphically displayed on the
system may be operated as an available dialogue request. For
example, if a graphical button states "Seek Up" on the display to
change to the next radio station, the query based system 801, may
allow the dialogue system 803 to output "seek up" or "next radio
station" as a possible command.
[0066] Furthermore, the query based system 801 may be used in
conjunction with a cloud based server 807 that utilizes a context
aggregator 809 to generate different HMI commands or in conjunction
with additional computing processes. By utilizing the context
aggregator 809 to generate a unique spoken dialogue system 803, the
aggregator 809 may receive location data 813 and other vehicle
information 811 to generate a dialogue for the query based system
801. In one example, the context aggregator 809 may recognize that
the vehicle has low fuel via vehicle data 811 received from the
engine ECU. By utilizing location data 813, the system may know
that a gas station is ahead. Thus the dialogue system 803 may ask
the user "Would you like to visit the Mobile gas station ahead?"
Furthermore, the visual HMI elements 805 may display the exact
location of the gas station on a map, along with other intuitive
information like the address or phone number. Although the
embodiment illustrated shows one example, any combination of the
various vehicle data and off-board data may be utilized in
conjunction with the query based system 801. In one example, the
user can query/search for a suitable lumbar and bolster adjustment
that are offered as an option on a multi-contour seat. The spoken
dialogue system can aggregate the data from a vehicle cloud server
and provide suitable help for a given query.
[0067] While exemplary embodiments are described above, it is not
intended that these embodiments describe all possible forms of the
invention. Rather, the words used in the specification are words of
description rather than limitation, and it is understood that
various changes may be made without departing from the spirit and
scope of the invention. Additionally, the features of various
implementing embodiments may be combined to form further
embodiments of the invention.
* * * * *