U.S. patent application number 15/784843 was filed with the patent office on 2018-06-07 for system and method for aggregating and reporting network traffic data.
The applicant listed for this patent is LEVEL 3 COMMUNICATIONS, LLC. Invention is credited to Raoul Alcala, Niclas Comstedt, Jeff Whitaker.
Application Number | 20180159750 15/784843 |
Document ID | / |
Family ID | 40229411 |
Filed Date | 2018-06-07 |
United States Patent
Application |
20180159750 |
Kind Code |
A1 |
Alcala; Raoul ; et
al. |
June 7, 2018 |
SYSTEM AND METHOD FOR AGGREGATING AND REPORTING NETWORK TRAFFIC
DATA
Abstract
A method for analyzing traffic in a communications network
includes sampling data packets at a plurality of network
interconnection points, wherein sampling the data packets includes
generating a plurality of sampled packet data in one or more
standardized formats, converting the sampled packet data from the
one or more standardized formats into a neutral format, and
aggregating the sampled packet data in the neutral format from the
plurality of network interconnection points. A system includes a
communications node operable to sample data packets flowing through
and generate sample packet data in a specified format, a collector
node operable to convert the sampled packet data into a neutral
format, the collector node further operable to map IP addresses of
the sampled packet data to corresponding prefixes in a routing
table; and an aggregator node operable to aggregate neutrally
formatted sampled packet data from a plurality of collector
nodes.
Inventors: |
Alcala; Raoul; (Superior,
CO) ; Comstedt; Niclas; (Broomfield, CO) ;
Whitaker; Jeff; (Broomfield, CO) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
LEVEL 3 COMMUNICATIONS, LLC |
Broomfield |
CO |
US |
|
|
Family ID: |
40229411 |
Appl. No.: |
15/784843 |
Filed: |
October 16, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14997005 |
Jan 15, 2016 |
9794142 |
|
|
15784843 |
|
|
|
|
14691483 |
Apr 20, 2015 |
|
|
|
14997005 |
|
|
|
|
12116354 |
May 7, 2008 |
9014047 |
|
|
14691483 |
|
|
|
|
60948960 |
Jul 10, 2007 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04L 43/04 20130101;
H04L 43/062 20130101; H04L 43/16 20130101; H04L 12/5602 20130101;
H04L 43/022 20130101; H04L 43/50 20130101; H04L 43/0876 20130101;
H04L 43/026 20130101; H04L 47/2425 20130101; Y02D 30/50 20200801;
Y02D 50/30 20180101 |
International
Class: |
H04L 12/26 20060101
H04L012/26; H04L 12/851 20130101 H04L012/851; H04L 12/54 20130101
H04L012/54 |
Claims
1. A system comprising: a service provider network comprising a
plurality of communication nodes operable to supply network
bandwidth and routing functionality to a plurality of autonomous
systems, the service provider network comprising: a plurality of
network interconnection points operable to route packets between
the service provider network and the plurality of autonomous
systems; a plurality of communication nodes operable to route
packets within the service provider network, wherein at least some
of the plurality of communication nodes comprises a sampling agent
operable to generate sampled packet data by sampling packets
flowing through each respective communication node, wherein the
sampling agent generates sampled packet data in accordance with a
sampling standard format comprising at least one of sFlow, Netflow,
and cflowd: one or more collector nodes operable to: receive the
sampled packet data from the sampling agent; obtain routing table
information from the collector node associated with the sampling
agent, wherein the routing table information comprises a list of
network prefixes associated with Internet protocol (IP) addresses;
and use the sampled packet data and the routing table to generate a
summary of sampled packets flowing through at least one of the
plurality of network interconnection points.
2. The system as recited in claim 1, further comprising: one or
more aggregator nodes operable to: receive the summary of sampled
packets from the one or more collector nodes; and use the summary
of sampled packets to derive a statistical measure of total traffic
flow throughout the service provider network associated with one or
more network-related attribute categories.
3. The system as recited in claim 2, wherein the one or more
aggregator nodes are further operable to: apply additional data to
the statistical measure of total traffic flow to generate a report
of traffic flow through the service provider network.
4. The system as recited in claim 1, wherein the service provider
network is a backbone network providing wholesale network services
to the plurality of autonomous systems.
5. The system as recited in claim 1, wherein the plurality of
autonomous systems comprises at least one of an internet service
provider (ISP), other service provider network, enterprise network,
cable company, content delivery network (CDN), and application
service provider (ASP).
6. The system as recited in claim 1, wherein the plurality of
network interconnection points comprises at least one of an
Internet exchange point (IXP), network access point (NAP), gateway,
point of presence (POP), and peering point.
7. The system as recited in claim 1, wherein the plurality of
communication nodes comprises at least one or a router and a
switch.
8. The system as recited in claim 1, wherein the sampling agent
generates sampled packet data by generating datagrams that include
at least one of a source Internet protocol (IP) address,
destination IP address, port number, protocol, and autonomous
system number (ASN) associated with the sampled packets.
9. The system as recited in claim 1, wherein the routing table
information comprises: an indication of whether a network prefix is
associated with a customer of the service provider network or a
non-customer of the service provider network.
10. The system as recited in claim 2, wherein the one or more
collector nodes are further operable to: convert the sampled packet
data from the sampling standard format into a neutral format
commonly used by the plurality of collector nodes.
11. The system as recited in claim 10, wherein the neutral format
is common for summaries generates by each collector node and
comprises a unified assembly of specified units of data from the
sampled packet data.
12. The system as recited in claim 2, wherein the one or more
collector node generates the summary of sampled packets in response
to an increase or decrease in traffic flow that exceeds a
respective traffic flow threshold.
13. The system as recited in claim 3, wherein the additional data
comprises an autonomous system registry comprising a mapping of
autonomous system numbers (ASNs) identified in the summary of
sampled packets to autonomous system data.
14. The system as recited in claim 13, wherein the autonomous
system data comprises at least one of an autonomous system name,
autonomous system number, autonomous number country, and autonomous
number continent.
15. The system as recited in claim 13, wherein the additional data
further comprises customer attribute data associated with customer
of the service provider network.
16. The system as recited in claim 15, wherein the customer
attribute data comprises at least one of customer preferences,
historical average traffic volume, and interface device types.
17. The system as recited in claim 2, wherein the one or more
network-related attribute categories comprises at least one of
autonomous systems, communities, customer, noncustomers, on-net
traffic, off-net traffic, peers, applications, and protocols.
18. The system as recited in claim 2, wherein the one or more
aggregator nodes derives the statistical measure of total traffic
flow by extrapolating the number of sampled packets associated with
a given network-related attribute category.
19. The system as recited in claim 18, wherein the step of
extrapolating comprises multiplying the total number of sampled
packets associated with the given network-related attribute
category with a sampling factor.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of and claims the benefit
of priority to U.S. nonprovisional application Ser. No. 14/997,005,
filed Jan. 15, 2016, titled "SYSTEM AND METHOD FOR AGGREGATING AND
REPORTING NETWORK TRAFFIC DATA," which is hereby incorporated by
reference herein for all purposes. Application Ser. No. 14/997,005
is a divisional of and claims the benefit of priority to U.S.
nonprovisional application Ser. No. 14/691,483, filed Apr. 20,
2015, titled "SYSTEM AND METHOD FOR AGGREGATING AND REPORTING
NETWORK TRAFFIC DATA," which is hereby incorporated by reference
herein for all purposes. Application Ser. No. 14/691,483 is a
continuation of and claims the benefit of priority to U.S.
nonprovisional application Ser. No. 12/116,354, filed May 7, 2008,
titled "SYSTEM AND METHOD FOR AGGREGATING AND REPORTING NETWORK
TRAFFIC DATA," which is hereby incorporated by reference herein for
all purposes. Application Ser. No. 12/116,354 claims the benefit of
priority to U.S. provisional application No. 60/948,960, filed Jul.
10, 2007, which is hereby incorporated by reference herein for all
purposes.
COPYRIGHT NOTICE
[0002] Contained herein is material that is subject to copyright
protection. The copyright owner has no objection to the facsimile
reproduction of the patent disclosure by any person as it appears
in the Patent and Trademark Office patent files or records, but
otherwise reserves all rights to the copyright whatsoever.
Copyright .COPYRGT. 2008 Level 3 Communications, LLC.
TECHNICAL FIELD
[0003] Embodiments of the present invention generally relate to
network communications. More specifically, embodiments relate to a
system and method for aggregating and reporting network traffic
data.
BACKGROUND
[0004] Network service providers want to understand flow of
communications traffic and traffic flow changes over the network
for purposes of capacity planning, marketing and other reasons. For
example, it is important to know if traffic through a gateway is
increasing, in order to know whether routers should be added to the
gateway. It may also be helpful to know whether traffic from a
particular customer, such as an Internet service provider (ISP), is
increasing or decreasing to understand how to provide better
service to the ISP. Some tools, such as sFlow and Netflow, are
available that attempt to provide information about traffic flow by
sampling packet data through IP networks.
[0005] For example, a Nefflow agent running on a router can provide
data such as packet source IP address, destination IP address and
port numbers. In addition to these, the autonomous system (AS) may
be identified by a sFlow agent. Because currently available tools
gather traffic flow information at a packet level, these tools
obtain relatively low-level information such as source or
destination IP addresses. However, these tools are unable to
provide information such the city or ISP that the packets are
originating from or going to. As such, currently available traffic
analysis tools are not capable of providing a view of traffic flow
or changes in traffic flow as it relates to other business aspects
of the network service provider.
[0006] It is with respect to these and other problems that
embodiments of the present invention have been created.
SUMMARY
[0007] Embodiments of systems and methods can use sampled packet
data to determine traffic flow statistics associated with one or
more attributes, such as, but not limited to geographic region,
network, community, application, protocol, autonomous system, or
customer. Traffic flow statistics can be measures of traffic volume
associated with an attribute. For example, traffic volume
measurements can be generated that indicate the traffic volume that
is inbound to or outbound from an autonomous system network. As
another example, traffic volume measurements can be generated that
reflect the volume of traffic that is on-net and the volume of
traffic that is off-net with respect to a selected network.
[0008] An embodiment of a method for analyzing traffic in a
communications network includes sampling data packets at a
plurality of network interconnection points, wherein sampling the
data packets includes generating a plurality of sampled packet data
in one or more standardized formats, converting the sampled packet
data from the one or more standardized formats into a neutral
format, and aggregating the sampled packet data in the neutral
format from the plurality of network interconnection points.
[0009] In at least one embodiment of the method the IP prefixes are
obtained from a routing table. The method may further include
converting the sampled packet data from one or more standardized
formats into a neutral format. Enriching the sampled packet data
may include mapping IP addresses in the sampled packet data to
geographic locations. Enriching the sampled packet data may include
mapping IP addresses in the sampled packet data to customers. The
plurality of standardized formats may include one or more of sFlow
format, Netflow format and cflowd format. The geographic locations
may include one or more of a city, a country, a continent or a
region.
[0010] An embodiment of the method further includes generating one
or more traffic flow reports based on the sampled packet data. The
one or more reports may include one or more of a peer distribution
report including One or more of traffic volume measurements of
traffic outbound from a peer network or traffic volume measurements
of traffic inbound to a peer network, a customer distribution
report including on or more of traffic volume measurements of
traffic outbound from one or more customer networks or traffic
volume measurements of traffic inbound to one or more of customer
networks, an autonomous system distribution report including
average traffic volume, traffic volume by region, on-net traffic
volume, off-net traffic volume, direction of traffic, next hop
autonomous systems, and upstream autonomous systems associated with
one or more autonomous systems, an applications report including
average traffic volume and region to region traffic volume
associated with network applications, an on-net distribution report
includes traffic volume measurements of traffic outbound from a
peer network that was also inbound to the peer network, an off-net
distribution report includes traffic volume' measurements of
traffic outbound from a peer network that was not also inbound to
the peer network, a region distribution report including average
traffic volume inbound to or outbound from selected regions, and a
city distribution report including average traffic volume from one
or more source cities to one or more destination cities.
[0011] In an embodiment of the method enriching the sample packet
data includes generating one or more sampled packet data summaries
at a collector node by mapping the sampled packet data to routing
table data from communication nodes associated with the collector
node. The method may further include communicating the one or more
sampled packet data summaries from the collector nodes to an
aggregator node configured to perform the aggregating.
[0012] An embodiment of a system includes a communications node
operable to sample data packets flowing through and generate sample
packet data in a specified format, a collector node operable to
convert the sampled packet data into a neutral format, the
collector node further operable to map IP addresses of the sampled
packet data to the network (or prefix) they belong to in a routing
table, and an aggregator node operable to aggregate neutrally
formatted sampled packet data from a plurality of collector nodes.
The IP prefixes may be associated with respective autonomous
systems, and wherein the collector node is further operable to map
sampled packet data to associated autonomous systems. The
aggregator node may be further operable to generate traffic flow
measures associated with autonomous systems communicating over the
communications network.
[0013] An embodiment of the system may further include an
autonomous system (AS) registry storing information about
autonomous systems including AS identifier and region. The system
may further include a customer attributes data store storing
customer AS identifiers. The aggregator node may be further
operable to map AS identifiers from the sampled packet data to
regions in the AS registry. Further still, the aggregator node may
be operable to generate traffic flow measurements associated with
network applications based on the neutrally formatted sampled
packet data. Still further, the aggregator node may be operable to
generate one or more reports relating traffic flow measurements to
one or more of a region, an autonomous system, a community, a
network application, or a network protocol. At least one of the one
or more reports may include traffic flow measurements for traffic
flowing from a selected community to at least one other
community.
[0014] In at least one embodiment of the system the aggregator is
further operable to generate a traffic flow measurement of outbound
traffic from a selected AS to each of a plurality of other AS's.
The aggregator may be further operable to generate a traffic flow
measurement of outbound traffic to a selected AS from each of a
plurality of other AS's. Further still, the aggregator may be
operable to generate a traffic flow measurement of on-net traffic
and a traffic flow measurement of off-net traffic for a selected
AS.
[0015] Another embodiment of a method for analyzing communications
traffic through a network includes sampling data packets
communicated through the network, categorizing each data packet
according one or more network-related attribute categories,
aggregating data packets of each of the attribute categories, and
determining data packet transmission statistics associated with
each of the one or more attribute categories based on the
aggregation. The one or more attribute categories may include one
or more of geographic region, network, community, application,
protocol, autonomous system, customer, on-net, or off-net.
Determining data packet transmission statistics may include
determining data packet volume transmitted to or from one or more
of a geographic region, a network, a community, an autonomous
system, a customer, an on-net provider, or an off-net provider.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 illustrates an operating environment suitable for
practicing traffic flow data aggregation and reporting in
accordance with various embodiments of the present invention.
[0017] FIG. 2 illustrates a network configuration for carrying out
traffic flow data aggregation and reporting in accordance with the
embodiment of FIG. 1.
[0018] FIG. 3A illustrates an exemplary peer outbound distribution
report generated from an aggregating and reporting system.
[0019] FIG. 3B illustrates an exemplary peer inbound distribution
report generated from an aggregating and reporting system.
[0020] FIG. 4A illustrates an exemplary on-net/off-net outbound
distribution report generated from an aggregating and reporting
system.
[0021] FIG. 4B illustrates an exemplary on-net/off-net inbound
distribution report generated from an aggregating and reporting
system.
[0022] FIG. 5A illustrates an exemplary customer outbound
distribution report generated from an aggregating and reporting
system.
[0023] FIG. 5B illustrates an exemplary customer inbound
distribution report generated from an aggregating and reporting
system.
[0024] FIG. 6A illustrates a report of an autonomous system
associated with a highest traffic volume on a network.
[0025] FIG. 6B illustrates an exemplary report of applications or
protocols associated with a highest traffic volume on a
network.
[0026] FIG. 7A illustrates an exemplary report 700 trends in volume
of traffic associated with selected applications or protocols on a
network.
[0027] FIG. 7B illustrates an exemplary city to city traffic flow
report generated from an aggregating and reporting system.
[0028] FIG. 8 is a flowchart illustrating a traffic flow
aggregating and reporting algorithm in accordance with an
embodiment of the present invention.
[0029] FIG. 9 illustrates a general purpose computing device upon
which one or more aspects of embodiments of the present invention
may be implemented.
[0030] While the invention is amenable to various modifications and
alternative forms, specific embodiments have been shown by way of
example in the drawings and are described in detail below. The
intention, however, is not to limit the invention to the particular
embodiments described.
DETAILED DESCRIPTION
[0031] Embodiments of systems and methods can use sampled packet
data to determine traffic flow statistics associated with one or
more network-related attribute categories, such as, but not limited
to geographic region, network, community, application, protocol,
autonomous system, customer, on-net or off-net, Traffic flow
statistics can be measures of traffic volume associated with an
attribute. For example, traffic volume measurements can be
generated that indicate the traffic volume that is inbound to or
outbound from an autonomous system network. As another example,
traffic volume measurements can be generated that reflect the
volume of traffic that is on-net and the volume of traffic that is
off-net with respect to a selected network.
[0032] Some embodiments of the present invention relate to systems
and methods for aggregating and reporting traffic flow data
captured in a communications network. Various embodiments sample
packets of data flowing through the communications network and
derive aggregated traffic flow from the sampled packets. Data in
sampled packets are mapped to routing data descriptive of routes in
the communications network, thus enriching the sampled packet data.
By enriching sampled packet data with routing data, traffic flow
can be derived in relation to relevant attributes, such as
autonomous systems, geographical regions, NSP customers, NSP
noncustomers, peers, on-net or off-net distribution, network
applications or protocols. For example, total traffic flow between
AS's, peers or communities can be determined. Reports can include
traffic flow statistics in relation to various attributes. For
example, traffic volume measurements can be used to report traffic
volume between AS's, peers or communities, as well as
on-net/off-net, customer and noncustomer distribution, or traffic
associated with selected applications or protocols.
[0033] A method for analyzing communications traffic through a
network includes sampling data packets communicated through the
network, categorizing each data packet according one or more
network-related attribute categories, aggregating data packets of
each of the attribute categories, and determining data packet
transmission statistics associated with each of the one or more
attribute categories based on the aggregation. The one or more
attribute categories may include one or more of geographic region,
network, community, application, protocol, autonomous system,
customer, on-net, or off-net. Determining data packet transmission
statistics may include determining data packet volume transmitted
to or from one or more of a geographic region, a network, a
community, an autonomous system, a customer, an on-net provider, or
an off-net provider.
[0034] Prior to describing one or more preferred embodiments of the
present invention, definitions of some terms used throughout the
description are presented.
Definitions
[0035] The term "network service provider" refers to an
organization or business that provides network access to one or
more customers. An NSP may operate, for example, a backbone network
and/or edge networks coupled to a plurality of other networks,
whereby the other networks can communicate with each other and the
Internet via the NSP network(s).
[0036] The term "customer" refers to an entity that uses services
provided by an NSP. For example, the customer may pay the NSP for
carrying traffic over the NSP's network.
[0037] A "node" is a uniquely addressable functional device on 0 e,
communicatively coupled to) a network. A node may be any type of
computer, server, gateway device, or other.
[0038] "Traffic", "communication traffic" or "network traffic"
refer to the flow of data or messages in a network.
[0039] A "module" is a self-contained functional component. A
module may be implemented in hardware, software, firmware, or any
combination thereof.
[0040] The terms "connected" or "coupled" and related terms are
used in an operational sense and are not necessarily limited to a
direct connection or coupling.
[0041] The phrases "in one embodiment," "according to one
embodiment," and the like generally mean the particular feature,
structure, or characteristic following the phrase is included in at
least one embodiment of the present invention, and may be included
in more than one embodiment of the present invention. Importantly,
such phases do not necessarily refer to the same embodiment.
[0042] If the specification states a component or feature may",
"can", "could", or "might" be included or have a characteristic,
that particular component or feature is not required to be included
or have the characteristic.
[0043] The terms `responsive and "in response to" includes
completely or partially responsive.
[0044] A computer program product can take the form of one or more
computer-readable media. The term "computer-readable media" is
media that is accessible by a computer, and can include, without
limitation, computer storage media and communications media.
Computer storage media generally refers to any type of
computer-readable memory, such as, but not limited to, volatile,
non-volatile, removable, or non-removable memory. Communication
media refers to a modulated signal carrying computer-readable data,
such as, without limitation, program modules, instructions, or data
structures.
Exemplary System
[0045] FIG. 1 illustrates an exemplary operating environment 100 in
which traffic flow aggregation and reporting may be carried out in
accordance with embodiments of the present invention. The operating
environment 100 includes a network service provider (NSP) network
102 that operably interconnects multiple autonomous systems (AS)
104. The NSP network 102 provides communication service between the
AS's 104. More specifically, the NSP network 102 supplies network
bandwidth and routing functionality to the AS's 104 to route data
packets between endpoints 106 logically located in the AS's
104.
[0046] In one embodiment, the NSP network 102 comprises a managed
backbone network providing wholesale network service. The NSP could
provide services commonly provided by an ISP. For example, the NSP
could provide email, web site hosting; caching, and content
serving. The AS's 104 may include, for example, Internet service
providers (ISPs) networks, other NSP networks, enterprise networks,
Regional Bell Operating Companies (RBOC) networks, cable company
networks, content distribution networks (CDNs), web sites, and
application service provider (ASP) networks. The endpoints 106 are
communications devices used by; for example, private users,
enterprises, or web sites. One or more of the AS's 104 may
interconnect with other networks or AS's 108 to facilitate
communication to and from the other networks/AS's 108. The NSP
network 102, autonomous systems 104, and other network/AS's 108 may
include any combination of wireless or wireline networks.
[0047] The autonomous systems 104 interconnect with the NSP network
102 at network interconnection points (NIP) 110. A NIP 110 may be,
for example, an Internet exchange point (IXP), a network access
point (NAP), a gateway, a point of presence (POP), a peering point,
or a regional switching point. Data packets are routed through the
NiPs 110 to and from autonomous systems 104 via the NSP network
102. The data packets can pertain to various different
applications, such as, but not limited to, simple mail transport
protocol (SMTP) applications (e.g., email), hypertext transport
protocol (HTTP) applications, telnet applications, or peer-to-peer
(p2p) applications,
[0048] In various embodiments, the AS's 104 may be peers of other
AS's 104 and the NSP network 102 and/or the AS's 104 may be
customers of the NSP. For example, an AS 104 may provide backbone
network service like the NSP network 102; or, the AS 104 may be a
website or web content host that subscribes to backbone service
provided by the NSP network 102. Some AS's 104 may act as both a
peer and customer to the NSP network 102.
[0049] The NSP network 102 includes numerous communication nodes
112 that include routing or switching functionality, for routing
packets through the NSP network 102 and to/from one or more AS's
104. For simplicity, only a few communication nodes 112, and only a
few AS's 104, are shown in the embodiment of FIG. 1. The
communication nodes 112 may be, by way of example, routers or
switches. In order to effectively handle communication traffic
flowing over the NSP network 102 between AS's 104, it is useful to
understand the nature of traffic flow into, out of, and/or through
the NSP network 102.
[0050] Accordingly, at least some of the communication nodes 112
include sampling agents 114 that sample packets flowing through the
communication nodes 112. The sampling agents 114 may sample packets
according to a sampling standard, such as, but not limited to
sFlow, Netflow, or cflowd. Each of these standards yield sample
packet data in a specified format that may differ from one standard
to another. Generally the sampling agents 114 sample 1 in N
packets, where N can be set by the manufacturer or configured by a
user (e.g., a network administrator). The particular sampling
protocol employed in a communication node 112 may depend upon the
make or model of the communication node 112. For example, Cisco.TM.
routers typically employ Netflow, Juniper.TM. routers employ
cflowd, and Forcel0m4 routers employ sFlow.
[0051] Typically, the sampling agents 114 generate datagrams or
other units of data that include specified information obtained
from the sampled packets. The sample packet data 119 generally
includes at least the source IP address, destination IP address,
port numbers, and protocol associated with the sampled packet. Some
sampling standards, such as sFlow, also obtain autonomous system
numbers associated with the sampled packet. Sampled packet data 119
are sent from the communication nodes 112 to collector nodes 116.
In addition, each collector node 116 obtains a routing table 117
from the respective communication node 112. In some embodiments the
routing table 117 is sent over a routing feed that is separate from
the sample packed data 119. The collector node 116 uses the sample
packet data 119 and the routing table 117 to generate a summary 121
of sampled packets flowing through the network interconnection
point 110.
[0052] To generate the summary 121, a collector node 116 first
converts the sample packet data 119 from the sampling standard(s)
used by the sampling agent 114 into a neutral format. For example,
sample packet data 119 in the sFlow format, the Netflow format or
the cflowd format are converted into a neutral format that is
commonly used by the collector nodes 116 (and later the aggregator
nodes 118). Typically the neutral format differs from the standard
formats, but this is not required. In one embodiment, the neutral
format is a unified assembly of specified units of data from the
sample packet data 119 in a format that is common across all
summaries 121 and collector nodes 116. In various embodiments,
after the collector nodes 116 receive sample packet data 119 from
the sample agents 114, the collector nodes 116 identify each of the
specified data (e.g., source and destination IP addresses) in the
sample packet data 119, extract the data, convert the data into
common units, as may be necessary, and assemble the data into the
neutral format.
[0053] The routing table 117 from a communication node 112
indicates available routes over which packets can be sent to reach
their destinations. The routing table 117 includes a list of
networks (or prefixes) to which IP addresses belong that the
collector node 116 correlates with destination IP addresses in the
sample packet data 119. In addition to the destination IP prefixes
of routes, the routing table includes AS numbers for all AS's in
the route. In various embodiments, the routing table 117 includes
information about communities associated with the routes. A
community is generally a group of network nodes that have some
common attribute. For example, a community may be a geographic
area, such as a city, a country, a continent or a region. Using the
community data, the collector node 116 can map IP addresses in the
sampled packet data 119 to communities identified in the routing
table 117.
[0054] The routing table 117 also indicates whether the IP prefixes
are in autonomous systems that are customers or not customers of
the NSP network 102. In addition, the routing table 117 can include
information about the destination router associated with each
route. By mapping IP address data in the sample packet data to
corresponding IP prefixes in the routing table 117, each collector
node 116 can generate one or more summaries 121 of packet flow in
the associate NIP 110. In some embodiments the format and contents
of the summaries 121, as well as the timing of generation, are
configurable by the user.
[0055] In one embodiment, the collector node 116 enriches sample
packet data using the data in the routing table 117. For example, a
collector node 116 could generate a summary including a mapping of
AS numbers to source and destination IP addresses of sampled
packets. As another example, the collector node 116 could generate
a summary 121 including a mapping of source and destination 1P
addresses to city, country, continent or region. The summaries 121
may be generated by the collector nodes 116 automatically or on
demand. For example, the summaries 121 may be generated
periodically (e.g., once daily, weekly or monthly). Alternatively
or in addition, summaries 121 may be generated in response to
certain events. For example, summary 121 generation may be
triggered by an increase or decrease in traffic flow that exceeds a
set threshold. One exemplary embodiment of enriched sample packet
data is shown below:
[nfr] sa:75.126.53.172 da:82.206.143.35 nh:4.68.122.158 ii:66 oi:85
pa:1 oc:46 fi:198857563 la:198861211 sp:80 dp:1590 fl:0x0 pr:6
to:0x0 ds:12179 ss:22351 s1:16 d1:24 BV:default
SR:75.126.0.0/16
SN:4.69.185.162
SS:1279 12179 36351
SC:3356:3 3356:22 3356:100 3356:123 3356:575 3356:2008
DR:82.206.143.0/24
DN:4.69.185.2
DS:22351
DC:3356:3 3356:22 3356:100 3356:123 3356:575 3356:2010
22351:4001
[0056] The summaries 121 that are generated by the collector nodes
116 are sent to one or more aggregator nodes 118. An aggregator
node 118 is operable to apply additional data to the summaries from
the network interconnection points 110 to generate reports of
traffic flow through the NSP network 102 as a whole and to further
correlate features of the traffic flow with AS's 104 that
interconnect with the NSP network 102. In the illustrated
embodiment, the aggregator node 118 uses an AS registry 120 and
customer attributes data 122 to derive other correlations between
data and further enrich the data in the traffic flow report.
[0057] The AS registry 120 provides information about autonomous
systems worldwide. Such AS information can include the AS name,
number, country, continent, region, and so on. The aggregator 118
can use the AS registry 120, for example, to map AS numbers
identified in the summaries 121 to AS names, countries and/or
regions. The customer attributes data 122 is typically an internal
database of the NSP network 102 that stores various types of data
about customers of the NSP. The customer data is gathered over time
based on the NSP's understanding of the customers. For example, but
without limitation, the customer attributes data 122 can store
customer preferences, historical average traffic volume, interface
device types or specifications, as well as customer AS numbers with
their associated customer name, which may differ from the publicly
known name in the AS registry 120.
[0058] In some embodiments the aggregator nodes 118 derive
statistics, such as a measure of total traffic flow, associated
with one or more network-related attribute categories, such as AS's
104, communities, customers, noncustomers, on-net, off-net, peers,
application, protocol. In an embodiment, total traffic flow can be
determined by extrapolating the number of sampled packets
associated with a given attribute category. This extrapolation can
involve multiplying the total number of sampled packets with a
particular characteristic (e.g., associated with a particular peer,
AS, customer, etc.) with the sampling factor used by the sampling
agent 114.
[0059] For example, if the sampling agent 114 samples one packet in
every `N` packets, and samples 20 packets associated with a
particular peer network, then the total packet flow attributed to
the peer network is derived by multiplying N times 20. As another
example, if 50 packets are sampled from a selected AS and 15
packets of the 50 are sent off the AS network and the remainder are
sent back on the AS network, N times 15 yields the off-net traffic
flow, and N times 35 yields the on-net traffic flow associated with
the selected AS. The aggregator nodes 118 can generate reports on
demand or automatically or both. The reports may be configured by
the user to present certain data in certain ways. Exemplary
embodiments of collector nodes, aggregator nodes, and their
functions and outputs are discussed further below.
[0060] For example, FIG. 2 illustrates an exemplary embodiment of a
traffic flow aggregation and reporting system 200 in accordance
with the embodiment of FIG. 1. The exemplary system 200 includes
four gateways: 202a, 202b, 202c, and 202d. In general, a gateway
202 is a point of contact between two networks. The gateways 202
typically, but not necessarily, perform protocol conversion between
the networks. Each gateway 202 includes multiple routers 204 and
two core switches 206. The numbers of gateways 202, routers 204,
and core switches 206 shown in FIG. 2 are for illustrative purposes
only; it will be understood by those skilled in the art that a
typical network may include more or fewer gateways, routers and
switches than those shown in FIG. 2.
[0061] Each gateway 202 includes two collectors 208. In this
particular example, the collectors 208 are implemented as server
computers. In this embodiment two collectors 208 are provided for
redundancy, but in general, more or fewer collectors may be
installed at each gateway. The collectors 208 at each gateway 202
are coupled to respective core switches 206 in the gateway 202.
Further, the core switches 206 are coupled to multiple routers 204.
The routers 204 include sampling agents that sample packet data
that flows through the gateway 202. Each collector 208 includes an
application that converts and summarizes sample packet data from
routers 204. In addition, a routing feed exists between each core
switch 206 and the collectors 208, whereby the routing tables,
sample packet data, and/or other data of the routers 204 can be
sent to the collectors 208. The collectors 208 use the routing
table data to summarize the sample packet data.
[0062] The system 200 includes two aggregators 210 implemented as
server computers in this particular example. Two aggregators 210
are used here for redundancy; however, in other implementations
more or fewer aggregators may be used. The aggregators 210 may or
may not be geographically distant from the collectors 208 or each
other. For example, one aggregator 210 may be located in Atlanta,
Ga., and another in Denver, Colo. The aggregators 210 use one or
more AS registry databases 212 and one or more customer attribute
databases 214 to map sampled packet data to other data relevant to
marketing, capacity planning and/or security. For example, data in
the sampled packet data may be mapped to AS's, peer networks,
network applications, network protocols, customers, geographic
regions, and/or communities, such as, but not limited to cities,
counties or countries.
[0063] FIGS. 3-7 illustrate exemplary reports that may be generated
in accordance with various embodiments. In these embodiments the
reports comprise tables; however, the tables are merely one example
of the manner of presentation. In addition to tabular form, traffic
flow data can be presented, for example, by charts (e.g., pie
charts, bar charts), graphs (e.g., trend line graphs), text,
spreadsheets, and histograms. Such reports can be presented on a
computer display, and/or printed on paper, or other output
mechanism. The reports also may be stored in foes and/or sent via
email to one or more users.
[0064] FIG. 3a illustrates an exemplary peer outbound distribution
report 300. The peer outbound distribution report 300 indicates the
traffic volume sent to one or more other peer autonomous systems by
a selected peer AS, referred to here as Peer AS `n`. The left
column 302 of the report 300 lists names or other identifiers of
peer autonomous systems and the right column 304 lists the
corresponding traffic volume sent to the associated peer AS
networks identified in the left column. The volume may be listed in
various units, such as Megabits, Gigabits, or as a percentage of
total volume sent by Peer AS `n`.
[0065] FIG. 3b illustrates an exemplary peer inbound distribution
report 306. The peer inbound distribution report 300 indicates the
traffic volume sent to Peer AS `n` from one or more other peer
autonomous systems. The left column 308 of the report 306 lists
names or other identifiers of peer autonomous systems and the right
column 310 lists the corresponding traffic volume sent from each of
the peer AS's identified in the left column to the Peer AS `n`, The
volume may be listed in various units, such as Megabits, Gigabits,
or as a percentage of total volume received by Peer AS `n`.
[0066] FIG. 4a illustrates an exemplary on-net/off-net outbound
distribution report 400. In general the report 400 indicates the
traffic volume sent by AS `n` that is also received by AS `n.
(on-net) and the traffic volume sent by AS `n` that is not received
by AS `n` (off-net). For example, in the case of on-net traffic,
the traffic is sent from a node (e.g., an endpoint) on the AS `n`
network onto the backbone network, and is directed to a destination
node that is also on the AS `n` network, so the backbone network
routes the traffic back onto the AS `n` network. By contrast,
off-net traffic is sent from the AS network and has a destination
on a different AS network, so the backbone network routes the
traffic onto the different AS network. Referring to report 400, the
left column 402 includes designations on-net and off-net. The right
column 404 indicates the volume of traffic sent from the AS *n`
network that is on-net and off-net, respectively. The volumes in
the right column 404 can be in various units such as Megabits,
Gigabits, or as a percentage of total outbound volume.
[0067] FIG. 4b illustrates an exemplary on-net/off-net inbound
distribution report 406. In general the report 406 indicates the
traffic volume received by AS `n` that is also sent by AS `n`
(on-net) and the traffic volume received by AS `n` that is not sent
by AS `rf (off-net). The report 406 includes a left column 408 that
includes designations on-net and off-net. The right column 410
indicates the volume of traffic received by the AS `n` network that
is on-net and off-net respectively. The volumes in the right column
410 can be in various units such as Megabits (Mb), Gigabits (Gb),
or as a percentage of total inbound volume.
[0068] FIG. 5a illustrates an exemplary customer outbound
distribution report 500. In general, the report 500 indicates
traffic volume that is sent by an AS (AS `n` in this example) to
selected customers of the NSP network that carries the traffic. The
report 500 includes a left column 502 that lists customer
identifiers (e.g., customer names). The right column 504 indicates
the traffic volume sent by AS `n` to the respective customers
identified in the left column 502. In the particular example of
FIG. 5, the top 10 customers are shown: however, any customers
could be selected for the report 500.
[0069] FIG. 5b illustrates an exemplary customer inbound
distribution report 506. In general, the report 506 indicates
traffic volume that is received by an AS (AS `n` in this example)
from selected customers of the NSP network that carries the
traffic. The report 506 includes a left column 508 that lists
customer identifiers (e.g., customer names). The right column 510
indicates the traffic volume received by AS `n` from the respective
customers identified in the left column 508. In the particular
example of FIG. 5, the top 10 customers are shown; however, any
customers could be selected for the report 506.
[0070] FIG. 6a illustrates a report 600 of the AS's associated with
the highest traffic volume on an NSP network. A left column 602
lists AS's by name (or other identifier). An `average volume`
column 604 lists the average volume sent or received by the
respective AS shown in the left column 602. A `top region` column
606 lists the region identifier and associated volume for the
region that the associated AS sends to or receives from. Exemplary
regions are Europe, North America or Asia. The volume shown in
column 606 could be a percentage of total volume or some other
units, such as Mb or Gb. An `on-net/off-net` column 608 indicates
whether most of the volume was on the respective AS's network or
off the respective AS's network, and what the percentage of volume
the on-net or off-net volume constituted.
[0071] In a `direction` column 610 indicates the direction of the
majority of traffic flow relative to the respective AS listed in
the left column 602. The direction is indicated by the terms
`source` (sent from the AS) and `sink` (received by the AS). A
`next hop AS's` column 612 identifies one or more AS's that the
traffic flow was sent to in the routing of the traffic. An
`upstream` AS's column 614 identifies one or more AS's that the
traffic was routed through prior to getting to the respective AS
listed in the left column 602. The next hop AS's and upstream AS's
can be determined from the routing table provided by the router or
switch. The AS's can be identified by any relevant identifiers,
such as, but not limited to, a name or number.
[0072] FIG. 6b illustrates an exemplary report 616 of the
applications or protocols associated with the highest traffic
volume on an NSP network. Exemplary applications or protocols
include, but are not limited to, Internet Protocol Version 4 (or
other version), Internet control message protocol (ICMP), Internet
group multicast protocol (IGMP), gateway to gateway protocol,
transmission control protocol (TOP), interior gateway protocol
(IGP), exterior gateway protocol (EGP), universal datagram protocol
(UDP), source demand routing protocol (SDMP), simple mail transport
protocol (SMTP), EIGRP, TCF, and multicast transport protocol
(MTP). The left column 618 lists application or protocol
identifiers, such as names, acronyms, version numbers, or
others.
[0073] An `average volume` column 620 indicates the average traffic
volume associated with the respective application/protocol listed
in the left column 618. Columns 622, 624, 626 and 628 list average
region to region volumes for selected regions. Specifically an
`Average EU to EU Volume` column 622 lists the average traffic
volume associated with the respective application/protocol that was
sent from Europe and received in Europe. An `Average EU to NA
Volume` column 624 lists the average traffic volume associated with
the respective application/protocol that was sent from Europe and
received in North America. An `Average NA to EU Volume` column 626
lists the average traffic volume associated with the respective
application/protocol that was sent from North America and received
in Europe. An `Average NA to NA Volume` column 628 lists the
average traffic volume associated with the respective
application/protocol that was sent from North America and received
in North America.
[0074] FIG. 7a illustrates an exemplary report 700 of the trends in
volume of traffic associated with selected applications or
protocols. The left column 702 lists application or protocol
identifiers, such as names, acronyms, version numbers, or others.
Columns 704, 706, 708, and 710 provide trend data for the
applications/protocols listed in the left column 702. For example,
an `Average Trended (Mb/s)` column 704 lists the 1 year trend in
average traffic flow in Mb/s for the respective
applications/protocol. An `Average CAGR (Mb/s)` column 706 lists
the compound average growth rate in Mb/s for the respective
applications/protocol. An `Average Trended (%)` column 708 lists
the 1 year trend in average traffic flow as a percentage for the
respective applications/protocol. An `Average CAGR (%)` column 706
lists the compound average growth rate as a percentage for the
respective applications/protocol.
[0075] FIG. 7b illustrates an exemplary city to city traffic flow
report 712. Other reports could show traffic flow from and to other
geographic areas such as countries, continents, or regions. In the
left column 714, the date(s) of the measurement or report are
listed. A `Source City` column 716 lists one or more source cities
from which traffic was sent. A `Destination City` column 718 lists
destination cities that the traffic was sent to from the respective
source cities in the `Source City` column 716. Cities may be
identified by name, abbreviation or other identifier. An `Average
Volume` column 720 indicates the traffic volume sent to the
respective destination city by the source city. In this embodiment,
the volume is indicated in units of Mb/s, but other units could be
used depending on the particular implementation.
[0076] The various reports described above and variations of those
reports can be used for marketing analysis, capacity planning,
security analysis, and others. For example, a city-to-city traffic
flow report (or other geographic regional flow report) can be used
to determine if traffic flow is overloading a gateway at a
particular city, or if extra bandwidth is available in the gateway.
The peer-to-peer distribution reports or peer-to-customer
distribution reports can suggest new services to provide or whether
to increase bandwidth or price for given customers or peers. A
report that shows the next hop autonomous systems and upstream
autonomous systems can indicate how far traffic is traveling to
reach its destination; if traffic is traveling very far (from a
topological, logical, or geographical basis) this could suggest the
need for additional bandwidth at different geographic or network
locations.
Exemplary Operations
[0077] FIG. 8 is a flowchart illustrating a traffic flow
aggregating, mapping and reporting algorithm 800 in accordance with
one embodiment. The algorithm 800 may be carried out by one or more
collector nodes and aggregator nodes discussed above or
alternatively on one or more other systems. The operations shown in
the algorithm 800 need not be carried out in the particular order
shown except where order is implied. The particular operations and
steps included in the operations may be rearranged, broken out, or
combined with others as may be suitable to a particular
implementation without straying from the scope of the
invention.
[0078] In a receiving operation 802, sampled packet data is
received. In one embodiment the sampled packet data includes data
from sampled packets, wherein the data is formatted according to a
predetermined standard format. Exemplary formats include, but are
not limited to, sFlow format, Netflow format, and cflowd format.
The sampled packet data may include, but is not limited to, source
IP address, destination IP address, port number(s), AS identifier,
network application or network protocol.
[0079] In a converting operation 804, the sampled packet data is
converted to a neutral format. In one embodiment, the converting
operation 804 involves identifying specified data in the sampled
packet data, such as source IP address, destination IP address,
port number(s), or AS number, and storing the specified data in
fields of a data structure organized in accordance with the neutral
format. The converting operation 804 may convert that data to other
units or encoding prior to storing the data in the data
structure.
[0080] In another receiving operation 806, one or more routing
tables are received from a network node. The routing table(s) set
forth a list of routes to specified destination IP addresses. The
routing table(s) may also include community data, such as city,
country, continent or region associated with the destination IP
addresses. The routing table(s) may further indicate AS numbers
and/or routers associated with the destination IP addresses.
Further still, the routing table(s) could indicate whether each of
the destination IP addresses are associated with a customer of the
NSP or not.
[0081] In mapping operation 808, destination 1P addresses from each
of the sampled packet data are mapped to destination prefixes in
the routing tables. In an associating operation 810, the community
data, AS data, and router data in the routing table(s) are
associated with the corresponding destination addresses.
[0082] In an aggregating operation 812, sample packet data and
related routing table data are aggregated from multiple network
interconnection points. In one embodiment, AS numbers in the sample
packet data are mapped to AS names found in an AS registry. In some
embodiments, customer attributes are mapped to corresponding sample
packet data to further enrich the data. The aggregating operation
812 derives one or more traffic flow measurements corresponding to
one or more traffic flows through the NSP network. For example, a
total flow measurement may indicate total traffic flow between one
or more peer networks or one or more AS networks. In addition,
traffic flows between geographic regions can be determined. Further
still, on-net and off-net traffic flow measurements may be
determined for one or more peer or AS networks. As yet another
example, traffic flows between peers and NSP customers and
noncustomers can be derived in the aggregating operation 812.
Numerous other traffic flow measurements may be derived in the
aggregating operation 812.
[0083] In a generating operation 814, one or more reports are
generated that show the traffic flow measurements, trends or other
traffic flow statistics associated with traffic flow. Exemplary
reports that the generating operation 814 could generate are shown
in FIGS. 3-7 and described above; however, the types, format, and
contents of reports are not limited to those shown above. In some
embodiments, reports are generated automatically, on demand or
both. The generating operation 814 may also send the reports to
specified individuals, such as network administrators or managers,
for analysis.
Exemplary Computing Device
[0084] FIG. 9 is a schematic diagram of a computing device 900 upon
which embodiments of the present invention may be implemented and
carried out. For example, one or more computing devices 900 may be
used to perform the sampling, collecting, aggregating, and
reporting operations described herein. As discussed herein,
embodiments of the present invention include various steps or
operations. A variety of these steps may be performed by hardware
components or may be embodied in machine-executable instructions,
which may be used to cause a general-purpose or special-purpose
processor programmed with the instructions to perform the
operations. Alternatively, the steps may be performed by a
combination of hardware, software, and/or firmware.
[0085] According to the present example, the computing device 900
includes a bus 901, at least one processor 902, at least one
communication port 903, a main memory 904, a removable storage
media 905, a read only memory 906, and a mass storage 907.
Processor(s) 902 can be any known processor, such as, but not
limited to, an Intel.RTM. Itanium.RTM. or Itanium 2.RTM.
processor(s), AMD.RTM. Opteron.RTM. or Athlon MP.RTM. processor(s),
or Motorola lines of processors. Communication port(s) 903 can be
any of an RS-232 port for use with a modem based dialup connection,
a 10/100 Ethernet port, a Gigabit port using copper or fiber, or a
USB port. Communication port(s) 903 may be chosen depending on a
network such a Local Area Network (LAN), Wide Area Network (WAN),
or any network to which the computing device 900 connects. The
computing device 900 may be in communication with peripheral
devices (not shown) such as, but not limited to, printers,
speakers, cameras, microphones, or scanners.
[0086] Main memory 904 can be Random Access Memory (RAM), or any
other dynamic storage device(s) commonly known in the art. Read
only memory 906 can be any static storage device(s) such as
Programmable Read Only Memory (PROM) chips for storing static
information such as instructions for processor 902. Mass storage
907 can be used to store information and instructions. For example,
hard disks such as the Adaptec.RTM. family of SCSI drives, an
optical disc, an array of disks such as RAID, such as the Adaptec
family of RAID drives, or any other mass storage devices may be
used.
[0087] Bus 901 communicatively couples processor(s) 902 with the
other memory, storage and communication blocks. Bus 901 can be a
PCI/PCI-X, SCSI, or USB based system bus (or other) depending on
the storage devices used. Removable storage media 905 can be any
kind of external hard-drives, floppy drives, `OMEGA.RTM. Zip
Drives, Compact Disc-Read Only Memory (CD-ROM), Compact
Disc-Re-Writable (CD-RW), Digital Video Disk-Read Only Memory
(DVD-ROM).
[0088] Embodiments of the present invention include various steps,
which are described in this specification. The steps may be
performed by hardware components or may be embodied in
machine-executable instructions, which may be used to cause a
general-purpose or special-purpose processor programmed with the
instructions to perform the steps. Alternatively, the steps may be
performed by a combination of hardware, software and/or
firmware.
[0089] Embodiments of the present invention may be provided as a
computer program product, which may include a machine-readable
medium having stored thereon instructions, which may be used to
program a computer (or other electronic devices) to perform a
process. The machine-readable medium may include, but is not
limited to, floppy diskettes, optical disks, compact disc read-only
memories (CD-ROMs), and magneto-optical disks; ROMs, random access
memories (RAMs), erasable programmable read-only memories (EPROMs),
electrically erasable programmable read-only memories (EEPROMs),
magnetic or optical cards, flash memory, or other type of
media/machine-readable medium suitable for storing electronic
instructions. Moreover, embodiments of the present invention may
also be downloaded as a computer program product, wherein the
program may be transferred from a remote computer to a requesting
computer by way of data signals embodied in a carrier wave or other
propagation medium via a communication link (e.g., a modem or
network connection).
[0090] Various modifications and additions can be made to the
exemplary embodiments discussed without departing from the scope of
the present invention. For example, while the embodiments described
above refer to particular features, the scope of this invention
also includes embodiments having different combinations of features
and embodiments that do not include all of the described features.
Accordingly, the scope of the present invention is intended to
embrace all such alternatives, modifications, and variations
together with all equivalents thereof.
* * * * *