Internet2 NetFlow: Weekly Reports: Week of 20091005

1. Introduction
2. Bulk TCP
   • Throughputs
   • Transfer Sizes
   • Durations
   • Selected Points from Distribution Graphs
   • Average Packet Sizes
   • Popular Applications
   • Fastest Measurement Flows with Unique AS Source and Destination
   • Fastest Non-Measurement Flows with Unique AS Source and Destination
   • Concurrency
3. Full Data Set
   • Aggregated Application Types
   • Detailed Application Types
   • IP Protocols Distribution
   • Average Packet Sizes
   • Important DSCP Values
   • ECN Traffic Statistics
   • Frequent Unidentified Ports

Introduction

During this week, data for the following day(s) were missing: Wednesday, Thursday, Friday, Saturday, Sunday. We multiplied all nominal quantities by 7/2 to estimate the amounts of various types of traffic. Percentages and distributions were not modified.

The data are split into two sections: bulk TCP data and the full data set. A "bulk TCP" flow is defined as a TCP flow that transferred more than 10MB of data. The first section only concerns these data. The second section studies the overall traffic composition.

All the numbers in this report are hyperlinked to plots that show their history (e.g., clicking on the percentage of octets of NNTP traffic will bring up a time-series plot that shows the history of this parameter).

Bulk TCP

During this week, bulk TCP traffic comprised 36.83% of octets and 18.93% of packets of the full data set traffic.

The distribution of bulk TCP throughputs is the most important piece of data in this report. Cumulative distribution function plots (1-CDF vs. throughput in bits/second) in semi-log and log-log scales are as follows:

Distribution of the amount of data transferred (in semi-log and log-log scale, 1-CDF vs. total trasfer size in octets) is presented below. It should be recognized that NetFlow collection mechanism is always configured so that flows (in the accounting sense) cannot last longer than a certain period of time. Therefore, the distribution of transfer sizes is to a certain extent skewed in the upper part.

The distribution of durations of bulk TCP flows (in seconds) is as follows (you may notice the cut-off phenomenon mentioned above):

The following table shows actual values from the above distribution plots that correspond to characteristic values (such as median, 90%, max, etc.).

Table 1. Selected Points from Distribution Graphs (Bulk TCPs)

We compute average packet size of each flow by dividing the number of octets in a flow by the number of packets. Distribution of average sizes of packets belonging to bulk TCP flows is as follows:

Table 2. Packet Sizes (Bulk TCP)

We show what applications transfer large amounts of data in the following table. Note that this is bulk TCP traffic only; full data set usage is presented in the next section.

Table 3. Aggregated Application Types (Bulk TCP)

The following are the fastest 10 measurement flows with unique source and destination AS numbers (i.e., for any given pair of source and destination AS numbers, no more than one fastest flow is shown).

Table 4. Fastest Bulk TCP Measurement Flows with Unique AS Source and Destination

The following are the fastest 10 non-measurement flows with unique source and destination AS numbers (i.e., for any given pair of source and destination AS numbers, no more than one fastest flow is shown). When unable to determine the application type, we give the source and destination port numbers.

Table 5. Fastest Bulk TCP Non-measurement Flows with Unique AS Source and Destination

We also compute the average concurrency of bulk TCP flows for the week (by adding durations of all captured flows and dividing the result by the by the duration of the week). This week's average number of concurrent bulk TCP flows: 1.676k.

Full Data Set

In addition to bulk TCP flows data, we provide statistics that characterize the overall composition of the complete data set (everything that transited the Abilene network this week).

The following table describes what kinds of traffic went through the network (multiple applications are aggregated into classes):

Table 6. Aggregated Application Types (Full Data Set)

This table is available additionally in the following more verbose version (no applications are aggregated into classes, but class composition is shown):

Table 7. Detailed Application Types (Full Data Set)

The following table summarizes use of most popular IPv4 protocols:

Table 8. IP Protocols Distribution (Full Data set)

We compute average packet size of each flow by dividing the number of octets in a flow by the number of packets. Distribution of (average) packet sizes is as follows:

Table 9. Packet Sizes (Full Data Set)

We only track DSCP values for which special treatment was defined by Internet2 QoS working group (and the default of DSCP=0):

Table 10. Important DSCP Values (Full Data Set)

We collect statistics about ECN-capable traffic:

Table 11. ECN-Capable Traffic

To facilitate detection of emerging applications, we present statistics about frequently encountered unidentified port numbers (no distinction is made in this table between TCP and UDP):

Table 12. Frequent Unidentified Ports