[Iccrg] TCP evaluation suite round-table

[Lachlan Andrew]

Greetings all,

(I sent this invitation to the  tmrg  mailing list awhile ago, but
there are probably people here who are also interested, but not on
that list.)

In order to compare different TCP algorithms, it is important to have
a standardized set of tests.  On 8-9 November, we're having a "round
table" discussion to make a small step in that direction.

We won't try to answer things like "what fairness metric is best", but
we can agree on some basic parameters.  The situation we're trying to
avoid is:

Group A finds that at  500Mbps,  flow 1 reaches 10% of its final
throughput after 30s
Group B finds that at  622Mbps,  flow 1 reaches  20% of its final
throughput after 20s

I'll try to have live video-conferencing via VRVS
<http://www.vrvs.org/Doc/faq.html> so that those who can't come in
person can still participate.  Unfortunately, our videoconferencing
room is small, and so physical attendance will probably be limited to
a dozen or so people.

**** If you're interested, please let me know in the next week or two. ***

As basic goals, I'd like to come away from the roundtable with:
- a set of bandwidths that are of interest, say 10, 155, 622, 2500 Mbps
- a set of buffer sizes that are of interest, like BDP or 16384 packets
- a set of distributions of RTT that are of interest
- an agreed notion of "convergence time"
           -- e.g., "the average over period x is within y% of the
final average
- an agreed notion of "time to converge to fairness"
           -- e.g., "the ratio of averages over period x is within y%
of the final ratio"
           -- should this metric depend on the final ratio achieved?
- an agreed notion of "intraflow variability"
          -- e.g., what timescales are of interest?
- an agreed set of traffic models for background traffic

Injong has added to that list:
- a measure of total link utilization
- fluctuation in utilization due to fluctuation in background traffic.
- What is the per-flow fair bandwidth share?

We've started some preliminary discussions on the TMRG mailing list to
sort out the "easy" issues, leaving time at the round table to discuss
the more contentious ones.  These are summarized at
<http://wil.cs.caltech.edu/mwiki/index.php?title=Benchmarking_parameters>.

It would be good to have a common set of scenarios which could be
tested by simulation, emulation and real networks.  Obviously, the
simulation is the most flexible, and so it may have a larger set of
tests, but we can at least simulate the emulated cases.

Ulterior motive: I'd like people also to simulate/emulate the
scenarios that can also be tested on WAN-in-Lab :)

I hope that we can get enough agreement from enough people to make
this effective.

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603