README.md
1### viamillipede:
2Fast, resiliant, network transparent pipe tranport.
3
4Viamillipede is client/server program built to improve pipe transport across networks by using multiple TCP sessions. It demultiplexes stdin into multiple buffered TCP connectons and then terminates the connections into stdout on another host. Order is guaranteed and the pipe is transparent to the source/sink programs. It is as simple to use as Netcat and can generate large throughputs.
5
6#### Problems With existing approaches:
7TCP connections are friable and IP is best effort to preserve its ecomony. TCP was not engineered for resiliance or longevity for a single flow. Relying on a single TCP connection to succeed or perform is not defendable in software.
8### typical pathology:
9 + `tar cf - / | /bin/dd obs=1m 2> /dev/null | /bin/dd obs=1m 2> /dev/null| /usr/local/bin/pipewatcher | ssh mal "tar xf- "`
10 + double serial penalty, note latent mistake causing 1B reads
11 + Extra pipe steps impose 'serial resistance' to throughput.
12 + desperately superstitious optimizations are premature and not generally applicable
13 + ssh is not the tool for every situation
14 + a fixed pipeline is not tuned for any system
15 + SMP ignorance, Cpu's either lonely forever or hotspotted.
16 + underused tx/rx interrupt endpoints, pcie lanes, NIC channel workers, memory lanes and flow control.
17 + networks are tuned against hot single tcp connections; that is hard to fix
18 + poor mss window scaling. Congestion controls aggressively collapse when network conditions are not pristine.
19 + large bandwidth latency product vs. contended lans; both penalized due to 'impedence mismatches')
20 + Poor buffer interactions eg: "Shoe shining" delays.
21 + NewReno alternatives are not often acceptable.
22 + Flows are stuck on one L1/L2/L3 path. This defeats the benefits of aggregation and multi-homed connections.
23 + Alternate Scatter gather transport is usually not pipe transparent and difficult to set up; eg: pftp, bittorrent, pNFS
24 + Your NOC will do maintenance in intervals shorter than your network transport windows.
25
26#### Goals and Features of viamillipede:
27+ Provide:
28 + Fast transparent delivery of long lived pipes across typical networks.
29 + Runtime SIGINFO inspection of traffic flow.`( parallelism, worker allocation, total throughput )`
30 + Resilience against dropped TCP connections and read links.
31+ Increase traffic throughput by:
32 + Using parallel connections that each vie for survival against adverse network conditions.
33 + Using multiple destination addresses with LACP/LAGG or separate Layer 2 adressing.
34 + Permit adequate buffering to prevent shoe shining.
35 + Return traffic is limited to ACK's to indicate correct operations
36+ Specified Traffic Shaping:
37 + Steer traffic to preferred interfaces.
38 + Use aggregated link throughput in a user specified sorted order.
39+ Future plans `(*)`
40 + Make additional 'sidechain' compression/crypto pipeline steps parallel.
41 + hard due to unpredictable buffer size dynamics
42 + sidechains could include any reversable pipe transparent program
43 + gzip, bzip2
44 + openssl
45 + rot39, od
46 + xdr/rpc marshalling for architecture independence
47 + serializing a struct is not ideal
48 + reverse channel capablity
49 + millipedesh ? millipederpc?
50 + specify rx/tx at the same time + fifo?
51 + is this even a good idea? Exploit generator?
52 + provide proxy trasport for other bulk movers: rsync, ssh OpenVPN
53 + error feedback path more that ack
54 + just run two tx/rx pairs?
55+ Error resiliance: TCP sessions are delicate things
56 + Restart broken TCP sessions on alternate transport automatically.
57 + Bypass dead links at startup; retry them later as other network topology changes are detected.
58 + self tuning worker count, side chain, link choices and buffer sizes, Genetic optimization topic? `(*)`
59 + checksums, not needed, but it's part of the test suite, use the 'checksums' transmitter flag to activate
60 + error injection via tx chaos <seed> option - break the software in weird ways, mostly for the test suite
61 + programmable checkphrase uses a 4 character checkphrase to avoid confusion rather than provide strong authenticaion
62
63+ Simple to use in pipe filter programs
64 + Why hasn't someone done this before?
65 + IBM's parallel ftp for SP2 z/os, bittorrent: not pipe transparent.
66 + Hide complexity of parallel programming model from users.
67
68`(*)` denotes work in progress, because "hard * ugly > time"
69#### Examples:
70
71### Simple operation
72 + Configure receiver with rx <portnum>
73 ` viamillipede rx 8834 `
74 + Configure transmitter with tx <receiver_host> <portnum>
75 ` echo "Osymandias" | viamillipede tx host1.yoyodyne.com 8834 `
76
77### Options:
78+ `rx <portnum> ` Become a reciever.
79+ `tx <host> <portnum> ` Become a transmitter and add transport graph link toward an rx host. Optionally provide tx muliple times to inform us about transport alternatives. We fill tcp queues on the first entries and then proceed down the list if there is more input than link throughput. It can be helpful to provide multiple ip aliases to push work to different nic channel workers and balance traffic across LACP hash lanes. Analysis of the network resources shold inform this graph. You may use multiple physical interfaces by chosing rx host ip's that force multiple routes.
80 + The source and destination machine may have multiple interfaces and may have:
81 + varying layer1 media ( ethernet, serial, Infiniband , 1488, Carrier Pidgeon, insects, ntb)
82 + varying layer2 attachment ( vlan, aggregation )
83 + varying layer3 routes ( multihomed transport )
84 + Use the preferred link. Should you saturate it, fill the next available link.
85```
86 tx host1.40g-infiniband.yoyodyne.com\
87 tx host1a.40g-infiniband.yoyodyne.com\
88 tx host1.internal10g.yoyodyne.com\
89 tx host1.internal1g.yoyodyne.com\
90 tx host1.expensive-last-resort.yoyodyne.com
91
92```
93+ verbose <0-20+>,
94 + transmitter or receiver
95 ` viamillipede rx 8834 verbose 5 `
96+ threads <1-16> control worker thread count
97 + (only on transmitter)
98 ` viamillipede tx foreign.shore.net 8834 threads 16 `
99+ checksum (only on transmitter) with threads
100 ` viamillipede tx foreign.shore.net 8834 checksum `
101+ chaos <clock divider> add error via chaos
102 + transmitter only
103 + periodically close sockets to simulate real work network trouble and tickle recovery code
104 + deterministic for how many operations to allow before a failure
105 + `viamillipede tx localhost 12334 chaos 1731`
106+ checkphrase <char[4]> provide lightweight guard agaist a stale or orphaned reciever,
107 + not a security mechanism
108 Transmitter and Reciever word[4] must match exactly.
109
110### Use case with zfs send/recv:
111+ Configure transmitter with tx <receiver_host> <portnum> and provide stdin from zfs send
112 ` zfs send dozer/visage | viamillipede tx foriegn.shore.net 8834 `
113+ Configure receiver with rx <portnum> and ppipe output to zfs recv
114 `viamillipede rx 8834 | zfs recv trinity/broken `
115
116### Use outboard crypto: viamillipede does not provide any armoring against interception or authentication
117+ use ipsec/vpn and live with the speed
118+ provide ssh tcp forwarding endpoints
119 + from the tx host:` ssh -N -L 12323:localhost:12323 tunneluser@rxhost `
120 + use mutiple port instances to get parallelism
121 * use a trusted peers tcp encapuslation tunnel to offload crypto
122+ use openssl in stream, take your crypto into your own hands
123 + ` /usr/bin/openssl enc -aes-128-cbc -k swordfIIIsh -S 5A `
124 + choose a cipher that's binary transparent
125 + appropriate paranoia vs. performance up to you
126 + enigma, rot39, morse?
127
128
