SSH over UDP: How to swallow an elephant

(this is a rehash of a message I sent to a private list.  Those of you who aren’t interested in theoretical network discussion, read no further.)

I just got back from a very successful network rollout (100mbps in 4 business days, to a government building no less) that ultimately failed because of the limitations of TCP and latency – the servers to which the data was being sent were too far away to have reasonable throughput.  After some thought, it seems that my problems could have been resolved if I was just streaming dumb packets (UDP) instead of TCP streams that required the ~150ms ACK round trip.  I know I can’t be the first to encounter this, and it would seem that in these edge cases some TCP-over-UDP method of packet delivery would be suitable despite the hackiness of the solution.

My goal: move large packet streams over mildly (max 5%) lossy network elements with long latency (max 600ms) but high bandwidth.  A 100mbps pipe is worthless if you’re trying to move a 10gig file across a trans-continental (or further) link with any realistic latency or even the slightest packet loss.  But if you’re prepared to re-send some data and hold things for a few seconds in buffer, it seems like a vast speed improvement could be obtained.

It seems that UDP would be an ideal way to transmit large blocks of data, with a very large buffer at each end and a non-sequential retransmission strategy for lost packets. This would not be simply re-writing TCP over UDP, with the inherent ACK path for each datagram.  It would be a larger re-write, with large blocks of datagrams collected and ACK’ed in reply packets, with re-transmits possible within the buffer pool and not just at the very end of the buffer pool.  This seems do-able, given that most modern machines have VERY large memory capabilities and the network is typically the weak link.  This might lead to “bursty” output while waiting for blocks to complete during retransmits, but I’d think that the output would be much larger in size over the same time period as that of a comparative TCP stream.

The goal here is not requiring kernel access, and no demand for any control over network elements in the path.  This should be 100% user-land accessible for installation on generic UNIX style hosts without root permissions.

Yes, I have done a bit of Googling on this, but there’s a flood of responses.  Iproxy seems to be for multicast.  bbftp is interesting with the multi-stream method, but is limited to file transfer and not generic TCP connections.  atou seems to require heavy kernel modifications on each side.  I found some ssh-over-UDP sites that are blurry in their details, and they seem to not be sophisticated at all – still blocking at the point where ACKs are requested back on a packet-by-packet basis, and not blasting out huge piles of data and then selectively backfilling if there are drops reported by the receiver.

It would seem to me that SSH would be a great place to shim this in.  The number of services that can run over SSH is growing, and the tunnel capability (both UDP and TCP) and port re-direction seem to be an already versatile set of methods that would benefit from such a shim component to increase bandwidth.  It also has the advantage of having native file transfer (scp) that is well-supported.

Anyone have any ideas on research on this that has already been done, or shim layers that already exist to take advantage of UDP’s fill-the-pipe methodology?  Looks like some people have done experiments, but the data is obscured (paywall) and/or it is unclear that what I’m looking for has actually been attempted.

JT

notes:
http://www.csm.ornl.gov/~dunigan/net100/bulk.html
http://www.csm.ornl.gov/~dunigan/net100/atou.html
http://doc.in2p3.fr/bbftp/
http://horms.net/projects/iproxy/
http://code.google.com/p/udptunnel/
http://publications.lib.chalmers.se/cpl/record/index.xsql?pubid=123799

update: Several people replied to me privately, with these for-pay options which seem to do pretty much what I’m talking about.  A free (open source) variation of this embedded in SSH might be a game-changer.  This seems well within the range of a graduate CS project.

http://www.dataexpedition.com/
http://www.asperasoft.com/images/Aspera_Technology_Capabilities_2010.pdf

Tags: , , , , ,

8 Responses to “SSH over UDP: How to swallow an elephant”

  1. Dave Täht Says:

    Are you moving major amounts of data across a single stream? If it is multiple streams, perhaps sctp might help. If it is a large dataflow that has to go to multiple locations, uftp is pretty neat.

    http://www.tcnj.edu/~bush/uftp.html

    I think it does FEC, but haven’t looked at it recently.

    Otherwise, UDP does not seem like a good idea, as anything that you do over it is likely to break something, somewhere else, in the network.

    Jim Gettys notes that “bufferbloat” is breaking TCP across the Internet by doing too much intermediate buffering: http://gettys.wordpress.com/category/bufferbloat/

  2. jtodd Says:

    It’s a single stream – several giant files that need to move quickly from point A to point B.
    UFTP looks interesting. It has a single host unicast option, which I’ll try out. I’d still love to see UDP supported in SSH, since that would be widely available and easily adaptable to other features of SSH (port forwarding, scp, etc.)

    What client applications support SCTP?

  3. jtodd Says:

    Another solution (commercial, hardware-based) that was suggested several times in private conversation was the platform from http://www.storagedna.com/

  4. jtodd Says:

    For other people who find this post, here are some additional pointers people sent to me. I’ve not had time to test these TCP over UDP perl scripts:

    Duat:
    http://code.google.com/p/duat/

    TCP over UDP:
    http://www.jankratochvil.net/project/tcpoverudp/

  5. jzp Says:

    UFTP has some bootstrapping issues. Our fixes to provide some CTS-style handshaking may or may not be acceptade by the maintainer.

  6. jtodd Says:

    I was also told about this: HyperIP, which is available as a VMWare image:

    http://www.vmware.com/appliances/directory/va/155643

    I did experiment a bit with the Aspera software last week on a licensed version, but apparently it was licensed only for the Mac “GUI client” and did not include a license for the “command line” client. WTF? I really, really hate licensed software head-pounding. There are options only availble in the command-line version that aren’t available in the GUI version. Anyway, it worked well enough for a single stream – we reached 80mbps, which was maximum for the upstream path (two stream-balanced 80mbps links.) But we could have gone to 160mbps if we were able to use the command line version, which allows files to be split and sent as multiple chunks, each using a different UDP stream. But the cost for this stuff is eye-popping – it’s not that difficult. This should be free, since it is a common problem with multi-hundred megabit and gigabit (and 10-gigabit) networking becoming commonplace now with transcontinental ACK latency, or steady low-level errors on radio/satellite connections that cause TCP to be useless.

    I still want to see this embedded in SSH. SSH would have only a few additional options:

    -U Send using UDP.
    -Q[N] Aggressiveness level for transmission – how much packet loss is tolerated before back-off, in percent. Default: 1 (or 1% packet loss)

    SCP could have more features, but rsync already provides a lot of the file-handling kind of stuff via SSH calls (which would be UDP-enhanced) so SCP would only need a few tweaks. Having options like this in SCP would be awesome:

    -U Send using UDP
    -s: Split files. Send Y chunks, with this session being chunk X. (i.e.: -s1:3 -s2:5 etc.) When last chunk is received, remote system will reassemble file.
    -f Freshen files. If partial files have been received from prior aborted transmissions, delete segments on remote system and start anew. Applies to full file or chunk specified with “-:
    -Q[N] Aggressiveness level for transmission – how much packet loss is tolerated before back-off, in percent. Default: 1 (or 1% packet loss)

    The “-l” command already exists to limit bandwidth.

  7. jtodd Says:

    Almost a year later, I’ve found another project which seems to fit the bill perfectly. It’s the “High Performance SSH” aka HPN-SSH project, run by the Pittsburgh Supercomputing Center (PSC). It appears fairly well maintained. Why isn’t this code part of OpenSSH? It seems to be a no-brainer if it does what it says it does. Right now, I don’t have the time to test, but in the next week or so that may change.

    http://www.psc.edu/networking/projects/hpn-ssh/

  8. jtodd Says:

    Update on the HPN-SSH tests: we tried this at TED 2012, or at least Bog tried it. He was unable to get the kernel tuned with MacOS 10.7 in a way that moved files at greater than (IIRC) around 20mbps. Clearly, not a win, but this may have been due to MacOS 10.7 changing some of the parameters to rely upon each other in the kernel that previously had not. The instructions for MacOS were for an older version of the software, so this was not a big win and the Aspera software was used along with a 1gbps uplink provided by CENIC (thanks, CENIC!)

    I have not had any time to really look at this in more detail, because I’m pretty much swamped with the day job. But still the question remains: why isn’t this part of SSH as a default? This would speed up transfers immensely. I have yet to hear anyone come up with a reason that would be suitably significant to block development on a project like this.

    I’ve also come up with another idea, which I had while discussing this issue with a CDN vendor yesterday: why isn’t there any CDN that has created a “helper” app that would run on the local user’s computer and intercept packets destined for certain IP addresses, and then shim them through a UDP accelerator? This could run almost entirely transparently, and if it had appropriate fallback mechanisms it would work under any network conditions, even where UDP was blocked. (It wouldn’t be able to help where proxies were configured, but that’s an edge case.) For game or media delivery, this would be _amazing_. This would truly be a “viral” application, meaning that anyone who used that content network could encourage users to use the helper app, and they would get the amazing performance transparently. In the game industry, that kind of thing spreads like wildfire in the community – whoever gets better performance on the network, wins more. So why no magic from a CDN? Well, chalk this up to yet another idea of mine that will make someone else millions of dollars that I won’t have time to do – it’s a long and sad list. (It’s also an idea that I’m sure someone else has already had, but… I don’t see it on the market, nor have I ever seen it, so I’ll call it “mine”.)

Leave a Reply