UDP-based reliable bulk data transfer

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

UDP-based reliable bulk data transfer

pupkin
Greetings,

I need to send large quantities of data reliably from a Zynq-based device to a PC.
I'm looking for a data transfer protocol which can get as close as possible to wire speed on a dedicated 1Gb ethernet link. It would be nice although not strictly required it it worked over arbitrary internet connections.

Currently I'm considering either porting http://enet.bespin.org/ or http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based protocol. I will need support for zero-copy with scatter-gather DMA.

Has anyone done this before? Any suggestions?

Regards
VP
Reply | Threaded
Open this post in threaded view
|

Re: ?==?utf-8?q? UDP-based reliable bulk data transfer

Nenad Pekez
Hi pupkin,

I am actually should do the same thing with the same SOC. And I am still considering different options. Have you tried to actually use any of these two protocols you posted (enet and udt)? I am really interested in how well they work and how close they get to 1Gb speed. And whether they can do the better job than TCP after all... 

Regards,
Nenad

-------- Original Message --------
Subject: [lwip-users] UDP-based reliable bulk data transfer
Date: Tuesday, April 25, 2017 09:16 CEST
From: pupkin <[hidden email]>
Reply-To: Mailing list for lwIP users <[hidden email]>
To: [hidden email]


 
Greetings,

I need to send large quantities of data reliably from a Zynq-based device to
a PC.
I'm looking for a data transfer protocol which can get as close as possible
to wire speed on a dedicated 1Gb ethernet link. It would be nice although
not strictly required it it worked over arbitrary internet connections.

Currently I'm considering either porting http://enet.bespin.org/ or
http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based protocol. I
will need support for zero-copy with scatter-gather DMA.

Has anyone done this before? Any suggestions?

Regards
VP




--
View this message in context: http://lwip.100.n7.nabble.com/UDP-based-reliable-bulk-data-transfer-tp29424.html
Sent from the lwip-users mailing list archive at Nabble.com.

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

 
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: UDP-based reliable bulk data transfer

goldsimon@gmx.de
In reply to this post by pupkin
I can't comment on the suggested protocols, but...

pupkin wrote:
> I need to send large quantities of data reliably from a Zynq-based device to
> a PC.

Although the name does not match: if you plan to implement a reliable
protocol over UDP, you could try and make it implement the new (git
master) API/protocol type "altcp", which cuts the link-time binding
between an application and TCP. Change requests to make this API fit
your needs are welcome!

> I will need support for zero-copy with scatter-gather DMA.

I'm in the progress of reviewing zero-copy scatter-gather I/O. It should
basically work, but specifically alignment of TX buffers remains to be
reviewed... Again, any input or change requests are welcome.

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: UDP-based reliable bulk data transfer

Patrick Klos-2
In reply to this post by pupkin
On 4/25/2017 3:16 AM, pupkin wrote:

> Greetings,
>
> I need to send large quantities of data reliably from a Zynq-based device to
> a PC.
> I'm looking for a data transfer protocol which can get as close as possible
> to wire speed on a dedicated 1Gb ethernet link. It would be nice although
> not strictly required it it worked over arbitrary internet connections.
>
> Currently I'm considering either porting http://enet.bespin.org/ or
> http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based protocol. I
> will need support for zero-copy with scatter-gather DMA.
>
> Has anyone done this before? Any suggestions?
>
> Regards
> VP

For what it's worth, I've done some tests on a dedicated 100Mb link and
they are quite reliable.  I suspect the same could be said about a
dedicated 1Gb link?  I ran a test once on a dedicated link between an
embedded system and a Windows computer with no lost packets for hours
(maybe even overnight - I forget the exact details).

That means that you should be able to implement a high speed UDP based
transfer protocol with very little overhead to make it "reliable".  I
would bet that creating a "window" of UDP packets (with sequence
numbers) will do what you want with the receiver sending back an ACK of
sorts for every so many packets or if it sees a sequence number missing.

I'm not sure about the zero-copy scatter-gather support - you'll
probably just have to try it.  The lwIP stack might scan your pbuf and
decide to copy the packet to a single buffer pbuf depending on the
capabilities of the ethernet interface.  Even so, your Zynq processor
will probably be able to handle the copies faster than a 1Gb link could
transfer the data?

Good luck!

Patrick Klos
Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: ?==?utf-8?q? UDP-based reliable bulk data transfer

Nenad Pekez
Hi Patrick,

did you actually port one of these 2 proposed protocols to your embedded system or implemented your own UDP reliable protocol? Also, did you get bitrates close to 100Mb?

Regards,
Nenad

-------- Original Message --------
Subject: Re: [lwip-users] UDP-based reliable bulk data transfer
Date: Tuesday, April 25, 2017 23:52 CEST
From: Patrick Klos <[hidden email]>
Reply-To: Mailing list for lwIP users <[hidden email]>
To: [hidden email]
References: <[hidden email]>


 
On 4/25/2017 3:16 AM, pupkin wrote:

> Greetings,
>
> I need to send large quantities of data reliably from a Zynq-based device to
> a PC.
> I'm looking for a data transfer protocol which can get as close as possible
> to wire speed on a dedicated 1Gb ethernet link. It would be nice although
> not strictly required it it worked over arbitrary internet connections.
>
> Currently I'm considering either porting http://enet.bespin.org/ or
> http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based protocol. I
> will need support for zero-copy with scatter-gather DMA.
>
> Has anyone done this before? Any suggestions?
>
> Regards
> VP

For what it's worth, I've done some tests on a dedicated 100Mb link and
they are quite reliable. I suspect the same could be said about a
dedicated 1Gb link? I ran a test once on a dedicated link between an
embedded system and a Windows computer with no lost packets for hours
(maybe even overnight - I forget the exact details).

That means that you should be able to implement a high speed UDP based
transfer protocol with very little overhead to make it "reliable". I
would bet that creating a "window" of UDP packets (with sequence
numbers) will do what you want with the receiver sending back an ACK of
sorts for every so many packets or if it sees a sequence number missing.

I'm not sure about the zero-copy scatter-gather support - you'll
probably just have to try it. The lwIP stack might scan your pbuf and
decide to copy the packet to a single buffer pbuf depending on the
capabilities of the ethernet interface. Even so, your Zynq processor
will probably be able to handle the copies faster than a 1Gb link could
transfer the data?

Good luck!

Patrick Klos
Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

 
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: ?==?utf-8?q? UDP-based reliable bulk data transfer

Patrick Klos-2
On 4/25/2017 6:23 PM, Nenad Pekez wrote:
> Hi Patrick,
>
> did you actually port one of these 2 proposed protocols to your
> embedded system or implemented your own UDP reliable protocol? Also,
> did you get bitrates close to 100Mb?

The point I was trying to make was that you barely need to do anything
to make a dedicated link reliable.  Some people may disagree, but I ran
a raw frame test for hours without loss.  That indicates to me that the
link itself is quite reliable.  Adding a sequence number, window and ACK
mechanism can be done quite simply. I was also saying that it would be a
simple protocol to implement, but I haven't implemented it myself (at
least not as a high speed data transfer protocol).

We had ENET on our device for other reasons, but have since removed it
and replaced it with a simpler "reliable UDP protocol" we wrote ourselves.

Patrick Klos
Klos Technologies, Inc.

>> On 4/25/2017 3:16 AM, pupkin wrote:
>> > Greetings,
>> >
>> > I need to send large quantities of data reliably from a Zynq-based
>> device to
>> > a PC.
>> > I'm looking for a data transfer protocol which can get as close as
>> possible
>> > to wire speed on a dedicated 1Gb ethernet link. It would be nice
>> although
>> > not strictly required it it worked over arbitrary internet connections.
>> >
>> > Currently I'm considering either porting http://enet.bespin.org/ or
>> > http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based
>> protocol. I
>> > will need support for zero-copy with scatter-gather DMA.
>> >
>> > Has anyone done this before? Any suggestions?
>> >
>> > Regards
>> > VP
>>
>> For what it's worth, I've done some tests on a dedicated 100Mb link and
>> they are quite reliable. I suspect the same could be said about a
>> dedicated 1Gb link? I ran a test once on a dedicated link between an
>> embedded system and a Windows computer with no lost packets for hours
>> (maybe even overnight - I forget the exact details).
>>
>> That means that you should be able to implement a high speed UDP based
>> transfer protocol with very little overhead to make it "reliable". I
>> would bet that creating a "window" of UDP packets (with sequence
>> numbers) will do what you want with the receiver sending back an ACK of
>> sorts for every so many packets or if it sees a sequence number missing.
>>
>> I'm not sure about the zero-copy scatter-gather support - you'll
>> probably just have to try it. The lwIP stack might scan your pbuf and
>> decide to copy the packet to a single buffer pbuf depending on the
>> capabilities of the ethernet interface. Even so, your Zynq processor
>> will probably be able to handle the copies faster than a 1Gb link could
>> transfer the data?
>>
>> Good luck!
>>
>> Patrick Klos
>> Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: UDP-based reliable bulk data transfer

pupkin
I agree, dedicated link is generally quite reliable. I'm more concerned about SW timing on both sides of the link. At 1Gb there is < 20us per packet. Getting close to capacity requires careful attention to throttling and flow control. If the rate is too slow I will under-utilize the bandwidth, if it is too fast, I will start losing packets. If the feedback loop is too tight, the flow is going to be jerky, if it is too sloppy, it will take longer time to settle. I've seen the code to do these kind of things properly and it was quite involved, I'd rather not reinvent this particular wheel if I can.

I'm looking at enet and it uses non-blocking UDP sockets with select().  I'm running lwIp 1.4.1 on bare metal with NO_SYS=1 and I'm trying to find a way to implement select() for the tx side. Unless I'm missing something obvious, there appears to be no mechanism for the driver to signal tx completion, is that right?

Regards,
VP

Patrick Klos-2 wrote
On 4/25/2017 6:23 PM, Nenad Pekez wrote:
> Hi Patrick,
>
> did you actually port one of these 2 proposed protocols to your
> embedded system or implemented your own UDP reliable protocol? Also,
> did you get bitrates close to 100Mb?

The point I was trying to make was that you barely need to do anything
to make a dedicated link reliable.  Some people may disagree, but I ran
a raw frame test for hours without loss.  That indicates to me that the
link itself is quite reliable.  Adding a sequence number, window and ACK
mechanism can be done quite simply. I was also saying that it would be a
simple protocol to implement, but I haven't implemented it myself (at
least not as a high speed data transfer protocol).

We had ENET on our device for other reasons, but have since removed it
and replaced it with a simpler "reliable UDP protocol" we wrote ourselves.

Patrick Klos
Klos Technologies, Inc.

>> On 4/25/2017 3:16 AM, pupkin wrote:
>> > Greetings,
>> >
>> > I need to send large quantities of data reliably from a Zynq-based
>> device to
>> > a PC.
>> > I'm looking for a data transfer protocol which can get as close as
>> possible
>> > to wire speed on a dedicated 1Gb ethernet link. It would be nice
>> although
>> > not strictly required it it worked over arbitrary internet connections.
>> >
>> > Currently I'm considering either porting http://enet.bespin.org/ or
>> > http://udt.sourceforge.net/ to lwIp or rolling my own UDP-based
>> protocol. I
>> > will need support for zero-copy with scatter-gather DMA.
>> >
>> > Has anyone done this before? Any suggestions?
>> >
>> > Regards
>> > VP
>>
>> For what it's worth, I've done some tests on a dedicated 100Mb link and
>> they are quite reliable. I suspect the same could be said about a
>> dedicated 1Gb link? I ran a test once on a dedicated link between an
>> embedded system and a Windows computer with no lost packets for hours
>> (maybe even overnight - I forget the exact details).
>>
>> That means that you should be able to implement a high speed UDP based
>> transfer protocol with very little overhead to make it "reliable". I
>> would bet that creating a "window" of UDP packets (with sequence
>> numbers) will do what you want with the receiver sending back an ACK of
>> sorts for every so many packets or if it sees a sequence number missing.
>>
>> I'm not sure about the zero-copy scatter-gather support - you'll
>> probably just have to try it. The lwIP stack might scan your pbuf and
>> decide to copy the packet to a single buffer pbuf depending on the
>> capabilities of the ethernet interface. Even so, your Zynq processor
>> will probably be able to handle the copies faster than a 1Gb link could
>> transfer the data?
>>
>> Good luck!
>>
>> Patrick Klos
>> Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users