Desired - reference netconn_write() to external memory

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Desired - reference netconn_write() to external memory

Alan L
I've 'inherited' an application where I need to pull GB's of data out of a tool
running on a AT91SAM7X256.  
Currently I simply play with...
- the data is read from external memory and buffered in RAM
- netconn_write() is called with COPY enabled
- the loop repeats

I don't have the processor memory to optimally up the pbuf and memory lwIP
options for a large data transfer application.

What came to mind today was the possibility to call netconn_write() with
NO_COPY, referencing an external memory location instead of pointing to an
internal buffer.

Any thoughts? - not worth it?
Gut feeling - would it be a large code rewrite/addition?
Have you lived with or solved similar problems doing large data transfers?

Thanks,
Alan



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Desired - reference netconn_write() to external memory

Kieran Mansley
On Tue, Oct 10, 2006 at 04:01:56PM +0000, Alan Lamphier wrote:

> What came to mind today was the possibility to call netconn_write() with
> NO_COPY, referencing an external memory location instead of pointing to an
> internal buffer.

If you can guarantee that the data will still be accessible should TCP
need to retransmit the packet some time later it should be fine to
call without requesting a copy.  The copy is necessary as normally
when the write call returns the caller is free to do what it likes
with the buffer that it has just passed in.  TCP may need to access
that memory many times, and so it has to copy it to make sure it's
still available.

I'm not sure about your hardware but depending on the speed of the
external memory compared to RAM, doing the copy may surprisingly help
performance.  If the external memory was slow, and TCP tried to read
from it many times (e.g. checksum, write to network, and retransmit)
it might have worked out faster to just read it from the slow external
memory into faster RAM once and then TCP can access it from there.
Something to consider anyway.

Hope that helps,

Kieran


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Desired - reference netconn_write() to external memory

Caglar Akyuz-2
In reply to this post by Alan L
Alan Lamphier yazmış:

> I've 'inherited' an application where I need to pull GB's of data out of a tool
> running on a AT91SAM7X256.  
> Currently I simply play with...
> - the data is read from external memory and buffered in RAM
> - netconn_write() is called with COPY enabled
> - the loop repeats
> I don't have the processor memory to optimally up the pbuf and memory lwIP
> options for a large data transfer application.
>
> What came to mind today was the possibility to call netconn_write() with
> NO_COPY, referencing an external memory location instead of pointing to an
> internal buffer.
>  
What kind of external memory do you mean? SAM7X does not have any
external memory bus. Are you using a parallel memory from GPIO?
> Any thoughts? - not worth it?
> Gut feeling - would it be a large code rewrite/addition?
> Have you lived with or solved similar problems doing large data transfers?
>
>  
Actually I'm unable to get your problem. If you are facing slow transfer
rate and want to increase it, then try to give some extra information.
For instance, I couldn't exceed 5 KBytes per second transfer rate with
TCP on a SAM7X while I can reach 800 KBytes per second with UDP one the
same machine.

> Thanks,
> Alan
>
>
>
> _______________________________________________
> lwip-users mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/lwip-users
>
>
>
>  



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Desired - reference netconn_write() to external memory

Alan L
Çağlar AKYÜZ <caglar@...> writes:

> > What came to mind today was the possibility to call netconn_write() with
> > NO_COPY, referencing an external memory location instead of pointing to an
> > internal buffer.
> >  
> What kind of external memory do you mean? SAM7X does not have any
> external memory bus. Are you using a parallel memory from GPIO?

In my current tests, I am retrieving files from external DataFlash.  Eventually
I will be retrieving bulk data from external NAND memory.

> Actually I'm unable to get your problem. If you are facing slow transfer
> rate and want to increase it, then try to give some extra information.
> For instance, I couldn't exceed 5 KBytes per second transfer rate with
> TCP on a SAM7X while I can reach 800 KBytes per second with UDP one the
> same machine.

I seem to be in the ~310KBytes per second range with TCP right now, but I need
yet to test with larger file size transfers.  My problem is that the limited
processor RAM makes it difficult to optimize lwIP options for large data
transfers, from what I understand.  I had been calling netconn_write(), with the
COPY option as the passing data buffer is reloaded with new data from DataFlash
just after the netconn_write() call - so the original data buffer would not be
intact for retransmissions.
 
My thought was the possibility to call netconn_write() with the NO_COPY option
and instead of passing a pointer to a RAM data buffer, pass a starting DataFlash
address.  Retransmissions could be re-read from DataFlash instead of holding
them in RAM.

Alan



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Caglar Akyuz-2
Alan Lamphier yazmış:

> Çağlar AKYÜZ <caglar@...> writes:
>
>  
>>> What came to mind today was the possibility to call netconn_write() with
>>> NO_COPY, referencing an external memory location instead of pointing to an
>>> internal buffer.
>>>  
>>>      
>> What kind of external memory do you mean? SAM7X does not have any
>> external memory bus. Are you using a parallel memory from GPIO?
>>    
>
> In my current tests, I am retrieving files from external DataFlash.  Eventually
> I will be retrieving bulk data from external NAND memory.
>  
Then, I guess, you use SPI interface to transfer data from DataFlash to
RAM. If this is the case I don't understand how you are going to
transfer from Dataflash directly.

>> Actually I'm unable to get your problem. If you are facing slow transfer
>> rate and want to increase it, then try to give some extra information.
>> For instance, I couldn't exceed 5 KBytes per second transfer rate with
>> TCP on a SAM7X while I can reach 800 KBytes per second with UDP one the
>> same machine.
>>    
>
> I seem to be in the ~310KBytes per second range with TCP right now, but I need
> yet to test with larger file size transfers.  My problem is that the limited
> processor RAM makes it difficult to optimize lwIP options for large data
> transfers, from what I understand.  I had been calling netconn_write(), with the
> COPY option as the passing data buffer is reloaded with new data from DataFlash
> just after the netconn_write() call - so the original data buffer would not be
> intact for retransmissions.
>  
Actually I think this will not improve the performance of MAC. Because
ACK'ing of TCP frames is one of the major drawbacks which degrades
performance ( thanks to Delayed ACK algorithm!). Secondly I agree with
you. You should use NOCOPY option. Try to segment your buffer into 2,
use one for Dataflash update while the other one is for MAC
transmission. Then, change the pointers and so on.
>  
> My thought was the possibility to call netconn_write() with the NO_COPY option
> and instead of passing a pointer to a RAM data buffer, pass a starting DataFlash
> address.
What is the interface of your Dataflash? I'm still getting hard time to
realize how you will address Dataflash directly.

Çağlar


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Kieran Mansley
On Wed, 2006-10-11 at 16:25 +0300, Çağlar AKYÜZ wrote:

> Actually I think this will not improve the performance of MAC. Because
> ACK'ing of TCP frames is one of the major drawbacks which degrades
> performance ( thanks to Delayed ACK algorithm!).

Only in a very limited set of circumstances will the delayed ACK
algorithm result in poorer performance.  Namely, when the TCP window has
been set to a very small value.  Changing this configuration option will
remove any bottleneck that the delayed ACK algorithm is imposing.  There
are TCP extensions such as the "quickack" mode in linux's TCP stack that
stop the use of delayed ACK during the slow start phase of TCP
connections, but the performance difference that this is likely to lead
to in lwIP is minimal.

ACKing of packets can be arguably said in fact to increase performance,
as without it the sender couldn't discover the maximum rate it could
send without packet loss, and so if it didn't wish to lose data would
have to be more conservative with how fast it sent packets.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Desired - reference netconn_write()to external memory

Alan L
In reply to this post by Caglar Akyuz-2
Çağlar AKYÜZ <caglar@...> writes:

> Then, I guess, you use SPI interface to transfer data from DataFlash to
> RAM. If this is the case I don't understand how you are going to
> transfer from Dataflash directly.

Yes, it would be SPI interface to the DataFlash.  It would not be a 'direct'
interface (sorry for the confusion) - the TCP handler would require
rewrite/additional code such that if it was passed a DataFlash address, it would
appropriately retrieve the data from DataFlash as needed.  I'm sure it could be
done, but most likely a big code change and it seems no gain for the trouble.

> Actually I think this will not improve the performance of MAC. Because
> ACK'ing of TCP frames is one of the major drawbacks which degrades
> performance ( thanks to Delayed ACK algorithm!). Secondly I agree with
> you. You should use NOCOPY option. Try to segment your buffer into 2,
> use one for Dataflash update while the other one is for MAC
> transmission. Then, change the pointers and so on.
> >  

Thank you for your ideas,
Alan





_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Kieran Mansley
In reply to this post by Kieran Mansley
On Wed, 2006-10-11 at 14:39 +0100, Kieran Mansley wrote:
> On Wed, 2006-10-11 at 16:25 +0300, Çağlar AKYÜZ wrote:
>
> > Actually I think this will not improve the performance of MAC. Because
> > ACK'ing of TCP frames is one of the major drawbacks which degrades
> > performance ( thanks to Delayed ACK algorithm!).
>
> Only in a very limited set of circumstances will the delayed ACK
> algorithm result in poorer performance.  Namely, when the TCP window has
> been set to a very small value.

Should have said "TCP window or send queue length" there.  Oops.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Caglar Akyuz-2
In reply to this post by Kieran Mansley
Kieran Mansley yazmış:

> On Wed, 2006-10-11 at 16:25 +0300, Çağlar AKYÜZ wrote:
>
>  
>> Actually I think this will not improve the performance of MAC. Because
>> ACK'ing of TCP frames is one of the major drawbacks which degrades
>> performance ( thanks to Delayed ACK algorithm!).
>>    
>
> Only in a very limited set of circumstances will the delayed ACK
> algorithm result in poorer performance.  Namely, when the TCP window has
> been set to a very small value.  Changing this configuration option will
> remove any bottleneck that the delayed ACK algorithm is imposing.  
I tried to increase recieve window but it always resulted in a poorer
performance with my SAM7X. ( Maybe I'm too low in memory?) . is TCP
window size related to somewhat pbufs? I don't know if I have to
increase number of buffers as well.
> There
> are TCP extensions such as the "quickack" mode in linux's TCP stack that
> stop the use of delayed ACK during the slow start phase of TCP
> connections, but the performance difference that this is likely to lead
> to in lwIP is minimal.
>  
Actually I didn't mean to say that problem is related to LWiP. I wanted
to say that delayed ACK algorithm is slowing TCP connections in our case
and my Windows machine do not provide me any way to change this setting.
( Thanks to Linux as you noted. )
> ACKing of packets can be arguably said in fact to increase performance,
> as without it the sender couldn't discover the maximum rate it could
> send without packet loss, and so if it didn't wish to lose data would
> have to be more conservative with how fast it sent packets.
>  
Thanks for this quote. I'll remember this.

Çağlar


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Tom Hennen
You may not have enough memory for the receive window you've specified.
 
TCP_WND doesn't actually allocate any memory by itself, you'll need to make sure you have enough memory in the pbuf pool or on the heap (if you can allocate incoming packets from the heap) to accommodate the promise made by TCP_WND.
 
Also remember TCP_WND refers to the TCP payload, you'll need to have space for the headers and other structures as well.
 
Tom
 
On 10/11/06, Çağlar AKYÜZ <[hidden email]> wrote:
Kieran Mansley yazmış:

> On Wed, 2006-10-11 at 16:25 +0300, Çağlar AKYÜZ wrote:
>
>
>> Actually I think this will not improve the performance of MAC. Because
>> ACK'ing of TCP frames is one of the major drawbacks which degrades
>> performance ( thanks to Delayed ACK algorithm!).
>>
>
> Only in a very limited set of circumstances will the delayed ACK
> algorithm result in poorer performance.  Namely, when the TCP window has
> been set to a very small value.  Changing this configuration option will
> remove any bottleneck that the delayed ACK algorithm is imposing.
I tried to increase recieve window but it always resulted in a poorer
performance with my SAM7X. ( Maybe I'm too low in memory?) . is TCP
window size related to somewhat pbufs? I don't know if I have to
increase number of buffers as well.
> There
> are TCP extensions such as the "quickack" mode in linux's TCP stack that
> stop the use of delayed ACK during the slow start phase of TCP
> connections, but the performance difference that this is likely to lead
> to in lwIP is minimal.
>
Actually I didn't mean to say that problem is related to LWiP. I wanted
to say that delayed ACK algorithm is slowing TCP connections in our case
and my Windows machine do not provide me any way to change this setting.
( Thanks to Linux as you noted. )
> ACKing of packets can be arguably said in fact to increase performance,
> as without it the sender couldn't discover the maximum rate it could
> send without packet loss, and so if it didn't wish to lose data would
> have to be more conservative with how fast it sent packets.
>
Thanks for this quote. I'll remember this.

Çağlar


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re: Desired - reference netconn_write() to external memory

Caglar Akyuz-2
Tom Hennen yazmış:
> You may not have enough memory for the receive window you've specified.
>  
> TCP_WND doesn't actually allocate any memory by itself, you'll need to
> make sure you have enough memory in the pbuf pool or on the heap (if
> you can allocate incoming packets from the heap) to accommodate the
> promise made by TCP_WND.
This explains why I can't improve performance just adjusting TCP_WND.  
Since I'm trying to optimize for throughput rather than  latency, I'm
minimizing the number of pbuf's in the pool and maximizing the each
one's memory space. From your words, I see that I should review this
strategy and take TCP_WND into account.


Thanks for the help
Çağlar


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users