Problem with sending big buffers to lwip_write and semaphoresduring big downloads.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with sending big buffers to lwip_write and semaphoresduring big downloads.

Hans.Forsstrom

Hello,
I have been using LWIP in a project for a couple of months now and I think I have been able to correct the main part of my bugs :-). We have ported LWIP onto our own software platform, so we had some problems with the porting layer to start with. For the moment we are running version 1.2, but I have also tested 1.1.1 and a newer version directly from CVS. We have created our own ethernet interface that works toward a USB CDC Subset solution, so the problem might be with other parts of our solution.


The first problem is sending big buffers to lwip_write. If I send in a buffer that are bigger then a ethernet frame (3000 bytes) then LWIP will send the first buffer but my second call to write will not work. Attached you will find a log from Wireshark running on a laptop connected via USB to our platform. The laptop are 10.0.01 and our platform has 10.0.0.2.


The other problem is with semaphores. This is quite a small problems as I seems to be able to work around it by streaming smaller amount of data. Anyway, If I connect to a server and starts streaming data over TCP via LWIP. Then everything works fine until about 100 MB of data when our platform assert based on more calls to
sys_sem_signal then to sys_arch_sem_wait.

Have anyone had simular problems and can give me some idea on where the problem might be?


Thank's for the help in advance!


BR,

Hans
******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an
electronic signature and/or legally binding agreement in the absence of an express
statement or Autoliv policy and/or procedure to the contrary.

This E-mail and any attachments hereto are Autoliv property and may contain legally
privileged, confidential and/or proprietary information. The recipient of this E-mail is
prohibited from distributing, copying, forwarding or in any way disseminating any
material contained within this E-mail without prior written permission from the author.
If you receive this E-mail in error, please immediately notify the author and delete this
E-mail. Autoliv disclaims all responsibility and liability for the consequences of any
person who fails to abide by the terms herein.
******************************

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

tcpip_dump_hangs.pcap Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers to lwip_write and semaphoresduring big downloads.

Kieran Mansley

On Wed, 2007-04-11 at 10:20 +0200, [hidden email] wrote:

> The first problem is sending big buffers to lwip_write. If I send in a
> buffer that are bigger then a ethernet frame (3000 bytes) then LWIP
> will send the first buffer but my second call to write will not work.

Can you describe in what way it doesn't work?  Does the lwip_write
function return an error for example?

> The other problem is with semaphores. This is quite a small problems
> as I seems to be able to work around it by streaming smaller amount of
> data. Anyway, If I connect to a server and starts streaming data over
> TCP via LWIP. Then everything works fine until about 100 MB of data
> when our platform assert based on more calls to sys_sem_signalthen to
> sys_arch_sem_wait.

I haven't heard of that particular problem before.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers to lwip_writeand semaphoresduring big downloads.

Hans.Forsstrom



Kieran Mansley <[hidden email]>
Sent by: lwip-users-bounces+hans.forsstrom=[hidden email]

2007-04-11 11:23

Please respond to
Mailing list for lwIP users <[hidden email]>

To
Mailing list for lwIP users <[hidden email]>
cc
Subject
Re: [lwip-users] Problem with sending big buffers to lwip_writeand         semaphoresduring big downloads.






On Wed, 2007-04-11 at 10:20 +0200, [hidden email] wrote:

>> The first problem is sending big buffers to lwip_write. If I send in a
>> buffer that are bigger then a ethernet frame (3000 bytes) then LWIP
>> will send the first buffer but my second call to write will not work.

>Can you describe in what way it doesn't work?  Does the lwip_write
>function return an error for example?

>Kieran

The first call to lwip_write returns 3000 and the second call doesn't return at all. We have updated the version of the platform so I have some trouble running some real test for you at the moment.

/Hans


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an
electronic signature and/or legally binding agreement in the absence of an express
statement or Autoliv policy and/or procedure to the contrary.

This E-mail and any attachments hereto are Autoliv property and may contain legally
privileged, confidential and/or proprietary information. The recipient of this E-mail is
prohibited from distributing, copying, forwarding or in any way disseminating any
material contained within this E-mail without prior written permission from the author.
If you receive this E-mail in error, please immediately notify the author and delete this
E-mail. Autoliv disclaims all responsibility and liability for the consequences of any
person who fails to abide by the terms herein.
******************************

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers to lwip_writeand semaphoresduring big downloads.

Kieran Mansley
On Wed, 2007-04-11 at 15:25 +0200, [hidden email] wrote:

> >Can you describe in what way it doesn't work?  Does the lwip_write
> >function return an error for example?
>
> >Kieran
>
> The first call to lwip_write returns 3000 and the second call doesn't
> return at all. We have updated the version of the platform so I have
> some trouble running some real test for you at the moment.
>
> /Hans

I tried to have a look at your packet capture, but it's an empty file I
think.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers to lwip_writeandsemaphoresduring big downloads.

Hans.Forsstrom

Sorry, I took the wrong file. This should work.

/Hans


Kieran Mansley <[hidden email]>
Sent by: lwip-users-bounces+hans.forsstrom=[hidden email]

2007-04-11 15:34

Please respond to
Mailing list for lwIP users <[hidden email]>

To
Mailing list for lwIP users <[hidden email]>
cc
Subject
Re: [lwip-users] Problem with sending big buffers to         lwip_writeandsemaphoresduring big downloads.





On Wed, 2007-04-11 at 15:25 +0200, [hidden email] wrote:
> >Can you describe in what way it doesn't work?  Does the lwip_write
> >function return an error for example?
>
> >Kieran
>
> The first call to lwip_write returns 3000 and the second call doesn't
> return at all. We have updated the version of the platform so I have
> some trouble running some real test for you at the moment.
>
> /Hans

I tried to have a look at your packet capture, but it's an empty file I
think.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an
electronic signature and/or legally binding agreement in the absence of an express
statement or Autoliv policy and/or procedure to the contrary.

This E-mail and any attachments hereto are Autoliv property and may contain legally
privileged, confidential and/or proprietary information. The recipient of this E-mail is
prohibited from distributing, copying, forwarding or in any way disseminating any
material contained within this E-mail without prior written permission from the author.
If you receive this E-mail in error, please immediately notify the author and delete this
E-mail. Autoliv disclaims all responsibility and liability for the consequences of any
person who fails to abide by the terms herein.
******************************

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

tcpip_big_buffer.pcap (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers to lwip_writeandsemaphoresduring big downloads.

Kieran Mansley
On Wed, 2007-04-11 at 15:42 +0200, [hidden email] wrote:
>
> Sorry, I took the wrong file. This should work.
>

That does indeed look odd.  There are no further packets sent/received
on the connection after this point?

It looks like you've got some deadlocking issue perhaps, where the
second lwip_send call is sitting blocked trying to get a lock of some
kind (although lwip infamously has very few locks!) which isn't being
released by another thread.

I'm afraid that you'll have to do some debugging.  If you can find out
where the blocked lwip_send call is waiting, that would probably help,
and try running the LWIP_DEBUG stuff on to get some more information
about what's going on.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers tolwip_writeandsemaphoresduring big downloads.

Hans.Forsstrom

Hello,
I'm not able to see anything more on wireshark running on my laptop, and I'm not able to run any tools on our target platform.

Yes, it looks like a deadlock problem. I haven't figured out why I only get it for data sizes over one TCP frame size though.

For the moment I will have to leave this, but I will try to do some more debugging later on. I'll try to use our "consol" for the LWIP_DEBUG printouts, but it has some limitations.

/Hans


Kieran Mansley <[hidden email]>
Sent by: lwip-users-bounces+hans.forsstrom=[hidden email]

2007-04-11 16:26

Please respond to
Mailing list for lwIP users <[hidden email]>

To
Mailing list for lwIP users <[hidden email]>
cc
Subject
Re: [lwip-users] Problem with sending big buffers         tolwip_writeandsemaphoresduring big downloads.





On Wed, 2007-04-11 at 15:42 +0200, [hidden email] wrote:
>
> Sorry, I took the wrong file. This should work.
>

That does indeed look odd.  There are no further packets sent/received
on the connection after this point?

It looks like you've got some deadlocking issue perhaps, where the
second lwip_send call is sitting blocked trying to get a lock of some
kind (although lwip infamously has very few locks!) which isn't being
released by another thread.

I'm afraid that you'll have to do some debugging.  If you can find out
where the blocked lwip_send call is waiting, that would probably help,
and try running the LWIP_DEBUG stuff on to get some more information
about what's going on.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an
electronic signature and/or legally binding agreement in the absence of an express
statement or Autoliv policy and/or procedure to the contrary.

This E-mail and any attachments hereto are Autoliv property and may contain legally
privileged, confidential and/or proprietary information. The recipient of this E-mail is
prohibited from distributing, copying, forwarding or in any way disseminating any
material contained within this E-mail without prior written permission from the author.
If you receive this E-mail in error, please immediately notify the author and delete this
E-mail. Autoliv disclaims all responsibility and liability for the consequences of any
person who fails to abide by the terms herein.
******************************

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big buffers tolwip_writeandsemaphoresduring big downloads.

Kieran Mansley
On Thu, 2007-04-12 at 08:29 +0200, [hidden email] wrote:

> Yes, it looks like a deadlock problem. I haven't figured out why I
> only get it for data sizes over one TCP frame size though.

If you send less than a frame size, chances are that the frame will go
out straight away.  If you send more than a frame size (particularly at
the start of the connection when the congestion window is small) the
second portion might be queued waiting for an acknowledgement.  If the
deadlock was a conflict between sending this second portion of the first
send, and the first portion of the second send, this would explain why
you get the difference in behaviour. i.e. I'd look at the send queue as
the first suspect.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with sending big bufferstolwip_writeandsemaphoresduring big downloads.

Hans.Forsstrom

Thanks, I was hoping for an answere like that. I will twist my brain around it later and dig into our send queue and see what I can find.


/Hans


Kieran Mansley <[hidden email]>
Sent by: lwip-users-bounces+hans.forsstrom=[hidden email]

2007-04-12 09:56

Please respond to
Mailing list for lwIP users <[hidden email]>

To
Mailing list for lwIP users <[hidden email]>
cc
Subject
Re: [lwip-users] Problem with sending big         bufferstolwip_writeandsemaphoresduring big downloads.





On Thu, 2007-04-12 at 08:29 +0200, [hidden email] wrote:

> Yes, it looks like a deadlock problem. I haven't figured out why I
> only get it for data sizes over one TCP frame size though.

If you send less than a frame size, chances are that the frame will go
out straight away.  If you send more than a frame size (particularly at
the start of the connection when the congestion window is small) the
second portion might be queued waiting for an acknowledgement.  If the
deadlock was a conflict between sending this second portion of the first
send, and the first portion of the second send, this would explain why
you get the difference in behaviour. i.e. I'd look at the send queue as
the first suspect.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an
electronic signature and/or legally binding agreement in the absence of an express
statement or Autoliv policy and/or procedure to the contrary.

This E-mail and any attachments hereto are Autoliv property and may contain legally
privileged, confidential and/or proprietary information. The recipient of this E-mail is
prohibited from distributing, copying, forwarding or in any way disseminating any
material contained within this E-mail without prior written permission from the author.
If you receive this E-mail in error, please immediately notify the author and delete this
E-mail. Autoliv disclaims all responsibility and liability for the consequences of any
person who fails to abide by the terms herein.
******************************

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users