TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
Hi,

I've implemeneted lwip stack in my device. I'm successfully able to ping both IPv4 and IPv6.

But I've problem with implementing webserver. Whenever I try to access webpage from PC to lwip based device, SYN and SYN+ACK packets are continuously getting Tx/Rx between PC and device but no ACK packet is obtained and thus 3-way handshaking process does not get completed.

So TCP request keeps re-transmitting.

Can any one please suggest me what can be the cause of handshaking issue or ACK packet not being received??

Any help would be highly appreciated.

Regards,
Mohsin
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
Can you at least explain what you are doing with what and post a capture
file ?

--


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
I'm implementing webserver using updated lwip stack.

Actually, before updating to the latest lwip stack from Master GIT Head, my webserver for IPv4 was working fine but now after integrating both IPv4/v6, webserver is not working.

For debugging, I'm printing pcb->states in tcp_process( ) before entering into any cases. For every http requests, I'm getting pcb->state=3 (SYN_RCVD).

And my device keeps re-transmitting SYN, ACK request to the PC.

So where can be the actual problem that I'm not getting. Right now I'm unable to post a capture file but soon I'll post it.

Regards,
Mohsin
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
In reply to this post by Sergio R. Caprile
I'm implementing webserver using updated lwip stack. Below I've attached the capture file for the same.

Actually, before updating to the latest lwip stack from Master GIT Head, my webserver for IPv4 was working fine but now after integrating both IPv4/v6, webserver is not working.
 
For debugging, I'm printing pcb->states in tcp_process( ) before entering into any cases. For every http requests, I'm getting pcb->state=3 (SYN_RCVD).
 
And my device keeps re-transmitting SYN, ACK request to the PC.

So where can be the actual problem that I'm not getting.
 
Regards,
Mohsin

IPv4+v6_capture_for_webserver.pcapng
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
In reply to this post by Sergio R. Caprile
There are 298 frames in your capture file.
Somewhere in the middle of it I see a 10.9.208.127 sending a TCP SYN to
port 80 to 10.9.208.210
Judging by the amount of crap it is sending, and the fact that it is
opening the connection, I assume 10.9.208.127 is your PC
Then, 10.9.208.210 must be your "box", which is running git head code on
a X processor with a X port, configured for X (SYS or NO_SYS) which is
OK and tested good because ? And, your webserver is X from X, using API
X which is also OK and tested good because ?

The TCP three-way handshake for opening is
SYN
SYN + ACK
ACK
I can see the first two frames, but there is no clue of the third one.
Since it has to be sent by the opener in response to the SYN+ACK, which
I can see, then the opener is either not seeing this frame or not liking
itm or broken.
Enabling checksum validation in wireshark, I see the (supposed to be)
lwIP box sends correct checksums, but the PC does not. (?) This is
probably due to checksum offloading and you taking the capture on the
same machine.

Can you do a plain "telnet mybox 80" from a known-to-be-good machine on
a not so loaded (read "just the two boxes and all the extra crap
removed") network and ellaborate on all the Xs ?

--


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
This post was updated on .
First of all thank you so much for your reply. Below I've tried to explain my problem in more detail along with the answers of all Xs (sorry for messing up whole queries and answers).

Sergio R. Caprile wrote
I assume 10.9.208.127 is your PC
Then, 10.9.208.210 must be your "box", which is running git head code on
a X processor with a X port, configured for X (SYS or NO_SYS) which is
OK and tested good because ?
Your assumption is correct in the sense that:
10.9.208.127 (Windows-7 based PC as CLIENT) ---> SYN
10.9.208.210 (Embedded device running git head LWIP stack as SERVER) --->SYN+ACK

Processor = Renesas H8S/2556 microcontroller and Port = 48556 configured for NO_SYS=0. Its OK and tested good because IPv4 is already implemented on it using stable release LWIP 1.4.1 stack since long back and its working fine (IPv4 ping and webserver tested OK for this version). Also for current git head, IPv4 and IPv6 ping is working.


And, your webserver is X from X, using API
X which is also OK and tested good because ?
Frankly I didn't get what you meant to ask by what "webserver is X from X", but I can say that the webserver is specifically designed according to application of the device whose http webpages are translated into C using "webgen" software and are accessed through webserver.
Webserver uses TCP Socket API and it is also OK and tested good because still same webserver code is working fine with the backup code of LWIP 1.4.1 stack (only IPv4).

One more thing is that with exactly same IP configurations, webserver code and same PC & device, webserver is working for one of my previous backup codes (stable released LWIP 1.4.1 and not git head updated).

So, I guess there may be some change in the functionality of tcp.c/tcp_in.c/tcp_out.c in current git head because of which my application is not working.

Can you do a plain "telnet mybox 80" from a known-to-be-good machine on
a not so loaded (read "just the two boxes and all the extra crap
removed") network?
Truly I have no knowledge of how to configure my device/stack for telnet as I'm beginner in all these concepts.


Regards,
Mohsin
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users



Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

goldsimon@gmx.de
Mohsin wrote:
> One more thing is that with exactly same IP configurations, webserver
> code and same PC & device, webserver is working for one of my previous
> backup codes (stable released LWIP 1.4.1 and not git head updated).
> So, I guess there may be some change in the functionality of
> tcp.c/tcp_in.c/tcp_out.c in current git head because of which my
> application is not working.

Looking at the traces, all TCP packets sent by your device seem to have
invalid *IP* checksums (where ICMP/IPv4 packets are OK). Since the TCP
packets might well be pbuf chains while the ping-reply packets are
(almost) surely single-pbuf-packets, I suspect a free-after-use issue
somewhere in your port/driver setup.

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
In reply to this post by Sergio R. Caprile
By "port" I meant who did the work of porting the stack to your processor.
Sorry, I don't fully agree with your definitions of "known to be good"
or "known to work". To me those are "supposed to be OK". Your web server
is not lwIP code, how are you certain it works OK ? You can't have two
unkown variables and believe both of them are true. In order to test the
port for good, you need a known to work application, and viceversa.
Some things might have happened on what you have there:
- The web server might have defects that weren't seen on 1.4.1 and are
evident in git head.
- The official release 1.4.1 might have been manipulated by the one
doing the port to make it work, so when you apply a git head it won't work.
As Simon pointed out (I missed that, only TCP checksum validation
activated on my Wireshark...), you are sending IP datagrams with bad
checksums. That is the reason why your browser does not see the SYN+ACK
and keeps sending SYNs.
What he describes is what I called "defects that weren't seen on 1.4.1
and are evident in git head" a couple lines above...

If we assume that the driver (the port) and the stack (the port) are OK,
you have to try to reduce the web server to the minimum required to
serve a static page, know what API is it using, and if you post the
minimum code someone will probably check it (I offer my help with the
raw API, don't know the others. Be specific in the subject on what API
you are using.
If we assume the web server is OK (why ?), then something is broken in
your port, that is, the lwIP code is modified and or the ethernet code
is buggy. Pickup one of the known to work applications, and see if it
works, and if it doesn't, find why.
You are using NO_SYS=0, hope you know what your are doing with tasks and
calling lwIP from just one of them. I can't help further on this, I'm a
NO_SYS=1 guy.
Good luck

PS: how come you are developing networking applications and don't know
how to do a plain telnet to a port ? you open a command shell or
whatever M$ calls it and write "telnet" followed by the port number, or
install PuTTY.

--


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

goldsimon@gmx.de
Sergio R. Caprile wrote:
> If we assume the web server is OK (why ?)

Well, I'll have to take the OP's side here: if it is a webserver
programmed with the socket API, this should in no way result in IP
checksum errors. This means if the socket webserver is the one
triggering this, the assumption that this is a bug in lower layers (lwIP
or the port/driver) is not a bad one.

However, to get to the cause of the problem, using a known-to-work raw
API app might indeed be the best.

> then something is broken in your port,

Even if it's not too likely, this could still be a bug in lwIP. It's
just way more likely to be a bug in your port/driver...

> You are using NO_SYS=0, hope you know what your are doing with tasks and
> calling lwIP from just one of them.

When using the sockets API, tasks should not be a problem unless one
socket is used from multiple threads at once.

> PS: how come you are developing networking applications and don't know
> how to do a plain telnet to a port ? you open a command shell or
> whatever M$ calls it and write "telnet" followed by the port number,

 From Windows 7 on (or was it Vista?), telnet is *not* included with
standard windows any more :-(

>   or install PuTTY.

Right. That's what I do. With that knownledge however, I think you'd
have to explain what to do with telnet in this case...


Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
In reply to this post by Sergio R. Caprile
Sergio R. Caprile wrote
Sorry, I don't fully agree with your definitions of "known to be good"
or "known to work". To me those are "supposed to be OK". Your web server
is not lwIP code, how are you certain it works OK ?
...
you need a known to work application, and viceversa.
Here is the capture file for the same webserver which is working for my backup based on lwIP 1.4.1 which may justify what you call and I believe as "known to be good".

020315_1740_(IPv4_webserver_working).pcapng

I'm also getting the webpages served by webserver and can see them in my browser. Below is the screen shot of webserver that is working.

Webserver_Snapshot_1.jpg

As Simon pointed out (I missed that, only TCP checksum validation
activated on my Wireshark...), you are sending IP datagrams with bad
checksums. That is the reason why your browser does not see the SYN+ACK
and keeps sending SYNs.
Though IP datagrams checksums are bad for above attached captured file, its webserver is working fine (as shown in screen shot above) which I'm also unable to judge.
 
PS: how come you are developing networking applications and don't know
how to do a plain telnet to a port ? you open a command shell or
whatever M$ calls it and write "telnet" followed by the port number, or
install PuTTY.
What I actually don't know about telnet is configuring stack/port for telnet (which I'm not sure about and might not be required at all). Otherwise I know this basic telnet using command prompt in Windows-7 PC and I also tried as u suggested to "telnet mybox 80" but connection gets closed and telnet is also not working. My box sends RST+ACK packets and connection gets terminated. Below I've also attached capture file of telnet test and screen shot that shows what command I'm entering to try the telnet.

telnet_test_cmd.jpg
IPv4_capture_for_telnet.pcapng

As per your suggestions, now my next goal is to test raw API based webserver with minimal application instead of my current webserver which is using Socket API.


[Note: I guess there is some problem at TCP layer in stack bcoz of which no ACK is received in response to SYN+ACK. If TCP connection will only not be established then how those TCP segments will be further processed and get access to socket/webserver?]


Regards,
Mohsin
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
In reply to this post by Sergio R. Caprile
  or install PuTTY.
> Right. That's what I do. With that knownledge however, I think you'd have
> to explain what to do with telnet in this case...

    haha, right, developing web server applications and not knowng HTTP
    either... oh, these kids nowadays... good point Simon, sometimes I
    forget I'm an old man...

Once you connect to the box to port 80, however you do it, you paste (the web server might not wait for you to type) the following

GET / HTTP/1.0<CR><CR>

Anyway, we just wanted to connect here and it served our purposes.

Mohsin:
your telnet capture: frames 75 and 173: you are not trying to connect to port 80 but to the telnet port (last time I used M$ stuff, the port number was entered plain text without brackets, as in any other shell in the Un*x world), that explains the RSTs.
Anyway, your PC is ignoring the RST+ACK and I bet that is because of the IP checksum error. Your previous stuff capture (1.4.1) shows IP checksums are correct, so as far as I'm concerned, that is the problem.

OK, summing up:

Web server uses socket API, so it can't be the problem (Simon's statement)
driver + vendor port seem to work OK with (vendor ?) 1.4.1 (you don't say where your 1.4.1 came from). IP checksums are OK
driver + vendor port don't work with git head. IP checksums are bad

My best guesses
1) Are you calculating IP checksums in software and in your hardware (driver)? Please check
2) I remember there were some changes on the checksum functions, perhaps something in your port collides with those changes or exposes a defect in them.
3) Your port can be trashing memory that was not in use by 1.4.1.

My suggestions:
1) Check for this
     #define CHECKSUM_GEN_IP                 1
and also check how your driver initializes the chip for checksums. Setting the above macro to the opposite value might be faster to check...
2 and 3) Check for these anywhere in your port and please post what you find
 * #define LWIP_CHKSUM <your_checksum_routine>
 *
 * Or you can select from the implementations below by defining
 * LWIP_CHKSUM_ALGORITHM to 1, 2 or 3.

I don't know anything but the raw API, so in my case I would use the echo application from the contrib tree..
You need to see what the IP checksum function calculates, compare with what is written in the pcb, and what goes out. One of those three should be wrong.
I would put a breakpoint at the echo and work my way down, but there is probably a closer place to start.

Good hunting.
-- 

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
Sergio R. Caprile wrote
My best guesses
1) Are you calculating IP checksums in software and in your hardware
(driver)? Please check
2) I remember there were some changes on the checksum functions, perhaps
something in your port collides with those changes or exposes a defect
in them.
For ethernet packets, I'm claculating checksum in hardware but for TCP and IP, its is in software only.

My suggestions:
1) Check for this
     #define CHECKSUM_GEN_IP                 1
I checked above Macro in my opt.h and it is set to 1.

2 and 3) Check for these anywhere in your port and please post what you find
 * #define LWIP_CHKSUM <your_checksum_routine>
 *
 * Or you can select from the implementations below by defining
 * LWIP_CHKSUM_ALGORITHM to 1, 2 or 3.
There is no implementation of above LWIP_CHKSUM routine in my ports. I could find it only in "lwip/sr/core/inet_checksum.c" file of my project which is mentioned in the file attached below:
lwip_chksum_routine.txt


One more think I wanted to ask about IP Checksum is that, for all those packets transmitted from my box having IP checksum error, I analyzed those packets in wireshark and found that there is error of only one bit for all IP checksum. I mean for one TCP packet, wireshark gives error as follows:
"Header checksum: 0x065e [incorrect, should be 0x065d]".
For another TCP packet, error was:
"Header checksum: 0x065c [incorrect, should be 0x065b]"
So I think there is difference of only one bit for every checksum calculated by my box and actual correct checksum. Hence for work around, can I decrease 1-bit at the end of every IP checksum calculation?


Regards,
Mohsin
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
In reply to this post by Mohsin
> For ethernet packets, I'm claculating checksum in hardware but for TCP and
> IP, its is in software only.


>> My suggestions:
>> 1) Check for this
>>      #define CHECKSUM_GEN_IP                 1

>I checked above Macro in my opt.h and it is set to 1.

Then you have just found your problem.
If you are calculating IP checksums in hardware, you must not calculate them in lwIP.
Set 
#define CHECKSUM_GEN_IP 0


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
Sergio R. Caprile wrote
> For ethernet packets, I'm claculating checksum in hardware but for TCP and
> IP, its is in software only.

>>* My suggestions:*
>>* 1) Check for this*
>>*      #define CHECKSUM_GEN_IP                 1*

>I checked above Macro in my opt.h and it is set to 1.

Then you have just found your problem.

If you are calculating IP checksums in hardware, you must not
calculate them in lwIP.

Set
*#define CHECKSUM_GEN_IP                 0*
Actually, I don't calculate IP checksums in hardware but in software only as I mentioned previously which is quoted above.
So, I have set  
            #define CHECKSUM_GEN_IP   1

Hence, if I set it to 0 then also it is not working and IP checksum comes out to be 0x000000.

Still I'm unable to find the cause of IP checksum error in my implementation


Regards,
Mohsin Kesarani
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

goldsimon@gmx.de
Mohsin wrote:
> Actually, I don't calculate IP checksums in hardware but in software

I think Sergio simply got confused by you saying "For ethernet packets,
I'm claculating checksum in hardware". Now I guess you meant the CRC
with that?

> Still I'm unable to find the cause of IP checksum error in my
> implementation

Try setting adding the line "#define LWIP_INLINE_IP_CHKSUM   0" to your
lwipopts.h. Maybe there's something wrong with the checksum-inlining
code on your CPU (isn't it a 16-bit one?)...

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Sergio R. Caprile
In reply to this post by Mohsin
If IP checksums come out as 0x0000 when you set them in lwIP for hardware calculation, that means your system seems to be correctly configured for that, lwIP sends 0x0000 and the link-layer driver is doing nothing (or at least it is not setting any bits...).

OK, so your IP checksums are calculated in software and they are going out wrong. Then [quoting myself]:

>> You need to see what the IP checksum function calculates, compare with what is
>> written in the pcb, and what goes out. One of those three should be wrong.

I guess the easiest test is to check for the output of the checksumming function and compare that to the IP checksum in the wire.
Put a breakpoint there, or a printf, or whatever.

See  src/core/inet_chksum.c, start with checking this:
#ifndef LWIP_CHKSUM
# define LWIP_CHKSUM lwip_standard_chksum
# ifndef LWIP_CHKSUM_ALGORITHM
#  define LWIP_CHKSUM_ALGORITHM 2
# endif
and then

>> Good hunting.

BTW, you could also just try changing the algorithm and see what happens.
Also check for a definition of
LWIP_CHKSUM_ALGORITHM in your port.
I see many changes to this file from 1.4.1 to git head, maybe you have just found something... maybe not. Sorry but you are the one with the problem and you have to hunt it down, as no one else seems to see what you are seeing in your system.

-- 



_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

Mohsin
In reply to this post by goldsimon@gmx.de
goldsimon@gmx.de wrote
Try setting adding the line "#define LWIP_INLINE_IP_CHKSUM   0" to your
lwipopts.h. Maybe there's something wrong with the checksum-inlining
code on your CPU
Thank you so much Simon. I added above line in my lwipopts.h and it solved the IP checksum error.
 (isn't it a 16-bit one?)...
Exactly, my box is based on 16-bit microcontroller. So was it the actual cause for this problem?

Now with that solution, atleast my webserver for IPv4 again got started. But for IPv6, my box sends RST+ACK for every TCP request.

Currently, below is the status of my box:
IP Configurations of my box:
IPv4 address                : 10.9.208.210
link-local IPv6 address  : fe80::200:21ff:fe12:324
Static IPv6 address       : fda8:6c3:ce53:a890::8

IP config of my PC:
IPv4 address                : 10.9.208.127
link-local IPv6 address  : fe80::64e4:e351:a09d:5bf4
Static IPv6 address       : fda8:6c3:ce53:a890::3


lwip IPv4+v6 Stack implementation status of my box:
Ping status:
IPv4-- working
IPv6-- working

Webserver Status:
IPv4-- working
IPv6-- not working. just sends RST+ACK for evry TCP request.

So, now I again came back to my original problem of RST response in case of IPv6. Although ping for IPv6 is working fine. If you can help me in this case then it would be highly appreciated.

Here is the packet capture from wireshark addressing this RST+ACK problem in IPv6 webserver case.
07041048_IPv4_v6.pcapng


Thanks & Regards,
Mohsin Kesarani




_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

Reply | Threaded
Open this post in threaded view
|

Re: TCP keeps re-transmitting but no ACK packet after SYN, SYN+ACK

goldsimon@gmx.de
Mohsin wrote:
>> (isn't it a 16-bit one?)...
>
> Exactly, my box is based on 16-bit microcontroller. So was it the actual
> cause for this problem?

I really don't know that. Inline checksum was introduced in 1.4.0, so there should be no difference to git master. That was just an idea to get it right... It hasn't been tested much on 16-bit though, I guess.

> Now with that solution, atleast my webserver for IPv4 again got started. But
> for IPv6, my box sends RST+ACK for every TCP request.

That might just be a problem of correctly configuring a TCP pcb to answer IPv6 requests.

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users