lwIP hangs on transfers with many short packets

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

lwIP hangs on transfers with many short packets

multiplex
This post was updated on .
Hi all,

We're having a problem that stumps us. Using lwIP1.3.0 (Xilinx Microblaze port), we are using lWIP to support a telnet CLI connection. As long as we don't have a lot to print, no problem; once we print a lot of data typically, this can be even packets containing 1 byte of payload, lwIP hangs (typically, after 30-40 packets). With larger chuncks, it just takes longer.

We are using socket mode, all threads are running at the same priority.

With the debugging turned on, we see that the cause appears to be inside tcp_enqueue which keeps going around in circles trying to allocate/deallocate something but hitting a boundary.

Any suggestions on how to resolve this ?
* right now, we just wait a few hundred milliseconds after each lwip_write - but this really slows down data transfer

KR,
Michael

P.S. Here is a typical trace
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0xmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
40711100
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0x4memp_malloc: out of memory in pool TCPIP_MSG_INPKT
0711280
tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x40memp_malloc: out of memory in pool TCPIP_MSG_INPKT
711180
tcpip_thread: PACKET 0x40711200
tcpip_thread: PACKET 0x407memp_malloc: out of memory in pool TCPIP_MSG_INPKT
111c0
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x4071memp_malloc: out of memory in pool TCPIP_MSG_INPKT
1100
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0x40711memp_malloc: out of memory in pool TCPIP_MSG_INPKT
280


----------------------

tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x40711180
tcp_write(pcb=0x4070c500, dmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
ata=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b94:1b95 (0x0)
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKET 0x40711280
tcpip_thread: PACKET 0x40711200
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b95:1b96 (0x0)
tcp_enqueue: chaining segments, new len 2
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b96:1b97 (0x0)
tcp_enqueue: chaining segments, new len 3
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b97:1b98 (0x0)
tcp_enqueue: chaining segments, new len 4
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b98:1b99 (0x0)
tcp_enqueue: chaining segments, new len 5
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b99:1b9a (0x0)
tcp_enqueue: chaining segments, new len 6
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9a:1b9b (0x0)
tcp_enqueue: chaining segments, new len 7
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9b:1b9c (0x0)
tcp_enqueue: chaining segments, new len 8
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9c:1b9d (0x0)
tcp_enqueue: chaining segments, new len 9
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9d:1b9e (0x0)
tcp_enqueue: chaining segments, new len 10
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9e:1b9f (0x0)
tcp_enqueue: chaining segments, new len 11
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1b9f:1ba0 (0x0)
tcp_enqueue: chaining segments, new len 12
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba0:1ba1 (0x0)
tcp_enqueue: chaining segments, new len 13
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba1:1ba2 (0x0)
tcp_enqueue: chaining segments, new len 14
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba2:1ba3 (0x0)
tcp_enqueue: chaining segments, new len 15
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba3:1ba4 (0x0)
tcp_enqueue: chaining segments, new len 16
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba4:1ba5 (0x0)
tcp_enqueue: chaining segments, new len 17
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba5:1ba6 (0x0)
tcp_enqueue: chaining segments, new len 18
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba6:1ba7 (0x0)
tcp_enqueue: chaining segments, new len 19
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba7:1ba8 (0x0)
tcp_enqueue: chaining segments, new len 20
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba8:1ba9 (0x0)
tcp_enqueue: chaining segments, new len 21
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1ba9:1baa (0x0)
tcp_enqueue: chaining segments, new len 22
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1baa:1bab (0x0)
tcp_enqueue: chaining segments, new len 23
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bab:1bac (0x0)
tcp_enqueue: chaining segments, new len 24
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bac:1bad (0x0)
tcp_enqueue: chaining segments, new len 25
tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x407112c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bad:1bae (0x0)
tcp_enqueue: chaining segments, new len 26
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x40e3f79c, len=37, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x40e3f79c, len=37, flags=0, apiflags=1)
tcp_enqueue: queueing 1bae:1bd3 (0x0)
tcp_enqueue: chaining segments, new len 63
tcpip_thread: PACKET 0x407112c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015ca90, len=2, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015ca90, len=2, flags=0, apiflags=1)
tcp_enqueue: queueing 1bd3:1bd5 (0x0)
tcp_enqueue: chaining segments, new len 65
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x40e3f6dc, len=4, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x40e3f6dc, len=4, flags=0, apiflags=1)
tcp_enqueue: queueing 1bd5:1bd9 (0x0)
tcp_enqueue: chaining segments, new len 69
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bd9:1bda (0x0)
tcp_enqueue: chaining segments, new len 70
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bda:1bdb (0x0)
tcp_enqueue: chaining segments, new len 71
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bdb:1bdc (0x0)
tcp_enqueue: chaining segments, new len 72
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bdc:1bdd (0x0)
tcp_enqueue: chaining segments, new len 73
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bdd:1bde (0x0)
tcp_enqueue: chaining segments, new len 74
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bde:1bdf (0x0)
tcp_enqueue: chaining segments, new len 75
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bdf:1be0 (0x0)
tcp_enqueue: chaining segments, new len 76
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be0:1be1 (0x0)
tcp_enqueue: chaining segments, new len 77
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be1:1be2 (0x0)
tcp_enqueue: chaining segments, new len 78
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be2:1be3 (0x0)
tcp_enqueue: chaining segments, new len 79
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be3:1be4 (0x0)
tcp_enqueue: chaining segments, new len 80
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be4:1be5 (0x0)
tcp_enqueue: chaining segments, new len 81
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be5:1be6 (0x0)
tcp_enqueue: chaining segments, new len 82
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be6:1be7 (0x0)
tcp_enqueue: chaining segments, new len 83
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be7:1be8 (0x0)
tcp_enqueue: chaining segments, new len 84
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be8:1be9 (0x0)
tcp_enqueue: chaining segments, new len 85
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1be9:1bea (0x0)
tcp_enqueue: chaining segments, new len 86
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bea:1beb (0x0)
tcp_enqueue: chaining segments, new len 87
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1beb:1bec (0x0)
tcp_enqueue: chaining segments, new len 88
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bec:1bed (0x0)
tcp_enqueue: chaining segments, new len 89
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bed:1bee (0x0)
tcp_enqueue: chaining segments, new len 90
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bee:1bef (0x0)
tcp_enqueue: chaining segments, new len 91
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bef:1bf0 (0x0)
tcp_enqueue: chaining segments, new len 92
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bf0:1bf1 (0x0)
tcp_enqueue: chaining segments, new len 93
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bf1:1bf2 (0x0)
tcp_enqueue: chaining segments, new len 94
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1bf2:1bf3 (0x0)
tcp_enqueue: chaining segments, new len 95
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x40e3f87c, len=19, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x40e3f87c, len=19, flags=0, apiflags=1)
tcp_enqueue: queueing 1bf3:1c06 (0x0)
tcp_enqueue: chaining segments, new len 114
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015ca90, len=2, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015ca90, len=2, flags=0, apiflags=1)
tcp_enqueue: queueing 1c06:1c08 (0x0)
tcp_enqueue: chaining segments, new len 116
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x40e3fa54, len=11, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x40e3fa54, len=11, flags=0, apiflags=1)
tcp_enqueue: queueing 1c08:1c13 (0x0)
tcp_enqueue: chaining segments, new len 127
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c13:1c14 (0x0)
tcp_enqueue: chaining segments, new len 128
tcp_slowtmr: processing active pcb
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c14:1c15 (0x0)
tcp_enqueue: chaining segments, new len 129
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c15:1c16 (0x0)
tcp_enqueue: chaining segments, new len 130
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x407112c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c16:1c17 (0x0)
tcp_enqueue: chaining segments, new len 131
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c17:1c18 (0x0)
tcp_enqueue: chaining segments, new len 132
tcpip_thread: PACKET 0x407112c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c18:1c19 (0x0)
tcp_enqueue: chaining segments, new len 133
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c19:1c1a (0x0)
tcp_enqueue: chaining segments, new len 134
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1a:1c1b (0x0)
tcp_enqueue: chaining segments, new len 135
tcpip_thread: PACKET 0x407112c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1b:1c1c (0x0)
tcp_enqueue: chaining segments, new len 136
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1c:1c1d (0x0)
tcp_enqueue: chaining segments, new len 137
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1d:1c1e (0x0)
tcp_enqueue: chaining segments, new len 138
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1e:1c1f (0x0)
tcp_enqueue: chaining segments, new len 139
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c1f:1c20 (0x0)
tcp_enqueue: chaining segments, new len 140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c20:1c21 (0x0)
tcp_enqueue: chaining segments, new len 141
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c21:1c22 (0x0)
tcp_enqueue: chaining segments, new len 142
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c22:1c23 (0x0)
tcp_enqueue: chaining segments, new len 143
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c23:1c24 (0x0)
tcp_enqueue: chaining segments, new len 144
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0xmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
40711200
tcpip_thread: PACKET 0x40711280
tcpip_thread: API messagmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
e 0x40b82988
tcp_write(pcb=memp_malloc: out of memory in pool TCPIP_MSG_INPKT
0x4070c500, data=0x4015caa8,memp_malloc: out of memory in pool TCPIP_MSG_INPKT
 len=1, apiflags=1)
tcp_enqmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
ueue(pcb=0x4070c500, arg=0x4memp_malloc: out of memory in pool TCPIP_MSG_INPKT
015caa8, len=1, flags=0, apimemp_malloc: out of memory in pool TCPIP_MSG_INPKT
flags=1)
tcp_enqueue: queuememp_malloc: out of memory in pool TCPIP_MSG_INPKT
ing 1c24:1c25 (0x0)
tcp_enqmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
ueue: chaining segments, newmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
 len 145
memp_malloc: out of memory in pool TCPIP_MSG_INPKT
tcpip_thread: PACKEmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
T 0x407112c0
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKETmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET memp_malloc: out of memory in pool TCPIP_MSG_INPKT
0x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
x40711200
tcpip_thread: PACKET 0x40711280
tcpip_thread: API messamemp_malloc: out of memory in pool TCPIP_MSG_INPKT
ge 0x40b82988
tcp_write(pcbmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
=0x4070c500, data=0x4015caa8memp_malloc: out of memory in pool TCPIP_MSG_INPKT
, len=1, apiflags=1)
tcp_enmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
queue(pcb=0x4070c500, arg=0xmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
4015caa8, len=1, flags=0, apmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
iflags=1)
tcp_enqueue: queumemp_malloc: out of memory in pool TCPIP_MSG_INPKT
eing 1c25:1c26 (0x0)
tcp_enmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
queue: chaining segments, nememp_malloc: out of memory in pool TCPIP_MSG_INPKT
w len 146
memp_malloc: out of memory in pool TCPIP_MSG_INPKT
tcpip_thread: PACKEmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
T 0x407112c0
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKETmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET memp_malloc: out of memory in pool TCPIP_MSG_INPKT
0x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
x40711200
tcpip_thread: PACKET 0x40711280
tcpip_thread: API messamemp_malloc: out of memory in pool TCPIP_MSG_INPKT
ge 0x40b82988
tcp_write(pcbmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
=0x4070c500, data=0x40e3fac4memp_malloc: out of memory in pool TCPIP_MSG_INPKT
, len=50, apiflags=1)
tcp_ememp_malloc: out of memory in pool TCPIP_MSG_INPKT
nqueue(pcb=0x4070c500, arg=0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
x40e3fac4, len=50, flags=0, memp_malloc: out of memory in pool TCPIP_MSG_INPKT
apiflags=1)
tcp_enqueue: qumemp_malloc: out of memory in pool TCPIP_MSG_INPKT
eueing 1c26:1c58 (0x0)
tcp_memp_malloc: out of memory in pool TCPIP_MSG_INPKT
enqueue: chaining segments, memp_malloc: out of memory in pool TCPIP_MSG_INPKT
new len 196
memp_malloc: out of memory in pool TCPIP_MSG_INPKT
tcpip_thread: PACKEmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
T 0x407112c0
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKETmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET memp_malloc: out of memory in pool TCPIP_MSG_INPKT
0x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
x40711200
tcpip_thread: PACKET 0x40711280
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015ca90, len=2, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015ca90, len=2, flags=0, apiflags=1)
tcp_enqueue: queueing 1c58:1c5a (0x0)
tcp_enqueue: chaining segments, new len 198
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x40e40814, len=7, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x40e40814, len=7, flags=0, apiflags=1)
tcp_enqueue: queueing 1c5a:1c61 (0x0)
tcp_enqueue: chaining segments, new len 205
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c61:1c62 (0x0)
tcp_enqueue: chaining segments, new len 206
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c62:1c63 (0x0)
tcp_enqueue: chaining segments, new len 207
tcpip_thread: PACKET 0x40711180
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c63:1c64 (0x0)
tcp_enqueue: chaining segments, new len 208
tcpip_thread: PACKET 0x40711180
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c64:1c65 (0x0)
tcp_enqueue: chaining segments, new len 209
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c65:1c66 (0x0)
tcp_enqueue: chaining segments, new len 210
tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x407111c0
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c66:1c67 (0x0)
tcp_enqueue: chaining segments, new len 211
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140memp_malloc: out of memory in pool TCPIP_MSG_INPKT

tcpip_thread: PACKET 0x40711240
memp_malloc: out of memory in pool TCPIP_MSG_INPKT

tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x40711280
tcpip_thread: API mmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
essage 0x40b82988
tcp_writememp_malloc: out of memory in pool TCPIP_MSG_INPKT
(pcb=0x4070c500, data=0x4015memp_malloc: out of memory in pool TCPIP_MSG_INPKT
caa8, len=1, apiflags=1)
tcmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
p_enqueue(pcb=0x4070c500, armemp_malloc: out of memory in pool TCPIP_MSG_INPKT
g=0x4015caa8, len=1, flags=0memp_malloc: out of memory in pool TCPIP_MSG_INPKT
, apiflags=1)
tcp_enqueue: memp_malloc: out of memory in pool TCPIP_MSG_INPKT
queueing 1c67:1c68 (0x0)
tcmemp_malloc: out of memory in pool TCPIP_MSG_INPKT
p_enqueue: chaining segments, new len 212
tcpip_thread: PACKET 0x407111c0
tcpip_thread: PACKET 0x40711200
tcpip_thread: PACKET 0x40711180
tcpip_thread: PACKET 0x40711140
tcpip_thread: PACKET 0x40711240
tcpip_thread: PACKET 0x40711100
tcpip_thread: PACKET 0x407112c0
tcpip_thread: PACKET 0x40711280
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: queueing 1c68:1c69 (0x0)
tcp_enqueue: chaining segments, new len 213
tcpip_thread: API message 0x40b82988
tcp_write(pcb=0x4070c500, data=0x4015caa8, len=1, apiflags=1)
tcp_enqueue(pcb=0x4070c500, arg=0x4015caa8, len=1, flags=0, apiflags=1)
tcp_enqueue: too long queue 89 (max 89)
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

goldsimon@gmx.de
multiplex wrote:

> Hi all,
>
> We're having a problem that stumps us. Using lwIP1.3.0 (Xilinx Microblaze
> port), we are using lWIP to support a telnet CLI connection. As long as we
> don't have a lot to print, no problem; once we print a lot of data
> typically, this can be even packets containing 1 byte of payload, lwIP hangs
> (typically, after 30-40 packets). With larger chuncks, it just takes longer.
>
> We are using socket mode, all threads are running at the same priority.
>
> With the debugging turned on, we see that the cause appears to be inside
> tcp_enqueue which keeps going around in circles trying to
> allocate/deallocate something but hitting a boundary.
>    
Do you mean "tcp_enqueue: too long queue 89 (max 89)"? If so, just
increase TCP_SND_QUEUELEN in your lwipopts.h. This is a limit on how
many pbuf shall be allowed in the queue of unacknowledged segments per
tcp pcb. Up to (and including) 1.3.2, lwIP created a new pbuf for each
call to tcp_write. This results in queuelen being quite high when
calling tcp_write (or the socket functions send/write) with many small
chunks.This has been improved in CVS HEAD already.

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Simon Goldschmidt wrote
Do you mean "tcp_enqueue: too long queue 89 (max 89)"
Yes, that was my impression. I tried increasing it by a factor of ten, but then nothing came out until I turned of Nagle. Then, it still disappeared into never never land. tcp_output reporting: nothing to send. Any other suggestions? Do I have to worry about the out of memory errors?

Thanks already for the suggestion.
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

goldsimon@gmx.de
multiplex wrote:
> Yes, that was my impression. I tried increasing it by a factor of ten, but
> then nothing came out until I turned of Nagle. Then, it still disappeared
> into never never land. tcp_output reporting: nothing to send. Any other
> suggestions?
Nothing to send? I don't see that statement in the log you sent with the
last mail.
>   Do I have to worry about the out of memory errors?
>    
Do you mean the "out of memory in pool TCPIP_MSG_INPKT" messages? Yes,
you definitively have to worry about that: every message like that means
one incoming packet is dropped! Try increasing MEMP_NUM_TCPIP_MSG_INPKT.

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Simon Goldschmidt wrote
Nothing to send? I don't see that statement in the log you sent with the
last mail.

Do you mean the "out of memory in pool TCPIP_MSG_INPKT" messages? Yes,
you definitively have to worry about that: every message like that means
one incoming packet is dropped! Try increasing MEMP_NUM_TCPIP_MSG_INPKT.
Nothing to send is new, after increasing the QUEUELEN x10. So maybe the ACKs are lost ? Will try your suggestion tomorrow (not a accessible option from the Xilinx SDK).

Thanks again, already.
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex
In reply to this post by goldsimon@gmx.de

Simon Goldschmidt wrote
Do you mean the "out of memory in pool TCPIP_MSG_INPKT" messages? Yes,
you definitively have to worry about that: every message like that means
one incoming packet is dropped! Try increasing MEMP_NUM_TCPIP_MSG_INPKT.
No joy. It takes a bit longer, we have no out-of-memory anymore, but it still freezes. Unfortunately, it does not happen when printing a trace (probably, the overhead of printing slows down everything and gives somebody some breathing room).

Will try to trigger is with a trace, then post.
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Tyrel Newton
How is your system configured?

- xilkernel or standalone
- socket or raw API
- emaclite or temac

Btw, don't expect much help from the Xilinx forums. Also, don't much
faith in the quality of the application-layer code that Xilinx provides,
including xilkernel and the associated sys_arch layer for lwIP and the
lwIP netif drivers for the emaclite and the temac.

Tyrel

On 5/20/2010 4:56 AM, multiplex wrote:

>
>
> Simon Goldschmidt wrote:
>    
>>
>> Do you mean the "out of memory in pool TCPIP_MSG_INPKT" messages? Yes,
>> you definitively have to worry about that: every message like that means
>> one incoming packet is dropped! Try increasing MEMP_NUM_TCPIP_MSG_INPKT.
>>
>>
>>      
> No joy. It takes a bit longer, we have no out-of-memory anymore, but it
> still freezes. Unfortunately, it does not happen when printing a trace
> (probably, the overhead of printing slows down everything and gives somebody
> some breathing room).
>
> Will try to trigger is with a trace, then post.
>    


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Tyrel Newton wrote
How is your system configured?

- xilkernel or standalone
- socket or raw API
- emaclite or temac
Xilkernel, socket, temac. And here is a trace.
Any suggestions ?
StupidTelnet.txt
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Kieran Mansley
On Thu, 2010-05-20 at 11:10 -0700, multiplex wrote:
> Any suggestions ?

I would use the LWIP_STATS code to identify if there is some resource
within lwIP that is running low.

Kieran


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Kieran Mansley wrote
On Thu, 2010-05-20 at 11:10 -0700, multiplex wrote:
> Any suggestions ?

I would use the LWIP_STATS code to identify if there is some resource
within lwIP that is running low.

Kieran
Thanks Kieran - Here are two printouts for the display_stats(). The first one is when everything is still going well; the second when it hangs (note: it is only lwIP that hangs - I still have serial - and if we disconnect and wait really really really long, the connection will closely nicely).

================= PRE - HANG =====================
LINK
        xmit: 1
        rexmit: 0
        recv: 131
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ETHARP
        xmit: 1
        rexmit: 0
        recv: 114
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 80

IP_FRAG
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

IP
        xmit: 0
        rexmit: 0
        recv: 18
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ICMP
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

UDP
        xmit: 0
        rexmit: 0
        recv: 18
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

TCP
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

MEM HEAP
        avail: 3d0900
        used: 4a80
        max: 4b80
        err: 0

MEM RAW_PCB
        avail: 4
        used: 0
        max: 0
        err: 0

MEM UDP_PCB
        avail: 4
        used: 1
        max: 1
        err: 0

MEM TCP_PCB
        avail: 20
        used: 0
        max: 1
        err: 0

MEM TCP_PCB_LISTEN
        avail: 8
        used: 1
        max: 1
        err: 0

MEM TCP_SEG
        avail: a01
        used: 0
        max: 0
        err: 0

MEM REASSDATA
        avail: 5
        used: 0
        max: 0
        err: 0

MEM NETBUF
        avail: 8
        used: 0
        max: 0
        err: 0

MEM NETCONN
        avail: 10
        used: 1
        max: 1
        err: 0

MEM TCPIP_MSG_API
        avail: 8
        used: 0
        max: 0
        err: 0

MEM TCPIP_MSG_INPKT
        avail: 320
        used: 0
        max: 1
        err: 0

MEM ARP_QUEUE
        avail: 1e
        used: 0
        max: 0
        err: 0

MEM SYS_TIMEOUT
        avail: 8
        used: 3
        max: 3
        err: 0

MEM PBUF_REF/ROM
        avail: 10
        used: 0
        max: 0
        err: 0

MEM PBUF_POOL
        avail: 800
        used: 40
        max: 41
        err: 0


================= POST - HANG ====================

LINK
        xmit: 116
        rexmit: 0
        recv: 572
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ETHARP
        xmit: 1
        rexmit: 0
        recv: 396
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 344

IP_FRAG
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

IP
        xmit: 113
        rexmit: 0
        recv: 163
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ICMP
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

UDP
        xmit: 0
        rexmit: 0
        recv: 61
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

TCP
        xmit: 34
        rexmit: 0
        recv: 102
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

MEM HEAP
        avail: 3d0900
        used: 4a80
        max: 5980
        err: 0

MEM RAW_PCB
        avail: 4
        used: 0
        max: 0
        err: 0

MEM UDP_PCB
        avail: 4
        used: 1
        max: 1
        err: 0

MEM TCP_PCB
        avail: 20
        used: 1
        max: 1
        err: 0

MEM TCP_PCB_LISTEN
        avail: 8
        used: 1
        max: 1
        err: 0

MEM TCP_SEG
        avail: a01
        used: 0
        max: 3
        err: 0

MEM REASSDATA
        avail: 5
        used: 0
        max: 0
        err: 0

MEM NETBUF
        avail: 8
        used: 1
        max: 1
        err: 0

MEM NETCONN
        avail: 10
        used: 2
        max: 2
        err: 0

MEM TCPIP_MSG_API
        avail: 8
        used: 0
        max: 0
        err: 0

MEM TCPIP_MSG_INPKT
        avail: 320
        used: 0
        max: 2
        err: 0

MEM ARP_QUEUE
        avail: 1e
        used: 0
        max: 0
        err: 0

MEM SYS_TIMEOUT
        avail: 8
        used: 4
        max: 4
        err: 0

MEM PBUF_REF/ROM
        avail: 10
        used: 0
        max: 0
        err: 0

MEM PBUF_POOL
        avail: 800
        used: 40
        max: 8a
        err: 0

==================== ANOTHER POST - HANG =======================

LINK
        xmit: 187
        rexmit: 0
        recv: 590
        fw: 0
        drop: 6
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ETHARP
        xmit: 1
        rexmit: 0
        recv: 329
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 320

IP_FRAG
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

IP
        xmit: 184
        rexmit: 0
        recv: 242
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

ICMP
        xmit: 0
        rexmit: 0
        recv: 0
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

UDP
        xmit: 0
        rexmit: 0
        recv: 74
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

TCP
        xmit: 63
        rexmit: 0
        recv: 168
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 5
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

MEM HEAP
        avail: 3d0900
        used: 3cc40
        max: 3cc40
        err: 0

MEM RAW_PCB
        avail: 4
        used: 0
        max: 0
        err: 0

MEM UDP_PCB
        avail: 4
        used: 1
        max: 1
        err: 0

MEM TCP_PCB
        avail: 20
        used: 1
        max: 1
        err: 0

MEM TCP_PCB_LISTEN
        avail: 8
        used: 1
        max: 1
        err: 0

MEM TCP_SEG
        avail: a01
        used: 2
        max: 4
        err: 0

MEM REASSDATA
        avail: 5
        used: 0
        max: 0
        err: 0

MEM NETBUF
        avail: 8
        used: 1
        max: 1
        err: 0

MEM NETCONN
        avail: 10
        used: 2
        max: 2
        err: 0

MEM TCPIP_MSG_API
        avail: 8
        used: 0
        max: 0
        err: 0

MEM TCPIP_MSG_INPKT
        avail: 320
        used: 0
        max: 1
        err: 0

MEM ARP_QUEUE
        avail: 1e
        used: 0
        max: 0
        err: 0

MEM SYS_TIMEOUT
        avail: 8
        used: 4
        max: 4
        err: 0

MEM PBUF_REF/ROM
        avail: 10
        used: 0
        max: 0
        err: 0

MEM PBUF_POOL
        avail: 800
        used: 41
        max: 95
        err: 0

Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Kieran Mansley

On 26 May 2010, at 06:54, multiplex wrote:

>
>
>
> Kieran Mansley wrote:
>>
>> On Thu, 2010-05-20 at 11:10 -0700, multiplex wrote:
>>> Any suggestions ?
>>
>> I would use the LWIP_STATS code to identify if there is some resource
>> within lwIP that is running low.
>>
>> Kieran
>>
>>
>
> Thanks Kieran - Here are two printouts for the display_stats(). The first
> one is when everything is still going well; the second when it hangs

Unfortunately there's nothing there to point a finger a what might have gone wrong.  All the resources seem fine: there is plenty of everything available, and it doesn't seem like anything has run out at any point.

> (note:
> it is only lwIP that hangs - I still have serial - and if we disconnect and
> wait really really really long, the connection will closely nicely).

Can you explain that in more detail?  Can you be precise about what you mean by disconnect and how long a really really really long time is?  If lwIP has hung, I wouldn't expect it to do anything no matter how long you wait.

Thanks

Kieran
_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Kieran Mansley wrote
On 26 May 2010, at 06:54, multiplex wrote:

>
>
>
> Kieran Mansley wrote:
>>
>> On Thu, 2010-05-20 at 11:10 -0700, multiplex wrote:
>>> Any suggestions ?
>>
>> I would use the LWIP_STATS code to identify if there is some resource
>> within lwIP that is running low.
>>
>> Kieran
>>
>>
>
> Thanks Kieran - Here are two printouts for the display_stats(). The first
> one is when everything is still going well; the second when it hangs

Unfortunately there's nothing there to point a finger a what might have gone wrong.  All the resources seem fine: there is plenty of everything available, and it doesn't seem like anything has run out at any point.

> (note:
> it is only lwIP that hangs - I still have serial - and if we disconnect and
> wait really really really long, the connection will closely nicely).

Can you explain that in more detail?  Can you be precise about what you mean by disconnect and how long a really really really long time is?  If lwIP has hung, I wouldn't expect it to do anything no matter how long you wait.

Thanks

Kieran
From the system logs, I get the message that the connection was shut down correctly - my interpretation was that, due to the other side disconnecting when you close the telnet session, at a certain we hit th timeout (which for TCP can be really long).

Too bad - nothing seemed wrong to me either. Any chance we might be losing ACKs? And we just start waiting, hoping for them to come back? If so - I am currently looking at switching to UDP.

One last thing. If we take Xilinx' lastezd and greatezd SDK 12.1 - we get 1.3.2 as well. Would that help, IYO?
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Kieran Mansley

On 26 May 2010, at 20:04, multiplex wrote:

>
> One last thing. If we take Xilinx' lastezd and greatezd SDK 12.1 - we get
> 1.3.2 as well. Would that help, IYO?

Yes.  I think I remember that in earlier versions of the SDK it was not passing received packets to lwIP properly, although I could have confused that with a different system.

Kieran
_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Tyrel Newton
In reply to this post by multiplex

> One last thing. If we take Xilinx' lastezd and greatezd SDK 12.1 - we get
> 1.3.2 as well. Would that help, IYO?

I would provide more insight on this, but I'm currently using ISE 11.4 with the emaclite (10/100 mac) peripheral, FreeRTOS, and the latest CVS head for lwIP. As I mentioned previously, I also re-wrote the lwIP driver provided by Xilinx because their driver was rather terrible and inefficient.

As I have been unimpressed with the application-level code provided by Xilinx, I'm betting their lwIP temac driver is not the best. The temac peripheral has a lot of customizable runtime settings and I'm betting the lwIP driver does not set them correctly or doesn't provide hooks to set them at all. For example, perhaps you are running out of some resource at the lwIP temac driver level, such as queue space for outgoing packets or buffer descriptors for the scatter-gather DMA stuff.

I highly doubt moving to 12.1 and 1.3.2 will fix your problem. Its very likely at the lwIP temac driver level, which Xilinx probably didn't update between releases (the 12.1 release heavily focused on support for the Spartan/Virtex-6 FPGAs).

Tyrel
_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

multiplex

Tyrel Newton wrote
> One last thing. If we take Xilinx' lastezd and greatezd SDK 12.1 - we get
> 1.3.2 as well. Would that help, IYO?

I would provide more insight on this, but I'm currently using ISE 11.4 with the emaclite (10/100 mac) peripheral, FreeRTOS, and the latest CVS head for lwIP. As I mentioned previously, I also re-wrote the lwIP driver provided by Xilinx because their driver was rather terrible and inefficient.

As I have been unimpressed with the application-level code provided by Xilinx, I'm betting their lwIP temac driver is not the best. The temac peripheral has a lot of customizable runtime settings and I'm betting the lwIP driver does not set them correctly or doesn't provide hooks to set them at all. For example, perhaps you are running out of some resource at the lwIP temac driver level, such as queue space for outgoing packets or buffer descriptors for the scatter-gather DMA stuff.

I highly doubt moving to 12.1 and 1.3.2 will fix your problem. Its very likely at the lwIP temac driver level, which Xilinx probably didn't update between releases (the 12.1 release heavily focused on support for the Spartan/Virtex-6 FPGAs).

Tyrel
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/lwip-users
Unfortunately, IIRC, emaclite does not support backtoback packets. Since we are sitting right behind a packet processor, we do expect this to happen - and we have to process non-TCP as well so cannot afford to drop packets at high rates. Has this restriction changed? Would you provide your driver, should we ask?
Reply | Threaded
Open this post in threaded view
|

Re: lwIP hangs on transfers with many short packets

Tyrel Newton

> Unfortunately, IIRC, emaclite does not support backtoback packets. Since we
> are sitting right behind a packet processor, we do expect this to happen -
> and we have to process non-TCP as well so cannot afford to drop packets at
> high rates. Has this restriction changed?

If you configure the emaclite with both the ping/pong buffers, its possible to receive/transmit packets back-to-back. The real limitation with the emaclite is that there is no DMA. This means the MicroBlace/PPC needs to manually move data in/out of the mac's frame buffers, which they are especially not good at because of the data alignment requirements. The main problem is that if the MicroBlaze/PPC is copying data, then it is not processing the contents of the packets (i.e. TCP/IP processing). There is really no way that an emaclite-based system could keep a continuous stream of packets flowing in either direction.

> Would you provide your driver,
> should we ask?


All I really did with the emaclite driver was strip out all of the configurability and cruff so that it is simply a data copier into and out of the emaclite's frame buffers. Boiled down to its simplest, I have only been able to get about 10 Mbps out of the emaclite using the following setup (this is the basic setup provided with the Spartan-3A DSP 1800 development board):
- 62.5 MHz MicroBlaze
- 125 MHz DDR2 w/ 8k BRAM caches
The primary bottleneck is definitely the 62.5 MHz MicroBlaze copying data. With the default emacite lwIP driver provided by Xilinx, I think the peak rate I saw was in the 1~2 Mbps range.

I would provide the driver, but if you are using the temac, I would definitely recommend staying with that. I would just verify that there is nothing silly going on. For example, with Xilinx's default lwIP/emaclite driver, the outgoing pbufs are copied to an intermediary buffer. This extra copy is completely unnecessary and severely limits the throughput.

Tyrel
_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users