Re: Raw TCP - PBuf segment not decreasing

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

Sergio R. Caprile
Instead of speculating on a slow receiver, I would see traffic on the
wire and check my tcp_sent() callbacks.
You stuff the TCP buffer with tcp_write() and it will eventually get
sent, you can speed things up as you are doing by calling tcp_output(),
and the stack will let you know when your receiver has ACKed by calling
you at what you provided for tcp_sent(). You should take advantage of
this callback.
I don't think pushing at 2KHz is a good idea, you should perhaps buffer
at your desired speed and send at TCP speed; I mean, queue when you want
to and unqueue at the callback. Perhaps "queue" can be tcp_write() and
"unqueue" tcp_output(), didn't try myself, but remember both must be on
the same context, no 2KHz interrupts please, in such case you will need
a decoupling mechanism.

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

LRA_
Thank you for the reply.
The example provided is not very good to explain how to use all that so I
probably did something wrong.

I actually didn't quite understand how to use tcp_sent() since I didn't
understand where the argument came from (the buffer to send). I will try to
find out more about how to use it in this case without OS.

What you mean by " but remember both must be on the same context" is that I
should not use tcp_write in normal run code and tcp_output inside interrupt
routine? Since this has no threads.



What I would try from this is:
- set up tcp_sent() callback
- Call tcp_write at 2Khz (or whatever).
- If tcp_write has the current buffer full, set new buffer for tcp_sent to
handle.
- Call tcp_output inside tcp_sent() callback. After that, let it call
tcp_write if there is extra data.






--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

Sergio R. Caprile
In reply to this post by Sergio R. Caprile
Well, perhaps TI was short on examples but you can learn from the apps
section and those in the contrib tree.

> I actually didn't quite understand how to use tcp_sent() since [...]

When you open a connection (think you can be a server), you set the
environment to serve it and then setup an argument to be passed by
calling tcp_arg() that depends entirely on you. If you are an http
server, you will alloc some memory structure to handle each connection,
and that will be passed on calls to callback functions.
If you don't need it, you don't use it.

The connection has a pcb that is also different from the listening pcb,
so the connection pcb is passed as an argument to callback functions.

> What you mean by " but remember both must be on the same context" [...]

You git it right.
All RAW API functions must be called from within the same execution
environment (perhaps more proper than 'context'), you can handle
everything on the main loop or some people choose to work only
interrupt-driven using the Ethernet Rx interrupt. Those functions are
not protected for reentrability so you must use them on the same
"thread", whether you have an OS or not. (execution environment...)


Most work requires sending some content in response to a request or an
event. You then put inside the TCP buffer as much as you can send, and
if more is left, then, when tcp_sent() informs you that you can send
more because the other end ACKed, you keep pushing until it is done.
There is the first "intention" to send the contents in some part of your
app, in response to the event or the request, and then the real-life
sending in response to the tcp_sent() callback.
Since you want to send at 2KHz, your "intention" will fire every 500us.
TCP might think otherwise, though.

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

LRA_
Hello

Again, thank you very much. I probably am just really bad at using these
sources but I find it hard to find all the info so your help is really
great.

So I have now a different setup, still just for testing. I think I
understand better but it looks like there is some concept I am not grasping.
So I have:
- the "main".
- The ISRs (ethernet ISR and timer ISR)

So ethernet ISR and timer ISR should be in the same context if they have the
same priority.
The main should be like a low priority task.

In the main I have a function to send data. The function either
a) call tcp_sent() and tcp_write() if no data is loaded. This between
disabling interrupts (all for now) so it's like a priority inversion and the
ISRs should not be where they shouldn't.
b) load up my buffer - a buffer that I created for the purpose.

In the timer ISR:
- Increment lwip timer by 5.
- Call tcp_output every 10 counts, so 50ms.
- Load tcp_write if there is data available and tcp_sent callback has
happened.


With this I can see an increase in throughput though I see many lost data -
my buffer fills up too quickly.
I have tried to instead just fill my buffer when TCP snd_buf is full
(tcp_write() returns err_mem).
I am trying to find the bottle neck with no success. In wireshark I see that
never more than 2 segments are sent without an ACK. Some cases it sends 2 at
the same time, get and ACK, send other 2 just 3ms later (all with len=1460).
So this seems high enough throughput on some bursts.

I already tried with nagle enabled and disabled. With nagle disabled I had
to buffer all data into tcp_write to not run out of snd_queuelen. But it
seems to me that, since I don't quite need crazy low latency, nagle should
be enabled.



I can see some issues with my test case even for just a test.
But not sure if I am in the right path.
This is very different from playing with tiny tests in Linux with Unix
sockets.



--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

LRA_
In reply to this post by Sergio R. Caprile
Well okay I did some more tests.
Also I changed where tcp_write loading from my buffer happens. Now it's in
tcp_sent instead of the timer interrupt that calls lwIPTimer.

Decided to use putty to read the data. I assume it's pretty fast for that.
This should remove a bottleneck on the PC side.
Had it running for 30minutes with no apparent loss. Wireshark shows 800kb/s
with 1s window.
The problem I see is that pcb->snd_queuelen creeps up. In that test it went
up to 13-16.
Tried a shorter test where I stopped the generation of messages (stopped
loading bytes into tcp_write) mid way and it never decreased.

Not sure if this is normal behavior, not sure whats causing it.



On a side note. I had a problem where adding a simple volatile uint32_t,
working as a boolean, to the message generator "if" caused Fault in
tcp_output when doing "add segment to tail of unacked list". It seems pcb
was NULL... Not sure if this was the compiler being silly.










--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

Sergio R. Caprile
In reply to this post by Sergio R. Caprile
I understand your frustration but it is vendor generated, not lwIP's
fault. Say "bye, thanks" to your vendor "docs" (or the lack of them)
and dive here:
        https://www.nongnu.org/lwip/2_1_x/        (or the version you're using)
The official source code tree has a directory named "apps" where you can
find working apps that will let you understand how to develop your own.
        https://www.nongnu.org/lwip/2_1_x/files.html
When in doubt, check how the gurus did it. SMTP is quite a piece.
Then there is the contrib tree, which carries all that stuff that has
been contributed but has not been elevated to the status of official app
(yet). Many of them are great starting points, particularly the
"tcpecho_raw" server, it is simple enough to grasp the idea.
        http://download.savannah.nongnu.org/releases/lwip/
... and finally the "examples" directory there has, well, LOTS of examples.

Regarding your "locks", I think you know what you are doing, and if you
did it right it should work, but we mortals tend to make mistakes and I
prefer to let the interrupt magic for a later time after all that is
happening has its proper explanation. So, if I were in your shoes, I
would signal on Ethernet Rx and Timer Tick and collect and send on main
loop; at least for now. For greater receive throughput you can queue on
Eth Rx and unqueue on main loop, the pbuf alloc and free functions can
be called from both contexts according to the docs and to the guys here
(I'm still flagging but I don't need no rx throughput for what I do
right now; a main loop with nothing else to do can tx/rx around 800KB+/s
on a 70MHz Cortex-M3 with no DMA, so don't rush to build a queue now)
Strange things like what you describe are easily explained by
reentrability issues, so...
You can play with sockets if you want, though they require an OS and
more memory.

Finally, tcp_write() just writes to a buffer; TCP will send when it
wants to.
Then tcp_output() is a way to suggest TCP to do it.
Calling tcp_write at the timer tick and not in the tcp_sent() callback
generates a dead time between the ACK and your timer, which can
translate in upto 50ms wasted. To keep TCP flowing, tcp_write() on
tcp_sent callback except if all the data for the former tick has been
sent; you will start again on next tick.
Before tcp_write(), you should (must) check for room with tcp_sndbuf().
It is OK to call tcp_output() after tcp_write() if you just pushed some
data. I never called tcp_output() just for the sake of it.
This is a high-performance example from the httpd server in times of
yore, haven't checked for recent incarnations.

    if (http_send(hs->pcb, hs)) {
      /* If we wrote anything to be sent, go ahead and send it now. */
      LWIP_DEBUGF(HTTPD_DEBUG | LWIP_DBG_TRACE, ("tcp_output\n"));
      tcp_output(hs->pcb);

and the send loop called by http_send() is something like this:
  do {
    LWIP_DEBUGF(HTTPD_DEBUG | LWIP_DBG_TRACE, ("Trying to send %d bytes
with flags: %u\n", len, apiflags));
    err = tcp_write(pcb, ptr, len, apiflags);
    if (err == ERR_MEM) {
      if ((tcp_sndbuf(pcb) == 0) ||
        (tcp_sndqueuelen(pcb) >= TCP_SND_QUEUELEN)) {
          /* no need to try smaller sizes */
          len = 1;
      } else {
        len /= 2;
      }
      LWIP_DEBUGF(HTTPD_DEBUG | LWIP_DBG_TRACE,
        ("Send failed, trying less (%d bytes)\n", len));
    }
  } while ((err == ERR_MEM) && (len > 1));

http_send() is called when starting to send "a file" and on the
connection's tcp_sent() callback.

OH! And there is the timing stuff... your main() should frequently call
the timer handling function that keeps all timed stuff (including TCP)
going:
        sys_check_timeouts()

Cheers

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

LRA_
Hello Sergio,

Sorry for taking long to reply, flu season hasn't been kind.

Nevertheless I did some tests. And you are correct, they are reentrability
issues. The periodic interrupt that called tcp_write was higher priority
than the ethernet interrupt, set them to the same and it's been working very
well. I had it running some tests so far and no issues from the tm4c test
code (let's see later trying to use it in the real code).

The issue of PBuf segment not decreasing is solved so I guess this can be
closed.
Like most searches hinted, it was really reentrability issues.


For the rest,
The port has everything pretty much on the ethernet interrupt. I probably
could change it, I hate it being there. The example I based myself on is the
raw TCP_Echo.
I am not using RTOS because it's an existing project, actually the best
supported RTOS is TI-RTOS which doesn't seem to even use LWIP, I prefer
FreeRTOS but oh well...

Btw, the processor is a 120Mhz ARM-M4 and I am not using DMA (for
reference).



I have to read more into your answer, the links you provided and the http
server example. For now it's rest for me.


 



--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Raw TCP - PBuf segment not decreasing

Nathan Hartman
On Sat, Mar 23, 2019 at 10:59 AM LRA_ <[hidden email]> wrote:

The port has everything pretty much on the ethernet interrupt. I probably
could change it, I hate it being there.

There's an article about how to get it out of the interrupt. Not sure what the performance is like but it's for TM4C12x:


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users