Reopening LWIP for new HTTP message

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Reopening LWIP for new HTTP message

H.A.
I am using LWIP 1.3.2 and software based on the Stellaris ser2enet (Cortex M3) example. It runs a web server  for configuration plus a simple http client based on the telnet client. All dedicated telnet processing has been removed so it is basically using raw data. Data comes in on a serial port and is assembled in a global buffer in the serial receive interrupt routine until complete. Data is than transmitted with tcp_write on port 80 to a server and flushed out with tcp_output straight from the interrupt routine. We can expect to have new data perhaps once per hour and it is normally around 130 bytes.

It works but the server shuts down the connection every 10 seconds. LWIP detects this and does a callback to telnet_error which tears down the old connection completely (4 TCP messages) and reconnects (3 TCP messages). As far as I can see these messages are all empty from a TCP point of view (they manipulate the flags). And they are all 60 bytes.

We could perhaps have a couple of hundred of these devices connected to a single server so I am a little bit worried about the unnecessary Ethernet traffic, disconnect/connect, when there really is no new data available.

Have tried several things like opening a new connection inside the serial receive routine instead from the error callback. Didn’t work. What is the best way to solve this. Opening the connection from somewhere else? How?
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

Kieran Mansley

On 7 Aug 2012, at 12:00, H.A. wrote:

> Have tried several things like opening a new connection inside the serial
> receive routine instead from the error callback. Didn’t work. What is the
> best way to solve this. Opening the connection from somewhere else? How?

What didn't work when you tried to open the connection?

Kieran
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

H.A.
Connection didn't open and no tcp output, nothing visible in Wireshark. OK, the open is a non-blocking command so it might take a "certain" amount of time for it to open. But it doesn't feel right to sit and wait for the connection to open inside an interrrupt routine. Any ideas?

Kieran Mansley wrote
On 7 Aug 2012, at 12:00, H.A. wrote:

> Have tried several things like opening a new connection inside the serial
> receive routine instead from the error callback. Didn’t work. What is the
> best way to solve this. Opening the connection from somewhere else? How?

What didn't work when you tried to open the connection?

Kieran
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

goldsimon@gmx.de
"H.A." <[hidden email]> wrote:
> Connection didn't open and no tcp output, nothing visible in Wireshark.
> OK,
> the open is a non-blocking command so it might take a "certain" amount of
> time for it to open. But it doesn't feel right to sit and wait for the
> connection to open inside an interrrupt routine.

Cretainly not! The raw API is an event-based API: you call tcp_connect() and return instead of watiting. Either your connected-callback (on successful connect) or your err-callback will be called later.

However, you should at least see a SYN packet going out when calling tcp_connect().

But as I mentioned before, the Stellaris port for lwIP I once had a look at was using lwIP in a strange way, so this *might* be a race condition of using lwIP from different interrupt-levels (or interrupt and main loop).

Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

H.A.
As I tried to mention before, only the serial receive routine has currently the knowledge of when a complete serial message is recevied and it than uses tcp_write and tcp_output to send out the message.

Do you mean that I should check if the connection is open in the interrupt routine? If open send the message. If not open, open the connection, return and let the connect callback send the message? Can the connection be closed between the check and the transmit inside the interrupt routine? Is this mix of different contexts really safe?

When I tried to open the connection inside the serial interrupt I did not see any SYN packages. Do I have to do an init one more time also?

When I let the error callback reconnect everything looks normal in Wireshark. I am just a little bit worried about the big amount of disconnect/connect messages if we use many devices.

Simon Goldschmidt wrote
Cretainly not! The raw API is an event-based API: you call tcp_connect() and return instead of watiting. Either your connected-callback (on successful connect) or your err-callback will be called later.

However, you should at least see a SYN packet going out when calling tcp_connect().

But as I mentioned before, the Stellaris port for lwIP I once had a look at was using lwIP in a strange way, so this *might* be a race condition of using lwIP from different interrupt-levels (or interrupt and main loop).

Simon

_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

Sylvain Rochet
Hi,

On Wed, Aug 08, 2012 at 06:14:52AM -0700, H.A. wrote:
>
> Do you mean that I should check if the connection is open in the interrupt
> routine? If open send the message. If not open, open the connection, return
> and let the connect callback send the message? Can the connection be closed
> between the check and the transmit inside the interrupt routine? Is this mix
> of different contexts really safe?

Of course not !

You have to read lwip/doc/rawapi.txt in order to understand how to use
lwIP with threads.

Sylvain

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

H.A.
Of course not what? I had at least 3 questions.

I do understand that the LWIP stack is not reentrant but the Stellaris code is using tcp_write and tcp_output from the serial interrupt and it is even sold a a commercial product. (serial to ethernet module using telnet).

So it must work somehow but I don't undestand how.
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

Sylvain Rochet
Hi,


On Wed, Aug 08, 2012 at 06:29:47AM -0700, H.A. wrote:
>
> Of course not what? I had at least 3 questions.

This is why I quoted only the relevant part.


> I do understand that the LWIP stack is not reentrant but the Stellaris code
> is using tcp_write and tcp_output from the serial interrupt and it is even
> sold a a commercial product. (serial to ethernet module using telnet).

Sorry to say that, but this is defective by design. Every lwIP raw API
call must be run from the lwIP context, no exception.


NO_SYS == 1 :

everything should be done in the "main()" context.


NO_SYS == 0 :

lwIP is running in its own thread (started from TCPIP API init function).
sequential (netconn) or socket API must be used outside of the lwIP thread.


> So it must work somehow but I don't undestand how.

By chance... actually it depends if you are in the "in watchdog we
trust" way of designing things or not.


Sylvain

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

Dave Wilson
In reply to this post by H.A.

H.A. wrote
As I tried to mention before, only the serial receive routine has currently the knowledge of when a complete serial message is recevied and it than uses tcp_write and tcp_output to send out the message.

Do you mean that I should check if the connection is open in the interrupt routine? If open send the message. If not open, open the connection, return and let the connect callback send the message? Can the connection be closed between the check and the transmit inside the interrupt routine? Is this mix of different contexts really safe?

When I tried to open the connection inside the serial interrupt I did not see any SYN packages. Do I have to do an init one more time also?
It sounds to me as if you have some serious context and race condition issues going on here. The Stellaris port of lwIP 1.3.2 is written such that you may only call the lwIP raw APIs from within the context of the Ethernet interrupt handler or, if using SafeRTOS or FreeRTOS, the lwIP thread. If you call from any other context, you will definitely see problems since the raw API is not protected against calling from multiple contexts and certainly not from multiple interrupt contexts.

If you look at the original code for the ser2enet application, you will see that the serial side writes data into a ring buffer when it is received. This buffer is then read from a handler function called from lwIPHostTimerHandler() which is itself called in the right context to allow you to call the raw API safely. You should ensure that you design your application similarly such that lwIP calls are only made from within lwIPHostTimerHandler().


Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

H.A.
It was quite some time I modified this code and now I had to update it again. You are right, I was wrong! The tcp_write / tcp_output are called from TelnetHandler which is called from lwIPHostTimerHandler.

I also think I found my reconnect bug and will test it now.
Reply | Threaded
Open this post in threaded view
|

Linkedin - Networking (attention off topic)

Funk, Artur
In reply to this post by Dave Wilson
I don't know if off topic is allowed or not, sorry If not :>

Would you mind to build a LwIP developers connection on linkedin?
Everyone who wants
to be contacted and connected by further mailing list users,
is invited to send his name/surename and location. Others
are allowed to add the person on linkedin. I would make a start:

Artur Funk Berlin




Best Regards

Artur

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Linkedin - Networking (attention off topic)

Richard Barry-2
I for one would rather keep all lwIP information and discussion in a
single dedicated place, not dispersed across multiple locations.


Regards,
Richard.

+ http://www.FreeRTOS.org
Designed for microcontrollers.  More than 7000 downloads per month.

+ http://www.FreeRTOS.org/trace
15 interconnected trace views. An indispensable productivity tool.



On 08/08/2012 15:20, Funk, Artur wrote:

> I don't know if off topic is allowed or not, sorry If not :>
>
> Would you mind to build a LwIP developers connection on linkedin?
> Everyone who wants
> to be contacted and connected by further mailing list users,
> is invited to send his name/surename and location. Others
> are allowed to add the person on linkedin. I would make a start:
>
> Artur Funk Berlin
>
>
>
>
> Best Regards
>
> Artur
>
> _______________________________________________
> lwip-users mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/lwip-users
>

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: Linkedin - Networking (attention off topic)

FreeRTOS info

I would rather keep all lwIP information and discussion in a
single dedicated place, not dispersed across multiple locations.


Regards,
Richard.

+ http://www.FreeRTOS.org
Designed for microcontrollers.  More than 7000 downloads per month.

+ http://www.FreeRTOS.org/trace
15 interconnected trace views. An indispensable productivity tool.



On 08/08/2012 15:20, Funk, Artur wrote:

> I don't know if off topic is allowed or not, sorry If not :>
>
> Would you mind to build a LwIP developers connection on linkedin?
> Everyone who wants
> to be contacted and connected by further mailing list users,
> is invited to send his name/surename and location. Others
> are allowed to add the person on linkedin. I would make a start:
>
> Artur Funk Berlin
>
>
>
>
> Best Regards
>
> Artur
>
> _______________________________________________
> lwip-users mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/lwip-users
>



_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Re opening LWIP for new HTTP message

goldsimon@gmx.de
In reply to this post by H.A.
dawilson <[hidden email]> wrote:

> It sounds to me as if you have some serious context and race condition
> issues going on here. The Stellaris port of lwIP 1.3.2 is written such
> that
> you may only call the lwIP raw APIs from within the context of the
> Ethernet
> interrupt handler or, if using SafeRTOS or FreeRTOS, the lwIP thread. If
> you
> call from any other context, you will definitely see problems since the
> raw
> API is not protected against calling from multiple contexts and certainly
> not from multiple interrupt contexts.


Dave, thanks so much for your support here. It's always hard to have an overview of all the lwIP ports available, so I think it's really nice to have someone around who can help with port-related problems.

Thanks again,
Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: Linkedin - Networking (attention off topic)

Funk, Artur
In reply to this post by FreeRTOS info
>I would rather keep all lwIP information and discussion in a single
dedicated place, not dispersed across multiple locations.

I agree, we shall keep all discussions in one place. You have
misunderstand the purpose. I mean just to build a connection not for
support or
discussions. The sense is to connect with other developers not with the
own boss or dilbert whatever ;)


Best Regards

Artur

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Linkedin - Networking (attention off topic)

goldsimon@gmx.de
In reply to this post by Funk, Artur
Please don't use the response function when your mail is not a response but a complete new topic: advanced email clients (like on the iPad:) sort mails by the referred IDs, so your LinkedIn-mail got sorted into another thread - this makes following discussions unnecessarily hard.

Simon


 "Funk, Artur" <[hidden email]>:

> I don't know if off topic is allowed or not, sorry If not :>
>
> Would you mind to build a LwIP developers connection on linkedin?
> Everyone who wants
> to be contacted and connected by further mailing list users,
> is invited to send his name/surename and location. Others
> are allowed to add the person on linkedin. I would make a start:
>
> Artur Funk Berlin
>
>
>
>
> Best Regards
>
> Artur
>
> _______________________________________________
> lwip-users mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/lwip-users

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Linkedin - Networking (attention off topic)

H.A.
I still can't get this to work. I open a "telnet like" raw connection to a server on port 80 and send a string. It works but after 10 seconds the server shuts down the connection. The telnetReceive detects this and tears down the connection completely, reopens the connection. Each time 7 empty packets are sent every 10 seconds. It works but with many devices this can generate a lot of traffic.

What I'm trying to do is to remove the open in the telnetReceive and reconnect only when i need inside the telnethandler which is called from the lwIPHostTimerHandler. According to Dave Wilson that should be the right context when working with the Stellaris port. In fact this is the only function called from this timer sice I have removed UPNP.  My idea is the the TelnetHandler should sort of poll the status of the connection and send the data when connected.

Tried many different ways but nothing works. Am I doing something completely wrong?
This code is placed at the end of the TelnetHandler. Can't even see a SYN in Wireshark.
This is not the serial interrupt handler as I incorrectly stated before. It reads the serial
ringbuffer.

//
// Reconnect port 1 if required.
//
    if(pState->eTCPState != STATE_TCP_CONNECTED)
    {
        TelnetOpen(pState->ulTelnetRemoteIP, pState->usTelnetRemotePort,
                          pState->usTelnetLocalPort, 1);
            // TelnetOpen(g_sParameters.sPort[1].ulTelnetIPAddr,
            // g_sParameters.sPort[1].usTelnetRemotePort,
            // g_sParameters.sPort[1].usTelnetLocalPort,1);
        return;
    }

    // Put string into the TCP buffer.
    tcp_write(pState->pConnectPCB, pucTemp, strlen((char *)pucTemp), 1);

    // Flush the data that has been written into the TCP output buffer
    tcp_output(pState->pConnectPCB);
    pState->ulLastTCPSendTime = g_ulSystemTimeMS;
    bRxMsgFlag = false;              // Clear some stuff for next time
    ucRxIndex = 0;
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect HTTP

H.A.
I do have this code before the reconnect. It is probably relevant
            //
            // Avoid repeated connect
            //
            if(pState->eTCPState == STATE_TCP_CONNECTING)
                return;

Am I in the wrong context for the stellaris port?


H.A. wrote
I still can't get this to work. I open a "telnet like" raw connection to a server on port 80 and send a string. It works but after 10 seconds the server shuts down the connection. The telnetReceive detects this and tears down the connection completely, reopens the connection. Each time 7 empty packets are sent every 10 seconds. It works but with many devices this can generate a lot of traffic.

What I'm trying to do is to remove the open in the telnetReceive and reconnect only when i need inside the telnethandler which is called from the lwIPHostTimerHandler. According to Dave Wilson that should be the right context when working with the Stellaris port. In fact this is the only function called from this timer sice I have removed UPNP.  My idea is the the TelnetHandler should sort of poll the status of the connection and send the data when connected.

Tried many different ways but nothing works. Am I doing something completely wrong?
This code is placed at the end of the TelnetHandler. Can't even see a SYN in Wireshark.
This is not the serial interrupt handler as I incorrectly stated before. It reads the serial
ringbuffer.

//
// Reconnect port 1 if required.
//
    if(pState->eTCPState != STATE_TCP_CONNECTED)
    {
        TelnetOpen(pState->ulTelnetRemoteIP, pState->usTelnetRemotePort,
                          pState->usTelnetLocalPort, 1);
            // TelnetOpen(g_sParameters.sPort[1].ulTelnetIPAddr,
            // g_sParameters.sPort[1].usTelnetRemotePort,
            // g_sParameters.sPort[1].usTelnetLocalPort,1);
        return;
    }

    // Put string into the TCP buffer.
    tcp_write(pState->pConnectPCB, pucTemp, strlen((char *)pucTemp), 1);

    // Flush the data that has been written into the TCP output buffer
    tcp_output(pState->pConnectPCB);
    pState->ulLastTCPSendTime = g_ulSystemTimeMS;
    bRxMsgFlag = false;              // Clear some stuff for next time
    ucRxIndex = 0;
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect HTTP

H.A.
This is really a question to Simon Goldschmidt.

When you reply to a post you always show the first part of the other persons email address as a quote.
When other people answer you just see the username. In my case I have a rather unusual name and I don't like the idea that someone can build a data base on me by doing some clever Internet searches. If someone wants my name I have no problems give it out on the forum but I would prefer to insert some spaces, punctuations etc. to avoid Internet searches and bots.

As said in the nabble help:
Is my email address protected?
Yes. All email addresses on Nabble are obfuscated.
Removing just the domain is not really obfuscating. At least not in my case. You could always try this partial address on all big mail sites. If the mail doesn't bounce, they know it is a valid address. I'm sure there are people that use scripts for that to generate spam. I have enough of that already.

Anyway, my LWIP problem is not yet solved. It works but I have too much traffic.

H.A. wrote
I do have this code before the reconnect. It is probably relevant
            //
            // Avoid repeated connect
            //
            if(pState->eTCPState == STATE_TCP_CONNECTING)
                return;

Am I in the wrong context for the stellaris port?


H.A. wrote
I still can't get this to work. I open a "telnet like" raw connection to a server on port 80 and send a string. It works but after 10 seconds the server shuts down the connection. The telnetReceive detects this and tears down the connection completely, reopens the connection. Each time 7 empty packets are sent every 10 seconds. It works but with many devices this can generate a lot of traffic.

What I'm trying to do is to remove the open in the telnetReceive and reconnect only when i need inside the telnethandler which is called from the lwIPHostTimerHandler. According to Dave Wilson that should be the right context when working with the Stellaris port. In fact this is the only function called from this timer sice I have removed UPNP.  My idea is the the TelnetHandler should sort of poll the status of the connection and send the data when connected.

Tried many different ways but nothing works. Am I doing something completely wrong?
This code is placed at the end of the TelnetHandler. Can't even see a SYN in Wireshark.
This is not the serial interrupt handler as I incorrectly stated before. It reads the serial
ringbuffer.

//
// Reconnect port 1 if required.
//
    if(pState->eTCPState != STATE_TCP_CONNECTED)
    {
        TelnetOpen(pState->ulTelnetRemoteIP, pState->usTelnetRemotePort,
                          pState->usTelnetLocalPort, 1);
            // TelnetOpen(g_sParameters.sPort[1].ulTelnetIPAddr,
            // g_sParameters.sPort[1].usTelnetRemotePort,
            // g_sParameters.sPort[1].usTelnetLocalPort,1);
        return;
    }

    // Put string into the TCP buffer.
    tcp_write(pState->pConnectPCB, pucTemp, strlen((char *)pucTemp), 1);

    // Flush the data that has been written into the TCP output buffer
    tcp_output(pState->pConnectPCB);
    pState->ulLastTCPSendTime = g_ulSystemTimeMS;
    bRxMsgFlag = false;              // Clear some stuff for next time
    ucRxIndex = 0;
Reply | Threaded
Open this post in threaded view
|

Re: Re connect HTTP

goldsimon@gmx.de


>
> This is really a question to Simon Goldschmidt.

So what's your question really? This is how my email program generates responses. Everyone subscribing to lwip-users can see your email.


> As said in the nabble help:

What does nabble have to do with lwip-users?

Simon
_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
12