Handle a broadcast storm

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Handle a broadcast storm

michele.corradin@socomec.com

Hello,

I’m experiencing a problem of watchdog in a STM32F407 board running LWIP and uCOS-III. Doing some debug I discovered that sometimes I receive up to 400 messages in less than 20ms locking my Ethernet interrupt longer than the watchdog refresh time. I have 3 receive buffers for the Ethernet MAC so it seems that really I’m receiving just a message after the other. Doing a wireshark check I noticed that effectively sometimes even my pc receives some broadcast message, normally up to 4-5 just one after the other and probably sometimes there are much more coming.

Is there any suggested way to deal with this type of condition? I’m not a big expert of network management: I did a search on web I was not able to find anything.

Thanks for the support

Michele

 

This message and any attachments are established exclusively for his or its recipients, and are confidential. Any use, diffusion or unauthorized publication is prohibited. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. SOCOMEC declines all responsibility concerning this message if it has been altered or tampered with. It normally contains no virus, but it is the responsibility of the recipient to ensure this. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization.

Ce message et ses éventuelles pièces jointes sont établis à l'intention exclusive de son ou de ses destinataires et sont confidentiels. Toute utilisation, diffusion ou publication non autorisée est prohibée. Merci d’informer immédiatement l’auteur par retour de message si vous avez reçu ce message par erreur, et supprimez ce message. SOCOMEC décline toute responsabilité au titre de ce message s'il a été altéré ou falsifié. Il ne contient normalement aucun virus, mais il est de la responsabilité de son destinataire de s'en assurer. L’organisation décline toute responsabilité en ce qui concerne les informations fournies et les avis exprimés dans le présent message.

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

Patrick Klos-2
On 3/21/2019 5:27 AM, CORRADIN Michele wrote:

Hello,

I’m experiencing a problem of watchdog in a STM32F407 board running LWIP and uCOS-III. Doing some debug I discovered that sometimes I receive up to 400 messages in less than 20ms locking my Ethernet interrupt longer than the watchdog refresh time. I have 3 receive buffers for the Ethernet MAC so it seems that really I’m receiving just a message after the other. Doing a wireshark check I noticed that effectively sometimes even my pc receives some broadcast message, normally up to 4-5 just one after the other and probably sometimes there are much more coming.

Is there any suggested way to deal with this type of condition? I’m not a big expert of network management: I did a search on web I was not able to find anything.

Thanks for the support

Michele


Hello Michele,

What kind of broadcast packets is your device receiving?  Are these broadcast packets actually IP and/or ARP packets?  If they aren't, maybe you can optimize the path in your device's ethernet driver to discard these uninteresting packets more efficiently?

Patrick Klos
Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

michele.corradin@socomec.com
It seems they are TCP packets

Thanks for your support
Michele

<http://lwip.100.n7.nabble.com/file/t1187/broadcast.jpg>



--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

goldsimon@gmx.de
Am 21.03.2019 um 17:44 schrieb [hidden email]:
> It seems they are TCP packets
>
> Thanks for your support
> Michele
>
> <http://lwip.100.n7.nabble.com/file/t1187/broadcast.jpg>

As always, please send pcaps, not screenshots.

Also, describe what we see, e.g. start with telling us which devices
have which IP / MAC address etc.

Having TCP using broadcast is strange. Having 255.255.255.255 as a
source address is even more strange.

Regards,
Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

Patrick Klos-2
On 3/21/2019 3:01 PM, [hidden email] wrote:

> Am 21.03.2019 um 17:44 schrieb [hidden email]:
>> It seems they are TCP packets
>>
>> Thanks for your support
>> Michele
>>
>> <http://lwip.100.n7.nabble.com/file/t1187/broadcast.jpg>
>
> As always, please send pcaps, not screenshots.
>
> Also, describe what we see, e.g. start with telling us which devices
> have which IP / MAC address etc.
>
> Having TCP using broadcast is strange. Having 255.255.255.255 as a
> source address is even more strange.

Adding to what Simon indicated, yes, those are certainly invalid TCP
packets (104 thru 114).

One question is what device is on IP address 172.17.8.175? (and why is
it sending out a broadcast TCP SYN packet?)

The next question is what device is responding? (i.e. what device has
the MAC address of 00:40:9d:80:44:e3 on your network?)  Based on the
OUI, it appears to be a board from Digi International.  That device
appears to have a TCP/IP stack that should never have let the (invalid)
broadcast TCP packet(s) get anywhere near the TCP stack. And why is it
responding with (at least) 9 TCP RST packets?

Yes, a PCAP file would be a little more useful / interesting.

Patrick Klos
Klos Technologies, Inc.


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

michele.corradin@socomec.com
Dear all,
thanks for all your replies and sorry if I didn't add enough information.
I'm not able to reply to your questions now, I have to check with ITD to try
to discover that device.
As you can see the problem I'm experiencing in my embedded board is that the
packets it is sending are very near each other and I was able to count up to
400 of them.
Again I'm not an expert of ethernet but is there a way to handle this type
of conditions? Do you have any suggestion? The best would be to filter at HW
level but I don't know if it's possible or I could do something disabling
temporarly Ethernet Irq but I would like to know if there is a common way to
handle these conditions to avoid just to try.
I attached a pcap file.
thanks again
Michele

broadcast_storm.pcapng
<http://lwip.100.n7.nabble.com/file/t1187/broadcast_storm.pcapng>  



--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

Patrick Klos-2
On 3/22/2019 3:17 AM, [hidden email] wrote:
Dear all,
thanks for all your replies and sorry if I didn't add enough information.
I'm not able to reply to your questions now, I have to check with ITD to try
to discover that device.
As you can see the problem I'm experiencing in my embedded board is that the
packets it is sending are very near each other and I was able to count up to
400 of them.
Again I'm not an expert of ethernet but is there a way to handle this type
of conditions? Do you have any suggestion? The best would be to filter at HW
level but I don't know if it's possible or I could do something disabling
temporarly Ethernet Irq but I would like to know if there is a common way to
handle these conditions to avoid just to try.
I attached a pcap file.
thanks again
Michele

broadcast_storm.pcapng
<http://lwip.100.n7.nabble.com/file/t1187/broadcast_storm.pcapng>   

Hello again Michelle,

The PCAP file really helps shed some light on your situation. 

Based on the OUI, the device that is sending out the bogus TCP broadcast is made by Microhard S.R.L - that may help you identify that device.  (they seem to make various vending machines if that helps?)

It turns out that the Digi Board device that is responding with the (even more bogus) TCP broadcasts is actually not one, but 5 devices from Digi, each one sending out a single response.

While your IT department should be concerned with removing bad devices from your network, you still want your device to be able to tolerate bad packets and not hang up when they happen. 

I'm not sure how you're seeing 400 packets in a short period of time?  Or is that count after a few minutes of running? 

Either way, you could attempt to filter out at least some of the bad packets as early as possible.  If you're using ethernet_input(), you could filter the bogus Digi packets with the following code (or something similar):
    case PP_HTONS(ETHTYPE_IP):
      if (!(netif->flags & NETIF_FLAG_ETHARP)) {
        goto free_and_return;
      }
      /* skip Ethernet header (min. size checked above) */
      if (pbuf_remove_header(p, next_hdr_offset)) {
        LWIP_DEBUGF(ETHARP_DEBUG | LWIP_DBG_TRACE | LWIP_DBG_LEVEL_WARNING,
                    ("ethernet_input: IPv4 packet dropped, too short (%"U16_F"/%"U16_F")\n",
                     p->tot_len, next_hdr_offset));
        LWIP_DEBUGF(ETHARP_DEBUG | LWIP_DBG_TRACE, ("Can't move over header in packet"));
        goto free_and_return;
      } else {
================= Insert begin ===============
        /* filter out packets with a broadcast source address */
        if ((p->payload[0] == 0x45) &&
            (p->payload[12] == 0xff) &&
            (p->payload[13] == 0xff) &&
            (p->payload[14] == 0xff) &&
            (p->payload[15] == 0xff))
            goto free_and_return;
================= Insert end ===============
        /* pass to IP layer */
        ip4_input(p, netif);
      }
      break;

This code is near line 185 in ethernet.c from LwIP 2.1.2.

It would be better if you could do a similar filter in your ethernet driver before it even gets into LwIP.

Good luck!

Patrick


_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

michele.corradin@socomec.com
Thank you very much for your analysis, I was not able to do that by myself! I
confirm that in the embedded device, even if I have set 3 rx buffer, there
is a while loop going through the rx buffer and it seems they are loaded
while I'm reading them so I'm reaching up to 400 while cicles in ethernet
irq.
I don't think it's possible to block them at HW level acting on MAC filters:
if it's not possible the workaround you propose can help but doesn't solve
my issue because a frame process takes about 20us in my HW with 8-9 for pool
buffer allocation so even if I filter out the pool part and I avoid to call
LWIP I can still have 11us*400 that is a lot of time.
Thanks again for your analysis, I will have a look at the STM32, and I will
post my proposed solution.
Michele



--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

goldsimon@gmx.de
Am 22.03.2019 um 16:02 schrieb [hidden email]:

> Thank you very much for your analysis, I was not able to do that by myself! I
> confirm that in the embedded device, even if I have set 3 rx buffer, there
> is a while loop going through the rx buffer and it seems they are loaded
> while I'm reading them so I'm reaching up to 400 while cicles in ethernet
> irq.
> I don't think it's possible to block them at HW level acting on MAC filters:
> if it's not possible the workaround you propose can help but doesn't solve
> my issue because a frame process takes about 20us in my HW with 8-9 for pool
> buffer allocation so even if I filter out the pool part and I avoid to call
> LWIP I can still have 11us*400 that is a lot of time.
> Thanks again for your analysis, I will have a look at the STM32, and I will
> post my proposed solution.

If you have performance problems, don't filter in ethernet_input() but
do it in the ethernet ISR as early as you can.

Keeping that aside: what do you want to achieve? You can implement the
best filter you can think of, but still someone could DoS your device by
sending a lot of *valid* packets passing the filter.

I'd probably disable RX for some time (e.g. until X pending RX frames
have been processed) to make it more stable. This can lead to lost RX
packets of course, but then again: if your device seems to slow to
handle traffic at wire speed, there's not much you can do...

But a very early filter that prevents even allocating pbufs for packets
that would be dropped later would be a good idea to not get packet loss
because of broadcast data. You could even limit broadcast RX to X
packets in a row...

Regards,
Simon

_______________________________________________
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: Handle a broadcast storm

michele.corradin@socomec.com
This post was updated on .
dear all,
thanks a lot for your support. Reading the answers and the different
suggestions, considering STM32 can only disable/enable broadcast messages I
decided to:
1 -  limit the number of frame processed in Irq: as I wrote, in my setup I
have 3 rx buffer defined but I was able cycle many times in the ethernet Irq
because in the meantime I read an Rx buffer it was filled again. With the
protection I added I can limit that amount
2 - based on Patrick and Simon suggestions I modified ethernetif_input and
low_level_input to return the first frame byte in order to use it to
understand the type of frame received
3 - If I'm filled by broadcast frames I disable broadcast reception and I
re-enable it after (in the link status task I have)
This solution is currently running: I verified broadcast disable and I
verified sometimes my protection is triggered.
Thanks again
Michele



#define LWIP_MAX_IRQ_RX_FRAME    10u

static void LwIP_SendSem( void )
{
   U32 zFrameCounter = NULL_U32;
   U8 zFrameType;
   U32 zFrameTypeCounter[eMaxFrameType] = {0u,0u,0u};

   while ( HalEth_IsRxPktValid() && (zFrameCounter < LWIP_MAX_IRQ_RX_FRAME))
   {
      zFrameType = ethernetif_input( &gNetIf[0] );
      zFrameCounter++;
      if (zFrameType == (U8)0xFF)
      {
         zFrameTypeCounter[eBroadcast]++;
      }
      else if (zFrameType == (U8)0x01)
      {
         zFrameTypeCounter[eMulticast]++;
      }
      else
      {
         zFrameTypeCounter[eUnicast]++;
      }
   }

   if (zFrameTypeCounter[eBroadcast] == LWIP_MAX_IRQ_RX_FRAME )
   {
      EnableBroadcastRx(FALSE);
   }
}

void EnableBroadcastRx(BOOL zEnable)
{
   ETH_Stop();
   /* Reset Rx Descriptors list to avoid problems in case frames arrive while we are disabling broadcast reception */
   ETH_DMARxDescChainInit(DMARxDscrTab, &Rx_Buff[0][0], ETH_RXBUFNB);
   if (zEnable == TRUE)
   {
      ETH->MACFFR &= ~ETH_MACFFR_BFD;
   }
   else
   {
      ETH->MACFFR |= ETH_MACFFR_BFD;
   }
   ETH_Start();
}




--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html

_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users