LWIP on a gigabit LAN

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

LWIP on a gigabit LAN

Jacob Gorm Hansen
hi,

I have been using UIP for a special purpose guest OS inside a Xen
virtual machine, but am upgrading to LWIP to see if I can get petter
performance. Has anyone tried LWIP over a gigabit link before, and how
likely am I to be able to saturate the link?

Thanks in advance,
Jacob



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

RE: LWIP on a gigabit LAN

Roger Cover
Howdy Jacob,

I have been using lwIP on a gigabit LAN for a while now. The highest
sustained throughput I can get is just over 8 gigabytes per second. My
application is sending UDP datagrams. The slow speed is not the fault of
lwIP. My application spends about 27% of its time operating my hardware
system, less than 2% inside functions in lwIP, and the remaining 71% of
its time inside the driver.

The driver I am using is amazingly inefficient. It is based on the
Xilinx (Virtex-2 Pro) driver for lwIP 0.6.3. I have updated it for
better performance and to operate with lwIP 1.2.0. If you have an
efficient driver you should be able to send a pretty high volume of data
through lwIP.

The quality of the driver you are using will be the major issue. You
should also plan on replacing the default checksum routine in lwIP with
an assembly routine (see the thread titled "Gigabit Ethernet and lwIP").

Regards,
Roger
-----Original Message-----
From: lwip-users-bounces+rcover=[hidden email]
[mailto:lwip-users-bounces+rcover=[hidden email]] On Behalf Of
Jacob Gorm Hansen
Sent: Tuesday, April 24, 2007 1:05 PM
To: [hidden email]
Subject: [lwip-users] LWIP on a gigabit LAN

hi,

I have been using UIP for a special purpose guest OS inside a Xen
virtual machine, but am upgrading to LWIP to see if I can get better
performance. Has anyone tried LWIP over a gigabit link before, and how
likely am I to be able to saturate the link?

Thanks in advance,
Jacob



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

RE: LWIP on a gigabit LAN

Jacob Gorm Hansen
On Tue, 2007-04-24 at 13:52 -0700, Roger Cover wrote:
> Howdy Jacob,
>
> I have been using lwIP on a gigabit LAN for a while now. The highest
> sustained throughput I can get is just over 8 gigabytes per second.

Surely you mean megabytes, right?  I am concerned mostly with TCP
bandwidth btw.

Thanks for your answer,
Jacob



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

RE: LWIP on a gigabit LAN

Roger Cover
Sorry, 8 megabytes.

-----Original Message-----
From: lwip-users-bounces+rcover=[hidden email]
[mailto:lwip-users-bounces+rcover=[hidden email]] On Behalf Of
Jacob Gorm Hansen
Sent: Tuesday, April 24, 2007 1:53 PM
To: Mailing list for lwIP users
Subject: RE: [lwip-users] LWIP on a gigabit LAN

On Tue, 2007-04-24 at 13:52 -0700, Roger Cover wrote:
> Howdy Jacob,
>
> I have been using lwIP on a gigabit LAN for a while now. The highest
> sustained throughput I can get is just over 8 gigabytes per second.

Surely you mean megabytes, right?  I am concerned mostly with TCP
bandwidth btw.

Thanks for your answer,
Jacob



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

RE: LWIP on a gigabit LAN

Jacob Gorm Hansen
On Tue, 2007-04-24 at 14:50 -0700, Roger Cover wrote:
> Sorry, 8 megabytes.

In my first tests I seem to be getting around 25-30 megabytes per
second. I suppose that to get better performance I need TCP Window
Scaling and perhaps SACK. I found mention of SACK in LwIP here:
http://edin.no-ip.com/html/?q=node/57 , but am not sure how to find the
patch for it. Has anyone experimented with Window Scaling?

Thanks,
Jacob



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

RE: LWIP on a gigabit LAN

Kieran Mansley
On Tue, 2007-04-24 at 23:57 +0200, Jacob Gorm Hansen wrote:
> On Tue, 2007-04-24 at 14:50 -0700, Roger Cover wrote:
> > Sorry, 8 megabytes.
>
> In my first tests I seem to be getting around 25-30 megabytes per
> second. I suppose that to get better performance I need TCP Window
> Scaling and perhaps SACK. I found mention of SACK in LwIP here:
> http://edin.no-ip.com/html/?q=node/57 , but am not sure how to find the
> patch for it. Has anyone experimented with Window Scaling?

It's been a while since I used lwIP for high performance networking, but
this is essentially how I first got involved in the project, and it
formed the thesis of my PhD.  I was able to saturate a gigabit link, but
this was using a rather specialised piece of hardware.

To get high performance from lwIP you will need to use the raw API.  You
will also need plenty of memory (for buffers to allow streaming) and
plenty of CPU.  Make sure your driver is able to do things like
interrupt moderation and does not induce any extra copies.  You could
modify lwIP to do a copy and checksum in a single operation to improve
efficiency.  In terms of high performance tuning, lwIP is not really
that different to any other stack.  For TCP you need to configure large
windows, large MTUs, and do everything you can to avoid any loss.  SACK
will help a little in the presence of loss, but not a great deal.
Window Scaling is useful to get larger windows.  

Using Xen throws a whole extra dimension of problems at you.  You may
find that the overhead of forwarding interrupts to your domain for
example is reducing your performance.  Xen can saturate a gigabit link,
but as I understand it, it uses more CPU than native linux would do.

Kieran



_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users
Reply | Threaded
Open this post in threaded view
|

Re: LWIP on a gigabit LAN

Jonathan Larmour
In reply to this post by Jacob Gorm Hansen
Jacob Gorm Hansen wrote:
> On Tue, 2007-04-24 at 14:50 -0700, Roger Cover wrote:
>> Sorry, 8 megabytes.
>
> In my first tests I seem to be getting around 25-30 megabytes per
> second. I suppose that to get better performance I need TCP Window
> Scaling and perhaps SACK. I found mention of SACK in LwIP here:
> http://edin.no-ip.com/html/?q=node/57 , but am not sure how to find the
> patch for it. Has anyone experimented with Window Scaling?

The patch is in that link! It appears to implement reception (but not
generation) of SACKs, and window scaling. It would be interesting to
experiment with, although perhaps my experience with window scaling is
limited, but the patch doesn't look entirely right, e.g.:

+        pcb->snd_scale = scale;
+        pcb->rcv_scale = 2;

It sets the receiver's scaling to 2, but that is never advertised in the
SYN (indeed no window scaling is supported for transmission in this patch).

And it all has rather non-deterministic consequences for memory use that
cannot be limited by the user. Still, this patch looks pretty close.

Jifl
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["The best things in life aren't things."]------      Opinions==mine


_______________________________________________
lwip-users mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/lwip-users