ixp425sar: An ATM device driver for the IXP425

(a brain dump)

Author:

Alex Zeffertt: ajz at cambridgebroadband.com

Licence:

GPL.

UPDATE:: Patches for CSR 1.3 Click here

UPDATE:: How to build big endian xscale toolchain Click here

Intro:

The ixp425sar is a linux ATM device driver for the IXP425 processor. The IXP425 is a network processor from Intel which contains an XScale core and 3 Network Processor Engines (NPEs), namely NPE-A, NPE-B, and NPE-C. These NPEs are general purpose processors - which run Intel proprietry microcode, downloaded by the core. Intel provide microcode for NPE-A which performs an ATM SAR function. They also provide code that runs on the XScale core which creates an API for ATM applications. This API is called CSR.

The CSR is not a userland API, it is a kernel mode API. Furthermore it does not marry with the ATM DD specification. Therefore, what is needed in order to leverage the linux-atm networking stack, is a thin layer which translates between Intel's CSR API and the linux-atm device driver API. The aim of this project is to create such a layer.

Project Status:

The driver is currently working for us. If you wish to participate in this project, please contact me.

Tested platforms:

We have successfully used the driver on the Richfield (IXDP425) platform using RedBoot 2.0, and linux-2.4.19-rmk7-ds2. We threw away the ADSL expansion card that came with this platform and replaced it with an OC3 expansion card (our own design) which uses the S/UNI Ultra phy. You will need to read the Configuration section (below) if you wish to use a different platform.

Build instructions:

Buy an IXDP425 platform from Intel

Download the driver:

                    [you@yours]$ export CVSROOT=yourlogin@cvs.ixp425.sourceforge.net:/cvsroot/ixp425
                    [you@yours]$ export CVS_RSH=ssh
                    [you@yours]$ cvs co ixp425sar

Get the following source code:

linux-2.4.19.tar.bz2
ftp://ftp.arm.linux.org.uk/pub/armlinux/kernel/v2.4/patch-2.4.19-rmk7.gz
ftp://source.mvista.com/pub/ds-patches/2.4/patch-2.4.19-rmk7-ds2.gz
CSR 1-2-1 code from http://www.intel.com/design/network/products/npfamily/ixp425swr1.htm

ixp425AccessLibrary-1_2_1.zip
ixp425LinuxDriversPatch-MVL2_1-1_2_1.zip

CSR patch required to work with linux-2.4.19 and ixp425sar patch-ixp425_xscale_sw
Kernel patch required for ixp425sar driver patch-linux

Create source tree:

Create a directory called build.

In build/linux unpack linux-2.4.19.tar.bz2

Then apply the ARM patch patch-2.4.19-rmk7

Then apply the IXP425 patch patch-2.4.19-rmk7-ds2

Next apply ixp425LinuxDrivers.patch from ixp425LinuxDriversPatch-MVL2_1-1_2_1.zip (don't worry about the patch failures - these will be cleared up in the next step)

Finally apply the patch "patch-linux" from this project.

In build/ixp425_xscale_sw unpack ixp425AccessLibrary-1_2_1.zip

Apply the patch "patch-ixp425_xscale_sw" from this project.

Finally check out the driver to build/ixp425_xscale_sw/src/codelets/ixp425sar/.

Customize driver (do not skip this step):

Microcode image fixup:

There is a bug in the NPE-A microcode in CSR 1.2.1. The problem is that during reception, the microcode cannot offload the ATM PHY at above 50 Mbit/s. The result is that if you hit the PHY hard with rx traffic, you get rx overflows in the PHY, which will start dropping cells. This leads to CRC errors on received AAL5 frames.

If the above is not a problem, undefine OVERRIDE_UCODE_IMAGE in ixp425sar.c. Otherwise read on.

This bug does not occur with the microcode images from CSR 1.01. The work around, therefore, is to replace the NPE-A microcode images from CSR 1.2.1 with those from CSR 1.0.1. I would like to just give you these images, but Intel wrote them so I cannot release them in a GPL'd package. Instead I shall explain how to create the images for yourselves.

Copy build/ixp425_xscale_sw/src/include/IxNpeMicrocode.h to build/ixp425_xscale_sw/src/codelets/ixp425sar_ucode.

Remove the #ifdef IX_NPE_MICROCODE_FIRMWARE_INCLUDED and its corresponding #endif

Replace the NPE-A DOWNLOAD MAPs with those from the version of IxNpeMicrocode.h that came with CSR 1.01. (I.e. replace the download maps for Version IDs 00010000, 00020000, 00030000 and 00040000 - the other download maps are for NPE-B and NPE-C and should be left alone)

Update the fields marked /* size of version */ in the MICROCODE IMAGE HEADER

Use the code in build/ixp425_xscale_sw/src/codelets/auto-offset.c to recalculate the MICROCODE IMAGE HEADER.

Note: this fix means that you cannot use AAL0. This is because the CSR 1.0.1 microcode does not support AAL0.

PHY selection:

My board uses a S/UNI ATM phy. Yours almost certainly doesn't. Even if it did, it would probably be memory mapped to a different address than ours. So comment out the line

#define USE_SUNI_PHY

This removes the dependency on build/linux/drivers/atm/suni.o

You may wish to implement your own phy driver - although this is not neccessary. But you should certainly change TX_PORT_RATE_BPS and RX_PORT_RATE_BPS to reflect your phy's line rate.

Fast Path:
You can enable fast path by defining IX_ETH_ACC_FPATH_AWARE in build/ixp425_xscale_sw/src/include/IxEthAccFpathDep.h. You probably want to disable it for the time being because it doesn't work.

The problem is that the Modifier in NPE-B is failing to strip the first 10 bytes which contain the LLC/SNAP header.

For the record - and in case anybody wants to try to get this to work - here's how to use fast path if enabled in the driver:

Open an AAL5 socket for rx only, but before you call bind set USE_FPATH_FLAG in qos.rxtp.spare. After the socket is opened RFC1483 frames received on that VC will be forwarded to NPE-B for transmission.

Like I said, at the moment this doesn't work. The Classifier in NPE-A forwards frames to NPE-B alright but the Modifier in NPE-B doesn't strip the bumf from the front of the frames before sending them out.

UTOPIA 1/2:
If your expansion board uses UTOPIA 1 then undefine IX_NPE_MPHY in build/ixp425_xscale_sw/src/include/IxNpeA.h. If it uses UTOPIA 2 then leave this defined. This will affect which NPE-A image gets loaded by the ATM DD. Note, however, that the ATM DD does not currently support multiple phy's even though CSR does.

Build kernel and driver:

(Note you may need to edit build/linux/Makefile and build/ixp425_xscale_sw/Makefile to point it at a different cross-compiler)

[you@yours]$ cd linux
[you@yours]$ make ixdp425_config
[you@yours]$ make oldconfig
[you@yours]$
[you@yours]$ ### These variables are needed by the CSR Makefiles
[you@yours]$ TOP=`pwd`
[you@yours]$ LINUX_SRC=${TOP}/linux
[you@yours]$ HARDHAT_BASE=/usr/local
[you@yours]$ PATH=${HARDHAT_BASE}/bin:${PATH}
[you@yours]$ IX_XSCALE_SW=${TOP}/ixp425_xscale_sw
[you@yours]$ TOOLENV=linux
[you@yours]$ export PATH LINUX_SRC HARDHAT_BASE IX_XSCALE_SW TOOLENV
[you@yours]$
[you@yours]$ make dep zImage modules

The ATM driver will be created in build/ixp425_xscale_sw/lib/linuxobjs/csr_codelets_ixp425sar.o. This module depends on build/ixp425_xscale_sw/lib/linuxobjs/csr.o. These modules should get copied to build/linux/drivers/csr/ - so if you then want to do a "make modules_install" the install tree should include these files and the modules.dep file will include dependencies for them.

Usage:

    insmod csr.o
    insmod csr_codelets_ixp425sar.o
    insmod ixp425_eth.o no_csr_init=1 # csr is already initialised by ATM driver!

Performance:

On my IXDP425 platform I have acheived 100 mbit/s bridging 512 byte frames from an RFC2684 encapsulated PVC on NPE-A to Ethernet on NPE-B. I have also acheived 100 mbit/s bridging in the opposite direction. I have not tested bidirectional throughput.

Undirectional throughput at 64 byte frames is around 20 mbit/s.

Todo/Bugs:

Fix FastPath - needs Intel support
Implement cleanup_module()
Enable more than 1 CBR VC at a time.

Thanks to:

    Russell king - Maintainer of ARM support for linux
    Deepak Saxena - Maintainer of IXP425 support for linux.
    The developers at http://gatewaymaker.sourceforge.net for providing a starting point.