Tomato – Adding Custom Packages - NVRAM vs JFFS vs USB Stick

I made the mistake last night of visiting the Tomato forum using my new (but actually second-hand) generation 1 iPad. It was late, and I desperately needed the sleep, but what the hell. Unfortunately I came across a forum thread titled My utilities web site revived.  Some two and a bit hours later (and well into the next day), I put the iPad down having read the thread from start to finish.  I gleaned so much useful and interesting bits of information from that that thread that I felt compelled to write a blogpost. This was mostly for my own future reference, but also to share the knowledge.

All I can say is that ‘rhester72’ is a porting/compilation stud and by the looks of it one generous and smart dude.  Tomato community are very lucky to have him. 

Me personally – well I’m one of the 4 billion java developer drones. I haven’t actively coded in C/C++ for a number of years. I miss these languages; mostly I miss  the forgotten skills to properly harness and control these languages – compiler directives / linking /make files / best practices etc.   My foray into iPad application development will at least see me leveraging C once more.

Anyway, returning to blog topic …

rhester72 has gone to the trouble/effort of compiling a number of very useful packages/applications for the Tomato environment (MIPS processor series / running 2.6 Linux kernel/and to a lesser extent 2.4 kernel). These packages add some nice bells and whistles that may come in very handy. For example,  the torrent client (transmission) means you can potentially turn off your power-sucking gaming box and leave it to your ~10 watt ASUS RT-N16 :)

The binaries come in two styles:

1. static linked
2. dynamic linked

Static linked binaries are generally much bigger in size, as the dependent libraries are linked/packaged directly in the resulting binary. It is not however always possible to produce a static binary for certain types of packages due to their architecture.

Dynamic linked binaries on the other hard should be smaller, as they are linked at runtime to dependent shared libraries.  The issue with these style of binaries is actually finding the the shared libraries themself at runtime:

• the shared library may actually be missing from the machine, or not available in the shared library search path (meaning the binary cannot be run)
• an incompatible version of the shared library may be located in the shared library search path (causing some type of conflict resulting in the binary not behaving correctly)

Dynamic linked binaries can however reduce memory footprint and save valuable space should multiple packages that you plan on running leverage same versions of specific shared libraries.

Space and memory permitting, it is often simpler to take the static binary.

Check out rhester72’s list of packages at the following URL:

http://www.multics.minidns.net/tomato/

Be sure to view the descriptions, notes, and most importantly the readme.

When using dynamically linked binaries, it is not always obvious what the required shared library dependences are. Fortunately, the “ldd” command can be used to help out.

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/atop
root@asus:/tmp# chmod u+x atop
root@asus:/tmp# ldd atop
        libncurses.so.5 => not found
        libm.so.0 => /lib/libm.so.0 (0x2aabf000)
        libz.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.0 => /lib/libc.so.0 (0x2aafc000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)
root@asus:/tmp#

 

Above, you can see that libncurses.so.5 and libz.so.1 are missing.  Lets rectify this …

 

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/lib/libncurses.so.5

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/lib/libz.so.1

root@asus:/tmp# ldd atop
        libncurses.so.5 => not found
        libm.so.0 => /lib/libm.so.0 (0x2aabf000)
        libz.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.0 => /lib/libc.so.0 (0x2aafc000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)

This still did not work; Why?

The answer is the shared library search path.

In Linux, Shared Libraries are searched (in order) from the following locations until a match is located:

1. LD_LIBRARY_PATH environment variable (if set)
2. A specific rpath location encoded in to the dynamically linked ELF binary (or shared library) at compilation time (if set)
3. System default paths defined in /etc/ld.so.conf

On my specific stock tomato instance, LD_LIBRARY_PATH is not set. 

Using the readelf command on the atop binary, I can see the Library rpath is set to /opt/lib:/opt/usr/lib.

The contents of ld.so.conf on my instance are:

root@asus:/tmp# cat /etc/ld.so.conf
/opt/lib
/opt/usr/lib
/lib
/usr/lib

 

Thus, I need to modify the instance so that the  libncurses.so.5 and libz.so.1 libraries can be found.  For the time being, I’m going to manually set LD_LIBRARY_PATH to the /tmp directory which is where I downloaded the files in the first place:

root@asus:/tmp# export LD_LIBRARY_PATH=/tmp

root@asus:/tmp# ldd atop
        libncurses.so.5 => /tmp/libncurses.so.5 (0x2aabf000)
        libm.so.0 => /lib/libm.so.0 (0x2ab12000)
        libz.so.1 => /tmp/libz.so.1 (0x2ab30000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2ab53000)
        libc.so.0 => /lib/libc.so.0 (0x2ab72000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)

As you can see above, the dependent libraries are now found, and we can at least attempt to invoke the atop binary!

Now that you know how to add the packages, the question is where to install them:

• Recall that the /tmp directory on the Tomato router is volatile; It is essentially a ramdisk created in available RAM of the router, and is blown away upon reboot.
• Non-volatile available/free flash memory on the router, if sufficient, can potentially be leveraged with important caveats.
• Another recommended choice (if available with your router) it to leverage an external USB stick/drive
• Possibly you may be able to even utilize network attached storage.

Flash memory is obviously very convenient, but it comes in a variety of sizes. The Linksys WRT54G series came in 2MB, 4MB and 8MB varieties. It is hard enough getting a distribution like Tomato on a router with such little flash memory, let alone using it for custom packages and the like.

The ASUS RT-N16 on the other hand has 32 MB of flash. Even with the full-blown VPN 2.6 Tomato bundle installed, there is still some 25MB available for potential JFFS2 use. There is also a special area in the flash memory known as the NVRAM segment. This is essentially the very last segment in the flash memory and is at minimum one “erase block” in size. Although the erase block size is typically 64KB (or 128KB as is the case with the RT-N16), the actual NVRAM is programmatically restricted to 32KB.

Tomato does support a special “nvram setfile2nvram” command that will allow you to store a very small file in any remaining NVRAM space (such that, within the 32KB not the full erase block segment size i.e. 128KB on the RT-N16).  However, this option should be leveraged as one of the last resorts. If you have free flash memory available, then JFFS2 is a better option.  Even better again is to leverage a USB stick if your router supports it. 

The reasons for this is best explained by OpenWRT developer/CoFounder MBM’s post at the following location: https://forum.openwrt.org/viewtopic.php?id=10083

To quote him directly:

“The flash chip is broken up into sections called erase blocks. On a 4M chip it's usually 64k and on an 8M chip it's 128k. Each erase block is rated at about 100,000-1,000,000 erase/write cycles depending on vendor.

This just means that on a 4M chip you have to have to erase 64k and rewrite it even if you only want to change one byte of the 64k. After that 64k block has been erased 100,000 times you risk failure where it won't store the data properly.

The problem with the NVRAM implementation is that it's exactly one erase block at the very end of the flash. When you boot, the NVRAM data is copied to a buffer in ram; with the exception of "nvram commit", all the nvram commands are using the copy in ram. When you do an "nvram commit" it writes the contents of ram to the flash. So, when you have a chip rated for 100,000 cycles, you'll probably have a failure around the 100,000th "nvram commit".
”

Although leveraging JFFS2 will still result in wear due to erase cycles, this file system is specifically designed with flash devices in mind.  It is engineered is such a way to make “wear-levelling more even and prevent erasures from being too concentrated”.

Once again though MBM  makes an important point:

“ Suppose we have a jffs2 filesystem with two types of files, files that never change and files that change frequently. Common sense says that the erase blocks containing the files that never change or contain free space will only be written once and will remain untouched while the blocks containing the other files will change frequently; wear leveling means that all of the blocks within the jffs2 filesystem will be used equally, so all of the blocks in the above example would be written to equally even if it means moving data that hasn't changed“

Thus, if you anticipate files being updated /writes occurring on the JFFS2 partition at a considerable rate, you are eventually going to ruin your flash chip.  In such scenarios you must absolutely use something like USB.  You must also be very careful that any custom packages you install are not repeatedly writing (e.g. log messages) to a location that is stored in the JFFS2 partition.

Here is an ugly script I wrote designed for Tomato 1.28 to get details of flash memory allocation/distribution; Note that I have enabled JFFS2 support on my router from the Tomato UI (Administration > JFFS page):

cat > /tmp/flashinfo.sh <<EOF
#!/bin/sh

# Tomato 1.28 Flash Info Script by Matt Shannon

echo "Router model: " \`nvram get t_model_name\`
uname -a
echo ""

dmesg | grep nvram
dmesg | grep jffs2
echo ""

cat /proc/mtd
echo ""

ERASEBLOCKSIZE=0x\`cat /proc/mtd | grep nvram | cut -f3 -d " "\`
echo "Erase Block size is" \`awk 'BEGIN{printf("%d", '\$ERASEBLOCKSIZE' / 1024)}\` kilobytes
echo ""

NVRAMSEGMENT=0x\`cat /proc/mtd | grep nvram | cut -f2 -d " "\`
echo "NVRAM Full Segment size is" \`awk 'BEGIN{printf("%d", '\$NVRAMSEGMENT' / 1024)}\` kilobytes

NVRAMSUMMARY=\`nvram show | tail -1\`
NVRAMUSED=\`echo \$NVRAMSUMMARY | cut -d "," -f2 | cut -d " " -f2\`
NVRAMFREE=\`echo \$NVRAMSUMMARY | cut -d "," -f3 | cut -d " " -f2\`
echo "NVRAM Actual Size Available to firmware is" \`awk 'BEGIN{printf("%d", ('\$NVRAMUSED' + '\$NVRAMFREE') / 1024)}\` kilobytes
echo "NVRAM Summary: \$NVRAMSUMMARY"
echo ""

JFFSSIZE=0x\`cat /proc/mtd | grep jffs2 | cut -f2 -d " "\`
echo "JFFS2 size is" \`awk 'BEGIN{printf("%d", '\$JFFSSIZE' / 1048576)}\` megabytes
echo ""
EOF

chmod 744 /tmp/flashinfo.sh

/tmp/flashinfo.sh

sample output:

Router model:  Asus RT-N16
Linux asus 2.6.22.19 #8 Tue Nov 30 14:58:27 EST 2010 mips GNU/Linux

0x01fe0000-0x02000000 : "nvram"
0x006e0000-0x01fe0000 : "jffs2"

dev:    size   erasesize  name
mtd0: 00040000 00020000 "pmon"
mtd1: 01fa0000 00020000 "linux"
mtd2: 005aec00 00020000 "rootfs"
mtd3: 01900000 00020000 "jffs2"
mtd4: 00020000 00020000 "nvram"

Erase Block size is 128 kilobytes

NVRAM Full Segment size is 128 kilobytes
NVRAM Actual Size Available to firmware is 32 kilobytes
NVRAM Summary: 852 entries, 21155 bytes used, 11613 bytes free.

JFFS2 size is 25 megabytes

 

Recall from above that rhester72’s dynamically linked binary atop hardcodes an rpath to /opt/lib:/opt/usr/lib. If you were to SSH in to your router however, you will likely find that the /opt directory is empty.  Rhester72 leverages an init script (Tomato UI > Administration > Scripts > Init)  to attempt to automatically bind the /opt location to a directory “opt” found in the /jffs mount location. (Note, when enabling JFFS2 in Tomato, the JFFS partition is automatically mounted at startup under the /jffs directory). Here is his script which will try for 30 seconds to bind the opt directory:

t=0
while [[ ! -d /jffs/opt && $t -lt 30 ]];do
sleep 1
t=$(($t+1))
done

if [ -d /jffs/opt ];then
mount -o bind /jffs/opt /opt
else
logger -t /jffs -p err did not mount within $t seconds
fi

If the bind fails, a message will be written to /var/log/messages.  To utilize such an approach you will need to do the following:

1. enable JFFS2
2. once the jffs partition is mounted, mkdir /jffs/opt
3. make appropriate subdirectories mkdir -p /jffs/opt/bin /jffs/opt/lib /jffs/opt/sbin /jffs/opt/usr/bin /jffs/opt/usr/lib /jffs/opt/usr/sbin /jffs/opt/usr/share

Refer to his readme.

Leveraging the above directory structure to store binaries and shared libraries should give you good results (remember /opt will be bound to /jffs/opt):

root@asus:/tmp/home/root# echo $PATH | tr ":" "\n"
/bin
/usr/bin
/sbin
/usr/sbin
/home/root
/mmc/sbin
/mmc/bin
/mmc/usr/sbin
/mmc/usr/bin
/opt/sbin
/opt/bin
/opt/usr/sbin
/opt/usr/bin

root@asus:/tmp/home/root# cat /etc/ld.so.conf | grep /opt
/opt/lib
/opt/usr/lib

As a final note, be careful with setting LD_LIBRARY_PATH in any profile.  Also it is worth inspecting the /etc/profile script, you will see that it attempts to source both /jffs/etc/profile and /opt/etc/profile (if they exist).

Thanks again to rhester72;