Today, guess what ...

dircolors – color ls unix directory listings – danger of overriding shell built-ins

2011-04-19T10:45:00.001+10:00

I ran in to a crazy issue today that stemmed from a built-in function override configuration setting that I have leveraged for the best part of a decade. I’m amazed that this is the first time I actually noticed it!

I have a environment setup script for one of my installations like the following ...

export MW_HOME=/u01/app/oracle/product/Middleware
export JAVA_HOME="`ls -d ${MW_HOME}/jdk16*`"
export PATH=$JAVA_HOME/bin:$PATH

The JDK path is not hardcoded as this is a pre-release environment, and the JDK is often patched and the directory name changes as a result.

A script I was invoking had a simple test like the following:

#!/bin/sh
if [ x${PATH} != x ]; then
PATH=$ORACLE_HOME/bin:$PATH
fi

When I ran the shell script from plain vt100 terminal, everything worked perfectly. When I fired the script up from an xterm running inside VNC, I hit an error like the following:

./test.sh: line 2: [: too many arguments

The reason for the error, the string JAVA_HOME ended up containing ANSI characters when run from xterm.

For example, from an XTERM

echo `ls -d /tmp` > file.txt

file.txt contained ... ^[[00m^[[00;34m/tmp^[[00m ^[[m

, whereas from the VT100 terminal:

echo `ls -d /tmp` > file.txt

file.txt contained ... /tmp

It turns out, I had a zsh function setup-for/overriding the ls command so as to use color listings when invoked from an xterm by passing the --color option. The function was as follows:

ls ()
{
[[ $TERM = (*xterm*|ansi|linux|dtterm) ]] && set -- --color "$@"
command ls -v "$@"
}

The fix was to change the --color option to be --color=auto so that the ANSI escape sequences were only output if the standard output is a terminal (but not for things like scripts etc).

ls ()
{
[[ $TERM = (*xterm*|ansi|linux|dtterm) ]] && set -- --color=auto "$@"
command ls -v "$@"
}

Moral of the story, be extremely careful using shell functions to override standard built-in commands, and make sure you understand the impact of switches!

Update

So my good mate and colleague Dragos states:

You should not rely on 'ls' (or any other command), as it can be an alias or function to anything else.

The fix is in your environment script to use /bin/ls or =ls, yes "equal sign ls"

There's a way to force filename expansion too:

JAVA_HOME=(${MW_HOME}/jdk16*)

Online Open Port Check Tool

2011-04-07T13:27:00.001+10:00

The RCCB / Residual Current Device in my place has a nasty habit of tripping whenever the house is vacant for more than a couple of days. For the remaining 99% of the time that I’m home, it never trips. I haven’t been able to isolate exactly what triggers it, but upon my return I’m faced with a fridge/freezer full of ruined food.

Anyway, given the electricity is completely lost when this RCCB triggers, I really need a simple tool that polls my router (like a ping) to see if it responds, and if not, gives me some type of warning.

I run OpenVPN on the router over TCP port 443, and it appears the various online port scanning tools are able to detect the presence of the port being open.

The following is a list of sites that I’ve found that allow one to scan a specific remote host and port (that differs to the host/proxy of the incoming client):

http://ping.eu/port-chk/
http://www.subnetonline.com/pages/network-tools/online-port-scanner.php
http://www.digitalcoding.com/tools/open-port-check.html
http://www.yougetsignal.com/tools/open-ports/

My favourite is the latter, as it allows you to potentially bookmark the location; for example, to check if google port 80 open:

http://www.yougetsignal.com/tools/open-ports/php/check-port.php?portNumber=80&remoteAddress=www.google.com

If anyone knows a site out there that offers remote port polling with SMS/email notification in the event the service is down, let me know!

I may have to build something otherwise on Google App Engine using the URL fetch capabilities

Here is the API http://code.google.com/appengine/docs/java/urlfetch/overview.html

Tools to locate class file in JAR / CLASSPATH

2011-03-29T18:24:00.001+10:00

There are a number of different approaches for obtaining the location (jar/directory) of a class at runtime. The following approach work pretty well…

Class c = ...
System.out.println(c.getProtectionDomain().getCodeSource().getLocation().toString());

You could also try VM debug options at startup such as:

-XX:-TraceClassLoading
Trace loading of classes.

-XX:-TraceClassLoadingPreorder
Trace all classes loaded in order referenced (not loaded).

In terms of static/offline location of classes, most people will suggest jarbrowser, jarminator, or brute force by leveraging a combination of the unix find command, unzip –l, and grep.

I’ve created a derivative of the offline approach. Essentially I have three scripts:

jarclasspath is a shell script/java class combo used to formulate a valid and complete list of libraries and directories that are referenced from the specified source JAR by way of META-INF/MANIFEST.MF Class-Path directives. A classpath containing a single JAR file could ultimately expand to hundreds of libraries, should that top-level archive specify Class-Path directives, and in-turn the dependent libraries provide Class-Path directives and so-on. See this post for more details regarding manifest classpath attributes.
jarcheck is a simple perl script that checks for presence of the specified class file in the specified JAR file/or directory. Note that Class-Path manifest directives can reference JAR files and/or directories. This tool supports searching both.
jarwhich is a demonstration shell script that shows how to leverage the jarclasspath and jarcheck scripts above. This particular script will search the CLASSPATH (and dependent libraries/directories based on Class-Path directives) to locate the requested class file.

Testing…

Using my UCM flavoured WebLogic Server as the basis for the test, I sourced $DOMAIN_HOME/bin/setDomainEnv.sh to set CLASSPATH.

This resulted in a CLASSPATH containing just 16 entries, however, all is not as it seems! :

% echo $CLASSPATH
/u01/app/oracle/product/Middleware/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar: ...
/u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/weblogic.jar: ...
/u01/app/oracle/product/Middleware/modules/features/weblogic.server.modules_10.3.4.0.jar:
...

% echo $CLASSPATH | tr ":" "\n" | wc -l
16

The weblogic.server.modules_10.3.4.0.jar referenced above in the CLASSPATH however has a Class-Path manifest directive referencing some 180 additional dependent JAR files:

% echo $CLASSPATH | tr ":" "\n" | grep weblogic.server.modules
/u01/app/oracle/product/Middleware/modules/features/weblogic.server.modules_10.3.4.0.jar

% unzip -l /u01/app/oracle/product/Middleware/modules/features/weblogic.server.modules_10.3.4.0.jar
Archive: weblogic.server.modules_10.3.4.0.jar
Length     Date   Time    Name
--------    ----   ----    ----
     8880 03-23-11 05:00   META-INF/MANIFEST.MF
--------                   -------
     8880                   1 file

% unzip -d /tmp /u01/app/oracle/product/Middleware/modules/features/weblogic.server.modules_10.3.4.0.jar META-INF/MANIFEST.MF
Archive: weblogic.server.modules_10.3.4.0.jar
inflating: /tmp/META-INF/MANIFEST.MF

% cat /tmp/META-INF/MANIFEST.MF|more
Manifest-Version: 1.0
Implementation-Vendor: BEA Systems
Implementation-Title: Oracle WebLogic Server Module Dependencies 10.3
Thu Oct 28 06:03:12 PDT 2010
Implementation-Version: 10.3.4.0
Feature-Name: weblogic.server.modules
Source-Repository-Change-Id: 1374366
Class-Path: weblogic.server.modules.wlsve_10.3.4.0.jar ../com.bea.core
.antlr.runtime_2.7.7.jar ../com.bea.core.descriptor.j2ee_1.5.0.0.jar
../com.bea.core.descriptor.j2ee.binding_1.5.0.0.jar ../com.bea.core.d
escriptor.wl_1.3.3.0.jar ../com.bea.core.descriptor.wl.binding_1.3.3.
0.jar ../com.bea.core.datasource6_1.9.0.0.jar ../com.bea.core.datasou
rce6.binding_1.9.0.0.jar ../com.bea.core.beangen_1.7.0.0.jar ../com.b
ea.core.descriptor.settable.binding_1.7.0.0.jar ../com.bea.core.diagn
ostics.accessor_1.5.0.0.jar ../com.bea.core.diagnostics.accessor.bind
ing_1.5.0.0.jar ../com.bea.core.management.core_2.8.0.0.jar ../com.be
a.core.management.core.binding_2.8.0.0.jar ../com.bea.core.ejbgen_1.1
.0.2.jar ../org.apache.ant_1.7.1/lib/ant-all.jar ../com.bea.core.repa
ckaged.apache.bcel_5.2.1.0.jar ../com.bea.core.repackaged.jdt_3.5.2.0
.jar ../com.bea.core.apache.commons.collections_3.2.0.jar ../com.bea.
core.apache.commons.lang_2.1.0.jar ../com.bea.core.apache.commons.poo
l_1.3.0.jar ../com.bea.core.apache.commons.io_1.0.0.0_1-4.jar ../com.
bea.core.apache.commons.fileupload_1.0.0.0_1-2-1.jar ../com.bea.core.
apache.dom_1.0.0.0.jar ../com.bea.core.apache.logging_1.0.0.0.jar ../
org.apache.openjpa_1.2.0.0_1-1-1-SNAPSHOT.jar ../com.bea.core.xml.xml
beans_2.1.0.0_2-5-1.jar ../com.bea.core.logging_1.8.0.0.jar ../com.be
a.core.bea.opensaml_1.0.0.0_6-1-0-0.jar ../com.bea.core.bea.opensaml2
_1.0.0.0_6-1-0-0.jar ../com.bea.core.monitoring.harvester.api_2.3.0.0
.jar ../com.bea.core.monitoring.harvester.jmx_2.3.0.0.jar ../com.bea.
.....

As you can see above, this short 16 entry CLASSPATH soon expands to something massive. In fact, it expanded to 412 unique entries (primarily JAR files, but also directories)!

Let us run some tests:

% ~/bin/jarwhich oracle.jdbc.driver.OracleDriver
/u01/app/oracle/product/Middleware/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar: oracle/jdbc/driver/OracleDriver.class
##Match Found.
/u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/ojdbc6.jar: oracle/jdbc/driver/OracleDriver.class
##Match Found.

% ~/bin/jarwhich javax.mail.internet.MimeUtility
/u01/app/oracle/product/Middleware/modules/javax.mail_1.1.0.0_1-4-1.jar: javax/mail/internet/MimeUtility.class
##Match Found.

And finally, here are the scripts:

jarclasspath

#!/bin/sh
#
# Copyright (c) 2011, Matt Shannon.
#
# NAME jarclasspath
#

if [ -z "${JAVA_HOME}" ]; then
printf "\n\nError: Ensure JAVA_HOME environment variable is set.\n"
exit 1
fi

# Checks error
checkerror()
{
RESULTCODE=$?
if [ ${RESULTCODE} -ne 0 ];then
exit 1
fi
}

JAVARUNCMD="${JAVA_HOME}/bin/java"
JAVACMPCMD="${JAVA_HOME}/bin/javac"

## note - need to be careful with dollar signs and backslashes!

cat > /tmp/JarClasspath.java <<EOF
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.io.IOException;

import java.util.List;
import java.util.ArrayList;

import java.util.jar.Attributes;
import java.util.jar.JarFile;
import java.util.jar.Manifest;

public class JarClasspath
{
public static void main(String[] args)
{
    if (args.length < 0)
    {
      System.err.println("Usage: arg0 = /path/to/jar");
      System.exit(-1);
    }

String sourceJar = args[0];

List<String> jars = new ArrayList<String>();

    try
    {
      File entry = new File(sourceJar);
      String entryPath = entry.getCanonicalPath();

      if (entry.isDirectory())
      {
        System.out.println(entryPath);
      }
      else if (entry.isFile())
      {
        if ((entryPath.toLowerCase().endsWith(".jar") ||
          entryPath.toLowerCase().endsWith(".zip")))
        {
          System.out.println(entryPath);
          jars.add(entryPath);
          printManifestClasspathEntries(jars, entry);
        }
      }
    }
    catch (Exception e)
    {
      e.printStackTrace();
      System.exit(-1);
    }
}

public static void printManifestClasspathEntries(
    List<String> jars,
    File sourceJar
) throws Exception
{
    File[] entries = getManifestClasspathEntries(sourceJar);
    int length = (entries == null) ? 0 : entries.length;
    for (int i = 0 ; i < length; i++)
    {
      File entry = entries[i];
      if (entry == null)
      {
        continue;
      }

String entryPath = entry.getCanonicalPath();

      if (entry.isDirectory())
      {
        System.out.println(entryPath);
      }
      else if (entry.isFile())
      {
        if ((entryPath.toLowerCase().endsWith(".jar") ||
          entryPath.toLowerCase().endsWith(".zip")))
        {
          if (! jars.contains(entryPath))
          {
            jars.add(entryPath);
            System.out.println(entryPath);
            printManifestClasspathEntries(jars, entry);
          }
        }
      }
    }
}

public static File[] getManifestClasspathEntries(File file)
throws IOException, FileNotFoundException
{
File[] results = null;

    InputStream is = null;
    try
    {
      is = new FileInputStream(file);

JarFile jarfile = new JarFile(file);

      Manifest manifest = jarfile.getManifest();
      if (manifest != null)
      {
        Attributes attributes = manifest.getMainAttributes();
        if (attributes != null)
        {
          String cp = attributes.getValue(Attributes.Name.CLASS_PATH);
          if (cp != null)
          {
            String[] entries = cp.split("\\\\s");
            int length = (entries == null) ? 0 : entries.length;
            if (length > 0)
            {
              results = new File[length];
              for (int i = 0 ; i < length; i++)
              {
                if ((entries[i] == null) || (entries[i].trim().length() == 0))
                {
                  continue;
                }
                File f = new File(file.getParentFile(), entries[i]);
                results[i] = f;
              }
            }
          }
        }
      }
    }
    finally
    {
      if (is != null)
      {
        try
        {
          is.close();
        }
        catch (Exception ignore)
        {
          ;
        }
        is = null;
      }
    }
    return results;
}
}
EOF

CP=/tmp

# Compile tool
${JAVACMPCMD} -classpath ${CP} -d "/tmp" /tmp/JarClasspath.java
checkerror

# Process arguments
for jar in "$@"; do
${JAVARUNCMD} -classpath ${CP} JarClasspath "${jar}"
checkerror
done

jarcheck

#!/usr/bin/perl -w
#
# Checks if specified class is found in provided jar/directory
#
use Getopt::Std;

$0 =~ /([^\/]+)$/ ; # Pattern match; 1 or more chars at end of perl script file name (after the last / if present)
$SCRIPT = $1 ; # Contains the subpattern from the first set of parentheses in the last pattern matched

# Get the class argument.
getopts('s');

unless (((scalar @ARGV) == 2) &&
(defined ($class = $ARGV[0])) && ($class =~ /^[\w\.\/]+$/) &&
(defined ($jar = $ARGV[1])) && ($jar =~ /^[\w\.\/\-]+$/))
{
die("Usage: $SCRIPT [-s] <class> <jar>\nwhere:",
      "\t<class> is the fully-qualified, dot or slash delimited name of a Java class.\n",
      "\t<jar> is the fully-qualified jar file or directory to search.\n",
      "\t-s tells jarcheck that the full class name is not provided\n");
}

# Get the partial name of the .class file for this class.
$classFile = $class ;
$classFile =~ s/\./\//g ;
$classFile .= ".class" ;

# If the jar is a file, Search it for the class file name.
if ((-f $jar) && (open(ARCHIVE, "unzip -l $jar|")))
{
while(defined ($line = <ARCHIVE>))
{
# escape any $ in the class file name
$escaped = $classFile;
$escaped =~ s/\$/\\\$/;

    # \b allows you to perform a "whole words only" search using a regular expression
if($line =~ /\b$escaped\b/)
    {
   print("$jar: ");
   if (defined($opt_s))
   {
     $line =~ /[_\w\/\$]+$classFile/;
     print($&);
   }
   else
   {
     print($classFile);
   }
   print("\n");

   exit(0);
}
}
}
# If the jar is infact a directory, see if the .class file is under it.
elsif(-d $jar)
{
if (-f "$jar/$classFile")
{
print("$jar/$classFile\n");
exit(0);
}
elsif (defined($opt_s))
{
$classFile =~ /[_\w]+\.class/;
$maxdepth = ($jar eq ".")? "-maxdepth 1":"";
$results = `find $jar $maxdepth | grep -w $&`;
if (!$?)
{
   print $results;
   exit(0);
}
}
}

exit(1);

jarwhich

#!/bin/sh
#
# Copyright (c) 2011, Matt Shannon.
#
# NAME jarwhich
#

if [[ $# -ne 1 ]]; then
printf "\nUsage: $0 <class>\n"
printf " where <class> is the fully-qualified, dot or slash delimited name of\n"
printf " the Java class to locate. e.g. oracle.jdbc.driver.OracleDriver\n"
exit
fi

if [ -z "${CLASSPATH}" ]; then
printf "\nError: Ensure CLASSPATH environment variable is set.\n"
exit 1
fi

DIRNAMECMD=`which dirname`
SCRIPT_DIR="`${DIRNAMECMD} $0`/"
SCRIPT_DIR="`cd \"${SCRIPT_DIR}\" && pwd`"

jarCheck()
{
"${SCRIPT_DIR}"/jarcheck "$1" "$2"
return
}

file=/tmp/out$$
echo $CLASSPATH | tr ":" "\n" | sort -u | xargs "${SCRIPT_DIR}"/jarclasspath | sort -u > $file

for line in `cat $file`; do
jarCheck $1 "${line}"
if [[ $? -eq 0 ]]; then
printf "##Match Found.\n"
# exit 0
fi
done

JAR : MANIFEST.MF Class-Path referencing a directory

2011-03-29T11:24:00.001+10:00

Leveraging Java "1.6.0_24" on Windows XP, I performed some quick tests to determine if a JAR's manifest (META-INF/MANIFEST.MF) Class-Path attribute could reference a directory, thereby automatically picking up any contained classes/jars within that directory.

The result... INTERESTING..

My directory tree contents were as follows:

/test
/test/test.jar
/test/lib
/test/lib/abc.jar

the "test.jar" found in the top level "test" directory contained a single file entry:
META-INF/MANIFEST.MF

the "abc.jar" found in the "lib" directory contained a single file entry; a class named "Testing" :-

public class Testing
{
public static void main(String args[])
{
System.out.println("found me");
}
}

To prove our Testing class can be located, we set test.jar's MANIFEST.MF contents initially to:

Manifest-Version: 1.0
Class-Path: lib/abc.jar
Created-By: 1.6.0_24 (Sun Microsystems Inc.)

(Note, following the Created-By: ... line, there are two newlines.)

Invoking the following java command line, we see the Testing class was successfully triggered:

C:\test>java -cp test.jar Testing
found me

I did some additional testing with relative and absolute paths in the MANIFEST.MF, the results of which were:

works: Class-Path: ./lib/abc.jar
works: Class-Path: /C:/test/lib/abc.jar
works: Class-Path: \C:\test\lib\abc.jar
fails: Class-Path: C:\test\lib\abc.jar

Next, I altered the MANIFEST.MF contents to:

Manifest-Version: 1.0
Class-Path: lib/
Created-By: 1.6.0_24 (Sun Microsystems Inc.)

and re-issued the java command:

C:\test>java -cp test.jar Testing
Exception in thread "main" java.lang.NoClassDefFoundError: Testing
Caused by: java.lang.ClassNotFoundException: Testing
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: Testing. Program will exit.

Thus, it appeared jar files in the lib directory would not be automatically included in the classpath.
To be overly thorough, I decided to run some additional tests, the results of which were:

fails: Class-Path: lib
fails: Class-Path: lib/
fails: Class-Path: ./lib
fails: Class-Path: ./lib/
fails: Class-Path: \C:\test\lib\
fails: Class-Path: \C:\test\lib
fails: Class-Path: /C:/test/lib/
fails: Class-Path: /C:/test/lib

At this point, I was of the opinion that a directory specified as part of a manifest Class-Path attribute/directive would simply be ignored.

WRONG!

I decided to extract (and subsequently delete) abc.jar. The contents of my directory tree were thus:

/test
/test/test.jar
/test/lib
/test/lib/Testing.class

I set the MANIFEST.MF contents to:

Manifest-Version: 1.0
Class-Path: lib/
Created-By: 1.6.0_24 (Sun Microsystems Inc.)

and re-issued the java command:

C:\test>java -cp test.jar Testing
found me

SUCCESS. It had located the class. What was even more interesting came out in the subsequent tests I performed:

fails: Class-Path: lib
fails: Class-Path: ./lib
fails: Class-Path: \C:\test\lib
fails: Class-Path: /C:/test/lib
works: Class-Path: lib/
works: Class-Path: ./lib/
works: Class-Path: \C:\test\lib\
works: Class-Path: /C:/test/lib/

Hence, if an explicit directory name is provided as part of the Class-Path attribute, it must have a trailing slash in order to be recognized!

If a directory entry ends with "." or ".." no trailing slash is required. For example, if test.jar was moved to /test/lib, and had its MANIFEST.MF Class-Path set to ../.. , then Testing.class would be found if it resided in "/".

As a final test, and just to be doubly certain that relative paths specified in a MANIFEST.MF are in no way influenced by the java invoking end-user's working directory, I ran the following:

(with MANIFEST.MF Class-Path set to lib/ and lib containing Testing.class)

C:\>java -cp test\test.jar Testing
found me

Malware exploiting Java Plug-In 1.6.0_22

2011-03-21T16:15:00.001+10:00

I learned a nasty lesson yesterday about the dangers of not having the very latest versions of browser plug-ins installed. The irony is that I work for the company that developed the Plug-in. I had installed the Sun (Oracle) JDK / JRE Standard Edition 6 Update 22 which was released only back in October 2010. This includes a Java Plug-In that will allow applets to be invoked from the browser.

Running the current latest version of Firefox (3.6.15) on Windows XP, I opened the top few sites returned from a Google search for UFC 128 title fight video in separate tabs. I was immediately called away to tend to my son, and returned to find my wife in front of the laptop with a multitude of Internet Explorer windows popped-up. I proceeded to chastise her and give her the ‘WTF are you doing’ / ‘What have you done’ spiel – only to be told she hasn’t pressed a key/or button.

Bugger!

Malware.

I immediately ripped the network cable from the back of the computer, and proceeded to start Task Manager along with a program called TaskInfo (by Iarsn) – and kill any and all processes that didn’t look right. Unfortunately I had recently got a bit lazy/blaze about what runs on my laptop and hadn’t taken too much notice regarding any new drivers/services that were installed (i.e. HP Universal Print Driver / Canon Scanner / OpenVPN network adapter etc). So I was struggling to work out what was legit and what was not.

In the few short minutes I was gone from the computer, the Malware well and truly set its hooks in installing crap such as Offerbox and various browser plug-ins/extensions. My AVG Anti-Virus Free Edition version 2011 unfortunately caught very little of this malware that was being installed.

Fortunately I also had installed on my machine two additional free programs. The first being CCleaner http://www.piriform.com/ccleaner, and the second, a very old version of Unlocker developed by Cedrick Collomb. Unlocker is an extremely useful tool that is capable of releasing various locks on files that are being held by system and application processes. Using CCleaner, I was able to see *some* of the additional startup programs that had been added. Using Unlocker, I was able to delete a lot of malware running, and also delete the “C:\Program Files\Java” directory as well. After removing some of the malware, I ran CCleaner’s Registry Cleaner tool which provided me some locations in the registry that were invalid (pointing to missing shared DLLS/applications paths broken etc). Those entries identified by CCLeaner which corresponded to Malware, gave me a starting point for my manual registry “cleaning”.

I also stumbled in to the Windows “Prefetch” directory that gave me a basic timeline of the crap that got installed on my machine when I was away, and also some other programs to try and identify and delete.

I cleaned and deleted as much as I could, and performed my first reboot (keeping the network cable disconnected). Firing up TaskInfo after the machine started, I could see some rundll32.exe processes that had spawned that were pointing to some weird DLL files in existing legitimate directories. I could see a file named “lusrmgrk.dll” that was both being invoked from the “C:\Windows” and “C:\Program Files\Pidgin” directory. I opened those directories and set the Folder View Model to detailed and enabled display of both “Date Modified” and “Date Created” columns. Sure enough, those files had been created/modified at exactly the time of this malware travesty. I deleted these files using Unlocker’s assitance. There were other strange files in the Windows directory with similar date/times that were listed in the Prefetch folder at around the time of the Malware installation; So those got deleted as well. I initially set about running some MD5 checksums on the files to see if anyone had a similar virus and reported the checksum, but didn’t get any hits. For example, the MD5 of “Lfeboa.exe”, which was 137216 bytes, was “014303FB5CF4F2F8A2EADD5EDD82427B”.

Up until this stage, I had not reconnected the network cable on the infected laptop. The Apple iPad was getting used to search google for md5 checksums and read various malware removal articles. It was at this point, I really hated Apple and what it stood for. Steve Jobs and/or the marketing geniuses at Apple decided the iPad would not need any external storage support (micro SD etc). I can only speculate this is purely self-motivated so that they can get the Apple fanboys to continually upgrade to a newer device with more capacity. What this meant though, is that I had no way of getting a file from the iPad to my infected computer without establishing a network connection (or jail-breaking the device and buying a camera kit). Thankyou Apple.

So the network connection on the PC had to be turned on (very very briefly). I downloaded two apps:

TDSSKiller (an anti-rootkit utility from Kaspersky Lab: http://support.kaspersky.com/faq/?qid=208283363)

and

Malwarebytes Anti-Malware (http://www.malwarebytes.org/mbam-download.php )

Malwarebytes needed to be briefly connected to the internet so that it could update its database, but there is a workaround in future for this: See issue #4: http://forums.malwarebytes.org/index.php?showtopic=10138

Having downloaded and installed these, the network connection was yanked and I let the programs do their jobs. Both apps found remaining malware on disk and were able to remove most* traces. I initially did the quick scan with Malwarebytes, but eventually did the full scan which detected an additional malware file.

Upon reboot, I fired up the taskinfo application, but could still see two rundll32.exe processes. However the processes were attempting to load DLLs that were not present on disk. I decided to run the Windows Malicious Software removal tool, but it did not detect anything.

My concern was that there was still some malware on disk that was causing the processes to be initiated.

Taskinfo also provides an option to view the parent process ID of a task. The parent process IDs for the rundll32 processes were pointing to a svchost process. And in particular, “C:\WINDOWS\System32\svchost.exe -k netsvcs”.

See the following article for a description of svchost: http://support.microsoft.com/kb/314056

Basically, Svchost is responsible for starting services at startup. If you were to view task manager, you would see there are a number of svchost processes, each of which is starting services from a particular service group. The “netsvcs” group encompasses some 20 or so services:

One of the services in the group was ultimately triggering the rundll32 processes. I just wasn’t sure how. It could be that a malware service had been added; or maybe an existing legitimate service had been compromised; or maybe an existing service had a dependency on some other service that was compromised etc.

Unfortunately I could not find an easy way to work out which service was triggering the rundll32 processes. So I decided to basically stop them in blocks and restart the computer and try and isolate which service was somehow responsible.

Bingo; Task Scheduler was somehow a part of this! I had read up about Malware creating and scheduling tasks to spawn their evil crap. I had looked in the C:\Windows\Tasks directory, but it was empty. This was from Windows Explorer with all options set to show hidden files. So I was concerned that the Task Scheduler process itself maybe compromised, or a dependant service (Remote Procedure Call).

I decided to do one final check from command prompt supplying the “/ah” option to the “dir” command. Wow; there were some .job files in the directory.

Using the “Xcacls.vbs” tool, http://support.microsoft.com/kb/825751/en-us I was able to give my local administrator user “full control” on the two .job files. Having done this, I was able to change the file permissions so they were no longer hidden etc.

At this point, the tasks became visible from Windows Explorer, and they could be deleted:

A quick reboot, and the processes were no more.

I’ll never know If I’ve removed all the Malware; But one thing is for sure, I’m going to be much more anal when it comes to Browser plug-ins.

And … if you are currently running JDK 1.6.0_23 or older, make sure you upgrade!!!!!!

http://www.oracle.com/technetwork/topics/security/javacpufeb2011-304611.html

Tomato – Adding Custom Packages - NVRAM vs JFFS vs USB Stick

2011-03-16T19:18:00.001+10:00

I made the mistake last night of visiting the Tomato forum using my new (but actually second-hand) generation 1 iPad. It was late, and I desperately needed the sleep, but what the hell. Unfortunately I came across a forum thread titled My utilities web site revived. Some two and a bit hours later (and well into the next day), I put the iPad down having read the thread from start to finish. I gleaned so much useful and interesting bits of information from that that thread that I felt compelled to write a blogpost. This was mostly for my own future reference, but also to share the knowledge.

All I can say is that ‘rhester72’ is a porting/compilation stud and by the looks of it one generous and smart dude. Tomato community are very lucky to have him.

Me personally – well I’m one of the 4 billion java developer drones. I haven’t actively coded in C/C++ for a number of years. I miss these languages; mostly I miss the forgotten skills to properly harness and control these languages – compiler directives / linking /make files / best practices etc. My foray into iPad application development will at least see me leveraging C once more.

Anyway, returning to blog topic …

rhester72 has gone to the trouble/effort of compiling a number of very useful packages/applications for the Tomato environment (MIPS processor series / running 2.6 Linux kernel/and to a lesser extent 2.4 kernel). These packages add some nice bells and whistles that may come in very handy. For example, the torrent client (transmission) means you can potentially turn off your power-sucking gaming box and leave it to your ~10 watt ASUS RT-N16 :)

The binaries come in two styles:

static linked
dynamic linked

Static linked binaries are generally much bigger in size, as the dependent libraries are linked/packaged directly in the resulting binary. It is not however always possible to produce a static binary for certain types of packages due to their architecture.

Dynamic linked binaries on the other hard should be smaller, as they are linked at runtime to dependent shared libraries. The issue with these style of binaries is actually finding the the shared libraries themself at runtime:

the shared library may actually be missing from the machine, or not available in the shared library search path (meaning the binary cannot be run)
an incompatible version of the shared library may be located in the shared library search path (causing some type of conflict resulting in the binary not behaving correctly)

Dynamic linked binaries can however reduce memory footprint and save valuable space should multiple packages that you plan on running leverage same versions of specific shared libraries.

Space and memory permitting, it is often simpler to take the static binary.

Check out rhester72’s list of packages at the following URL:

http://www.multics.minidns.net/tomato/

Be sure to view the descriptions, notes, and most importantly the readme.

When using dynamically linked binaries, it is not always obvious what the required shared library dependences are. Fortunately, the “ldd” command can be used to help out.

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/atop
root@asus:/tmp# chmod u+x atop
root@asus:/tmp# ldd atop
        libncurses.so.5 => not found
        libm.so.0 => /lib/libm.so.0 (0x2aabf000)
        libz.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.0 => /lib/libc.so.0 (0x2aafc000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)
root@asus:/tmp#

Above, you can see that libncurses.so.5 and libz.so.1 are missing. Lets rectify this …

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/lib/libncurses.so.5

root@asus:/tmp# wget http://www.multics.minidns.net/tomato/PRECOMPILED/lib/libz.so.1

root@asus:/tmp# ldd atop
        libncurses.so.5 => not found
        libm.so.0 => /lib/libm.so.0 (0x2aabf000)
        libz.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.0 => /lib/libc.so.0 (0x2aafc000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)

This still did not work; Why?

The answer is the shared library search path.

In Linux, Shared Libraries are searched (in order) from the following locations until a match is located:

LD_LIBRARY_PATH environment variable (if set)
A specific rpath location encoded in to the dynamically linked ELF binary (or shared library) at compilation time (if set)
System default paths defined in /etc/ld.so.conf

On my specific stock tomato instance, LD_LIBRARY_PATH is not set.

Using the readelf command on the atop binary, I can see the Library rpath is set to /opt/lib:/opt/usr/lib.

The contents of ld.so.conf on my instance are:

root@asus:/tmp# cat /etc/ld.so.conf
/opt/lib
/opt/usr/lib
/lib
/usr/lib

Thus, I need to modify the instance so that the libncurses.so.5 and libz.so.1 libraries can be found. For the time being, I’m going to manually set LD_LIBRARY_PATH to the /tmp directory which is where I downloaded the files in the first place:

root@asus:/tmp# export LD_LIBRARY_PATH=/tmp

root@asus:/tmp# ldd atop
        libncurses.so.5 => /tmp/libncurses.so.5 (0x2aabf000)
        libm.so.0 => /lib/libm.so.0 (0x2ab12000)
        libz.so.1 => /tmp/libz.so.1 (0x2ab30000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2ab53000)
        libc.so.0 => /lib/libc.so.0 (0x2ab72000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaa8000)

As you can see above, the dependent libraries are now found, and we can at least attempt to invoke the atop binary!

Now that you know how to add the packages, the question is where to install them:

Recall that the /tmp directory on the Tomato router is volatile; It is essentially a ramdisk created in available RAM of the router, and is blown away upon reboot.
Non-volatile available/free flash memory on the router, if sufficient, can potentially be leveraged with important caveats.
Another recommended choice (if available with your router) it to leverage an external USB stick/drive
Possibly you may be able to even utilize network attached storage.

Flash memory is obviously very convenient, but it comes in a variety of sizes. The Linksys WRT54G series came in 2MB, 4MB and 8MB varieties. It is hard enough getting a distribution like Tomato on a router with such little flash memory, let alone using it for custom packages and the like.

The ASUS RT-N16 on the other hand has 32 MB of flash. Even with the full-blown VPN 2.6 Tomato bundle installed, there is still some 25MB available for potential JFFS2 use. There is also a special area in the flash memory known as the NVRAM segment. This is essentially the very last segment in the flash memory and is at minimum one “erase block” in size. Although the erase block size is typically 64KB (or 128KB as is the case with the RT-N16), the actual NVRAM is programmatically restricted to 32KB.

Tomato does support a special “nvram setfile2nvram” command that will allow you to store a very small file in any remaining NVRAM space (such that, within the 32KB not the full erase block segment size i.e. 128KB on the RT-N16). However, this option should be leveraged as one of the last resorts. If you have free flash memory available, then JFFS2 is a better option. Even better again is to leverage a USB stick if your router supports it.

The reasons for this is best explained by OpenWRT developer/CoFounder MBM’s post at the following location: https://forum.openwrt.org/viewtopic.php?id=10083

To quote him directly:

“The flash chip is broken up into sections called erase blocks. On a 4M chip it's usually 64k and on an 8M chip it's 128k. Each erase block is rated at about 100,000-1,000,000 erase/write cycles depending on vendor.

This just means that on a 4M chip you have to have to erase 64k and rewrite it even if you only want to change one byte of the 64k. After that 64k block has been erased 100,000 times you risk failure where it won't store the data properly.

The problem with the NVRAM implementation is that it's exactly one erase block at the very end of the flash. When you boot, the NVRAM data is copied to a buffer in ram; with the exception of "nvram commit", all the nvram commands are using the copy in ram. When you do an "nvram commit" it writes the contents of ram to the flash. So, when you have a chip rated for 100,000 cycles, you'll probably have a failure around the 100,000th "nvram commit".
”

Although leveraging JFFS2 will still result in wear due to erase cycles, this file system is specifically designed with flash devices in mind. It is engineered is such a way to make “wear-levelling more even and prevent erasures from being too concentrated”.

Once again though MBM makes an important point:

“ Suppose we have a jffs2 filesystem with two types of files, files that never change and files that change frequently. Common sense says that the erase blocks containing the files that never change or contain free space will only be written once and will remain untouched while the blocks containing the other files will change frequently; wear leveling means that all of the blocks within the jffs2 filesystem will be used equally, so all of the blocks in the above example would be written to equally even if it means moving data that hasn't changed“

Thus, if you anticipate files being updated /writes occurring on the JFFS2 partition at a considerable rate, you are eventually going to ruin your flash chip. In such scenarios you must absolutely use something like USB. You must also be very careful that any custom packages you install are not repeatedly writing (e.g. log messages) to a location that is stored in the JFFS2 partition.

Here is an ugly script I wrote designed for Tomato 1.28 to get details of flash memory allocation/distribution; Note that I have enabled JFFS2 support on my router from the Tomato UI (Administration > JFFS page):

cat > /tmp/flashinfo.sh <<EOF
#!/bin/sh

# Tomato 1.28 Flash Info Script by Matt Shannon

echo "Router model: " \`nvram get t_model_name\`
uname -a
echo ""

dmesg | grep nvram
dmesg | grep jffs2
echo ""

cat /proc/mtd
echo ""

ERASEBLOCKSIZE=0x\`cat /proc/mtd | grep nvram | cut -f3 -d " "\`
echo "Erase Block size is" \`awk 'BEGIN{printf("%d", '\$ERASEBLOCKSIZE' / 1024)}\` kilobytes
echo ""

NVRAMSEGMENT=0x\`cat /proc/mtd | grep nvram | cut -f2 -d " "\`
echo "NVRAM Full Segment size is" \`awk 'BEGIN{printf("%d", '\$NVRAMSEGMENT' / 1024)}\` kilobytes

NVRAMSUMMARY=\`nvram show | tail -1\`
NVRAMUSED=\`echo \$NVRAMSUMMARY | cut -d "," -f2 | cut -d " " -f2\`
NVRAMFREE=\`echo \$NVRAMSUMMARY | cut -d "," -f3 | cut -d " " -f2\`
echo "NVRAM Actual Size Available to firmware is" \`awk 'BEGIN{printf("%d", ('\$NVRAMUSED' + '\$NVRAMFREE') / 1024)}\` kilobytes
echo "NVRAM Summary: \$NVRAMSUMMARY"
echo ""

JFFSSIZE=0x\`cat /proc/mtd | grep jffs2 | cut -f2 -d " "\`
echo "JFFS2 size is" \`awk 'BEGIN{printf("%d", '\$JFFSSIZE' / 1048576)}\` megabytes
echo ""
EOF

chmod 744 /tmp/flashinfo.sh

/tmp/flashinfo.sh

sample output:

Router model: Asus RT-N16
Linux asus 2.6.22.19 #8 Tue Nov 30 14:58:27 EST 2010 mips GNU/Linux

0x01fe0000-0x02000000 : "nvram"
0x006e0000-0x01fe0000 : "jffs2"

dev: size erasesize name
mtd0: 00040000 00020000 "pmon"
mtd1: 01fa0000 00020000 "linux"
mtd2: 005aec00 00020000 "rootfs"
mtd3: 01900000 00020000 "jffs2"
mtd4: 00020000 00020000 "nvram"

Erase Block size is 128 kilobytes

NVRAM Full Segment size is 128 kilobytes
NVRAM Actual Size Available to firmware is 32 kilobytes
NVRAM Summary: 852 entries, 21155 bytes used, 11613 bytes free.

JFFS2 size is 25 megabytes

Recall from above that rhester72’s dynamically linked binary atop hardcodes an rpath to /opt/lib:/opt/usr/lib. If you were to SSH in to your router however, you will likely find that the /opt directory is empty. Rhester72 leverages an init script (Tomato UI > Administration > Scripts > Init) to attempt to automatically bind the /opt location to a directory “opt” found in the /jffs mount location. (Note, when enabling JFFS2 in Tomato, the JFFS partition is automatically mounted at startup under the /jffs directory). Here is his script which will try for 30 seconds to bind the opt directory:

t=0
while [[ ! -d /jffs/opt && $t -lt 30 ]];do
sleep 1
t=$(($t+1))
done

if [ -d /jffs/opt ];then
mount -o bind /jffs/opt /opt
else
logger -t /jffs -p err did not mount within $t seconds
fi

If the bind fails, a message will be written to /var/log/messages. To utilize such an approach you will need to do the following:

enable JFFS2
once the jffs partition is mounted, mkdir /jffs/opt
make appropriate subdirectories mkdir -p /jffs/opt/bin /jffs/opt/lib /jffs/opt/sbin /jffs/opt/usr/bin /jffs/opt/usr/lib /jffs/opt/usr/sbin /jffs/opt/usr/share

Refer to his readme.

Leveraging the above directory structure to store binaries and shared libraries should give you good results (remember /opt will be bound to /jffs/opt):

root@asus:/tmp/home/root# echo $PATH | tr ":" "\n"
/bin
/usr/bin
/sbin
/usr/sbin
/home/root
/mmc/sbin
/mmc/bin
/mmc/usr/sbin
/mmc/usr/bin
/opt/sbin
/opt/bin
/opt/usr/sbin
/opt/usr/bin

root@asus:/tmp/home/root# cat /etc/ld.so.conf | grep /opt
/opt/lib
/opt/usr/lib

As a final note, be careful with setting LD_LIBRARY_PATH in any profile. Also it is worth inspecting the /etc/profile script, you will see that it attempts to source both /jffs/etc/profile and /opt/etc/profile (if they exist).

Thanks again to rhester72;

Quick & simple VPN setup guide: using OpenVPN on a ‘Tomato’ router

2011-03-14T18:39:00.001+10:00

Before the advent of custom firmware on consumer-priced/graded routers (Linksys/Netgear etc), obtaining secure remote access to a network resource using stock firmware was somewhat an art. Circa 2002, I remember geeking it up in front of my work colleagues by remotely accessing my home machine’s desktop to download mp3 over Napster. Back then, I was using a combination of port-forwarding, VNC, and SSH; amazingly this setup was able to be tunnelled through work’s HTTP proxy server.

Even today, many people still leverage such approaches. The following links give a bit of a history and overview:

http://members.shaw.ca/nicholas.fong/vnc/

http://nodivisions.com/tech/vnc/

Modern custom router firmwares (DD-WRT / Tomato / etc) make it even simpler by bundling some of the required software. Here is a link that describes an updated approach to the above:

http://blog.dreamdevil.com/index.php/2008/09/06/remote-desktop-through-ssh-with-putty-and-tomato-firmware/

Anyway…

A friend whom runs a small business on a tight technology budget recently asked me what I would suggest that would allow his employees remote access to the office “files”. He and his colleagues are regularly travelling for days at a time and require access both at customer sites, and also from hotels. Most of the sites he visits do allow him to connect his laptop to the customer network; but not all. Thus, we needed a solution that would wherever possible allow him to leverage the customer’s office internet connection. For those sites that prevent such access, he would leverage a 3G solution using either a USB dongle, or 3G Wi-Fi Hotspot.

Immediately, OpenVPN sprung to mind as a possible solution – mostly cause it was free, and also had client support on Windows, Mac, and Linux. The icing on the cake however was the fact that the DD-WRT and Tomato router firmwares happened to provide special VPN builds that bundled the OpenVPN server application. Thus, assuming we could locate a router with sufficient grunt that could run such a firmware, there would be no need to have a dedicated separate OpenVPN server machine.

Basically, three main routers sprung to mind:

Asus RT-N16

Linksys (Cisco) E3000

Netgear WNDR3700

The first two routers (Asus & Linksys) are supported by Tomato. All three routers are supported by DD-WRT.

I’ve had great success with Tomato in the past, so I suggested to my friend to purchase the Linksys E3000. This can be purchased here in Australia for about $170.

It just so happened that I had an Asus RT-N16 at home, hence I was able to test out the VPN configuration on my home environment, before messing with my friends office network. I liked it so much (having a VPN), that I decided to keep it and refine it!

The main decisions when leveraging OpenVPN are:

1) Does your router have a static WAN IP address?

If you router does not have a static IP address, then you are at the mercy of your internet provider whom may or may not reissue you the same IP address upon lease expiration/reboot etc. It is best to play it safe and configure your router to use a Dynamic DNS. Sign up for a free account at an appropriate provider (e.g. http://www.dyndns.com), and then configure your router with the appropriate DDNS account details. Your router will subsequently register your dynamic IP address with the provider whenever your WAN connection comes up/changes. This way, you can configure your OpenVPN client configuration with a static hostname (e.g. todayguesswhat.dyndns.org)

2) Do I TUN (network TUNnel) or TAP (network tap)?

TAP > runs at layer 2 (OSI model) - bridge mode ; effectively the VPN client will appear to be on the same network subnet as the destination. Packets broadcast on destination network will be received by client.

TUN > runs at layer 3 (OSI model) – router mode; effectively the VPN client is on a different network to the destination; routing rules are used to allow the client to access the destination network. Packets broadcast on destination network won’t be received by VPN client.

3) What protocol: TCP or UDP?

SgtPepper (Tomato guru) over at the LinksysInfo forum clarified/corrected my understanding of what this option means. Initially I assumed incorrectly that the choice between TCP and UDP came down to what application layer network protocols you planned on using over the VPN. Such that, if you wanted FTP/Telnet/SSH HTTP/HTTPS/SMTP/IMAP/SMB-over-TCP/NFS-over-TCP/ you chose TCP, whereas VoIP and Network Games, then UDP. WRONG!

To Quote SgtPepper directly “Choosing TCP vs UDP should have nothing to do with the application type you're using. OpenVPN tunnels TCP and UDP traffic over whichever protocol you choose. Tunneling TCP over TCP is extremely inefficient, so TCP should only be chosen if you absolutely have to. That should only be if you have to go through an HTTP proxy, trick a firewall, or have a very flaky connection and have problems with UDP. If you have the option, you should absolutely 100% use UDP.”

4) What port for the server to listen on?

The default port is 1194. But if 443 is available, I would suggest using this. The use of port 443 is particularly pertinent to TCP-based OpenVPN configurations whereby client access to the VPN may require tunnelling through a proxy server. Proxy servers are much more likely to accept traffic destined for port 443 versus something like 1194. Port 443 is the default port used by secure HTTP (aka HTTPS), and as such most proxy servers should allow it unimpeded.

5) What LAN subnet/segment should the router be leveraging?

Most routers ship with a default IP address of 192.168.1.1, and subnet mask 255.255.255.0. Chances are, one of the destinations you visit will also be leveraging such a topology. The problem occurs when the destination network is the same as the actual client network you are connecting from. For example, if my network at home is 192.168.1.x, and I’m at a client site that is using 192.168.1.x, and I attempt to make VPN connection to home, the routing tables get completely messed up. There are probably some crazy network mask and routing rule/metric options you can leverage to work around such a situation, but my advice is to avoid it in the first place.

For my home VPN, I decided to leverage TAP (interface type), TCP (protocol), 443 (port), and 192.168.192.1/24 (CIDR Notation).

The reason I chose TAP is to theoretically support broadcast traffic, but mostly so as to appear on the same network as my destination. If your client has no business to receive the broadcast traffic, and you are expecting large amounts of broadcast traffic on the destination network subnet, then it is best to switch to TUN.

The reason I chose TCP is that OpenVPN natively supports tunnelling of the TCP traffic over a HTTP proxy. This means that if we are stuck behind a HTTP proxy without direct internet access, we can still hopefully access the VPN by tunnelling through the proxy server. Refer to SgtPepper’s quote from above however regarding efficiency and only choose this option if you have to!

I chose port 443 likewise to increase my odds of a proxy server connection working, and also so that it looks legitimate from an auditing perspective.

I chose 192.168.192.x/255.255.255.0 network for my LAN, as this network has less chance of being leveraged on a client site (unlike for example 192.168.1.x).

One interesting point to make is that Tomato natively supports two OpenVPN server processes running at the same time. Thus there is nothing stopping you running TCP and TAP for one instance, and UDP with TUN/TAP on the other instance.

Now… on to the VPN setup:

If you are running a Linksys E3000, I suggest obtaining and installing the following firmware (or newer):

tomato-E3000USB-1.28.9054MIPSR2-beta-vpn3.6.bin

the above firmware can be installed directly over the top of the factory firmware.

If you are running an Asus RT-N16, I suggest obtaining and installing the following firmware (or newer):

tomato-K26USB-1.28.9054MIPSR2-beta-vpn3.6.trx

Note: With the RT-N16, to upgrade from a stock firmware you must first flash the router with the following:

http://www.dd-wrt.com/routerdb/de/download/Asus/RT-N16/-/dd-wrt.v24-14896_NEWD-2_K2.6_mini_RT-N16.trx/3763

Next, you must download and install the OpenVPN software specific for your OS (client)

For Windows, this is currently: http://swupdate.openvpn.net/community/releases/openvpn-2.1.4-install.exe

As we only need the client components and RSA certificate management scripts, be sure to DESELECT “OpenVPN Service”.

Next, we will generate the various keypairs…

CD /D "%ProgramFiles%\OpenVPN\easy-rsa"

## create vars.bat and openssl.cnf from templates
init-config.bat

##   Update the values in vars.bat as appropriate, specifically:
##   KEY_COUNTRY, KEY_PROVINCE, KEY_CITY, KEY_ORG, KEY_EMAIL
##   If requiring stronger keys, change KEY_SIZE from 1024 to 2048
##   Use WordPad to edit the file, as it leverages UNIX line breaks
"%ProgramFiles%\Windows NT\Accessories\wordpad.exe" "%ProgramFiles%\OpenVPN\easy-rsa\vars.bat"

## Invoke vars.bat to set environment
vars

## CAUTION - deletes any existing keys / reset SERIAL and INDEX.TXT
clean-all

## Construct Certificate Authority keypair, set commonName to be CA
build-ca

## Construct server keypair, set commonName to VPNServer
build-key-server server

## Construct client keypair for user 'matt', set commonName as appropriate (e.g. matt)
build-key client_matt

...

## Construct client keypair for user 'louise', set commonName as appropriate (e.g. louise)
## build-key client_louise

## NOTE - if you have specified a challenge password in your certificate when you created it,
## you will be required to provide that password should you ever want to request to revoke the certificate

##   The keys directory will contain private keys / certificates / index file stating
##   issued certificates / serial file containing next serial number to leverage etc.
##   !!! KEEP THEM IN A SAFE PLACE. !!!!

## Generate Diffie Hellman parameters
## This will generate dh1024.pem in keys folder (or dh2048.pem files, depending on KEY-SIZE variable)
build-dh

Next, we will configure Tomato…

From the Tomato Administration UI, choose “VPN Tunnelling” > "Server”

Refer to screenshots for detailed settings.

Basic Settings:
Start with WAN: <checked>
Interface Type: TAP
Protocol: TCP
Port: 443
Firewall: Automatic
Authorization Mode: TLS
Extra HMAC authorization (tls-auth): Disabled
Client address pool: DHCP <checked>

Advanced Settings:
Poll Interval: 0
Direct clients to redirect internet traffic: <NOT checked>
Respond to DNS: <NOT checked>
Encryption cipher: Use Default
Compression: Adaptive
TLS Renegotiation Time: -1
Manage Client-Specific Options: <checked>
Allow Client<->Client: <checked>
Allow Only These Clients: <NOT checked>

Custom Configuration**:
script-security 3
auth-user-pass-verify /etc/verify.sh via-env

Keys
Certificate Authority: <paste contents of ca.crt>
Server Certificate: <paste contents of server.crt from -----BEGIN CERTIFICATE----- through-----END CERTIFICATE----- inclusive>
Server Key: <paste contents of server.key>
Diffie Hellman parameters: <paste contents of dh1024.pem>

All the above OpenVPN configuration options ultimately get stored in NVRAM. Tomato will dynamically generate the appropriate configuration file and keys/certificates in the “/tmp/etc/openvpn” area based on these NVRAM values.

If you were to SSH/Telnet in to the router, and issue an “nvram show | grep vpn_server1” command, you should see the various configuration values stored in NVRAM from above.

The dynamically constructed files are as follows:

/tmp/etc/openvpn/server1/ [ca.crt | config.ovpn | dh.pem | server.crt | server.key]
/tmp/etc/openvpn/fw/server1-fw.sh

The server1-fw.sh file contains the iptables entries.

** You will notice the “Advanced” tab > “Custom Configuration” option for the Server VPN Tunnelling is populated with a script-security and auth-user-pass-verify entry. These options are adding an additional level of security by requiring the VPN client not only hold a valid keypair, but also present a valid username/password. The client supplied username/password is provided to a custom script (that we must create) named verify.sh. This script must return exit status 0 in order for the VPN client connection to be successful (assuming they had a valid keypair in the first place).

Unfortunately the /tmp folder is erased and recreated every time the router is rebooted. (/etc is a symbolic link to /tmp/etc). We need a mechanism to ensure shell scripts for custom authentication survive reboot. There are three options:

1) enable the JFFS feature and essentially leverage the unused portion of the router's NVRAM and turn it in to a mountable and writable space http://tomatousb.org/doc:jffs

2) leverage init scripts in the tomato UI to recreate the various shell scripts required in the /tmp directory at boot time.

3) use the "nvram setfile2nvram <filename>" command to save small files in nvram. The files will be automatically restored on start-up.

We will leverage the latter option (#3) for our custom authentication script; SSH/telnet in to the router as root and issue the following:

cd /etc

cat > /etc/generate.sh <<EOF
#!/bin/sh

HASHPASS=\`echo -n "\$1\$2" | md5sum | sed s'/\ -//'\`
i=0
while [ \$i -lt 10 ]; do
HASHPASS=\`echo -n \$HASHPASS\$HASHPASS | md5sum | sed s'/\ -//'\`
i=\`expr \$i + 1\`
done
echo \$1:\$HASHPASS
exit 1
EOF

chmod 755 /etc/generate.sh

nvram setfile2nvram /etc/generate.sh

cat > /etc/verify.sh <<EOF
#!/bin/sh
# echo "\${username}"
hash=\`/etc/generate.sh "\${username}" "\${password}"\`

USERS=\`cat /etc/vpnusers\`
for u in \$USERS; do
test "\${hash}" == "\${u}" && exit 0
done
exit 1
EOF

chmod 755 /etc/verify.sh

nvram setfile2nvram /etc/verify.sh

To generate password hashes for the users (first argument is username, second argument is password):

/etc/generate.sh matt test1234 >> /etc/vpnusers

Persist the vpnusers file:

nvram setfile2nvram /etc/vpnusers

To test the verify script:

export username=matt
export password=test1234
/etc/verify.sh
echo $?

If the output is 0 from the above command, the matt/test1234 credential was found in the vpnusers file. If the output is 1, something is broken!

The hard part is now done. Reboot the router!

The final task is creation of the client configuration file and adding the client keypair and CA:

1) Make a directory "config" under "%ProgramFiles%\OpenVPN" if not already present.

2) Within the "config" directory, make a subdirectory, e.g "homevpn"

3) Copy to the "homevpn" directory ca.crt, and the appropriate client keypair (e.g. client_matt.key / client_matt.crt).

4) Within the "config" directory, create an openvpn client config file, e.g. "homevpn.ovpn"; The contents of "homevpn.ovpn" based on the above server configuration above are as follows:

# The hostname/IP and port of the server. You can have multiple remote entries to load balance between the servers.
remote todayguesswhat.dyndns.org 443

# Specify that we are a client and that we will be pulling certain config file directives from the server.
client

ns-cert-type server

# On most systems, the VPN will not function unless you partially or fully disable the firewall for the TUN/TAP interface.
dev tap21

# Are we connecting to a TCP or UDP server?
proto tcp

# Keep trying indefinitely to resolve the host name of the OpenVPN server. Useful for machines which are not permanently connected to the internet such as laptops.
resolv-retry infinite

# Most clients don't need to bind to a specific local port number.
nobind

# Try to preserve some state across restarts.
persist-key
persist-tun

# --float tells OpenVPN to accept authenticated packets from any address, not only the address which was specified in the --remote option.
# Useful if you're using round-robin DNS. Also useful if your server has a dynamic IP address which the ISP could change.
# I use float so I can connect from inside AND outside my router.
float

# If the pushed routes appear not to be added on windows hosts, add the following:
# route-delay 30

# SSL/TLS parms.
ca "C:\\Program Files\\OpenVPN\\config\\homevpn\\ca.crt"
cert "C:\\Program Files\\OpenVPN\\config\\homevpn\\client_matt.crt"
key "C:\\Program Files\\OpenVPN\\config\\homevpn\\client_matt.key"

# Enable compression on the VPN link.
# Don't enable this unless it is also
# enabled in the server config file.
comp-lzo

# Set log file verbosity.
verb 3

# Silence repeating messages
mute 20

# prompt for username and password
auth-user-pass

You should hopefully now be able to establish a VPN connection! Good luck.

javac – file not found – wildcards and spaces in directory names

2011-02-10T16:33:00.001+10:00

Calling the javac command from Windows XP with the source files argument containing a wildcard in quotes – e.g. javac "*.java" will result in the error:

javac: file not found:

Whereas with no quotes – e.g. javac *.java , no error is produced.

Consider a source files argument containing both a path and wildcard combined – e.g. javac C:\temp\*.java , this is perfectly fine and no error is produced.

The following however will fail –e.g. javac "C:\my documents\*.java" , due to the wildcard in quotes issue above.

Some may argue that this is perfectly reasonable, given that java does not allow spaces in package names. But what if the java files above in the “my documents” folder are top-level classes without package qualification – such as the file below:

e.g.

class Test
{
public static void main(String[] args)
{
System.out.println("Hello World!");
}
}

Consider a simple java kit that you deliver to folks with the following structure:-

MyApplication\

bin\compile.bat
lib\utilities.jar
lib\resources.jar
src\com\acme\myapp\Main.java
src\com\acme\myapp\utils\Utils.java

If the kit above is extracted to some folder/directory without spaces, it should be pretty easy to compile from the command-line. Whereas if it were extracted to the standard Windows desktop location as shown below, more care must be taken:

C:\Documents and Settings\Administrator\Desktop\MyApplication\

bin\compile.bat
lib\utilities.jar
lib\resources.jar
src\com\acme\myapp\Main.java
src\com\acme\myapp\utils\Utils.java

One will not be able to run javac "C:\Documents and Settings\Administrator\Desktop\MyApplication\src\com\acme\myapp\*.java" . They must instead change directory to an appropriate location whereby the javac source files argument can be supplied without including quotes. For example:

cd /D "C:\Documents and Settings\Administrator\Desktop\MyApplication"
javac src\com\acme\myapp\*.java

Below is a sample windows java compile batch script that I’ve created that should work with the application structure above regardless of what destination directory the kit is extracted to:-

compile.bat

@ECHO off

REM Changes made to an Environment Variable after SETLOCAL has been issued are local to the batch file.
REM Issuing an ENDLOCAL command will restore the previous environment variables.
SETLOCAL

REM Commands
SET JAVACMPCMD=%JAVA_HOME%\bin\javac.exe
SET JARCMD=%JAVA_HOME%\bin\jar.exe

REM library files and source location
SET PROJECT_LIB=%~dp0..\lib
SET PROJECT_SRC=%~dp0..\src
REM source base package path
SET BASE_PKG_PATH=%PROJECT_SRC%\com\acme\myapp

REM classes directory
SET PROJECT_CLASSES=%~dp0..\classes

REM generated library destination path and file
SET DEST_JAR_FILE=%~dp0..\myapp.jar

REM classpath for compilation
SET CP=%PROJECT_LIB%\utilities.jar;%PROJECT_LIB%\resources.jar

REM ************
REM MAIN PROGRAM
REM ************

REM Verify whether required variables are set
IF "%JAVA_HOME%x"=="x" GOTO errjava

REM Preparation ...
IF EXIST "%DEST_JAR_FILE%" DEL /q "%DEST_JAR_FILE%"
IF EXIST "%PROJECT_CLASSES%" RMDIR /s /q "%PROJECT_CLASSES%"
MKDIR "%PROJECT_CLASSES%"

ECHO.
ECHO Compiling code ...

PUSHD "%BASE_PKG_PATH%"
"%JAVACMPCMD%" -target 1.6 -classpath "%CP%" -d "%PROJECT_CLASSES%" *.java utils\*.java
SET ERR=%ERRORLEVEL%
POPD
IF NOT "%ERR%"=="0" GOTO errcomp

ECHO.
ECHO Creating jar file ...

"%JARCMD%" cfM "%DEST_JAR_FILE%" -C "%PROJECT_CLASSES%/" .
IF NOT "%ERRORLEVEL%"=="0" GOTO errjar

REM Cleanup ...
RMDIR /s /q "%PROJECT_CLASSES%"
GOTO end

REM JAVA_HOME is not set
:errjava
ECHO.
ECHO Set the following environment variables:
ECHO.
ECHO JAVA_HOME (Absolute path to jdk directory)
GOTO end

:errcomp
ECHO.
ECHO Compilation did not succeed.
ECHO Note: If receiving read errors, ensure folder path does not contain whitespace.
GOTO end

:errjar
ECHO.
ECHO Jar creation did not succeed.
GOTO end

:end
ECHO.

vi – range replace – indent block of lines

2011-01-08T13:38:00.001+10:00

example 1:

from line 10 through 50, insert two spaces at the start of the line

<esc>:10,50s/^/ /

example 2:

from line 10 through 50, replace the two spaces at the start of the line with empty

<esc>:10,50s/^ //

Additional Batch PDFill AutoIt Scripts

2010-12-29T09:51:00.001+10:00

Merge all JPG image files in the source directory to a single PDF file in the output directory:

; Script by Matt Shannon - Dec 2010

$srcDirectory = 'C:\Documents and Settings\Administrator\Desktop\source'
$destDirectory = 'C:\Documents and Settings\Administrator\Desktop\output'

; change current directory to src directory
FileChangeDir($srcDirectory)
$search = FileFindFirstFile("*.jpg")

If $search = -1 Then
MsgBox(0, "Error", "No files/directories matched the search pattern")
Exit
EndIf

RunPDFill()

$dest = ''

While 1
$file = FileFindNextFile($search)
If @error Then ExitLoop
; MsgBox(4096, "File:", $file)

$dest = StringReplace($file, ".jpg", -1, 0) & ".pdf"

AddImg($srcDirectory, $file)

WEnd

; Close the search handle
FileClose($search)

SaveToPDF($destDirectory, $dest)

Func RunPDFill()

Run('C:\Program Files\PlotSoft\PDFill\PDFill_PDF_Tools.exe', "", @SW_MAXIMIZE)
WinWaitActive("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")

ControlClick("[REGEXPTITLE:PDFill PDF Tools 7.0.*]", "", "[CLASS:Button; TEXT: 9. Convert Images to PDF]")
WinWaitActive("[TITLE:Free PDF Tools: Convert images to PDF]")

; set paper size output to A4
ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:ComboBox; INSTANCE:1]")
Send("A{ENTER}")

; set margins to 0
ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Edit; INSTANCE:3]")
Send("{HOME}{SHIFTDOWN}{END}{SHIFTUP}{DEL}0")

ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Edit; INSTANCE:4]")
Send("{HOME}{SHIFTDOWN}{END}{SHIFTUP}{DEL}0")

ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Edit; INSTANCE:5]")
Send("{HOME}{SHIFTDOWN}{END}{SHIFTUP}{DEL}0")

ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Edit; INSTANCE:6]")
Send("{HOME}{SHIFTDOWN}{END}{SHIFTUP}{DEL}0")

EndFunc

Func AddImg($srcdir, $srcfile)

; add an image
ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Button; TEXT:Add an Image]")
WinWaitActive("[TITLE:Select Image files to add into PDF]")

; send image location followed by enter
ControlClick("[TITLE:Select Image files to add into PDF]", "", "[CLASS:Edit; INSTANCE:1]")
Send($srcdir & "\" & $srcfile & "{ENTER}")

; wait for window to return
WinWaitActive("[TITLE:Free PDF Tools: Convert images to PDF]")

EndFunc

Func SaveToPDF($destdir, $destfile)

; click save-as
ControlClick("[TITLE:Free PDF Tools: Convert images to PDF]", "", "[CLASS:Button; TEXT:Save As ...]")
WinWaitActive("[TITLE:Save all the images as ... ]")

; send pdf output location followed by enter
ControlClick("[TITLE:Save all the images as ... ]", "", "[CLASS:Edit; INSTANCE:1]")
Send($destdir & "\" & $destfile & "{ENTER}")

; wait for adobe to open - and then close it down
WinWaitActive("[TITLE:" & $destfile & " - Adobe Reader]")
WinClose("[TITLE:" & $destfile & " - Adobe Reader]")

; wait for pdf image tools to return - and then close it down
WinWaitActive("[TITLE:Free PDF Tools: Convert images to PDF]")
WinClose("[TITLE:Free PDF Tools: Convert images to PDF]")

; wait for main pdf tools to return - and then close it down
WinWaitActive("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")
WinClose("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")

EndFunc

Merge all PDF files in the source directory to a single PDF file in the output directory:

; Script by Matt Shannon - Dec 2010

$srcDirectory = 'C:\Documents and Settings\Administrator\Desktop\source'
$destDirectory = 'C:\Documents and Settings\Administrator\Desktop\output'

; change current directory to src directory
FileChangeDir($srcDirectory)
$search = FileFindFirstFile("*.pdf")

If $search = -1 Then
MsgBox(0, "Error", "No files/directories matched the search pattern")
Exit
EndIf

RunPDFill()

$dest = ''

While 1
$file = FileFindNextFile($search)
If @error Then ExitLoop
; MsgBox(4096, "File:", $file)

$dest = StringReplace($file, ".pdf", -1, 0) & "_merged.pdf"

AddPdf($srcDirectory, $file)

WEnd

; Close the search handle
FileClose($search)

SaveToPDF($destDirectory, $dest)

Func RunPDFill()

Run('C:\Program Files\PlotSoft\PDFill\PDFill_PDF_Tools.exe', "", @SW_MAXIMIZE)
WinWaitActive("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")

ControlClick("[REGEXPTITLE:PDFill PDF Tools 7.0.*]", "", "[CLASS:Button; TEXT: 1. Merge PDF Files]")
WinWaitActive("[TITLE:Free PDF Tools: Merge PDF Files]")

EndFunc

Func AddPdf($srcdir, $srcfile)

; Add a PDF File
ControlClick("[TITLE:Free PDF Tools: Merge PDF Files]", "", "[CLASS:Button; TEXT:Add a PDF File]")
WinWaitActive("[TITLE:Add PDF files to be Concated]")

; send image location followed by enter
ControlClick("[TITLE:Add PDF files to be Concated]", "", "[CLASS:Edit; INSTANCE:1]")
Send($srcdir & "\" & $srcfile & "{ENTER}")

; wait for window to return
WinWaitActive("[TITLE:Free PDF Tools: Merge PDF Files]")

EndFunc

Func SaveToPDF($destdir, $destfile)

; click save-as
ControlClick("[TITLE:Free PDF Tools: Merge PDF Files]", "", "[CLASS:Button; TEXT:Save As ...]")
WinWaitActive("[TITLE:Save the concated file as ... ]")

; send pdf output location followed by enter
ControlClick("[TITLE:Save the concated file as ... ]", "", "[CLASS:Edit; INSTANCE:1]")
Send($destdir & "\" & $destfile & "{ENTER}")

; wait for adobe to open - and then close it down
WinWaitActive("[TITLE:" & $destfile & " - Adobe Reader]")
WinClose("[TITLE:" & $destfile & " - Adobe Reader]")

; wait for pdf image tools to return - and then close it down
WinWaitActive("[TITLE:Free PDF Tools: Merge PDF Files]")
WinClose("[TITLE:Free PDF Tools: Merge PDF Files]")

; wait for main pdf tools to return - and then close it down
WinWaitActive("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")
WinClose("[REGEXPTITLE:PDFill PDF Tools 7.0.*]")

EndFunc

Batch Convert Images to PDF using PDFill and AutoIt

2010-12-24T13:57:00.001+10:00

I’m in the processing of completely digitalizing my filing cabinet and shredding all but the most important paper-based records. I recently purchased a Canon PIXMA MX870 Multifunction printer/scanner unit purely for its Auto Document Feeder with duplex scanning support. It turns out, the ADF duplex support isn’t all I had hoped with it regularly complaining about paper jams. This really irritates me Canon! I’m furious actually.

Anyway back to the digital filing cabinet …

I’m leveraging three really cool pieces of software:

FileCenter by Lucion http://lucion.com/filecenter-overview.html
PDFill PDF tools (FREE) by PlotSoft http://www.pdfill.com/pdf_tools_free.html
AutoIt http://www.autoitscript.com/autoit3

FileCenter is providing the interface to my new digital filing cabinet providing file management capabilities along with scanning and OCR (optical character recognition) features. It has a number of really useful bells and whistles including smart file (re)naming capabilities that really help with the overall organization of the filing cabinet. The standard edition which I initially purchased (on special now for $50) seems to do its job quite well. The only downside is the supplied OCR engine is sometimes not up to the task. The FileCenter Pro Plus edition (on special now for $199) ships with an Advanced OCR engine that seems to have a much better job at extracting text from the various scans/images I supply it. I may persist with Standard Edition for a little longer as money is tight – but if Lucion want to give me a free upgrade please contact me :). Lucion provides a 30 day trial of FileCenter Pro Plus with no limitations whatsoever – so check it out. It’s a great piece of software.

PDFill PDF tools are FREE and are without doubt some of the most useful and powerful tools I have ever come across. I use these tools regularly to do things like:

Merging PDF files together
Reordering pages of PDF files
Converting images to PDF
Rotating PDF pages

Whilst FileCenter appears to provide many of these capabilities, the level of customization and tweaking that the PDFill PDF tools provide is unmatched. The only downside to these tools is they are all GUI based requiring lots of clicks and user input and don’t appear to provide any command-line based control for doing tasks like batch conversion/processing.

This is where the final piece of software “AutoIt v3” comes in to its own. The guys that wrote this are studs to release it for free. This tool/scripting language allows one to construct little batch file/shell script like programs that can automatically control and interact with Windows applications without any end-user input required. Thus, I can write a little program for example that simulates mouse clicks and keyboard input from a user.

I have a number a JPG image files that are essentially scans of various paper-based documents that I want to convert to PDF. PDFill provides a tool "Convert Images to PDF" that allows me to manually navigate to the appropriate source image, and then specify the output PDF file along with the PDF output page size / margins / image layout etc etc. This is quite a laborious repetitive task that I decided to automate by using an AutoIT v3 script.

The script essentially searches for JPG files in a particular source directory, then iterates through each image file returned supplying it automatically to the PDFill tool. Appropriate output options / output file etc get set using a combination of automated mouse gestures and keyboard input. Upon conversion completion, the script detects Adobe Reader automatically launching to preview the output and closes it down.

Without further adieu – I give you the automated batched PDFill conversion script :-

; Script by Matt Shannon - Dec 2010

$srcDirectory = 'C:\Documents and Settings\Administrator\Desktop\scans'
$destDirectory = 'C:\Documents and Settings\Administrator\Desktop\output'

; change current directory to src directory
FileChangeDir($srcDirectory)
$search = FileFindFirstFile("*.jpg")

If $search = -1 Then
MsgBox(0, "Error", "No files/directories matched the search pattern")
Exit
EndIf

While 1
$file = FileFindNextFile($search)
If @error Then ExitLoop
; MsgBox(4096, "File:", $file)

$dest = StringReplace($file, ".jpg", -1, 0) & ".pdf"

ConvertToPDF($srcDirectory, $file, $destDirectory, $dest)

WEnd

; Close the search handle
FileClose($search)

Func ConvertToPDF($srcdir, $srcfile, $destdir, $destfile)