Search this blog ...

Saturday, September 15, 2012

JAXB - XSD to Java Map/HashMap example using xjc, bindings and XmlAdapter

I’ve been assigned the task of implementing for my specific product team a common RESTful API that is invoked as part of a cloud on-boarding process.  The spec provided to me describes the data structures comprising the request and responses – all of which will be encoded in JSON.  In an ideal world, I would leverage something like JAX-RS (using something like Jersey RI). But, alas, that would be too easy.  Instead I must host this API somehow on top of my product’s existing service-based architecture framework.  I went searching around for a JAXB equivalent for JSON – that would allow some type of JSON to Java binding. I was hoping I would find some type of JSON schema definition concept, an xjc and schemgen equivalent etc.  I came up short on my search for such tools, but I did discover that Jackson and Jersey can support de/serialization from/to JSON of Java objects that are annotated using JAXB (java.xml.bind.annotation). This was a welcome discovery and it meant I could set about trying to model the specification’s data structures with XSD.

My plan was to create the XML schema up front, and then leverage xjc to create the set of JAXB-annotated Java classes that map to the elements/types defined in the schema.  Everything was coming along nicely.  Whenever I got stuck on the XSD front, I would simply try and model the concept using some basic java classes, then fire up the schemagen tool to view the schema it generated, and incorporate the techniques/result back in to my own XSD. I completed the XSD, and invoked xjc and out came my auto-generated JAXB-annotated classes.  However, some of the classes and properties were not what I was expecting.  Where had my java.util.Map based properties gone?

If you take a simple sample like the following – a Person object with a map property of all their worldly gadgets (iPad/iPhones etc):

Person.java

import java.util.Map;

public class Person
{
  private String name;
  private Map<String, Gadget> gadgets;

  public void setName(String name)
  {
    this.name = name;
  }

  public String getName()
  {
    return name;
  }

  public void setGadgets(Map<String, Gadget> gadgets)
  {
    this.gadgets = gadgets;
  }

  public Map<String, Gadget> getGadgets()
  {
    return gadgets;
  }
}

Gadget.java

public class Gadget
{
  private String make;

  public void setMake(String make)
  {
    this.make = make;
  }

  public String getMake()
  {
    return make;
  }
}

Running schemagen on the above, you get the following schema:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<xs:schema version="1.0" xmlns:xs="
http://www.w3.org/2001/XMLSchema">

  <xs:complexType name="gadget">
    <xs:sequence>
      <xs:element name="make" type="xs:string" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="person">
    <xs:sequence>
      <xs:element name="gadgets">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="entry" minOccurs="0" maxOccurs="unbounded">
              <xs:complexType>
                <xs:sequence>
                  <xs:element name="key" minOccurs="0" type="xs:string"/>
                  <xs:element name="value" minOccurs="0" type="gadget"/>
                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
      <xs:element name="name" type="xs:string" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

If I then feed the above schema back to xjc, the JAXB-annotated classes that get generated look like the following (javadoc removed to save on space):

Person.java

import java.util.ArrayList;
import java.util.List;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlType;

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "person", propOrder = {
    "gadgets",
    "name"
})
public class Person {

    @XmlElement(required = true)
    protected Person.Gadgets gadgets;
    protected String name;

    public Person.Gadgets getGadgets() {
        return gadgets;
    }

    public void setGadgets(Person.Gadgets value) {
        this.gadgets = value;
    }

    public String getName() {
        return name;
    }

    public void setName(String value) {
        this.name = value;
    }


    @XmlAccessorType(XmlAccessType.FIELD)
    @XmlType(name = "", propOrder = {
        "entry"
    })
    public static class Gadgets {

        protected List<Person.Gadgets.Entry> entry;

        public List<Person.Gadgets.Entry> getEntry() {
            if (entry == null) {
                entry = new ArrayList<Person.Gadgets.Entry>();
            }
            return this.entry;
        }

        @XmlAccessorType(XmlAccessType.FIELD)
        @XmlType(name = "", propOrder = {
            "key",
            "value"
        })
        public static class Entry {

            protected String key;
            protected Gadget value;

            public String getKey() {
                return key;
            }

            public void setKey(String value) {
                this.key = value;
            }

            public Gadget getValue() {
                return value;
            }

            public void setValue(Gadget value) {
                this.value = value;
            }

        }

    }

}

Gadget.java

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlType;

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "gadget", propOrder = {
    "make"
})
public class Gadget {

    protected String make;

    public String getMake() {
        return make;
    }

    public void setMake(String value) {
        this.make = value;
    }
}

As can be seen above in the generated Person class, the Map was not reinstated. Instead a new Person.Gadgets class was created containing a list of the new Person.Gadgets.Entry class. To be fair, the xjc tool treated the schema at face value.  How was it to know that this structure should be modelled by a Map.  Ideally, it would be nice if only a few instructions in a bindings file (supplied to xjc tool along with the schema file) were sufficient to auto-generate all required JAXB-annotated classes with full Map support.  Unfortunately this is not the case. Instead a bindings file must be created that targets appropriate elements in the schema and overrides their baseType with a fully-qualified custom Map subclass.  Custom java files must be hand-created for the Map subclass, and also an XmlAdapter subclass which contains the logic to unmarshal/marshal to/from the Map subclass.


What proceeds is a fully-worked example based on the Person / Gadget scenario above that restores Map support for set/getGadgets() methods of the Person class. It also has a few extra features thrown in including subclassing of Gadget. Credit for this technique must go to Aaron Anderson @ adventuresintechology.blogspot.com.

First, the xml schema.

schema.xsd

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="
http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"
targetNamespace="
http://todayguesswhat.blogspot.com/test" xmlns:test="http://todayguesswhat.blogspot.com/test">

<xs:element name="Person">
  <xs:complexType>
   <xs:sequence>
    <xs:element name="name" type="xs:string"/>
    <xs:element name="gadgets" type="test:GadgetMapModeller" minOccurs="0"/>
   </xs:sequence>
  </xs:complexType>
</xs:element>

<xs:complexType name="GadgetMapModeller">
  <xs:sequence>
    <xs:element name="entry" minOccurs="0" maxOccurs="unbounded">
     <xs:complexType>
      <xs:sequence>
        <xs:element name="key" type="xs:string"/>
        <xs:element name="value" type="test:Gadget"/>
      </xs:sequence>
     </xs:complexType>
   </xs:element>
  </xs:sequence>
</xs:complexType>

<xs:complexType name="Gadget">
  <xs:sequence>
   <xs:element name="make" type="xs:string"/>
   <xs:element name="model" type="xs:string"/>
   <xs:element name="year" type="xs:int"/>
  </xs:sequence>
</xs:complexType>

<xs:complexType name="Computer">
  <xs:complexContent>
   <xs:extension base="test:Gadget">
    <xs:sequence>
     <xs:element name="speed" type="xs:int"/>
     <xs:element name="cpu" type="xs:string"/>
    </xs:sequence>
   </xs:extension>
  </xs:complexContent>
</xs:complexType>

</xs:schema>

The important bindings file that overrides the generated type of the gadgets element to the new Map subclass – GadgetMap<String, Gadget>

bindings.xml

<jaxb:bindings xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.1">

<jaxb:bindings schemaLocation="schema.xsd">
 
  <jaxb:bindings node="//xs:element[@name='Person']//xs:element[@name='gadgets']">
   <jaxb:property>
    <jaxb:baseType name="com.blogspot.todayguesswhat.test.model.GadgetMap&lt;String,Gadget&gt;" />
   </jaxb:property>
  </jaxb:bindings>

</jaxb:bindings>

</jaxb:bindings>

The new Map subclass with XmlJavaTypeAdapter JAXB annotation defining the name of the adapter for unmarshalling/marshalling  to/from this type.

GadgetMap.java

package com.blogspot.todayguesswhat.test.model;

import java.util.HashMap;

import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;

@XmlJavaTypeAdapter(GadgetMapAdapter.class)
public class GadgetMap<String,Gadget> extends HashMap<String,Gadget>
{
}

The XmlAdapter that does the important conversion from one type to another:

GadgetMapAdapter.java

package com.blogspot.todayguesswhat.test.model;

import java.util.Map;

import javax.xml.bind.annotation.adapters.XmlAdapter;

public class GadgetMapAdapter extends XmlAdapter<GadgetMapModeller, GadgetMap<String,Gadget>>
{
  @Override
  public GadgetMap<String,Gadget> unmarshal(GadgetMapModeller modeller)
  {
    GadgetMap<String,Gadget> map = new GadgetMap<String,Gadget>();
    for (GadgetMapModeller.Entry e : modeller.getEntry())
    {
      map.put(e.getKey(), e.getValue());
    }
    return map;
  }

  @Override
  public GadgetMapModeller marshal(GadgetMap<String,Gadget> map)
  {
    GadgetMapModeller modeller = new GadgetMapModeller();
    for (Map.Entry<String,Gadget> entry : map.entrySet())
    {
      GadgetMapModeller.Entry e = new GadgetMapModeller.Entry();
      e.setKey(entry.getKey());
      e.setValue(entry.getValue());
      modeller.getEntry().add(e);
    }
    return modeller;
  }
}

A few test classes.

MarshalTest.java

package com.blogspot.todayguesswhat.test.model;
 
import java.util.List;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.Marshaller;
 
public class MarshalTest
{
  public static void main(String[] args) throws Exception
  {
    JAXBContext jc = JAXBContext.newInstance(Person.class);
    ObjectFactory factory = new ObjectFactory();

    Person person = factory.createPerson();
    person.setName("Matt Shannon");

    GadgetMap<String, Gadget> map = new GadgetMap<String, Gadget>();

    Gadget gadget1 = new Gadget();
    gadget1.setMake("Apple");
    gadget1.setModel("iPod");
    gadget1.setYear(2002);

    Computer gadget2 = new Computer();
    gadget2.setMake("Lenovo");
    gadget2.setModel("Thinkpad X230");
    gadget2.setYear(2012);
    gadget2.setCpu("Intel i5-3320M");
    gadget2.setSpeed(2600);

    map.put("my ipad", gadget1);
    map.put("my laptop", gadget2);

    person.setGadgets(map);

    Marshaller marshaller = jc.createMarshaller();
    marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
    marshaller.marshal(person, System.out);
  }
}

UnmarshalTest.java

package com.blogspot.todayguesswhat.test.model;
 
import java.io.File;

import java.util.Map;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.Unmarshaller;
 
public class UnmarshalTest
{
  public static void main(String[] args) throws Exception
  {
    JAXBContext jc = JAXBContext.newInstance(Person.class);

    Unmarshaller u = jc.createUnmarshaller();

    Person person = (Person) u.unmarshal(new File(args[0]));
    logPerson(person);
  }

  public static void logPerson(Person p)
  {
    if (p != null)
    {
      System.out.println("Person [name=" + p.getName() + "]");
      GadgetMap<String, Gadget> map = p.getGadgets();
      if (map != null)
      {
        for (Map.Entry<String,Gadget> entry : map.entrySet())
        {
          Gadget g = entry.getValue();
          System.out.println("   " + entry.getKey() + " : " + gadgetToString(g));
        }
      }
    }
  }

  public static String gadgetToString(Gadget g)
  {
    String result = null;
    if (g != null)
    {
      result = "[make="+g.getMake()+"][model="+g.getModel()+"][year="+g.getYear()+"]";
      if (g instanceof Computer)
      {
        Computer c = (Computer) g;
        result = result + "[cpu="+c.getCpu()+"][speed="+c.getSpeed()+"]";
      }
    }
    return result;
  }
}

An XML file leveraged by the unmarshalling test

person.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Person xmlns="
http://todayguesswhat.blogspot.com/test">
    <name>Louise</name>
    <gadgets>
        <entry>
            <key>work laptop</key>
            <value xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:type="Computer">
                <make>Lenovo</make>
                <model>Thinkpad X120e</model>
                <year>2011</year>
                <speed>1600</speed>
                <cpu>AMD E-350</cpu>
            </value>
        </entry>
        <entry>
            <key>my phone</key>
            <value>
                <make>Apple</make>
                <model>iPhone 3g</model>
                <year>2008</year>
            </value>
        </entry>
    </gadgets>
</Person>

And some windows bat files to invoke the various commands

clean.bat

@ECHO off
SET PROJECT_GENSRC=%~dp0\gensrc
IF EXIST "%PROJECT_GENSRC%" RMDIR /s /q "%PROJECT_GENSRC%"
SET PROJECT_JAVADOC=%~dp0\javadoc
IF EXIST "%PROJECT_JAVADOC%" RMDIR /s /q "%PROJECT_JAVADOC%"
SET PROJECT_CLASSES=%~dp0\classes
IF EXIST "%PROJECT_CLASSES%" RMDIR /s /q "%PROJECT_CLASSES%"
pause.

jaxb.bat

@ECHO off
SET JAVA_HOME=C:\Oracle\Middleware\jdk160_24
SET JAXBCMD=%JAVA_HOME%\bin\xjc.exe
SET PROJECT_GENSRC=%~dp0\gensrc
IF EXIST "%PROJECT_GENSRC%" RMDIR /s /q "%PROJECT_GENSRC%"
MKDIR "%PROJECT_GENSRC%"
"%JAXBCMD%" -no-header -d "%PROJECT_GENSRC%" -p com.blogspot.todayguesswhat.test.model -xmlschema "%~dp0\schema.xsd" -b "%~dp0\bindings.xml"
pause.

image

image

image

compile.bat

@ECHO off
SET JAVA_HOME=C:\Oracle\Middleware\jdk160_24
SET JAVACMPCMD=%JAVA_HOME%\bin\javac.exe
SET SRC_BASE_PKG=src\com\blogspot\todayguesswhat\test\model
SET GENSRC_BASE_PKG=gensrc\com\blogspot\todayguesswhat\test\model
SET PROJECT_CLASSES=%~dp0\classes
IF EXIST "%PROJECT_CLASSES%" RMDIR /s /q "%PROJECT_CLASSES%"
MKDIR "%PROJECT_CLASSES%"
"%JAVACMPCMD%" -d "%PROJECT_CLASSES%" %SRC_BASE_PKG%\*.java %GENSRC_BASE_PKG%\*.java
pause.

image

javadoc.bat

@ECHO off
SET JAVA_HOME=C:\Oracle\Middleware\jdk160_24
SET JAVADOCCMD=%JAVA_HOME%\bin\javadoc.exe
SET PROJECT_SRC=%~dp0\src
SET PROJECT_GENSRC=%~dp0\gensrc
SET PROJECT_JAVADOC=%~dp0\javadoc
IF EXIST "%PROJECT_JAVADOC%" RMDIR /s /q "%PROJECT_JAVADOC%"
MKDIR "%PROJECT_JAVADOC%"
"%JAVADOCCMD%" -sourcepath "%PROJECT_GENSRC%;%PROJECT_SRC%" -d "%PROJECT_JAVADOC%" -subpackages com.blogspot.todayguesswhat.test.model -protected
pause.

image

image

 

image

 

image

 

image

run-marshal-test.bat

@ECHO off
MODE 120,50
SET JAVA_HOME=C:\Oracle\Middleware\jdk160_24
SET JAVACMD=%JAVA_HOME%\bin\java.exe
SET PROJECT_CLASSES=%~dp0\classes
"%JAVACMD%" -classpath "%PROJECT_CLASSES%" com.blogspot.todayguesswhat.test.model.MarshalTest
pause.

image

image

run-unmarshal-test.bat

@ECHO off
MODE 120,50
SET JAVA_HOME=C:\Oracle\Middleware\jdk160_24
SET JAVACMD=%JAVA_HOME%\bin\java.exe
SET PROJECT_CLASSES=%~dp0\classes
"%JAVACMD%" -classpath "%PROJECT_CLASSES%" com.blogspot.todayguesswhat.test.model.UnmarshalTest "%~dp0\person.xml"
pause.

image

image

Click here to download a zip of this JAXB sample.

Friday, August 31, 2012

Windows 7 Replacement for UserAccounts.CommonDialog in VBScript

After 6 years of mostly trouble-free development/engineering, I finally retired my Windows XP-based Dell Latitude D620 from active work duty.  I had been holding out for a business laptop with USB 3.0 to become available on the internal procurement site, and were finally able to obtain a Lenovo X230 with an Ivy Bridge i5-3320M processor.  (Un)fortunately this new machine is running Windows 7 x64, and for that matter a bastardized version full of all the resource hungry corporate mandated bloat.  This is the first time I have seen/used Windows 7 (having managed to also completely avoid Vista). The first thing I find myself doing is trying to make Windows 7 feel and behave like Windows XP again.  After installing classic shell to get the old start button functionality back (http://classicshell.sourceforge.net/) and turning off all of the visual effects (aka – ‘Adjust for best performance’ setting), my desktop is starting to resemble and feel like the ugly but reliable XP again.  Now I’m slowly working through my kit bag of productivity scripts that I created for XP and trying to get these to function in Windows 7.

One of the more frequent scripts I leverage is a simple VBScript for upload and download of a file by invoking the command-line FTP utility shipped with Windows.  See the following article I wrote for the full original XP supported source code: http://todayguesswhat.blogspot.com.au/2010/06/vbscript-ftp-upload-sample-leverages.html

I found out that the UserAccounts.CommonDialog class/control is not available in Windows 7. I leveraged this control to allow the user to select a file for upload.  Original VBScript code shown below:

Function ChooseFile(initialDir)
  Set cd = CreateObject("UserAccounts.CommonDialog")

  cd.InitialDir = initialDir
  cd.Filter = "ZIP files|*.zip|Text Documents|*.txt|Shell Scripts|*.*sh|All Files|*.*"
  ' filter index 4 would show all files by default
  ' filter index 1 would should zip files by default
  cd.FilterIndex = 1
  If cd.ShowOpen = True Then
    ChooseFile = cd.FileName
  Else
    ChooseFile = ""
  End If
  Set cd = Nothing
End Function

For Windows 7, I’ve kludged together code to replace the above method using techniques/articles/suggestions borrowed from multiple parties.  If there is a cleaner mechanism to navigate and select a file using VBScript in Windows 7, please let me know :)

The code I developed/hacked-together creates a temporary powershell script that spawns System.Windows.Forms OpenFileDialog, and then writes the chosen file out to a temporary output text file. The VBScript then reads in the value from the output text file and returns that in the function.  Code is as follows:

Function ChooseFile (ByVal initialDir)

  Set shell = CreateObject("WScript.Shell")

  Set fso = CreateObject("Scripting.FileSystemObject")

  tempDir = shell.ExpandEnvironmentStrings("%TEMP%")

  tempFile = tempDir & "\" & fso.GetTempName

  ' temporary powershell script file to be invoked
  powershellFile = tempFile & ".ps1"

  ' temporary file to store standard output from command
  powershellOutputFile = tempFile & ".txt"

  'input script
  psScript = psScript & "[System.Reflection.Assembly]::LoadWithPartialName(""System.windows.forms"") | Out-Null" & vbCRLF
  psScript = psScript & "$dlg = New-Object System.Windows.Forms.OpenFileDialog" & vbCRLF
  psScript = psScript & "$dlg.initialDirectory = """ &initialDir & """" & vbCRLF
  psScript = psScript & "$dlg.filter = ""ZIP files|*.zip|Text Documents|*.txt|Shell Scripts|*.*sh|All Files|*.*""" & vbCRLF
  ' filter index 4 would show all files by default
  ' filter index 1 would should zip files by default
  psScript = psScript & "$dlg.FilterIndex = 4" & vbCRLF
  psScript = psScript & "$dlg.Title = ""Select a file to upload""" & vbCRLF
  psScript = psScript & "$dlg.ShowHelp = $True" & vbCRLF
  psScript = psScript & "$dlg.ShowDialog() | Out-Null" & vbCRLF
  psScript = psScript & "Set-Content """ &powershellOutputFile & """ $dlg.FileName" & vbCRLF
  MsgBox psScript
 
  Set textFile = fso.CreateTextFile(powershellFile, True)
  textFile.WriteLine(psScript)
  textFile.Close
  Set textFile = Nothing

  ' objShell.Run (strCommand, [intWindowStyle], [bWaitOnReturn])
  ' 0 Hide the window and activate another window.
  ' bWaitOnReturn set to TRUE - indicating script should wait for the program
  ' to finish executing before continuing to the next statement

  Dim appCmd
  appCmd = "powershell -ExecutionPolicy unrestricted &'" & powershellFile & "'"
  MsgBox appCmd
  shell.Run appCmd, 0, TRUE

  ' open file for reading, do not create if missing, using system default format
  Set textFile = fso.OpenTextFile(powershellOutputFile, 1, 0, -2)
  ChooseFile = textFile.ReadLine
  textFile.Close
  Set textFile = Nothing
  fso.DeleteFile(powershellFile)
  fso.DeleteFile(powershellOutputFile)

End Function

UPDATE – May 2013

Some commenters have suggested leveraging BrowseForFolder.  At least for me, this produces strange behaviour on Windows 7 and may return -2147467259 (80004005) error code for certain file types (for example txt files) - but not others (e.g. zip).  I would NOT recommend it.

Here is a a new and improved version which is must faster than above and should be backward compatible with XP:-

Set shell = CreateObject( "WScript.Shell" )
defaultLocalDir = shell.ExpandEnvironmentStrings("%USERPROFILE%") & "\Desktop"
Set shell = Nothing

file = ChooseFile(defaultLocalDir)
MsgBox file

Function ChooseFile (ByVal initialDir)
    Set objWMIService = GetObject("winmgmts:\\.\root\cimv2")

    Set colItems = objWMIService.ExecQuery("Select * from Win32_OperatingSystem")
    Dim winVersion

    ' This collection should contain just the one item
    For Each objItem in colItems
        'Caption e.g. Microsoft Windows 7 Professional
        'Name e.g. Microsoft Windows 7 Professional |C:\windows|...
        'OSType e.g. 18 / OSArchitecture e.g 64-bit
        'Version e.g 6.1.7601 / BuildNumber e.g 7601
        winVersion = CInt(Left(objItem.version, 1))
    Next
    Set objWMIService = Nothing
    Set colItems = Nothing

    If (winVersion <= 5) Then
        ' Then we are running XP and can use the original mechanism
        Set cd = CreateObject("UserAccounts.CommonDialog")
        cd.InitialDir = initialDir
        cd.Filter = "ZIP files|*.zip|Text Documents|*.txt|Shell Scripts|*.*sh|All Files|*.*"
        ' filter index 4 would show all files by default
        ' filter index 1 would show zip files by default
        cd.FilterIndex = 1
        If cd.ShowOpen = True Then
            ChooseFile = cd.FileName
        Else
            ChooseFile = ""
        End If
        Set cd = Nothing    

    Else
        ' We are running Windows 7 or later
        Set shell = CreateObject( "WScript.Shell" )
        Set ex = shell.Exec( "mshta.exe ""about: <input type=file id=X><script>X.click();new ActiveXObject('Scripting.FileSystemObject').GetStandardStream(1).WriteLine(X.value);close();resizeTo(0,0);</script>""" )
        ChooseFile = Replace( ex.StdOut.ReadAll, vbCRLF, "" )

        Set ex = Nothing
        Set shell = Nothing
    End If
End Function   

Tuesday, July 17, 2012

File Download Java Servlet example - 2GB overflow workaround

I discovered a few days back an issue with our product on HTTP downloads > 2GB.  It appears to be a simple overflow on the HttpServletResponse.setContentLength method. You can probably excuse the API designers circa 1997 assuming a 32-bit signed Integer with max value 231-1 (2147483647 bytes) would be sufficient.  The Gigabit ethernet standard did not come for another year (1998)!

Here is the exception seen when you provide a long value greater than 2147483647 bytes to the setContentLength(int) method:

java.net.ProtocolException: Exceeded stated content-length of: '-XXXX' bytes
        at weblogic.servlet.internal.ServletOutputStreamImpl.checkCL(ServletOutputStreamImpl.java:200)

Below is a sample download servlet with workaround for the 2gb limitation.  It has been tested on Firefox 3.6 against WebLogic Server 10.3.6 with a 2.2GB download and worked perfectly.


import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;


public class DownloadServlet
  extends HttpServlet
{
  @SuppressWarnings("compatibility:1533750721037291976")
  private static final long serialVersionUID = 1L;

  protected void doGet(HttpServletRequest request,
    HttpServletResponse response)
    throws ServletException, IOException
  {
    doPost(request, response);
  }

  protected void doPost(HttpServletRequest request,
    HttpServletResponse response)
    throws ServletException, IOException
  {
    // if no file parameter specified, download hosts file
    String file = request.getParameter("file");
    file = (file == null || file.length() == 0) ? "/etc/hosts" : file;

    File fileObj = new File(file);
    if ((!fileObj.exists()) || (!fileObj.isFile()) || (!fileObj.canRead()))
    {
      throw new IOException("'file' '" + file + "' cannot be read.");
    }

    ServletContext context = getServletConfig().getServletContext();

    String mimetype = context.getMimeType(file);
    response.setContentType(mimetype == null ? "application/octet-stream" :
        mimetype);

    long length = fileObj.length();
    if (length <= Integer.MAX_VALUE)
    {
      response.setContentLength((int)length);
    }
    else
    {
      response.addHeader("Content-Length", Long.toString(length));
    }

    response.setHeader("Content-Disposition",
        "attachment; filename=\"" + fileObj.getName() + "\"");

    ServletOutputStream out = response.getOutputStream();
    InputStream in = null;
    byte[] buffer = new byte[32768];
    try
    {
      in = new FileInputStream(fileObj);

      int bytesRead;
      while ((bytesRead = in.read(buffer)) >= 0)
      {
        out.write(buffer, 0, bytesRead);
      }
    }
    finally
    {
      if (in != null)
      {
        in.close();
      }
    }
  }
}

Saturday, July 7, 2012

WebLogic HTTPS One-Way SSL Tutorial

One-way SSL is the mode which most "storefronts" run on the internet so as to be able to accept credit card details and the like without the customer’s details being sent effectively in the clear from a packet-capture perspective.  In this mode, the server must present a valid public certificate to the client, but the client is not required to present a certificate to the server.

Two-way SSL is far less common on the internet and is often leveraged for client certificate-based authentication.

In terms of configuring WebLogic Server to support one-way SSL, it is just a matter of setting up an identity keystore containing a valid private key and associated public certificate signed by a certificate authority.  If the client has no knowledge/trust of the certificate authority that signed the server’s public certificate, the client must manually import/accept the CA’s public certificate.

WebLogic in fact ships with two demo keystores for testing purposes containing X509 artifacts, namely DemoIdentity.jks and DemoTrust.jks (both found in $WL_HOME/server/lib). The latter is intended for Two-Way SSL purposes, though is also frequently supplied to java clients that may want to connect to the WebLogic instance due to it containing the CN=CertGenCAB… WebLogic demo CA public certificate that issued the certificate in the DemoIdentity keystore.

keytool -list -v -keystore $WL_HOME/server/lib/DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase

Keystore type: JKS
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: demoidentity
Creation date: Jun 30, 2012
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=xxxxxxxxxx, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 75e1d5b02ee5b71fd3b1431f483f49f1
Valid from: Fri Jun 29 06:45:36 PDT 2012 until: Wed Jun 30 06:45:36 PDT 2027
Certificate fingerprints:
         MD5:  BB:49:16:85:4E:84:66:09:8A:68:83:A4:A7:06:AB:80
         SHA1: 1F:67:9C:78:C7:64:B4:0B:2F:A7:AA:B2:69:70:9D:47:47:D5:A0:A8
         Signature algorithm name: MD5withRSA
         Version: 1

...

keytool -list -keystore $WL_HOME/server/lib/DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 4 entries

certgenca, Mar 22, 2002, trustedCertEntry,
Certificate fingerprint (MD5): 8E:AB:55:50:A4:BC:06:F3:FE:C6:A9:72:1F:4F:D3:89
wlsdemocanew2, Jan 24, 2003, trustedCertEntry,
Certificate fingerprint (MD5): 5B:10:D5:3C:C8:53:ED:75:43:58:BF:D5:E5:96:1A:CF
wlsdemocanew1, Jan 24, 2003, trustedCertEntry,
Certificate fingerprint (MD5): A1:17:A1:73:9B:70:21:B9:72:85:4D:83:01:69:C8:37
wlscertgencab, Jan 24, 2003, trustedCertEntry,
Certificate fingerprint (MD5): A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE

The WebLogic demo CA public certificate that issued the certificate in the DemoIdentity keystore …

keytool -printcert -file $WL_HOME/server/lib/CertGenCA.der

Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 08:54:45 PDT 2002 until: Tue Oct 25 08:54:45 PDT 2022
Certificate fingerprints:
         MD5:  A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
         SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
         Signature algorithm name: MD5withRSA
         Version: 3

...

In order to leverage the above keystores, it is just a matter of connecting to the Administration Console, then expand Environment and select Servers.
Choose the server for which you want to configure the identity and trust keystores, and select Configuration > Keystores. 
You would then fill in the relevant fields (keystore fully qualified path, type (JKS), and keystore access password). e.g.

Identity store location: /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/DemoIdentity.jks
Identity store type: JKS
Identity store password: DemoIdentityKeyStorePassPhrase

Trust store location: /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/DemoTrust.jks
Trust store type: JKS
Trust store password: DemoTrustKeyStorePassPhrase

Next, you need to enable the SSL Listen Port for the server:

Home >Summary of Servers >XXX > General
SSL Listen Port: Enabled (Check)
SSL Listen Port: XXXX

Finally, you need to tell WebLogic the alias and password in order to access the private key from the Identity Store:

Home >Summary of Servers >XXX > SSL
Identity and Trust Locations: Keystores
Private Key Alias: demoidentity
Private Key Passphrase: DemoIdentityPassPhrase

Note - you are likely to find that the Demo Identity and Demo Trust keystores are preconfigured out of the box!


For production setups, you should be generating your own key of required strength, and get the associated public certificate signed by a “real” certificate authority.  A “real” certificate authority could be hosted locally given a private client base, or it could be one of the main internet players (verisign and co) if the instance was to be accessed by the general public.

The steps below are a hybrid approach that leverage production techniques, but using the demo WebLogic certificate authority and openssl to perform the signing.

First, lets create our identity keystore along with a new keypair ...

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1
KEY_ALIAS=identity
KEY_PASSWORD=welcome1

keytool -genkeypair -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -keyalg RSA -keysize 1024 -validity 365 -keypass "$KEY_PASSWORD" -alias "$KEY_ALIAS" -dname "CN=`hostname -f`"

Generating 1,024 bit RSA key pair and self-signed certificate (SHA1withRSA) with a validity of 365 days
        for: CN=xxxxxxxxxx.xx.xxxxxx.com
[Storing /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/identity.jks]

The above certificate is currently self-signed (such that, issuer = subject , private key signed its associated public certificate)

For a production server we want to get our public certificate signed by a valid certificate authority (CA).
As we are just testing, we will use the WebLogic demo CA instead.

We first need to get a certificate signing request ready ...

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1
KEY_ALIAS=identity
SIGNING_REQ_FILE="$WL_HOME/server/lib/identity_cert_signing_request.pem"

keytool -certreq -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -alias "$KEY_ALIAS" -file "$SIGNING_REQ_FILE"

Certification request stored in file </u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/identity_cert_signing_request.pem>
Submit this to your CA

cat "$SIGNING_REQ_FILE"

-----BEGIN NEW CERTIFICATE REQUEST-----
MIIBYjCBzAIBADAjMSEwHwYDVQQDExhhZGMyMTAwODY2LnVzLm9yYWNsZS5jb20wgZ8wDQYJKoZI
hvcNAQEBBQADgY0AMIGJAoGBAIOiOavPqXv3JDjYCsHyTmfprmSACDpF5dpozI2i2U8YVJaDYHtK
PYPdKu+KG7hpmxHlI6nHF2aqTid0sYqZjQDeYj16dHyLR6kh0SnAj3fA+I7ev9SFffyQ4cdpHwfV
1jfEkyPwNSQxz5X4UiRyd3bwCXI4rEPSjRUVpjPC8Q9LAgMBAAGgADANBgkqhkiG9w0BAQUFAAOB
gQAcieNA+waTBjDHRIrxDw+hSwjiB5OYaA6+smiEO8WYwNQkJifsr9OEc7SKeHmM2pAKGDUl9bKO
GHkv9FEKShNbAFXIRl0mItTwhIAkmqxKxZBv7HKmdM+lah5jpOQWmH1W4NRPbbGs/bdgYXHwEeYs
yCuaZsp90G73MxlPMgebFw==
-----END NEW CERTIFICATE REQUEST-----

Now we need to get the certificate signed.  As mentioned previously, we will use the WebLogic demo CA along with the openssl tool to generate the signed certificate.

We first need to convert the WebLogic demo CA private key from DER to PEM format in order to use the openssl tool.
The private key as shown below is encrypted ....

openssl asn1parse -inform DER -in CertGenCAKey.der

    0:d=0  hl=4 l= 384 cons: SEQUENCE         
    4:d=1  hl=2 l=  26 cons: SEQUENCE         
    6:d=2  hl=2 l=   9 prim: OBJECT            :pbeWithMD5AndDES-CBC
   17:d=2  hl=2 l=  13 cons: SEQUENCE         
   19:d=3  hl=2 l=   8 prim: OCTET STRING      [HEX DUMP]:0011223344556677
   29:d=3  hl=2 l=   1 prim: INTEGER           :05
   32:d=1  hl=4 l= 352 prim: OCTET STRING      [HEX

DUMP]:2A89DCA6697D9F...

16

The private key is protected by password based encryption - employing DES secret-key encryption in cipher-block chaining mode, where the secret key is derived from a password with the MD5 message-digest algorithm.

The password used for encryption of this key is "password"

Let's convert the key from DER format to PEM format supplying the encryption password …

CA_KEY_DER="$WL_HOME/server/lib/CertGenCAKey.der"
CA_KEY_PEM="$WL_HOME/server/lib/CertGenCAKey.pem"

openssl pkcs8 -inform DER -in "$CA_KEY_DER" -passin pass:password -outform PEM -out "$CA_KEY_PEM"

cat "$CA_KEY_PEM"

-----BEGIN RSA PRIVATE KEY-----
MIIBOgIBAAJBAK+kgUpzTgIJnC8rktjUQsYUy3X9JJP8OZokh0Pm2KH3bOK49CNI
/EmCFs1v2Gy2sOfWVP2HPY7uTEI1Yli3slsCAwEAAQJAM8oRxV4SVk93GgGDHyQX
PEjNZVDrBCeO5IB40fCzV64b33n7nCnrnemS8eJwvOs6+4dN4CpogCWKrMKkdRdS
gQIhAN56PT3B7/jS2D6HVgXATF+ThJ2HkbL7CDMPVkJE00sLAiEAyhussDRxqESf
w+EibP34Effs6LJP73BeI43yIgq6R/ECIF75HfWkOdYl5AxlZ9KSscfHSMCa5Bbc
1TY+4NrvWKfbAiEAmnpEp/mt1dt98g4mbgBAmBtT7Rg9mNneWnaoV/SzJUECICjQ
Lsm4mCSPorIOlGsk2TQPH0MzJsk4TziNnsB08jBO
-----END RSA PRIVATE KEY-----


We also need the CA’s public certificate in PEM format …

CA_CERT_DER="$WL_HOME/server/lib/CertGenCA.der"
CA_CERT_PEM="$WL_HOME/server/lib/CertGenCA.pem"

openssl x509 -inform DER -in "$CA_CERT_DER" -outform PEM -out "$CA_CERT_PEM"

cat "$CA_CERT_PEM"

-----BEGIN CERTIFICATE-----
MIICGDCCAcKgAwIBAgIQI0tVWdH6Dz/1yCvf7QMqhzANBgkqhkiG9w0BAQQFADB5
MQswCQYDVQQGEwJVUzEQMA4GA1UECBMHTXlTdGF0ZTEPMA0GA1UEBxMGTXlUb3du
MRcwFQYDVQQKEw5NeU9yZ2FuaXphdGlvbjEZMBcGA1UECxMQRk9SIFRFU1RJTkcg
T05MWTETMBEGA1UEAxMKQ2VydEdlbkNBQjAeFw0wMjEwMjQxNTU0NDVaFw0yMjEw
MjUxNTU0NDVaMHkxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdNeVN0YXRlMQ8wDQYD
VQQHEwZNeVRvd24xFzAVBgNVBAoTDk15T3JnYW5pemF0aW9uMRkwFwYDVQQLExBG
T1IgVEVTVElORyBPTkxZMRMwEQYDVQQDEwpDZXJ0R2VuQ0FCMFwwDQYJKoZIhvcN
AQEBBQADSwAwSAJBAK+kgUpzTgIJnC8rktjUQsYUy3X9JJP8OZokh0Pm2KH3bOK4
9CNI/EmCFs1v2Gy2sOfWVP2HPY7uTEI1Yli3slsCAwEAAaMmMCQwDgYDVR0PAQH/
BAQDAgIEMBIGA1UdEwEB/wQIMAYBAf8CAQEwDQYJKoZIhvcNAQEEBQADQQBCOCsQ
9QWvGvAikjBBOjrTFqZBNJYJsv5+mVZ90pVwmzEDihw8dbI3ubMYz5saXkKSy3rc
SvJK9nZJkHvb88hZ
-----END CERTIFICATE-----

Once we have our CA key and certificate in PEM format, lets do the certificate signing ...

CA_KEY_PEM="$WL_HOME/server/lib/CertGenCAKey.pem"
CA_CERT_PEM="$WL_HOME/server/lib/CertGenCA.pem"
CA_SERIAL_FILE="$WL_HOME/server/lib/CA.srl"
SIGNING_REQ_FILE="$WL_HOME/server/lib/identity_cert_signing_request.pem"
SIGNED_CERT="$WL_HOME/server/lib/identity_cert_signed.pem"

openssl x509 -req -days 365 -in "$SIGNING_REQ_FILE" -inform PEM -CA "$CA_CERT_PEM" -CAkey "$CA_KEY_PEM" -out "$SIGNED_CERT" -outform PEM -CAcreateserial -CAserial "$CA_SERIAL_FILE"

Signature ok
subject=/CN=xxxxxxxxxx.xx.xxxxxx.com
Getting CA Private Key

keytool -printcert -file "$SIGNED_CERT"

Owner: CN=xxxxxxxxxx.xx.xxxxxx.com
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: d80dbb0b65dcc3b8
Valid from: Fri Jul 06 20:41:08 PDT 2012 until: Sat Jul 06 20:41:08 PDT 2013
Certificate fingerprints:
         MD5:  2E:2A:00:AD:A9:38:E9:FA:B7:C6:A8:85:33:74:72:B3
         SHA1: 27:0D:9C:A9:39:37:92:E3:68:A2:61:E8:65:D3:3F:B9:D1:86:A8:E2
         Signature algorithm name: SHA1withRSA
         Version: 1

As can be seen above, the issuer and owner are now different.  The certificate is no longer self-signed, but rather signed by our CA.
Now we are ready to imported the signed public certificate back in to our identity keystore ...

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1
KEY_ALIAS=identity
SIGNED_CERT="$WL_HOME/server/lib/identity_cert_signed.pem"

keytool -import -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -alias "$KEY_ALIAS" -file "$SIGNED_CERT"

You should see the following error when you perform the above command:

keytool error: java.lang.Exception: Failed to establish chain from reply

keytool prevents us from importing a certificate should it not be able to verify the full signing chain. As we leveraged the WebLogic demo CA, we need to import the CA's public certificate in to our keystore as a trusted certificate authority prior to importing our signed certificate.

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1
CA_ALIAS=wlsdemoca
CA_CERT_PEM="$WL_HOME/server/lib/CertGenCA.pem"

keytool -import -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -trustcacerts -alias "$CA_ALIAS" -file "$CA_CERT_PEM"

Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 08:54:45 PDT 2002 until: Tue Oct 25 08:54:45 PDT 2022
Certificate fingerprints:
         MD5:  A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
         SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
         Signature algorithm name: MD5withRSA
         Version: 3

...

Trust this certificate? [no]:  y
Certificate was added to keystore
[Storing /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/identity.jks]

Add the "-noprompt" option to prevent the “Trust this certificate” prompt.

Now lets retry our request to import the signed certificate ...

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1
KEY_ALIAS=identity
SIGNED_CERT="$WL_HOME/server/lib/identity_cert_signed.pem"

keytool -import -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -alias "$KEY_ALIAS" -file "$SIGNED_CERT" -noprompt

Certificate reply was installed in keystore
[Storing /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/identity.jks]

Let's verify the keystore contents ...

KEYSTORE_FILE=$WL_HOME/server/lib/identity.jks
KEYSTORE_PASSWORD=welcome1

keytool -list -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD"

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

Alias name: identity
Creation date: Jul 6, 2012
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=xxxxxxxxxx.xx.xxxxxx.com
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: d80dbb0b65dcc3b8
Valid from: Fri Jul 06 20:41:08 PDT 2012 until: Sat Jul 06 20:41:08 PDT 2013
Certificate fingerprints:
         MD5:  2E:2A:00:AD:A9:38:E9:FA:B7:C6:A8:85:33:74:72:B3
         SHA1: 27:0D:9C:A9:39:37:92:E3:68:A2:61:E8:65:D3:3F:B9:D1:86:A8:E2
         Signature algorithm name: SHA1withRSA
         Version: 1
Certificate[2]:
Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 08:54:45 PDT 2002 until: Tue Oct 25 08:54:45 PDT 2022
Certificate fingerprints:
         MD5:  A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
         SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
         Signature algorithm name: MD5withRSA
         Version: 3

...

*******************************************
*******************************************

Alias name: wlsdemoca
Creation date: Jul 6, 2012
Entry type: trustedCertEntry

Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 08:54:45 PDT 2002 until: Tue Oct 25 08:54:45 PDT 2022
Certificate fingerprints:
         MD5:  A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
         SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
         Signature algorithm name: MD5withRSA
         Version: 3

...

Next we go to the Administration Console and ensure for our Server that:

  1. “Custom Identity and Java Standard Trust” keystore configuration option is chosen
  2. The custom identity keystore path, type, and password is correctly set
  3. Java standard trust store password is set (“changeit”)
  4. The private key alias and password is set
  5. The SSL listen port is set

Identity store location: /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/identity.jks
Identity store type: JKS
Identity store password: welcome1

Identity and Trust Locations: Keystores
Private Key Alias: identity
Private Key Passphrase: welcome1

As far as the trust keystore is concerned (and given we are leveraging one-way SSL), we just set the instance to utilize the standard trust keystore supplied with the JDK ($JAVA_HOME/jre/lib/security/cacerts   - password “changeit”).


Clients that will be connecting to our SSL protected server will likely have to import our CA’s public certificate in to their local trust keystore (cacerts etc) given that we are using a demo CA with a certificate chain unlikely to be known/resolvable.

Failure to do this from a Java client will likely lead to an exception such as:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
  PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

As an alternative to using the JDK’s standard trust keystore, another option is create a separate keystore with just the CA’s public certificate chain.  You then provide explicit SSL trust store system properties on the java command line:

KEYSTORE_FILE=$WL_HOME/server/lib/trust.jks
KEYSTORE_PASSWORD=welcome1
CA_ALIAS=wlsdemoca
CA_CERT_PEM="$WL_HOME/server/lib/CertGenCA.pem"

keytool -import -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD" -trustcacerts -alias "$CA_ALIAS" -file "$CA_CERT_PEM" -noprompt

Certificate was added to keystore
[Storing /u01/app/oracle/product/Middleware/wlserver_10.3/server/lib/trust.jks]

Let's verify the keystore contents ...

KEYSTORE_FILE=$WL_HOME/server/lib/trust.jks
KEYSTORE_PASSWORD=welcome1
keytool -list -v -keystore "$KEYSTORE_FILE" -storepass "$KEYSTORE_PASSWORD"

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: wlsdemoca
Creation date: Jul 9, 2012
Entry type: trustedCertEntry

Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 08:54:45 PDT 2002 until: Tue Oct 25 08:54:45 PDT 2022
Certificate fingerprints:
         MD5:  A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
         SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
         Signature algorithm name: MD5withRSA
         Version: 3

...

Invoke Java with explicit SSL trust store elements set ...

SSL_TRUST_STORE="-Djavax.net.ssl.trustStore=$WL_HOME/server/lib/trust.jks"
SSL_TRUST_PASSWORD="-Djavax.net.ssl.trustStorePassword=welcome1"

$JAVA_HOME/bin/java "${SSL_TRUST_STORE}" "${SSL_TRUST_PASSWORD}" -cp ... XXX

Wednesday, June 27, 2012

Decrypt / Dump contents of CWALLET.SSO (Oracle file based credential store)

When using a file-based credential store with Oracle, credentials ultimately get stored in a wallet file (cwallet.sso)

Very little if any info exists on how to dump the contents of the wallet.  At best, most people leverage the trusty orapki command to get an overview of what’s inside as far as the maps and keys, but actual password information is never divulged.

For example:

$MW_HOME/oracle_common/bin/orapki wallet display -wallet ~/cwallet.sso
Oracle PKI Tool : Version 11.1.1.6.0
Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Oracle Secret Store entries:
dip@#3#@cn=odisrv
ODSMMap@#3#@ODSMKey.Wallet
oracle.wsm.security@#3#@enc-csf-key
oracle.wsm.security@#3#@keystore-csf-key
oracle.wsm.security@#3#@sign-csf-key
Trusted Certificates:
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Secure Server Certification Authority,O=RSA Data Security\, Inc.,C=US
Subject:        CN=Entrust.net Secure Server Certification Authority,OU=(c) 1999 Entrust.net Limited,OU=www.entrust.net/CPS incorp. by ref. (limits liab.),O=Entrust.net,C=US
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=Entrust.net Secure Server Certification Authority,OU=(c) 2000 Entrust.net Limited,OU=www.entrust.net/SSL_CPS incorp. by ref. (limits liab.),O=Entrust.net
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=Entrust.net Certification Authority (2048),OU=(c) 1999 Entrust.net Limited,OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.),O=Entrust.net

Until now that is :)

Note - wallet and jps-config-dump file must reside in same location for this sample code to function!!  In the example below, choose either the domain wallet, or the bootstrap wallet.

PATH_TO_WALLET="$DOMAIN_HOME/config/fmwconfig/bootstrap/cwallet.sso"
PATH_TO_JPS="$DOMAIN_HOME/config/fmwconfig/bootstrap/jps-config-dump.xml"

or …

PATH_TO_WALLET="$DOMAIN_HOME/config/fmwconfig/cwallet.sso"
PATH_TO_JPS="$DOMAIN_HOME/config/fmwconfig/jps-config-dump.xml"

cat > "${PATH_TO_JPS}" <<EOF
<?xml version="1.0" encoding="UTF-8" standalone='yes'?>
<jpsConfig xmlns="http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd
jps-config-11_1.xsd" schema-major-version="11" schema-minor-version="1">
   <serviceProviders>
      <serviceProvider type="CREDENTIAL_STORE" name="credstoressp" class="oracle.security.jps.internal.credstore.ssp.SspCredentialStoreProvider">
         <description>Credential Store Service Provider</description>
      </serviceProvider>
   </serviceProviders>
   <serviceInstances>
      <serviceInstance provider="credstoressp" name="credstore">
         <property value="file:${PATH_TO_WALLET}" name="location"/>
      </serviceInstance>
   </serviceInstances>
  <jpsContexts default="test">
    <jpsContext name="test">
      <serviceInstanceRef ref="credstore"/>
    </jpsContext>
  </jpsContexts>
</jpsConfig>
EOF

cat > /tmp/DumpWallet.java <<EOF
import java.io.File;

import java.util.Hashtable;

import oracle.security.jps.JpsContext;
import oracle.security.jps.JpsContextFactory;

import oracle.security.jps.service.credstore.Credential;
import oracle.security.jps.service.credstore.CredentialFactory;
import oracle.security.jps.service.credstore.CredentialMap;
import oracle.security.jps.service.credstore.CredentialStore;
import oracle.security.jps.service.credstore.GenericCredential;
import oracle.security.jps.service.credstore.PasswordCredential;

public class DumpWallet
{
  private static final byte[] HEX = new byte[] {
    '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' }; 
 
  public static void main(String[] args)
  {
    try
    {
      System.setProperty("oracle.security.jps.config", (args.length > 0) ? args[0] : "jps-config.xml");
      // System.setProperty("java.security.debug", "all");
      JpsContextFactory ctxFactory  = JpsContextFactory.getContextFactory();
      JpsContext ctx = ctxFactory.getContext();
      CredentialStore store = ctx.getServiceInstance(CredentialStore.class);
      listAll(store);
    }
    catch (Exception e)
    {
      e.printStackTrace();
    }
  }
 
  private static void listAll(CredentialStore store) throws Exception
  {
    System.out.println("Dumping store contents ...");
    for (String map : store.getMapNames())
    {
      System.out.println("\n" + "### Map: " + map);
      CredentialMap credMap = store.getCredentialMap(map);
      if (credMap != null)
      {
        int i = 1;
        for (String key : credMap.keySet())
        {
          System.out.println(" " + i++ + ". + Key: " + key);
          
          Credential cred = credMap.getCredential(key);
          System.out.println("  class = " + cred.getClass().getName());
          System.out.println("  desc  = " + cred.getDescription());

          if (cred instanceof PasswordCredential)
          {
            PasswordCredential pc = (PasswordCredential)cred;
            System.out.println("  name  = " + pc.getName());
            System.out.println("  pass  = " + new String(pc.getPassword()));
            System.out.println("  expires   = " + pc.getExpiryTime());
          }
          else if (cred instanceof GenericCredential)
          {
            GenericCredential gc = (GenericCredential)cred;
            Object c = gc.getCredential();
            String type = (! c.getClass().isArray())
              ? c.getClass().getName()
              : ("Array of " + c.getClass().getComponentType().getName());

            System.out.println("  type  = " + type);
            if (c instanceof String)
            {
              System.out.println("  cred  = " + c);
            }
            else if ( c instanceof Hashtable)
            {
              Hashtable ht = (Hashtable)c;
              for (Object htkey : ht.keySet())
              {
                Object htVal = ht.get(htkey);
                if (htVal instanceof char[])
                {
                  System.out.println("  cred  = (" + htkey + ", " + new String((char[])htVal) + ")");
                }
                else
                {
                  System.out.println("  cred  = (" + htkey + ", " + htVal + ")");
                }
              }
            }
            else if (c instanceof javax.crypto.spec.SecretKeySpec)
            {
              javax.crypto.spec.SecretKeySpec secret = (javax.crypto.spec.SecretKeySpec) c;
              System.out.println("  algorith  = " + secret.getAlgorithm());
              System.out.println("  format  = " + secret.getFormat());
              System.out.println("  key material as hex = " + bytesAsHex(secret.getEncoded()));
            }
            else if (c instanceof byte[])
            {
              System.out.println("  byte array as hex = " + bytesAsHex((byte[])c));
            }
            System.out.println("  expires   = " + gc.getExpiryTime());
          }
          else
          {
            System.out.println("  toStr = " + cred.toString());
          }
        }
      }
    }
  }

  public static final String bytesAsHex(byte[] bytes)
  {
    StringBuffer sb = new StringBuffer();
    for (int i = 0; i < bytes.length; i++)
    {
      sb.append((char)(HEX[(bytes[i] & 0x00F0) >> 4])).append((char)(HEX[bytes[i] & 0x000F])).append(" ");
    }
    return sb.toString();
  }
}
EOF

CP=/tmp
CP=$CP:$MW_HOME/oracle_common/modules/oracle.jps_11.1.1/jps-api.jar
CP=$CP:$MW_HOME/oracle_common/modules/oracle.jps_11.1.1/jps-common.jar
CP=$CP:$MW_HOME/oracle_common/modules/oracle.jps_11.1.1/jps-internal.jar
CP=$CP:$MW_HOME/oracle_common/modules/oracle.idm_11.1.1/identitystore.jar
CP=$CP:$MW_HOME/oracle_common/modules/oracle.osdt_11.1.1/osdt_xmlsec.jar
CP=$CP:$MW_HOME/oracle_common/modules/oracle.pki_11.1.1/oraclepki.jar

$JAVA_HOME/bin/javac -cp $CP /tmp/DumpWallet.java

$JAVA_HOME/bin/java -cp $CP DumpWallet "${PATH_TO_JPS}"

Sample Output :-

Dumping store contents ...

### Map: oracle.wsm.security
1. + Key: sign-csf-key
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = signing key alias/password
  name  = orakey
  pass  = welcome1
  expires   = null
2. + Key: enc-csf-key
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = encryption key alias/password
  name  = orakey
  pass  = welcome1
  expires   = null
3. + Key: keystore-csf-key
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = keystore access password
  name  = n/a
  pass  = welcome1
  expires   = null
4. + Key: test-appid-key
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = null
  name  = weblogic
  pass  = welcome1
  expires   = null

### Map: IDCCS
1. + Key: ldap:1340771431089
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = null
  name  = ldap:1340771431089
  pass  = MjcxN0M2ODREOEQ0RjZERg==
  expires   = null
2. + Key: db:1340771431083
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = null
  name  = db:1340771431083
  pass  = MTlDQTE1N0EzQzE3REY1OA==
  expires   = null
3. + Key: hash:1340771431089
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = null
  name  = hash:1340771431089
  pass  = MTk1MTk1QkQ4OUE2QzJBNw==
  expires   = null
4. + Key: proxy:1340771431089
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = null
  name  = proxy:1340771431089
  pass  = M0IwMUZBQjNGQTUxNzk0OA==
  expires   = null

In a WebLogic domain, you will find that there is also a cwallet.sso found in $DOMAIN_HOME/config/fmwconfig/bootstrap.

Dumping store contents ...

### Map: fks
1. + Key: master.key.0
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = javax.crypto.spec.SecretKeySpec
  algorith  = AES
  format  = RAW
  key material as hex = CB 45 4F B0 F8 26 FF 04 31 9F 48 DD 43 42 69 C7
  expires   = null
2. + Key: current.key
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = java.lang.String
  cred  = master.key.0
  expires   = null

### Map: IntegrityChecker
1. + Key: kss
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = Array of byte
  byte array as hex = AB 18 CD 76 6C 39 FE 46 A0 6D 1C F0 BC 8D 97 3A D1 64 BC 80 2C 33 64 8E AE C9 B1 63 88 BE 23 7C 37 2F 63 9D 55 2B 5E 8F 1E 08 0A 73 F1 A8 15 83 8F 24 3D 19 B8 79 6E 75 B2 1C 7F DB 72 FC AE BA 72 A3 62 62 27 29 EE DE
  expires   = null

If you ever decide to reassociate the credential store with LDAP (e.g. using the reassociateSecurityStore command), you will find that the bootstrap cwallet.sso will contain credentials to access the LDAP store.  Here is what my domain wallet files look like when using an LDAP credential store :-

$DOMAIN_HOME/config/fmwconfig/cwallet.sso :

Dumping store contents ...

### Map: dip
1. + Key: cn=odisrv
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = DIP Password
  name  = cn=odisrv,cn=Registered Instances,cn=Directory Integration Platform,cn=Products,cn=OracleContext
  pass  = YNDMSU1wP1WergcX
  expires   = null

### Map: ODSMMap
1. + Key: ODSMKey.Wallet
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = ODSM Key store password
  name  = ODSM
  pass  = 0000000000
  expires   = null

$DOMAIN_HOME/config/fmwconfig/bootstrap/cwallet.sso :

Dumping store contents ...

### Map: BOOTSTRAP_JPS
1. + Key: bootstrap_q6ShJcm89vO8N2oVoSFqTLuW6Sg=
  class = oracle.security.jps.internal.credstore.PasswordCredentialImpl
  desc  = bootstrap user name and password
  name  = cn=orcladmin
  pass  = welcome1
  expires   = null

### Map: fks
1. + Key: master.key.0
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = javax.crypto.spec.SecretKeySpec
  algorith  = AES
  format  = RAW
  key material as hex = 1C B5 89 2A 45 F2 BA A0 E5 C1 A8 F6 DE 6E FC 5A
  expires   = null
2. + Key: current.key
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = java.lang.String
  cred  = master.key.0
  expires   = null

### Map: IntegrityChecker
1. + Key: kss
  class = oracle.security.jps.internal.credstore.GenericCredentialImpl
  desc  = null
  type  = Array of byte
  byte array as hex = FD 01 1E 54 D5 84 3B 9D AF DA 62 62 22 BB 7E A9 0C DB 08 A3 D9 71 9F A6 03 96 7F DF 29 69 37 55 60 5D 0E 32 EE 3A D0 D6 F2 A9 FD 58 DB 82 87 A0 98 D2 78 6A 47 48 E9 6B 86 3E 68 77 BC 17 01 B6 A0 BD 29 A2 3B E7 B7 73
  expires   = null

 

Have fun!