Java SFtp (SSH) with support for auto-resume of interruted transfer - J2SSH toolkit patch

Opensource Java SFtp (SSH) library with support for resume of interrupted file transfer..

SFtp interrupt transfer background: The SFtp (SSH) protocol inherently _always_ supports auto-resume of transfer, which is quite unlike the standard Ftp protocol, where support of the REST command is not necessarily implemented on Ftp servers. So if SFtp clients does not support auto-resume of interrupted transfers, it should be considered lacking functionality on part of the client rather than on the server.

Objective: Intended as a quick tutorial for people, who just want to enable auto-resume SFtp transfer for a single file and really don't need a full-featured-bells-and-whistles SFtp client supporting resume of file transfer. Personally I used it as a suppelement to the unix command 'rsync' to transfer latest updates of log files from BEA installations in (clustered) environments.

Basicly this text describes how to bring the J2SSH toolkit up-to-speed to support SFtp auto-resume. j2ssh-0.2.9-src.zip (or later versions) may be found at sourceforge.net/projects/j2ssh.

To successfully manage resume of file transfer, you will need:

Example to restart file transfer from offset 0 (in case the file contents changes)
Example to auto-resume file transfer (from previously position)

To get the show on the road, you may retrieve the original j2ssh-0.2.9-src.zip and patch it yourself. For your convenience, I've includeded a patched version j2ssh-0.2.9-autoresume-patch.jar which includes the patched files - AND - the original distribution, with all information and copyrights. Naturally distribution is provided as-is and should be be used at own risk - without support.

Standard use of the J2SSH to just get a single file:

SFtpGetFile.java


package dk.topsecurity;

import java.io.*;
import java.lang.*;
import com.sshtools.j2ssh.*;
import com.sshtools.j2ssh.authentication.*;

//Code to be used at own risk. 
//
//Code by courtesy of : http://www.spindriftpages.net/blog/dave/2007/11/27/sshtools-j2ssh-java-sshsftp-library/comment-page-1/
//

public class SFtpGetFile {

  public static void main(String[] argv) {

    String host = "127.0.0.1", userName="guest01", password="guest01", filePathRemote="/readme.1st", filePathLocal="readme.1st.local";
    int port = 22;

    try {

      SshClient ssh = new SshClient();
      ssh.connect(host, port);

      System.out.println("Authenticate");

      PasswordAuthenticationClient passwordAuthenticationClient = new PasswordAuthenticationClient();
      passwordAuthenticationClient.setUsername(userName);
      passwordAuthenticationClient.setPassword(password);
      int result = ssh.authenticate(passwordAuthenticationClient);
      if(result != AuthenticationProtocolState.COMPLETE){
         throw new Exception("Login to " + host + ":" + port + " " + userName + "/" + password + " failed");
      }

      System.out.println("Open the SFTP channel");

      SftpClient client = ssh.openSftpClient();

      System.out.println("Get the file");

      client.get(filePathRemote,filePathLocal);

      System.out.println("disconnect");

      client.quit();
      ssh.disconnect();
    }
    catch(IOException iox) {
      iox.printStackTrace();
    }
    catch(Exception ex) {
      ex.printStackTrace();
    }
  }
}

Run with :

getFile.cmd



java -classpath .;commons-logging.jar;j2ssh-0.2.9-autoresume-patch.jar dk.topsecurity.SFtpGetFile

This example will retrieve just a single file from the server and store it locally in the default directory of the application. Fairly straightforward.

The trick to support auto-resume is to basicly start the transfer at a pre-specified offset rather than from initial offset 0L. Each time a SFtp block transfer is initiated, the SSH client transmits a binary command-data-block (in the code for SshFxpRead.java) containing the read command SSH_FXP_READ and the remote-file-offset, from where to read.

Support for SFtp auto-resume is introduced by changes in files:

com.sshtools.j2ssh.sftp.SftpFileInputStream.java
com.sshtools.j2ssh.SftpClient.java
com.sshtools.j2ssh.io.UnsignedInteger64.java

The most significant API change is :

SFtpClient.java


...
    public FileAttributes resume(String remote, String local,
        FileTransferProgress progress)
        throws IOException, TransferCancelledException {
        String remotePath = resolveRemotePath(remote);
        FileAttributes attrs = stat(remotePath);

        if (progress != null) {
            progress.started(attrs.getSize().longValue(), remotePath);
        }

        long resumeOffset = 0;
        File localFile = new File(local);
        RandomAccessFile randomAccessFile = new RandomAccessFile(localFile, "rw");
        if(localFile.exists()) {
           resumeOffset = (int)randomAccessFile.length();
           randomAccessFile.seek(resumeOffset);
        }

        SftpFileInputStream in = new SftpFileInputStream(sftp.openFile(
                    remotePath, SftpSubsystemClient.OPEN_READ), resumeOffset);
        transferFile(in, randomAccessFile, progress);

        if (progress != null) {
            progress.completed();
        }

        return attrs;
    }
...

This basicly just introduces the RandomAccessFile and initializing the SSH transfer handler with an initial read starting offset.

To support SFtp resume-of-transfer you may need to add a new constructor to class SftpFileInputStream, which can actually take an initial starting offset.

SftpFileInputStream.java


...
    public SftpFileInputStream(SftpFile file, long resumeOffset) throws IOException {
        if (file.getHandle() == null) {
            throw new IOException("The file does not have a valid handle!");
        }

        if (file.getSFTPSubsystem() == null) {
            throw new IOException(
                "The file is not attached to an SFTP subsystem!");
        }

        this.file = file;
        position = UnsignedInteger64.add(position, resumeOffset);
    }
...

For this to work, the class UnsignedInteger64 need to be able to handle long values.

UnsignedInteger64.java


...
    public static UnsignedInteger64 add(UnsignedInteger64 x, long l) {
        return new UnsignedInteger64(x.bigInt.add(BigInteger.valueOf(l)));
    }
...

And an equaivalent example to auto-resume SFtp transfer of the previous file would be :

SFtpResumeFile.java



package dk.topsecurity;

import java.io.*;
import java.lang.*;
import com.sshtools.j2ssh.*;
import com.sshtools.j2ssh.authentication.*;

//Code to be used at own risk. 
//
//Code by courtesy of : http://www.topsecurity.dk
//

public class SFtpResumeFile {

  public static void main(String[] argv) {
  
    String host = "127.0.0.1", userName="guest01", password="guest01", filePathRemote="/readme.1st", filePathLocal="readme.1st.local";
    int port = 22;

    try {

      SshClient ssh = new SshClient();
      ssh.connect(host, port);

      System.out.println("Authenticate");

      PasswordAuthenticationClient passwordAuthenticationClient = new PasswordAuthenticationClient();
      passwordAuthenticationClient.setUsername(userName);
      passwordAuthenticationClient.setPassword(password);
      int result = ssh.authenticate(passwordAuthenticationClient);
      if(result != AuthenticationProtocolState.COMPLETE){
         throw new Exception("Login to " + host + ":" + port + " " + userName + "/" + password + " failed");
      }

      System.out.println("Open the SFTP channel");

      SftpClient client = ssh.openSftpClient();

      System.out.println("Auto-resume the file from last position (creating from new if necessary)");

      client.resume(filePathRemote,filePathLocal);

      System.out.println("disconnect");
 
      client.quit();
      ssh.disconnect();
    }
    catch(IOException iox) {
      iox.printStackTrace();
    }
    catch(Exception ex) {
      ex.printStackTrace();
    }
  }

}

Run with :

resumeFile.cmd



java -classpath .;commons-logging.jar;j2ssh-0.2.9-autoresume-patch.jar dk.topsecurity.SFtpResumeFile

This should auto-resume transfer from last known position.

Running... may result in output here...

resumeFile.cmd output



2009-03-05 01:01:27 com.sshtools.j2ssh.transport.cipher.SshCipherFactory 
INFO: Loading supported cipher algorithms
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.kex.SshKeyExchangeFactory 
INFO: Loading key exchange methods
2009-03-05 01:01:27 com.sshtools.j2ssh.configuration.ConfigurationLoader initialize
INFO: JAVA version is 1.5.0_06
2009-03-05 01:01:27 com.sshtools.j2ssh.configuration.ConfigurationLoader initialize
INFO: Extension C:\Program Files\Java\jre1.5.0_06\lib\ext\dnsns.jar being added to classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.util.ExtensionClassLoader add
INFO: Adding C:\Program Files\Java\jre1.5.0_06\lib\ext\dnsns.jar to the extension classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.configuration.ConfigurationLoader initialize
INFO: Extension C:\Program Files\Java\jre1.5.0_06\lib\ext\localedata.jar being added to classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.util.ExtensionClassLoader add
INFO: Adding C:\Program Files\Java\jre1.5.0_06\lib\ext\localedata.jar to the extension classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.configuration.ConfigurationLoader initialize
INFO: Extension C:\Program Files\Java\jre1.5.0_06\lib\ext\sunjce_provider.jar being added to classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.util.ExtensionClassLoader add
INFO: Adding C:\Program Files\Java\jre1.5.0_06\lib\ext\sunjce_provider.jar to the extension classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.configuration.ConfigurationLoader initialize
INFO: Extension C:\Program Files\Java\jre1.5.0_06\lib\ext\sunpkcs11.jar being added to classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.util.ExtensionClassLoader add
INFO: Adding C:\Program Files\Java\jre1.5.0_06\lib\ext\sunpkcs11.jar to the extension classpath
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.publickey.SshKeyPairFactory 
INFO: Loading public key algorithms
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.compression.SshCompressionFactory 
INFO: Loading compression methods
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.hmac.SshHmacFactory 
INFO: Loading message authentication methods
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.TransportProtocolCommon startTransportProtocol
INFO: Starting transport protocol
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.TransportProtocolCommon run
INFO: Registering transport protocol messages with inputstream
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.TransportProtocolCommon negotiateVersion
INFO: Negotiating protocol version
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.TransportProtocolCommon negotiateVersion
INFO: Protocol negotiation complete
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.TransportProtocolCommon beginKeyExchange
INFO: Starting key exchange
2009-03-05 01:01:27 com.sshtools.j2ssh.transport.kex.DhGroup1Sha1 performClientExchange
INFO: Starting client side key exchange.
2009-03-05 01:01:28 com.sshtools.j2ssh.transport.AbstractKnownHostsKeyVerification verifyHost
INFO: Verifying 127.0.0.1 host key
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.TransportProtocolClient verifyHostKey
INFO: The host key signature is  valid
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.TransportProtocolCommon completeKeyExchange
INFO: Completing key exchange
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.cipher.SshCipherFactory newInstance
INFO: Creating new blowfish-cbc cipher instance
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.cipher.SshCipherFactory newInstance
INFO: Creating new blowfish-cbc cipher instance
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol onServiceInit
INFO: Registering connection protocol messages
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.Service start
INFO: ssh-connection has been requested
2009-03-05 01:01:29 com.sshtools.j2ssh.transport.AsyncService onStart
INFO: Starting ssh-connection service thread
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol openChannel
INFO: Channel 0 is open [session]
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol openChannel
INFO: Channel 1 is open [sftp]
2009-03-05 01:01:29 com.sshtools.j2ssh.sftp.SftpSubsystemClient initialize
INFO: Initializing SFTP protocol version 3
2009-03-05 01:01:29 com.sshtools.j2ssh.subsystem.SubsystemChannel startSubsystem
INFO: Starting sftp subsystem
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol sendChannelRequest
INFO: Sending subsystem request for the session channel
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol sendChannelRequest
INFO: Waiting for channel request reply
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol sendChannelRequest
INFO: Channel request succeeded
2009-03-05 01:01:29 com.sshtools.j2ssh.sftp.SftpSubsystemClient initialize
INFO: Server responded with version 3
2009-03-05 01:01:29 com.sshtools.j2ssh.sftp.SftpSubsystemClient getOKRequestStatus
INFO: Received response
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol closeChannel
INFO: Local computer has closed channel 1[sftp]
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol onStop
INFO: Closing all active channels
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.ConnectionProtocol onStop
INFO: thread has 2 active channels to stop
2009-03-05 01:01:29 com.sshtools.j2ssh.connection.Channel close
INFO: Finializing channel close

Basicly this example have demonstrated how to enable SFtp auto-resume transfer :

reference to non-commercial opensource Java SFtp (SSH) library
listing of java working example
how to run auto-resume SFtp

I respectfully include reference to inspirational ressource on the net :

http://www.spindriftpages.net/blog/dave/2007/11/27/sshtools-j2ssh-java-sshsftp-library/comment-page-1/

/topsecurity.dk 2009/03