Cyberduck

We have put some more love into duck, the CLI version of Cyberduck. There are now several new options to facilitate automation jobs.

• --retry <count> allows to repeat failed connections.
• -y will dismiss all prompts and warnings with Yes by default.
• --throttle <bytes/second> allows to throttle transfers.

plus the most interesting to improve throughput is

• --parallel <connections>

to open multiple concurrent connections for transfers (#1135). For downloads, larger files will be segmented into multiple parts (#4498) to transfer with multiple connections and assembled again when complete. This feature will also find its way into version 4.7 of Cyberduck.

We have readied a command line interface (CLI) version of Cyberduck for power users. It is named duck and runs in your shell on Linux and OS X or your Windows command line prompt.

The CLI version includes the power features of Cyberduck ready to be used in scripts and automated deployments. Edit files on remote servers, download, upload and copy between servers with FTP, SFTP or WebDAV plus support for cloud storage Amazon S3 & OpenStack Swift deployments. Let’s looks at some examples what in particular duck is useful for.

Copy a file from an FTPS server to a Rackspace Cloud container

Note the URI syntax protocol://container/key used for OpenStack Swift, Microsoft Azure and Amazon S3.

osaka:~ dkocher$ duck --copy ftps://dkocher@cyberduck.io/update.cyberduck.io/Cyberduck-4.6.1.zip rackspace://sandbox/Cyberduck-4.6.1.zip


FTP-SSL connection opened…
Login cyberduck.io with username and password. No login credentials could be found in the Keychain.
Username (dkocher):
Login as dkocher
Password:
Cloud Files connection opened…
Login identity.api.rackspacecloud.com with username and password. No login credentials could be found in the Keychain.
Username (dkocher): dkocher
Login as dkocher
API Key:
[▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮ ] 37.4 MiB (39,239,680 bytes) of 55.1 MiB (67%, 694.9 KB/sec, 27 seconds remaining)


Upload a website

Recursively upload a folder to a remote location. An overwrite prompt will ask the action to take for existing files or can be specified using the --existing option.

osaka:duck.sh dkocher$ duck --username dkocher --upload ftps://cyberduck.io/duck.sh/ output/


FTP-SSL connection opened…
Login cyberduck.io with username and password. No login credentials could be found in the Keychain.
Username (dkocher):
Login as dkocher
Password:
Login successful…
The remote file duck.sh already exists. Choose what action to take:

Resume Append existing files (resume)
Compare Skip files that match size, modification date or checksum (compare)
Rename existing Rename existing files with timestamp appended to filename (rename)
Overwrite Overwrite all files (overwrite)
Cancel (cancel)
Skip Skip transfer of files that already exist (skip)
Rename Rename transferred files with a timestamp appended to the filename (similar)

Action [resume, compare, rename, overwrite, cancel, skip, similar]: compare
[▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮] 1.0 MiB (1,067,009 bytes) of 1.0 MiB (100%, 88.6 KB/sec)


Edit a remote file over SSH

Connect using public key authentication using --identity for the private key.

osaka:~ dkocher$ duck --username dkocher --identity ~/.ssh/cyberduck.io-rsa --edit sftp://cyberduck.io/etc/httpd/vhosts.d/duck.sh.conf


List all buckets in a given region in S3

Note the --region parameter that limits the output to containers that are in the Frankfurt (eu-central-1) AWS data center.

osaka:~ dkocher$ duck --username AKIAJOC4AH3QDQUB3C7Q --password ugQE+uj8tZyywd9hjoyZ5H5EaPcMuJ4Pk2C7jF85 --list s3:/// --region eu-central-1

List files in a Rackspace container

Using the --retry option to retry the task on a networking failure.

osaka:~ dkocher$ duck --list rackspace://sandbox/ --retry
Cloud Files connection opened…
Login identity.api.rackspacecloud.com with username and password. No login credentials could be found in the Keychain.
Username (dkocher): rackcloud
Login as rackcloud
API Key:
Login successful…
Cyberduck-4.6.1.zip


Check out the output of duck --help and the Cyberduck Wiki for more help about available commands and options.

To provide a seamless setup we provide Homebrew packages on OS X, package repositories for YUM and APT on Linux and a Chocolatey package on Windows. Besides there are regular installers for all platforms available if you use no package manager.

Amazon Web Services has announced its second region in Europe opened in Frankfurt. Update to the latest 4.6 beta build or later to create and access S3 buckets in this new region. Support eu-central-1 region required substantial work to implement the new AWS4-HMAC-SHA256 authentication version and SHA256 content checksum for uploads in the Jets3t S3 framework used.

Cyberduck 4.5 is fully compatible with OS X 10.10.

With version 4.5 and later, notifications are displayed in the system taskbar on Windows. This applies to notifications when transfers have finished or a connection has been established or failed. It is most useful when editing files in an external editor application with a confirmation on screen that the upload has completed after saving the file in the editor. This new default behaviour can be disabled Control Panel → Notification Area Icons.

We have just finished with the implementation of mutual (two-way) TLS connections (#5883) for secured FTP and WebDAV (HTTPS) connections. When a server requests a client certificate for authentication, a prompt is displayed to choose a certificate with a private key that matches the given issuer name requested from the server. Matching certificates are searched for in the Keychain on OS X or the Windows Certificate Manager respectively.

 

 

To configure your server for mutual TLS authentication, refer to corresponding documentation.

• Apache HTTPD Client Authentication and Access Control
• ProFTPD TLS Client Authentication
• The configuration options validate_cert, ca_certs_file in vsftpd
• Mutual SSL authentication should also work with OpenStack Keystone deployment using PKI deployments.

This feature is currently available in the latest 4.5 beta build.

Interoperability with different providers of Swift (OpenStack Object Storage) has been a field of attention in the latest maintenance release of Cyberduck. We are now defaulting to use version 2.0 (Keystone) for authentication with the OpenStack Identity Service with consecutive fallback strategies for different options of tenant specifiers. To use the  version 1.0 (devauth) authentication method with a legacy OpenStack deployment, download the Openstack Swift (v1) profile.

We have made a number of improvements supporting two factor authentication with one time passcode for SFTP. The latest minor update of Cyberduck 4.4 has been tested to work flawlessly with installations running the Google Authenticator PAM module with OpenSSH keyboard interactive authentication or similar setups.

Give it a try in the latest beta build available for Mac & Windows.

Update. Documentation now available in the wiki.

We are about to reintroduce support for Windows Azure Storage on a new solid foundation after we dumped support some time ago. The new implementation in Cyberduck is fast and supports features such as editing metadata, editing ACLs on containers and copying shared signature URLs for giving temporary access to users with a time limited signed URL to files in a container.

We plan to enable Windows Azure Storage by default in version 4.5. If you want to give it a try just now, install the latest snapshot build and install the Azure connection profile.

We have integrated large upload support (5GB+) in 4.4.1 contributed by Joel Wright. OpenStack Swift requires files larger than 5GB to be split into segments on upload. This is quite cumbersome to implement as it is not transparently handled by the Swift server. By default Cyberduck is configured to use segmented uploads for files larger than 2GB which has the advantage to allow the resume of interrupted transfers.

As corporate backing of OpenStack continues to be strong we continue to improve support for Swift features and interoperability in Cyberduck.