Getting Started with SFTP Gateway 2.0 | Thorn Technologies

Getting Started with SFTP Gateway 2.0

Introduction

SFTP Gateway is a pre-configured SFTP server that will transfer uploaded files to an Amazon S3 bucket. To start, you need to create a user and an SSH key. This can be done through the SFTP Gateway admin interface. After that, you can upload your files to the SFTP server which in turn will push them to an Amazon S3 bucket. The SFTP server will not hold on to your files -- once uploaded, you can view them directly in S3.

This guide will focus on getting started with a Highly Available (HA) SFTP Gateway setup for testing.

Before you begin

Subscribe

You first need to subscribe to the SFTP Gateway product. Doing so allows your AWS account to use the SFTP Gateway AMI.

Click here to open the AWS Marketplace page for SFTP Gateway.

Click the Continue to Subscribe button.

continue-to-subscribe.png

Click the Accept Terms button.

accept-terms.png

Once the Terms of Use have been accepted, you will have access to the SFTP Gateway AMI. The software is billed at $0.05 per hour for each instance. This is in addition to the cost of the EC2 instance itself.

You don't need to purchase an Annual Subscription. But, this is a cost-savings option worth considering when running SFTP Gateway full-time in production.

Key Pair

You will need to create an EC2 Key Pair in order to SSH into your instances.  Although most SFTP Gateway configuration is via the web interface, there may be times when you need command line access.

To create an EC2 key pair in your AWS account:

  1. Login to AWS and go to the EC2 console

  2. In the navigation panel, under the Network & Security section, go to Key Pairs

  3. Click Create Key Pair

  4. Enter a name for the new key pair and click Create

    Note: When you click Create, your browser will download a private key file. This must be kept secure in a place where you can always find it. If this file is lost or deleted, it is difficult to regain access to your EC2 instance.

Spinning up an SFTP Gateway stack

To start an HA stack of SFTP Gateway:

  1. Download the HA CloudFormation template

  2. Log into AWS and go to the CloudFormation console

  3. Click Create Stack

  4. Choose Upload A template to Amazon S3, browse to the template that you downloaded earlier, and click Next

    Note: This will create a cf-templates bucket in your S3 that will store the template for later use. To use the template from the S3 bucket, navigate to the bucket, open the template details, and copy the link. Then in CloudFormation, select Specify an Amazon S3 template URl and paste the template link in the test box.

  5. Enter the details for the stack:

    • Stack name.

    • Default bucket name: The name of a new or existing S3 bucket in your AWS account.

    • Desired Capacity: The number of EC2 instances you woud like to have in your Auto Scaling group.

    • Disk Volume Size: This must be 32 GB (the volume size of the AMI) or higher. You can always increase this at a later time (see Resizing an EC2 instance volume).

    • EC2 Type: A t2.small instance is generally sufficient for testing.

    • EFS Encryption: SFTP Gateway (HA) uses Elastic File System (EFS) to store files. The EFS volume can be encrypted for compliance reasons.

    • InputCIDR: An IP range that allows inbound SSH and SFTP traffic to your EC2 instance. We recommend obtaining your computer's public IP from http://checkip.dyndns.org/, and then appending /32 (a CIDR range of a single address). Although you can use 0.0.0.0/0 to allow all traffic, this weakens your security posture. 

    • Key Pair. Choose the SSH key pair you created in the Before you begin section. You will need the private key in order to SSH into the server. For more information on public and private keys see, SSH Key Pairs.

    • VPC ID Range: Pick a Class C private IP address range. CloudFormation will provision resources inside this subnet range.

    • Web Admin Password: This will be used to log into the user management web console.

  6. Stack Options: The stack options page can be left as is. Scroll to the bottom of the page and click next.

  7. Review and create stack.

    • You must check the box that reads I acknowledge that AWS CloudFormation might create IAM resources. to give CloudFormation explicit permission to create IAM resources.

The stack creation progress can be monitored by selecting the stack and viewing the Events tab. If any errors occur during creation that will appear in the event log.

Access Admin Interface

To access SFTP Gateway admin interface, go to the output tab of the stack in your AWS CloudFormation console and copy the hostname value (Fig-1).


Fig-1

Paste the url in the address bar of your browser (Fig-2).


Fig-2

Login to the user interface

Use admin as your username and the password that you chose during the set up process. Click “Sign In”.
You can learn more about the user interface here.

Create new user

To add a new user, click on the  button in the top right corner. You will be taken to the “Create User” form (learn more).

  1. Enter a username.

  2. Select “Generate new SSH key pair” to generate a new key pair for the user (If this is selected the private key will be downloaded when the user is saved), or “Upload user-provided SSH key” to upload an existing public key.

  3. Click “Save”. You will be presented with connection instructions that you can copy and paste into an email to the user, along with their new private key (if generated).

Without any other configurations, this will allow you to sftp files to the uploads directory in the default S3 bucket that you specified during the set up process.

SFTP via command line (Linux/Mac)

Find the private key in your Downloads directory, the name of the file is <username>.key. You will need to adjust permissions:

chmod 600 <username>.key

If you skip this step, you will see this warning: UNPROTECTED PRIVATE KEY FILE.
Next, log in to the SFTP Gateway as the new user:

sftp -i <username>.key <username>@<public_dns>

where <username> is the username of the user you created and <public_dns> is the Public DNS you copied from AWS console earlier in the tutorial (Fig-1).
Once you log in, you will be able to transfer files to S3 the following way:


Fig-3

Note: the files will get transferred to S3 and will not remain in the uploads folder.

SFTP via FileZilla

To transfer files to S3 using FileZilla, first connect to the SFTP Gateway server:

  1. Open FileZilla;
  2. Go to Site Manager;
  3. Click “New Site”;
  4. Choose protocol: “SFTP - SSH File Transfer Protocol”;
  5. For the host, use Public DNS copied from AWS console earlier in the tutorial (Fig-1);
  6. Choose logon type: “Key file”;
  7. For the user, use the username you created earlier;
  8. For the key file, point to the <username>.key file you generated earlier;
  9. Click “Connect”.


Fig-4

Now you can drag and drop your files to the uploads directory. Note: the files will get transferred to S3 and will not remain in the uploads folder.

View your files on S3

Once you are done transferring the files, go to S3 in your AWS console and find the bucket that you set up to be the default bucket (if you forget the name, you can see it on the settings page in the admin UI). There you will see a list of users. Each user will have an uploads directory where you can find the files you just transferred.