SFTP Data Transfer

Suggest Edits

Introduction

SFTP, or Secure File Transfer Protocol, is a method for transferring files over networks. It provides a secure environment for exchanging data by encrypting both commands and file content, ensuring confidentiality and integrity during transmission.

SFTP offers several benefits, particularly for handling large file transfers where low latency is not critical. It prioritizes security and reliability over speed, making it an ideal choice when the protection and integrity of your data are important. The robust encryption mechanisms of SFTP safeguard your files against unauthorized access, while its error-checking features guarantee the integrity of the transferred data.

One of the key advantages of SFTP is its widespread support across different operating systems and platforms. This versatility makes it a convenient option when working with diverse technology ecosystems. SFTP's straightforward command line interface allows users to transfer files without requiring extensive technical knowledge or additional software installations on most popular operating systems. Alternatively, a familiar browser-like environment is available through multiple free and popular apps.

By utilizing SFTP for large file transfers, you can ensure a hassle-free process while maintaining the security of your data. Its seamless integration and dependable performance make it a practical solution for efficient and protected file exchange.

Access Control and Accounts Information

Users

Siprocal generates user accounts which have access to all files on their SFTP folder, and access only to that folder. Although called a "user", the SFTP interface is not intended to be used exclusively by individual collaborators of the partner, but for groups with the same access privileges to the data exposed on these folders.

For Example, let's say that your company wishes to share confidential Financial data with Siprocal which only members of the Finance team should have access to internally, and also Marketing data from the Sales team. Siprocal would generate Finance and Sales users to be used by the members from each department, respectively, instead of a single user for the partner. But two members within each group would share their access.

The way that users are given access to the server is through their credentials:

  • Their username
  • Their private key

The private key is a file that serves as the key (hence the name) to access the server. For this reason, this key must be shared exclusively through secure channels and sent directly to the people with access privilege to the files within the folder.

More details on how to use these credentials will be given on the Accessing the SFTP Server section

Server

Siprocal's SFTP server is mounted on top of the AWS Transfer Family and AWS S3 services, with all availability and resiliency guarantees this entail. Each user access maps securely to an isolated folder which cannot be accessed by any other user on the server.

Accessing and Using the SFTP Server

SFTP Server Architecture

Siprocal's user folder follows a predictable, standard pattern:

~/
├─inbox/
| └─[WORKFLOW FOLDER]/
├─outbox/
| └─[WORKFLOW FOLDER]/
└─HELLO_[USERNAME].txt

Every server has two folders, inbox and outbox at the root and a HELLO_*.txt file with the name of the partner, client or vendor. These folders' directions are from the standpoint of the partner:

  • Siprocal will send files to / The Partner should read their files from inbox
  • The Partner should put their on / Siprocal will read files from outbox.

Within inbox/outbox there will be dedicated folders for each workflow, meaning a specific file exchange protocol directed towards a particular goal.

IMPORTANT: A user has complete write and read access to files within their folder and the partner is responsible for ensuring that collaborators with access privileges follow the read/write protocol correctly on their side.

Command Line Access

The SFTP server can be readily accessed through the sftp command line application on MacOS and most Linux system distributions, and through sftp.exe on Windows 10 and above.

To get access to the server, use the given credentials (setup process described below) on the command (Linux/MacOS):

sftp -i PRIVATE_KEY_FILEPATH  [email protected]

Graphical Interface

For users who might wish to open and visualize the contents of the SFTP folder using a graphical application, we suggest FileZilla, a free and popular SFTP client applications.

To access a server on FileZilla using a private key, we suggest using the method described on this video

Programmatic Access

Technical teams or technically-oriented collaborators should be able to access SFTP servers programmatically through their usual development environments and production systems. Most programming languages provide either native or popular libraries for handling SFTP connections. A few selected libraries are listed below for reference:

User Setup

Getting Access Keys

Get in touch with your organization's Account Manager or Customer Success representative and ask them to generate credentials for a specified use case for a trusted individual within your organization.

Our Data team will get in touch with them to share the credentials privately

First Access

Using one of the methods described above, this trusted individual should open the SFTP folder assigned to them and verify that the inbox and outbox folders are preset, if they contain the workflow-specific subfolder and the HELLO_*.txt file on the root of the folder match their username.

RECOMMENDED - Update Public Key

SFTP uses a security scheme based on a "key pair", which is composed of two files: one called "public key" and another called "private key". The "public key" doesn't need to be confidential, there's no loss in letting it being publicly known, hence the name.

The private key, on the other hand, needs to be kept a secret because any individual with the "private key" file can pass as the user to the server.

Even though the initial private key is sent in a private exchange between the Data team and the organization representative, it's nevertheless recommended that the representative, with the support from their technical team, create a new key pair. We suggest following this tutorial and sticking with RSA-4096, but the three mentioned - RSA, ECDSA and ED25519 - are supported.

Once this new key pair is generated, the public key should be sent to Siprocal's data team, which will act as a new "lock" on the server that only the new private key can open. This ensures that no one else other than the organization's internal team has a copy of the file that controls access to the server.

Best Practices

Even though the SFTP exchange can be used as setup on the steps described above, there are a few practices that we recommend are followed to optimize the value of this file exchange channel:

  • Do not share private keys with multiple people, but let a single person be responsible for taking data out of the server and putting it in, preferrably through some automated system
  • Periodically rotate private keys: every quarter or year, generate a new key pair and share the new public key with your Account Manager or Customer Success Representative, which will request the change to the Data team
  • Always backup data on internal systems: This protocol is expected to be used for data exchange, not storage. Unless otherwise agreed, Siprocal will reserve the right to remove older files on these servers.

Standard Use Cases for SFTP Server

Updated 8 months ago