Connecting to a Filesystem (s\FTP, Google, Local)

iPaaS Documentation Connection Guides
1

Table Of Contents

  • Introduction
  • Step 1: Create a Filesystem
  • Step 2: Test the connection
  • Step 3: Interacting with the filesystem
    • Step 3a: Creating an incoming configuration
    • Step 3b: Creating an outgoing configuration

Introduction

A filesystem represents a (remote) environment that is used to host files. The most common example of a filesystem would be an FTP directory.

Alumio offers a multitude of filesystem types to connect to:

  • FTP

  • FTP over SSH

  • Google Cloud Storage

  • Local filesystem

For this guide, we will be connecting to an sFTP server. The steps are described below.

Step 1: Create a Filesystem

  • Navigate to Connections-> Filesystems and create a new Filesystem configuration by hitting the + icon on the right upper side of the screen.
  • Fill in the Name field - please note that while the description field is required, it will automatically be generated based on the name.
  • Select the following prototype: FTP over SSH
  • Please fill out the mandatory fields to connect to the given system. Our example requires the following fields; hostname*, username,* and password fields.
  • Also select in which way files should be logged in the filesystem. You can choose multiple options at once.
  • It’s also possible to log file contents. By default, a checkbox is set to enable that option that allows you to log what exactly will be stored in the filesystem. If you deal with such data as images and don’t won’t to put it into the filesystem, disable the checkbox.
  • Once you’ve filled out all required fields, hit Save.

image
image2868×1300 204 KB

Step 2: Test the connection

  • Navigate back to the filesystem overview if you aren’t already on the page
    • Hint: You can find all configured filesystems by navigating to Storages → Filesystem
  • On the desired filesystem, press the … icon under the Actions and hit Test connection.
  • Your Alumio instance will now verify if Alumio can set up a connection. The result will be shown as a notification.

Note: More information about connection testing can be found here

image
image2864×1298 220 KB

Step 3: Interacting with the filesystem

Now that you have set up the filesystem, you can interact with it through an incoming or outgoing configuration. Incoming configurations allow you to read data from a filesystem, while an outgoing configuration will enable you to write data to a filesystem.

Step 3a: Creating an incoming configuration

Click here to review the general guide on incoming configurations in order to get familiar with their purpose.

First, choose the relevant filesystem subscriber prototype:

  • Filesystem: listing subscriber - Select this subscriber to retrieve a list of files from a filesystem.
  • Filesystem: file subscriber - Select this subscriber to read individual files from a filesystem.

For this guide, we will be selecting the filesystem: file subscriber. Now it’s time to fill in the required fields.

  • Firstly, select the in step 1 created filesystem under the Filesystem field.
  • Define one or more Patterns
    • Patterns provide Alumio with instructions on which files should be read. If you want to read XML files from the ‘Documents’ directory, you will define the following pattern: Documents/.xml*. The * serves as a wildcard.
  • Select the required Parser
    • The Parser is responsible for translating the files to JSON objects.
  • Optionally, you may configure the rest of the fields.
    • Limit the number of files - limits the number of files read each time the incoming configuration is executed. Helpful in working with a high number of files.
    • Limit the number of parsed items - limits the number of records read each time the incoming configuration is executed. Helpful in working with big data sets.
    • Plugins - plugins enable extra functionality for this incoming configuration.
      • Add file information to data - adds meta information to the entity data
      • Execute operations on files - used to copy, move or delete files during various stages in the process
      • Track queue progress - required if either of the Limit… options are used.

Step 3b: Creating an outgoing configuration

Click here to review the general guide on outgoing configurations in order to get familiar with their purpose.

First, choose the relevant filesystem publisher prototype: Write files to a filesystem. Now it’s time to fill in the required fields.

  • Firstly, select the in step 1 created filesystem under the Filesystem field.
  • The Path can be customized to create a filename based on the required criteria dynamically.
    • Optionally, you may configure any mapper transformers to create the desired result.
  • The Contents field defines the body of the file which is created.
    • Optionally, you may configure any mapper transformers to create the desired result.
  • You may create multiple files in multiple locations on the same filesystem, by adding another configuration using the Add a file to write option.

1 Like