Adding Transport Only HTTP Data Feeds

This feature is only available for Archer on-premises installations.

Complete the following tasks to add a Transport Only HTTP Data Feed.

On this page

Task 1: Add a transport data feed

  1. From the menu, click Admin menu > Integration > Data Feeds.

  2. Click Add to create a new data feed.
  3. In the General Information section, do the following:
    1.  Enter the name and description of the data feed.

      Note: The alias populates automatically when you set the name of the data feed for the first time and save the data feed. You can modify the alias after you save the data feed. The remaining fields in the General Information section are read-only and populate when the data feed is created, updated, and run.

    2. Select Select to make the data feed active.
  4. In the Feed Information section, do the following:
    1. In the Feed Type field, select Transport Only. See Adding Standard Data Feeds to add a Standard data feed.
    2. In the Target Path field, enter the directory path that the exported source file data is saved to.
    3. In the Service Account Name field, enter the user account associated with the data feed. If the user does not exist, you can create a new user. Enter the username. See Data Feeds Service Account for more information on the Service Account Name.

  5. Do one of the following:

    • To continue configuring the data feed, go to the next task.
    • To finish setting up the feed later, click Save or Save and Close.

Task 2: Define the transport method

  1. Go to the Source Connection tab of the data feed.

  2. From the Source Locale list, select a locale.
  3. From the Transport Method list, select a transport type.

  4. In the Transport Configuration section, complete the File Type, Action Type, and File Filter fields.

    The following table describes the fields.

    Available selections

    Description

    Single Data File

    References a single data file.

    Manifest File

    Points the Data Feed Manager to a file that contains a list of files to process.

    Zip File

    References a ZIP file.

    Note: All file names within a ZIP file must consist of characters from the code page 437 character set.

    Get

    Uses the GET type of HTTP request. This type adds the parameters on the query string.

    Put

    Uses the POST type of HTTP request. This type includes the parameters as form parameters on the request.
    • Path: The path to the external source from which data is imported when you run the data feed
    • File Filter: Specifies which files in the path the data feed processes. This field can contain a single file filter or a list of file filters separated by semicolons. The data feed only processes data included in the File Filter field.

      • Note: When you set the File Filter for a ZIP file, enter the ZIP file as the first file type in the path. In the case of ZIP files, the data feed searches through the parent directory first and then searches through the file inside the specified ZIP file, based on the filter criteria.

      Example: Archer.zip;data.csv

    • Encryption Type: This option is only available for ZIP files. Select an Encryption Type from the list and enter a password to complete the encryption. Choose one of the following Encryption Types for your data feed:
      • None
      • WinZip
      • Rijndael / Advanced Encryption Standard (AES)

  5. In the Logon Properties section, enter the applicable credentials for logging on to the HTTP site. You can specify whether the HTTP site allows public access or restricts access to the data. For restricted access, select:

    • Use the Job Engine Service Account
    • Use specific account to make HTTP request, and enter your authorized account credentials.

  6. In the Data Request Properties section, complete the following fields.

    The following table describes the fields.

    Option

    Description

    Data Request URI

    Specifies the uniform resource identifier (URI) of the HTTP or HTTPS site that contains the data you want to import. This field also allows you to specify a port, for example, http://company-server:8080/httpFeed/.

    Header Parameters

    Specifies the key/value pair that may be required as part of your Get or Put operation in Header Parameters.

    To add another header parameter, click Add. To remove a header parameter, click Remove in the row of that header parameter.

    Post Data

    Specifies the posting data.

    This field is available only if you have selected Put as the Action Type.

  7. (Optional) In the Proxy Option field, select the applicable proxy option. You can specify whether the data feed passes through a proxy to access the source data. Set the configuration for the System Proxy in the Control Panel or configure and provide parameters to access another proxy.

  8. (Optional) Complete the applicable fields if you selected a proxy option. Provide the name, port ID, and domain of the proxy server and the user credentials to log onto the proxy server.

  9. (Optional) The data feed creates a local copy of the source data for further processing. In the Post Processing - Local copy of source file section, select from the following options to specify how the data feed handles the local copy of the source data after processing the source data.

    The following table describes the options for post processing the local copy of the source data.

    Option

    Description

    Delete

    Deletes the processed source file when the data feed successfully completes. The data feed also deletes any local copy of the source information.

    Rename

    Saves the source file under a new name when the data feed successfully completes. In File Path and Name, specify the new name for the file and the location to save the file.

    To save the data, ensure that the account running the Job Engine service can access the path of the destination file.

    If you select this option, use filename tokens for specifying the location or name of the file.

    Filename tokens

    Filename tokens are available for post processing when you want to save the source information and specify a location or name for the file. When you select the Rename option, you can use tokens to generate unique names automatically for the files.

    Here are the usable tokens for renaming data files.

    • Now. Insert a user-defined date format within the new filename. Possible formats include Now(MM/dd/yyyy) or Now(MM-dd-yyyy). See the Microsoft .Net Framework Developer Center for available custom date/time formats.

    • DataFileDirectoryName. Update the filename with the directory name, including the drive, of your file.

    • DataFileName. Insert the original filename, excluding the directory name and extension.

    • DataFileExtension. Insert the file extension, such as .csv, in the new filename.

    • DataFileFullName. Insert the fully qualified filename. This data includes the drive, directory, filename, and extension of the original file.

    For example, if the data file came from the following location, C:\DataFeed\Source\ESL\processed\ThreatData.csv, any files renamed using tokens provide the following output.

    Example 1

    • Input Tokens: {DataFileDirectoryName}\success\{DataFileName}_{Now(MM.dd.yyyy)}.{DataFileExtension}
    • Output: C:\DataFeed\Source\ESL\processed\success\ThreatData_01.31.2008.csv

    Example 2

    • Input Tokens: \\DFSRepository\{Now(yyyy)}\{Now(MM)}\{DataFileName}_success.{DataFileExtension}
    • Output: \\DFSRepository\2008\01\ThreatData_success.csv

  10. If you selected the Rename post-processing option, enter the location and name of the new file you want to save in the File Path and Name field.

  11. Do one of the following:

    • To continue configuring the data feed, go to the next task.
    • To finish setting up the feed later, click Save or Save and Close.

Task 3: Define the data feed schedule

You can set up data feeds to run automatically at regular intervals. This reduces the time and effort required to import data from an external file. You can initiate data feeds at various times and configure them to run in regular increments for an indefinite period of time.

You can also run the data feed immediately.

To prevent excess server load, schedule data feeds on a staggered basis. You can schedule a maximum of 10 data feeds to run at a time. If more than 10 data feeds are scheduled, each remaining data feed run as the previous one completes.

A reference feed allows you to specify another feed. This indicates to the Data Feed Service that this feed will start running as soon as the referenced feed completes successfully. A successful data feed run processes all input data, completes all expected record updates, and does not report any failures in the Run Details Report.

  1. Go to the Run Configuration tab > Schedule section.

  2. Do one of the following to schedule your data feed.

    Run on Schedule

    You can configure your data feed to run on a defined schedule.

    The following table describes the fields in the Run on Schedule option.

    Field

    Description

    Start Date

    Specifies the date on which the data feed schedule begins.

    Start Time

    Specifies the time the data feed starts running.

    Time Zone

    Specifies the time zone in which the data feed schedule begins.

    Recurring

    Specifies the interval in which the data feed runs, for example, Minutely, Hourly, Daily, Weekly, or Monthly.

    • Minutely. Runs the data feed by the interval set.

      • For example, if you specify 45 in the Every list, the data feed runs every 45 minutes.

    • Hourly. Runs the data feed by the interval set, for example, every hour (1), every other hour (2) and so forth.
    • Daily. Runs the data feed by the interval set, for example, every day (1), every other day (2) and, so forth.
    • Weekly. Runs the data feed based on a specified day of the week, for example, every other Monday (2), every third Monday (3), and so forth.
    • Monthly. Runs the data feed based on a specified week of the month, for example, on the first Monday of every month, on the second Tuesday of every third month, and so forth.

    Every

    Specifies the interval of the frequency in which the data feed runs.

    On

    Specifies the frequency of the days of the week on which the data feed runs.

    Weekday

    Specifies the days of the week on which the data feed runs.

    Run After

    Runs a specified data feed before the current one. The Data Feed Service starts the current data feed after the referenced data feed completes successfully.

    For example, you can select to have a Threats data feed run immediately after your Assets data feed finishes. From the Reference Feed dropdown, select the data feed that runs before the current data feed.

    Run Now

    Click the Run Now button in the toolbar on the Manage Data Feed page to run the data feed manually.

  3. To save the data feed, click Save or Save and Close.

  4. Do one of the following:

    • To continue configuring the data feed, go to the next task.
    • To finish setting up the feed later, click Save or Save and Close.