Data Feeds

Data Feed Manager is a flexible, code-free tool for aggregating data in Archer. Use the tool to:

  • Configure multiple, dynamic data feeds, and manage those feeds without relying on programming resources.
  • Build and configure dynamic integrations with external enterprise systems and files. From Data Feed Manager, you can build a transport path between Archer and an external source and then map the data from that source to an existing target application or questionnaire in Archer.
  • Configure the data feed to run on a schedule. After the initial configuration, the data feed runs automatically with no need for you to intervene.

You can integrate data using Data Feed Manager for:

  • Network and asset discovery data
  • Vulnerability scan results
  • Performance scorecards
  • Incident reports
  • Audit results and recommendations

Because Archer is vendor neutral and content independent, you can use Archer as a point of consolidation for enterprise data of any type for supporting analysis and process management. With a centralized view of data from point solutions, databases, spreadsheets, and other sources, you can access content more easily that is relevant to your job functions. Re-purpose data to support a variety of business processes.

A data feed must be both active and valid to run. As you configure your data feed, Data Feed Manager validates the information for you. If it is not valid, an error message appears. You can save the data feed and correct the errors later. However, the data feed does not process until you have corrected the errors and the data feed validates.

On this page

Data feed types

Important: To avoid potential conflicts with other data feeds, it is recommended that you use a different service user for each data feed. For more information, see Data Feed Service Account.

Data Feed Manager supports standard and transport data feeds.

The following table describes each type of data feed.

Feed Type

Description

Standard

Brings data from an external source into an application or questionnaire. This data feed type requires that you:

  • Define the fields and data format.
  • Map the fields in the source file to the target.
  • Configure a transport type that successfully imports the source data into another application or questionnaire. For example, perform a report-based search for an application or questionnaire that contains the source data that you want to import into another application or questionnaire.
  • Select a Data Feed Service Account.

You can specify the following:

  • Whether to send subscription notifications to specified users or groups when records are modified.
  • Whether to send a notification to specified users or groups when a data feed job completes, identifying a successful or failed completion.
  • The locale format of your source data. For example, different characters might be used to indicate a decimal place.

Transport Only

Creates the specified data file that subsequent, standard data feeds can use as input and that can be reviewed to understand the data structure that a transport configuration returns. For example, to review the configuration of the output file that an HTTP data feed returns, run the data feed as a Transport Only feed. The output allows you to determine how to configure the XSLT to generate source fields to use for data mapping.

  • Ensure that a user account for the data feed and a target path for the separate data file exist, but no additional data configuration.

Data feed transporter types

The Data Feed Service (DFS) architecture accommodates the definition of various data retrieval mechanisms.

The following table describes the out-of-the-box transporters.

Transporter

Description

Archer Web Services

Accesses the Web Services API and retrieves data from an instance of Archer This transporter is used in Archer-to Archer data feeds.

Database Query

Returns results using an SQL query

File

Retrieves delimited data files, including support for multi-file manifests

FTP

Retrieves data files using the FTP protocol

HTTP

Runs a GET or POST to retrieve data from an HTTP or HTTPS site.

JavaScript

Runs a user-provided JavaScript file

If the result of that run is a data set, it is transformed and processed into the platform as normal.

Mail Monitor

Retrieves content from monitored email accounts

RSS

Retrieves records from a configured RSS feed

Supported and unsupported field types for data mapping

Supported Field Types

  • Attachment
  • CAST Detail
  • Cross-Reference
  • Date
  • External Links
  • Image
  • IP Address
  • Matrix
  • Numeric
  • Record Permissions
  • Related Records
  • Sub-Form
  • Text
  • Tracking ID
  • User/Groups List
  • Values List

Note: For User/Groups List and Record Permissions, for the source input username, the data field always tries to find a match in the User list first. If no match is found, then it will try to find a match in the Groups list.

Unsupported Field Types

  • Access History
  • CAST Score Card
  • First Published Date
  • History Log
  • Last Updated Date
  • MRDC (Must be populated through reference fields.)
  • Record Status
  • Scheduler
  • System-generated Related Record that points to a Questionnaire
  • Voting

Schema sources

The source for the schema of your data feed depends on which transporter you are using. The following table identifies and describes the schema sources that are available for each of the out-of-the-box transporters.

Important: The process of loading a source definition for a data feed times out at five minutes. You may want to consider using a smaller set of source data when you set up the feed.

Source

Description

Execute Search

Runs the search in Archer and detects the source schema from the results

Recommended approach for an Archer-to-Archer data feed. Loads the source fields directly from the report. When using this scheme, complete all required information on the Transport and Navigation tabs.

Execute Query

Runs the query specified on the Transport tab and detects the source schema from the resulting record set

Using this option may trigger actions in the database associated with this query.

Sample File

Uses a skeleton of your actual source data file. For example, if you are importing data from a .csv file, the source data file is a .csv file that includes the column names from your source data. If you are importing data from an .XML file, the source data file includes the structure of your .XML without the actual field values.

When you select the sample file, the Source Fields section populates with the fields specified in the sample data file.

For the Archer Web Services Transporter, select a file from an external location that contains the data in the same format as the report format.

Load URL

Loads the contents at the target URL and detects the source schema from the contents

Using this option may trigger actions associated with accessing the target URL.

Standard Schema

Uses the standard mail schema

Updating locked records

Archer has an important feature that prevents the updating or altering of a locked record. A record becomes locked when a user has opened it in Edit mode for the purpose of modifying it.

However, it is important to note that records can be updated through the RESTful and Web APIs, as well as through data feeds, even when a user has locked them. The following are examples of typical APIs that can update user-locked records:

  • PUT content (RESTful )
  • UpdateRecord (Web Services)
  • UpdateRecords (Web Services)

Key Fields

You can use key fields from the source data to identify matching records in a target application or questionnaire. Keys defined within Data Feed Manager are different from keys defined within the Application Builder.

The following table lists the valid key field types from a target application or questionnaire.

Text-Based Field Types

List-Based Field Types

Text

Values Lists

Numeric

Record Permissions

Date

User Groups

IP Address

Sub-form Fields

Tracking ID ("System ID" only)

 

The following restrictions apply to key fields:

  • You can only use the Tracking ID as a key field if you configure the field as System ID. If you configure the field as Application ID, then you cannot use Tracking ID as a key field.
  • If you map a source field for a sub-form and that source field value is blank, then the data feed will not process the sub-form record.

Simple keys and combination keys

Simple keys include only one field. Each key has a different order number, and the data feed processes the keys in the specified order. The data feed updates existing records from the source data in the target application or questionnaire using a simple key.

For example, each person has a unique Individual Taxpayer Identification Number (ITIN) or Social Security Number (SSN). Simple keys that consist of either the ITIN or SSN fields can identify records based on only those fields.

Combination keys allow you to select multiple fields to create a single key. By establishing a combination key, the order of each field within the key matches.

For example, first name and last name are two simple keys. By combining both the first name and last name fields into a combination key, the data feed uses the combination key as a single key to identify records.

Use of multiple keys

Archer allows users to define multiple keys and define the order of processing to identify matching records. After you set the order number of each key, the data feed scans the source data for matches to each key in the specified order. If a key is a combination key, all fields in the key from the source data must match the corresponding fields in the target record. The data feed continues the following process until it finds a match or until it processes all keys.

  • If the first key from the source data matches the corresponding field in the target record, the source data updates the target record data.
  • If the data feed does not find a match for the first key, it attempts to find a match using each consecutive key.
  • If the data feed does not find a match or no keys are left to scan, then it creates a new application or questionnaire record.

For example, you can configure keys in the following order: ITIN, first name, and last name.

  • If the data feed identifies a matching record using the ITIN, a key unique to each user, it stops processing the records.
  • If the data feed cannot find matching records based on the first key, ITIN, it uses the second key, first name, to find a match.
  • If the data feed cannot find matching records based on the second key, first name, it uses the third and last key, last name, to find a match.
  • If there are still no matching records, the data feed creates a new application or questionnaire record using the source data.

Matching criteria for list-based field types

The following table describes the matching criteria for list-based field types.

Option

Description

MatchExact

Specifies that the values in the target record field and the values in the source data field must match to update the target record. If the match is not exact, the data feed creates a new record.

For example, a field in the target application record contains a list with the values Red, Blue, and Green, and the source data field contains a list with the values Blue, Green, and Red. Since the list in the target application record matches the exact values in the source data list, then the data feed updates the target record.

MatchAny

Specifies that at least one value in the source data field must match at least one value in the target record field for the data feed to update the target record. If no values match between both fields, the data feed creates a new record.

For example, if a target application record has the values Blue and Green selected in the key field, and the mapped field in the source data includes only the value Blue, the data feed updates the record because at least one of the values matches.

MatchAll

Specifies that all values in the target record field must contain all values included in the source data field for the data feed to update the target record. If the target record field does not include all source data values, the data feed creates a new record.

For example, if the target application record has the values Red, Blue, and Green selected in the key field, and the mapped field in the source data includes the values Red and Green, the data feed updates the record. However, if the target record source data has the values Red, Blue, and Yellow, the data feed does not update the record.

Data Feed Service Account

A data feed Service Account is an account that the system specifically uses to run a data feed. The Service Account user also creates and updates content in a data feed. When configuring a data feed, users can either choose an existing Service Account or create a new Service Account. Users can use the same Service Account to run every data feed, but for troubleshooting purposes, set up different Service Accounts for each data feed. Users cannot log into Archer with a data feed Service Account. History Log fields display field changes made by data feed Service users. Associating a unique data feed Service Account to each feed clarifies which data feed applied the update.

Data feed communication

The Data Feed Manager can be configured to retrieve or receive data from various external data sources using a variety of transport protocols. When given the option, it is recommended that you select secured versions over unsecured versions.

To strengthen data feed security, it is recommended that the Data Feed Manager require data feed paths to be specified as relative paths.

Note: Relative path entry is set up as the default.

BatchContentSave data feed token

Data feeds leveraging the BatchContentSave token should be used with caution. It is recommended that you use this token for high-volume ingestion of enrichment content. It is not recommended for content progressing through workflows. Content changes made by a BatchContentSave enabled feed are not tracked within the system History Log fields (though field audit information is retained).