Data Feeder for SOAR¶
-This package contains the QRadar SOAR (Resilient) plugin to the Data Feed extension. The Data Feed Extension allows you to maintain “replica” data for QRadar SOAR incidents, artifacts, tasks, notes, attachments and so on, in another QRadar SOAR organization in the same or different QRadar SOAR platform, as well as another CP4S environment. The updates are performed in near real-time.
-Refer to the documentation on the Data Feed extension for use cases supported and configuration options. Also refer to the other Data Feed plugins which can be used in combination.
+This package contains the QRadar SOAR (Resilient) plugin to the Data Feed suite of solutions. The Data Feed plugin allows you to maintain “replica” data for QRadar SOAR incidents, artifacts, tasks, notes, attachments and so on, in another QRadar SOAR organization in the same or different QRadar SOAR platform, as well as another QRadar Suite environment. The updates are performed in near real-time.
+Refer to the documentation on the Data Feed app for use cases supported and configuration options. Also refer to the other Data Feed plugins which can be used in combination.
Features¶
-
@@ -420,7 +420,9 @@
Transfer incident data between two Organizations within the same QRadar SOAR instance.
Transfer incident data to more than one QRadar SOAR instance at the same time.
Synchronized incident data objects include: artifacts, attachments, notes, milestones, tasks and datatables.
+Synchronize selective incident data based on a list of fields to exclude.
Choice of databases to retain synchronization information: SQLite or PostgreSQL.
+V2.0 introduces bidirectional synchronization. See V2.0 Changes for more information.
Features
History¶
-4/2024
+5/2024
+2.0.0
+Bi-directional synchronization between source and destination SOAR organizations
+
+4/2024
1.1.0
Enhancements for datatable select/multi-select fields. Fixes for synchronizing task attachments and notes. Threaded updates for better performance.
+
+V2.0 Changes¶
+2.0.0 introduces bidirectional changes. This can be used during a transitional period
+of moving to a new SOAR platform or QRadar Suite. This solution uses a shared PostgreSQL database and requires the SOAR data feeder plugin to be installed in both SOAR organization environment. One environment is considered the ‘source’ and the other as the ‘destination’.
+The following incident objects are synchronized:
+
+
+
+Incident Object
+Source Synchronization
+destination Synchronization
+Notes
+
+
+
+Incident
+Bidirectional
+Bidirectional
+destination incident changes (ex. severity, date occurring, incident closing) are sync’d back to the source incident. New incidents in the destination SOAR will synchronize to the source.
+
+Notes
+Bidirectional
+Bidirectional
+New destination notes synchronize to the source incident.
+
+Artifacts
+Bidirectional
+Bidirectional
+New destination artifacts synchronize to the source incident.
+
+Tasks
+Bidirectional
+Bidirectional
+Changes to tasks (ex. owner, due date, task completion) sync bidirectionally.
+
+Task Notes
+Bidirectional
+Bidirectional
+
+
+Task Attachments
+Bidirectional
+Bidirectional
+
+
+Attachments
+Bidirectional
+Bidirectional
+
+
+Milestones
+Bidirectional
+Bidirectional
+
+
+Datatables
+Bidirectional
+Bidirectional
+A custom rule is needed for each datatable to synchronize. Make sure the datatable exists in both the source and destination SOAR organizations.
+
+
+
+
+See sections Installation and Troubleshooting below for key issues when using bidirectional synchronization.
+
4/2024
5/2024
2.0.0
Bi-directional synchronization between source and destination SOAR organizations
4/2024
1.1.0
Enhancements for datatable select/multi-select fields. Fixes for synchronizing task attachments and notes. Threaded updates for better performance.
V2.0 Changes¶
+2.0.0 introduces bidirectional changes. This can be used during a transitional period +of moving to a new SOAR platform or QRadar Suite. This solution uses a shared PostgreSQL database and requires the SOAR data feeder plugin to be installed in both SOAR organization environment. One environment is considered the ‘source’ and the other as the ‘destination’. +The following incident objects are synchronized:
+Incident Object |
+Source Synchronization |
+destination Synchronization |
+Notes |
+
---|---|---|---|
Incident |
+Bidirectional |
+Bidirectional |
+destination incident changes (ex. severity, date occurring, incident closing) are sync’d back to the source incident. New incidents in the destination SOAR will synchronize to the source. |
+
Notes |
+Bidirectional |
+Bidirectional |
+New destination notes synchronize to the source incident. |
+
Artifacts |
+Bidirectional |
+Bidirectional |
+New destination artifacts synchronize to the source incident. |
+
Tasks |
+Bidirectional |
+Bidirectional |
+Changes to tasks (ex. owner, due date, task completion) sync bidirectionally. |
+
Task Notes |
+Bidirectional |
+Bidirectional |
++ |
Task Attachments |
+Bidirectional |
+Bidirectional |
++ |
Attachments |
+Bidirectional |
+Bidirectional |
++ |
Milestones |
+Bidirectional |
+Bidirectional |
++ |
Datatables |
+Bidirectional |
+Bidirectional |
+A custom rule is needed for each datatable to synchronize. Make sure the datatable exists in both the source and destination SOAR organizations. |
+
See sections Installation and Troubleshooting below for key issues when using bidirectional synchronization.
+Installation¶
-The integration package contains Python components that are called by the QRadar SOAR platform. These components run in the Resilient Circuits integration framework. -The package also includes QRadar SOAR customizations that will be imported into the platform later. -You perform these installation procedures at the QRadar SOAR integration server.
-Apps which are installed through the SOAR UI for App Host incorporate the UI components (rules and workflows) and no additional app installation is needed.
+This app can be used in either an App Host or an Integration Server configuration. Refer to the sections below for the steps to perform.
App Host Installation¶
+Apps which are installed through the SOAR UI for App Host incorporate the UI components (rules and workflows) and no additional app installation is needed.
To install or uninstall an App or Integration on the IBM SOAR platform, see the documentation at ibm.biz/soar-docs.
To install or uninstall an App on IBM Cloud Pak for Security, see the documentation at ibm.biz/cp4s-docs and follow the instructions above to navigate to Orchestration and Automation.
@@ -462,7 +531,10 @@ When installing on an integration server, install this base Data Feeder app prior to installing the SOAR plugin Data Feeder.
+If adding support for PostgreSQL, install the package as:
+Filter on the incident owner.
This functionality has been tested with QRadar SOAR instances >= v48. There is presently an issue with v37.0 restricting the live synchronization of a newly deleted artifact. If this capability is critical to your requirements, use QRadar SOAR version >=v37.1.
-The target QRadar SOAR platform must be at the same version or greater than the source QRadar SOAR platform.
-The target QRadar SOAR organization must have the same set of custom fields, incident types, playbooks (tasks and phases) in order to synchronize incident data. Use the export/import functionality under
Administrator Settings
.
-The target QRadar SOAR organization should have the same users and groups defined. For any user or group not found, incident and task ownership as well as member lists will be left empty.
+The destination QRadar SOAR platform must be at the same version or greater than the source QRadar SOAR platform.
+The destination QRadar SOAR organization must have the same set of custom fields, datatables, incident types, playbooks (tasks and phases) in order to synchronize incident data. Use the export/import functionality under
Administrator Settings
.
+The destination QRadar SOAR organization should have the same users and groups defined. For any user or group not found, incident and task ownership as well as member lists will be left empty.
To synchronize datatables in real time, create rules specifying the
feed_data_resilient
message destination in order to changes.
App Host Installation
Integration Server¶
-If adding support for PostgreSQL, upgrade the package as:
+
+
[sudo] pip install --upgrade rc_data_feed_plugin_resilientfeed-<version>.tar.gz[postgres]
@@ -502,7 +574,7 @@ Configuration[resilient_feed]
class=ResilientFeed
- # provide configuration information to the target QRadar SOAR and Organization
+ # provide configuration information to the destination QRadar SOAR and Organization
host=localhost
#proxy_host=
api_key_id=
@@ -513,7 +585,7 @@ Configurationorg=
cafile=false
# identify a sqlite db file to retain mapping between QRadar SOAR instances.
- sqlite_sync_file=/path/to/file
+ #sqlite_sync_file=/path/to/file
# postgresql db connection if sqlite_sync_file is not used
postgresql_connect=Driver={PostgresSQL Driver};Server=127.0.0.1;DB=<db>;Port=5432;connectTimeout=0
postgresql_uid=<acct>
@@ -526,14 +598,16 @@ Configuration#exclude_incident_fields=
# optionally include references within the incident to source org_id and incident_id. Values true/false
sync_reference_fields=True
- # true|false - specify whether to delete the target incident if the source incident is deleted. Default: false
+ # true|false - specify whether to delete the destination incident if the source incident is deleted. Default: false
delete_incidents=false
+ # destination syncs changes back to source (new artifacts, notes, etc.). Only editing an incident or a task is supported
+# sync_role_source = True
matching_incident_fields¶
-Use this capability to filter which incidents and it’s tasks, notes, artifacts, etc. are sent to the target organization. Below are a few examples for how to use this capability.
+Use this capability to filter which incidents and it’s tasks, notes, artifacts, etc. are sent to the destination organization. Below are a few examples for how to use this capability.
@@ -592,7 +666,7 @@ ResilientFeed Classhost, #proxy_host, api_key_id, api_key_secret, #email, #password, port, org, cafile
-Specify the connection values similar to the [resilient]
section for connection to the target SOAR environment
+Specify the connection values similar to the [resilient]
section for connection to the destination SOAR environment
sqlite_sync_file
/path/to/file
@@ -623,12 +697,16 @@ ResilientFeed ClassOptional semicolon separated list of fields and field sections to exclude when synchronizing an incident.
sync_reference_fields
-true|false
-Specify True
to add information to the target incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
+True|False
+Specify True
to add information to the destination incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
delete_incidents
true|false
-Specify ‘True’ to delete the target incident and it’s data when the source incident is deleted
+Specify ‘True’ to delete the destination incident and it’s data when the source incident is deleted
+
+sync_role_source
+true|false
+Specify ‘True’ to identify this SOAR organization as the source SOAR. ‘False’ represents the destination SOAR. The default is ‘True’.
@@ -639,50 +717,54 @@ Requirements
Integration Server¶
-If adding support for PostgreSQL, upgrade the package as:
+-
+
[sudo] pip install --upgrade rc_data_feed_plugin_resilientfeed-<version>.tar.gz[postgres]
Configuration[resilient_feed]
class=ResilientFeed
- # provide configuration information to the target QRadar SOAR and Organization
+ # provide configuration information to the destination QRadar SOAR and Organization
host=localhost
#proxy_host=
api_key_id=
@@ -513,7 +585,7 @@ Configurationorg=
cafile=false
# identify a sqlite db file to retain mapping between QRadar SOAR instances.
- sqlite_sync_file=/path/to/file
+ #sqlite_sync_file=/path/to/file
# postgresql db connection if sqlite_sync_file is not used
postgresql_connect=Driver={PostgresSQL Driver};Server=127.0.0.1;DB=<db>;Port=5432;connectTimeout=0
postgresql_uid=<acct>
@@ -526,14 +598,16 @@ Configuration#exclude_incident_fields=
# optionally include references within the incident to source org_id and incident_id. Values true/false
sync_reference_fields=True
- # true|false - specify whether to delete the target incident if the source incident is deleted. Default: false
+ # true|false - specify whether to delete the destination incident if the source incident is deleted. Default: false
delete_incidents=false
+ # destination syncs changes back to source (new artifacts, notes, etc.). Only editing an incident or a task is supported
+# sync_role_source = True
Configuration#exclude_incident_fields= # optionally include references within the incident to source org_id and incident_id. Values true/false sync_reference_fields=True - # true|false - specify whether to delete the target incident if the source incident is deleted. Default: false + # true|false - specify whether to delete the destination incident if the source incident is deleted. Default: false delete_incidents=false + # destination syncs changes back to source (new artifacts, notes, etc.). Only editing an incident or a task is supported +# sync_role_source = True
matching_incident_fields¶
-Use this capability to filter which incidents and it’s tasks, notes, artifacts, etc. are sent to the target organization. Below are a few examples for how to use this capability.
+Use this capability to filter which incidents and it’s tasks, notes, artifacts, etc. are sent to the destination organization. Below are a few examples for how to use this capability.
ResilientFeed Classhost, #proxy_host, api_key_id, api_key_secret, #email, #password, port, org, cafile
-Specify the connection values similar to the [resilient]
section for connection to the target SOAR environment
+Specify the connection values similar to the [resilient]
section for connection to the destination SOAR environment
sqlite_sync_file
/path/to/file
@@ -623,12 +697,16 @@ ResilientFeed ClassOptional semicolon separated list of fields and field sections to exclude when synchronizing an incident.
sync_reference_fields
-true|false
-Specify True
to add information to the target incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
+True|False
+Specify True
to add information to the destination incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
delete_incidents
true|false
-Specify ‘True’ to delete the target incident and it’s data when the source incident is deleted
+Specify ‘True’ to delete the destination incident and it’s data when the source incident is deleted
+
+sync_role_source
+true|false
+Specify ‘True’ to identify this SOAR organization as the source SOAR. ‘False’ represents the destination SOAR. The default is ‘True’.
@@ -639,50 +717,54 @@ Requirements
host, #proxy_host, api_key_id, api_key_secret, #email, #password, port, org, cafile
Specify the connection values similar to the [resilient]
section for connection to the target SOAR environment
Specify the connection values similar to the [resilient]
section for connection to the destination SOAR environment
sqlite_sync_file
/path/to/file
ResilientFeed ClassOptional semicolon separated list of fields and field sections to exclude when synchronizing an incident.
sync_reference_fields
true|false
Specify True
to add information to the target incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
True|False
Specify True
to add information to the destination incident to maintain the original org id incident id, sync host and incident create date. Fields are df_org_id
, df_inc_id
, df_host
and df_create_date
, respectively
delete_incidents
true|false
Specify ‘True’ to delete the target incident and it’s data when the source incident is deleted
Specify ‘True’ to delete the destination incident and it’s data when the source incident is deleted
sync_role_source
true|false
Specify ‘True’ to identify this SOAR organization as the source SOAR. ‘False’ represents the destination SOAR. The default is ‘True’.
Setup Steps¶
-
-
Ensure QRadar SOAR version requirements are met for both the source and destination instances.
-Perform the manual duplication of custom fields, incident types, phases and tasks, etc. by exporting these configurations from the QRadar SOAR source organization and importing them to the target QRadar SOAR organization.
-Manually recreate the users and groups needed in the target QRadar SOAR organization.
-Configure the app.config settings with the settings for the target QRadar SOAR organization and, optionally, the criteria for the types of incidents to synchronize and fields to exclude.
+Ensure QRadar SOAR version requirements are met for both the source and destination instances.
+Perform the manual duplication of custom fields, incident types, phases and tasks, etc. by exporting these configurations from the QRadar SOAR source organization and importing them to the destination QRadar SOAR organization.
+Manually recreate the users and groups needed in the destination QRadar SOAR organization.
+Configure the app.config settings with the settings for the destination QRadar SOAR organization and, optionally, the criteria for the types of incidents to synchronize and fields to exclude.
Run
resilient-circuits run
to confirm connectivity to both instances of QRadar SOAR (withreload=False
).
-The best way to test is to set
reload=False
under[feeds]
in your app.config file, and in the source QRadar SOAR organization, run theData Feeder: Sync Incidents
rule to synchronize a small number of incidents. Change the message destination of theData Feeder: Sync Incidents
function tofeed_data_resilient
as it may be used by another Data Feeder plugin.
+The best way to test is to set
reload=False
under[feeds]
in your app.config file, and in the source QRadar SOAR organization, run theData Feeder: Sync Incidents
rule to synchronize a small number of incidents. Change the message destination of theData Feeder: Sync Incidents
function tofeed_data_resilient
as it may be used by another Data Feeder plugin.
If using other Data Feeder plugins (ex. odbcfeed), change the Data Feeder rules to include the
feed_data_resilient
message destination. Restart resilient-circuits in order for the message destination change to take effect.
Bidirectional Synchronization¶
+Bidirectional synchronization is performed with the data feeder SOAR plugin installed in both SOAR/QRadar Suite organization environments, sharing the same database to track the relationship between incidents, tasks, notes, etc. When using App Host, this database must be PostgreSQL. PostgreSQL is also suggested in an Integration Server environment based on it’s robustness and performance.
+If bidirectional updates are unnecessary, then you only install this app in the source SOAR environment.
+When configuring each plugin, one SOAR organization is designated as the source (see app.config setting: sync_role_source) with the other designated as the destination. These identifiers are important to ensure database lookups are performed correctly when updating tasks, or creating new artifacts or notes in the destination SOAR organization.
+The database plays a key role to ensure there is no duplication of incidents, tasks, notes when creating or updating data. Special logic called debounce is used to ensure updates made from the either SOAR organization do not infinitely bounce back and forth between the two organizations. To that end, if there are repeated failures to persist data to the database, the app will shut down. Additional information can be found in the Troubleshooting section.
+Not all incident data changes made in the destination SOAR organization is returned to the source SOAR organization incident. For instance, changes to GDPR breach information may not synchronize. However, changes to the owner, severity, etc. and closing the incident will synchronize.
+Considerations¶
-
-
If real-time synchronization remains in place, changes in the source QRadar SOAR data will overwrite any changes made in the target QRadar SOAR organization data.
-Deleting a source incident, task, artifact, etc. will also delete the matching target information.
+For non bidirectional synchronization environments, if real-time synchronization remains in place, changes in the source QRadar SOAR data will overwrite any changes made in the destination QRadar SOAR organization data.
+Deleting a source incident, task, artifact, etc. will also delete the matching destination information.
Synchronization of incidents may fail if newer required fields were created that were not present on these older incidents. Same is true for newer ‘on close’ created fields. This can be overcome by changing the fields from
required
tooptional
.
-Unofficial timing tests shows creating incident data can take .3-.5 seconds each data type. Consider the time it will take for all your incident data to synchronize when using
reload=True
.
-Review the permissions given to target QRadar SOAR user or api_key. They must have the appropriate values to read and write incidents, artifacts, attachments, comments, milestones, tasks, and datatables. In addition, some incidents have wiki entries associated with them. Finally, older versions of QRadar SOAR also intermix simulation incidents. Provide the permission to create those as necessary.
+Unofficial timing tests shows creating incident data can take .3-.5 seconds per each data type. Consider the time it will take for all your incident data to synchronize when using
reload=True
.
+Review the permissions given to destination QRadar SOAR user or api_key. They must have the appropriate values to read and write incidents, artifacts, attachments, comments, milestones, tasks, and datatables. In addition, some incidents have wiki entries associated with them. Finally, older versions of QRadar SOAR also intermix simulation incidents. Provide the permission to create those as necessary.
Limitations¶
-
-
This solution is not intended for bidirectional synchronization.
-Unfortunately, the create date of the original incident is lost when the target incident is synchronized.
+Unfortunately, the create date of the original incident is lost when the destination incident is synchronized.
Timer data cannot be synchronized.
Presently, artifacts with custom artifact types cannot synchronize.
-Deleting a source incident task attachment presently doesn’t synchronize.
Incident email messages (via the built-in Inbound Email Connectors) do not synchronize.
-It’s presently not possible to restrict synchronization of incident elements such as tasks, artifacts, notes, etc. -when using
reload=all
and the actionData Feeder: Sync Incidents
.