Home

Main Window

This window contains all data management tools.

• Add/Edit/Delete - actions with users.
• Refresh - refresh the user list. You do not need to press this button. Added just in case...
• Info - show downloaded users and what exactly and how many were downloaded.
• Feed - show recently downloaded data
• Channels - open Channels form
• Saved posts - download saved posts
• Download - download operations
• Stop - stop downloading
• View - change view mode
• LOG - program error log and information

Buttons

Info menu

Info

This form will show downloaded items (sorted by latest download date). Each line of data contains information about how many images and videos have been downloaded for the user. If no media is downloaded, the user will not appear in this form.

Double-clicking on the data line will open the content folder.

• View - view modes:
  • Session - show downloaded results for this session only;
  • All - show downloaded results for all users, sorted by last downloaded date.
• Up / Down - list navigation buttons
• Find - find the selected user in the main window
• Refresh - update information list. You don't need to press this button. Added just in case...
• Clear - clear the information list of the current download session.

In the Session view, the list is updated automatically.

Hot keys:

• F2 - Down
• F3 - Up
• F4 - Find
• F5 - Refresh

Users download queue

This is a visualization of users' download queue.

Hot keys: Del - cancel the download of the selected user Ctrl+F - find the selected user in the main window

Missing posts

This form shows posts that have not been downloaded. This form is opened by clicking on the 'Missing posts' button in the 'Info' menu in the main window.

Buttons:

• Delete - delete selected posts
• Update - check the missing posts again and refill the list
• Download ALL - download all missing posts (not working, in development)
• Info - show post information

Context menu:

• Download - download selected post
• Open post - open post in browser
• Open user folder - open user folder
• Show post information - show post information
• Find user - find user in the main window
• Delete post - delete selected post

Hot keys:

• F1 - Info

User metrics

You can open this form by clicking on the 'User metrics' button in the 'Info' menu in the main window.

Here you can get summary information for each user you have, such as: total size, number of files and their total size (by file type), last download date, etc.

• Calculate - start calculation
• Cancel - cancel calculation
• View- view modes and options

Feed

This form will show the recently downloaded data (images and videos). Each image/video contains download information, file address, user information, etc.

Double-clicking on an image will open the original image in an external viewer.

You can also add posts to your Favorite feed and/or special feed(s) that you can create.

Buttons:

• Refresh - the feed will be updated to receive new downloaded items and/or remove missing items
• Clear - clear list of recent downloads
• Delete selected - delete marked items from the disk

Context menu:

• Open picture/video - open picture/video in external viewer
• Open user (with folder image) - open user folder
• Open user (with globe) - open user site
• Open post - open post in browser
• Add to Favorite - this post will be added to your favorite feed
• Add to special feed...- this post will be added to the feed(s) you select (a selection form will open)
• Remove from Favorite - this post will be removed from your favorite feed
• Remove from special feed... - this post will be removed from the feed(s) you select (a selection form will open)
• Find user - find user in the main window
• Information - show post information
• Delete picture/video - delete picture/video

Load sessions:

• Load current session
• Load last session - load last saved session data
• Select loading session - load the data of the session you have chosen (the session selection form will open)
• Load Favorite - load your Favorite feed
• Load special feed - load one or more of your special feeds
• Add checked to Favorite - add checked posts to your Favorite feed
• Remove checked from Favorite - remove checked posts from your Favorite feed
• Add checked to special feed... - add checked posts to one or more of your feeds
• Remove checked from special feed... - remove checked posts from one or more of your feeds
• Clear Favorite - remove all posts from your Favorite feed (just clearing the feed; files will not be deleted)
• Clear special feed... - remove all posts from one or more of your feeds (just clearing the feeds; files will not be deleted)
• Delete special feed...- delete one or more of your feeds (files will not be deleted)
• Delete daily feed (by list) - delete one or more session feeds (a selection form will open; files will not be deleted)
• Delete daily feed (by date) - delete session feeds by the date (the date selection form will open; files will not be deleted)

Change view:

• Downloads - show downloaded regular users
• Subscriptions - show downloaded subscriptions

Hot-keys:

• F5 - refresh
• Ctrl+G - go to a specific page

The buttons are only displayed if the Store session data option is enabled

Hot keys

• F5 - Refresh
• F3 - Previous page
• F4 - Next page

You can read about the settings here.

Channels

The channels form will open.

More about channels here.

Subscriptions

You can add profiles in subscription mode (if available). This mode means that files will not be downloaded. Instead, a video preview (screenshot) will be loaded. You can choose what to download, open a post, etc. The download goes through a standalone downloader. You can see downloaded subscriptions in the feed.

Saved posts

SITES REQUIREMENTS

Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the Saved posts user textbox. Click OK. Go to the main window. If this option is not displayed in the site settings, it doesn't need to be configured.

This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the !Saved posts folder.

You may see a message like Host [Site] is not ready or Host [Site] is unavailable. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time.

Download

• Download selected (F5) - start downloading one or more selected users.
• Download all: ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!
  • Download all (F6) - Download all users marked Ready for download from all sites.
  • Download all site users - Download all users marked Ready for download from specific sites.
  • Download all FULL - Download all users from all sites. The Ready for download option will be ignored.
  • Download all site users FULL - Download all users from specific sites. The Ready for download option will be ignored.
• Download (advanced) - Filter the users you want to download and download them.
• Standalone downloader - Just a downloader for single videos.
• Add a new download group - Create a new download group (read more here).
• Silent mode - Temporarily disabling notifications.
• Automation - Automatic download settings (read more here).
• Pause automation - Pause the automation process.
• Stop - stop all download operations.

HotKeys

Pause automation

This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.

• 1 hour - pause automation for 1 hour
• 2 hour - pause automation for 2 hours
• 3 hour - pause automation for 3 hours
• 4 hour - pause automation for 4 hours
• 6 hour - pause automation for 6 hours
• 12 hour - pause automation for 12 hours
• Until - you will be prompted to enter the time you want to pause the task(s)
• Unlimited - pause the task until you turn it off or close the program
• Disable - disable pause

View

• View - users view modes.
  • The first block specifies how users will be displayed in the form.
    • Large/Small images - with icons.
    • List / Details - simple, without icons.
  • The second block specifies what types of user will be displayed in the form.
    • Show users - show regular users in user list.
    • Show subscriptions - show subscriptions in user list.
  • The third block specifies the user sites that will be displayed in the form.
    • All - default
    • Specific sites - select sites to be displayed
  • The fourth block specifies which users will be displayed in the form.
    • All - default.
    • Regular - only show users who are not marked as Favorite or Temporary.
    • Temporary - only show users marked as temporary.
    • Favorite - only show users marked as favorite.
    • Deleted - only show users whose profiles have been deleted
    • Suspended - only show users whose profiles have been suspended
    • Labels - show users labeled with one or more selected label
    • No Labels - only show users without a label
    • Excluded labels - exclude from displaying users with the selected labels (this is suitable for situations where you want to hide a user but don't want to delete)
    • Ignore excluded labels - Ignore excluding labels. This allows you to show users that should be excluded from display without resetting the current selection of excluded labels.
    • Show groups instead of labels - show sites groups instead of labels (when filtering users by labels)
    • Advanced filter - advanced filter of users you want to display
  • The fifth block specifies the date range of the last download, whose users will be displayed in the form.
    • Limit dates (not in range) - filter users whose last download date is not in the selected date range
    • Limit dates (in range) - filter users whose last download date is in the selected date range

LOG

The error log will be opened by clicking on this button. The log contains program and download errors. The log is saved automatically when you close the program, but you still can save, copy, or clear it (using the special buttons on the log form) if you want.

Bug report

You can submit a bug report anonymously, or prepare it for GitHub or email. You can find it in the top right corner of the main window (message icon).

Fields:

• Describe the bug or write your message - a clear and concise description of what the bug is.
• Profile URL
• Post URL
• To Reproduce - steps to reproduce the behavior: do something, see error.
• Expected behavior - a clear and concise description of what you expected to happen.
• Log data
• Files - attach files to your message (only works with anonymous message).

Buttons:

• Anon message - send an anonymous message. The developer will not be able you contact you back. You can attach files (images, photos) to your message. If you would like a response from the developer, response, please add your contact details (email, Discord, etc.). You can also use this button to say something nice to the developer. 😊
• email - create a message to send via email.
• GitHub - create a MarkDown message to post to GitHub.
• Copy - create a message and copy to your clipboard.

Hot keys

• F1 - program information and checking for a new version
• Insert - add user
• Del - delete one or more selected users
• Enter - open user folder
• F3 - edit user
• F5, Ctrl+F5 - start downloading one or more selected users (pressing with Ctrl excludes this data from the feed)
• F6, Ctrl+F6 - start downloading all your users marked as Ready for download (pressing with Ctrl excludes this data from the feed)
• Ctrl+F - open search form
• Ctrl+NumericKey - download group with this number
• Alt+A - open scheduler
• Alt+P - open progress form

Double-click the user to open the content folder.

Download separate video

Read here

Tray

• Silent mode - Temporarily disabling notifications.
• Pause automation - Pause the automation process.
• Channels
• Feed
• Downloader
• Show/Hide - show/hide SCrawler
• Close - close the program
• Close (no script) - close the program ignore the script (if you configured the script in the settings)

Search users

Ctrl+F to start searching

Users list

Add user

In the main window, click the Add button to open the user creation form.

Paste the copied URL into the User name field. Ignore if the username is set to something other than the actual User name. For some sites, this is necessary for internal purposes. If the site is identified, don't change the User name manually.

If you want to add a folder as a user, you need to use the following pattern: D:\Data\.

• User name - Paste the user profile URL here.
• Friendly name - if configured, this name will be displayed in the user list instead of the username (if this field is green, it means that the user site name is shown). F button to set the friendly name you configured; S button to set site name.
• Special path - specify a path other than the default path for user data. If you set an absolute path (for example, D:\Data\), that path will be used as the user's path. If you set a relative path (for example, SomeFolder, SomeFolder\SomeFolder2), then this path will be created relative to the root folder for this site (configured in the site settings).
• Subscription - read about subscriptions here
• Sites combo box - site pointers.
• Options - additional user options.
• Account - if you've configured more than one account for the selected site, this field will be active. You can select the account through which the user will be downloaded.
• Temporary - mark user as Temporary.
• Favorite - mark user as Favorite.
• Download images/videos - for the created user, only the selected media types will be downloaded.
• Ready for download - download this user in the operation Download all.
• Get user media only - download all posts posted by this author.
• Add by list - add users' profiles by list (new line as separator (Shift+Enter to insert new line)).
  • If you add users by the URL, you must check the Auto detect site checkbox;
  • If you are adding users by username, you must select the site in the combo box.
• Auto detect site - add users by the URLs.
• Description - some information you want to keep. If the Add by list checkbox is checked, you must enter a list of users in this textbox (Shift+Enter to insert new line);
• Labels - labels associated with this user (used to filter users in the main window).
• Color - you can select custom user highlighting in the main window and feed.
• Script - If the checkbox is checked, script to be executed after the user download is complete. If the text is empty, the script from the global settings will be executed. How to use the script. If neither Temporary nor Favorite is checked, the user will be displayed in the regular site group.

In collection editing mode checkboxes have three states. When loaded, an indeterminate state means that profiles have different values for this option. When you save, checkboxes that are in indeterminate state will not be applied.

Labels

In this form, you can create labels and select them for users.

Click the + button to add a new one.

You can pick up one or more labels in this form.

Hot keys

• F2 - open advanced options
• F4 - open labels form for pick up a label

User colors

• Users whose profile has been deleted are marked in red.
• Users whose profile has been suspended are marked in yellow.
• Collections in which all profiles have been deleted are marked in red.
• Collections that contain active profiles and deleted profiles are marked in blue.

Context menu

• Download data - download this user/collection data.
• Download data limited - download this user/collection data only the specified number of posts.
• Download data to the date - download this user/collection data up to selected date.
• Edit user information - open a form to change user parameters.
• Delete user/collection - delete one or more selected users/collections.
• Erase data - delete all user data and/or download history
• Copy data to another folder - allows you to copy user data to another folder.
• Change favorite - if the user was marked as favorite, he becomes a regular; otherwise, he becomes the favorite.
• Change temporary - if the user was marked as temporary, he becomes a regular; otherwise, he becomes the temporary.
• Change ready for download - download this user in the operation Download all.
• Change labels - change user labels.
• Change script usage - change script usage for selected users.
• Add to collection - add user to collection.
• Merge collection files - move files to collection root folder.
• Change folder - change user data folder
• Change image - change user icon.
• Open contains folder - open selected user/collection folder.
• Open site - open user profile.
• Information - some user information.

Change user location

This window gives you the option to select the user's exact location.

When you change the locations of a collection, the Collection name textbox also appears. You must select the location of the collection and set the collection name!

If the Global path checkbox is checked, the selected location will be added to the global locations for quick selection in the future.

Redownload user

Method 1

Right-click on the user, then click Erase data. Then download this user again.

Method 2

Delete the user and add it again.

Method 3

1. Open the user settings folder.
2. Find the ....._Data.xml and ....._Posts.txt files.
3. Delete these files.
4. Restart SCrawler.
5. Download these users again.

Deleting a user

SCrawler has advanced deletion options. It can delete a user/collection with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options.

graph TB
style C fill:#f00,color:white,font-weight:bold
subgraph Delete user
A[Delete]
A -->|With ban|A1[Enter ban reason]
A -->|Without ban|B[Process]
A -->|User only|Q[Delete user/collection from SCrawler]
A1 --> B
B -->|Is user| C(Delete files)
C--> Q
Q --> Z[Exit]
end
subgraph Delete collection
B-->|Is collection| CC[Collection deletion confirmation message]
CC-->|Delete| C
CC-->|Split| E[Split collection]
CC-->|Cancel| Z
E-->EE[Extract users and add them to SCrawler as regular users]
EE-->Q
end

Loading

Collections

I've developed collections for users with multiple profiles, but you can use them however you like.

To add a user to the collection, click Add to collection in the context menu. Add a new one by clicking the + button, or select an existing one.

When you add a user to a collection, the user's data will be moved to the collection folder with the name you specified.

You can select multiple users and a collection to add users to this collection.

Merging collection files

When you add a user to the collection, a new folder will be created in the collection folder named by pattern Site_UserName. That is, each profile in the collection is stored separately. When you merge, all files in the profiles' folders are moved to the root of the collection.

Deleting a collection

1. Select the collection you want to delete.
2. Click the Delete button or right-click on the collection and select Delete user / collection.
   • Delete - The collection and all its files will be permanently deleted.
   • Split - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
   • Cancel - Cancel operation.

Go to a diagram that illustrates all the available options.

Collections example

You have two users: SomeUserName1 (from Reddit) and SomeUserName1 (from Twitter).

1. Creation of the First collection collection.
2. The folder First collection will be created in the collections folder (you specify it in the program settings).
3. The SomeUserName1 (Reddit) folder will be moved to the First collection folder and renamed to Reddit_SomeUserName1.
4. The SomeUserName1 (Twitter) folder will be moved to the First collection folder and renamed to Twitter_SomeUserName1.

If you want to merge the collection.

Files from the Reddit_SomeUserName1 and Twitter_SomeUserName1 folders will be moved to the First collection folder.

Virtual mode

You can create a virtual collection. This means that this collection will only exist in SCrawler. The collection folder will not be created. The user files of users included in this type of collection, will not be moved.

You can also virtually add a user to a real collection. In this case, the user's files will not be moved to the collection folder.

Adding a user to a virtual collection is possible only in virtual mode.

Import users

Settings-Import users

You can import users that are not in SCrawler from the specific folder. You can also scan existing SCrawler directories to find missing users.

This feature is not for importing users from the site. It's more like searching for missing users.

Black list

The program has the Black list, which included users you have added to. It is only a blacklist of the program and no impact on the site!

Users in this list will be skipped when downloading channels. If you add a new user to the program and this user is already blacklisted, you will receive a notification about. The notification includes the username and the reason (if specified) for deletion.

The program implements two ways to add users to this list:

• with a reason (you can specify the reason for the ban for cases if you try to add this user again);
• with no reason.

Move the SCrawler data to a new location.

1. Close SCrawler.
2. Move the data to a new location.
3. In the SCrawler settings folder, find the Settings.xml file.
   1. Change the value of the GlobalPath node.
   2. For each site in this file (under the Sites node) where it exists, change the value of the Path node to the new location. Example: Sites - Instagram - Path.
   3. Save this file and close.
4. Open the Users.xml file (in the SCrawler settings folder).
   1. Check the SpecialPath values to make sure they still point to the correct locations. If not, replace these values with the correct ones. To find all non-empty SpecialPath values, you can use a simple regular expression: SpecialPath="[^"]+".
   2. Save this file and close.
5. Open SCrawler.