3.0.0.5

Added options for Reddit channel and user
Fixed minor bugs and typos

.github/ISSUE_TEMPLATE/bug_report.md

@@ -26,8 +26,10 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Release information (please complete the following information):**
+ - OS [e.g. Windows 10, Windows 11]
  - Architecture [e.g. x86, x64]
  - Version [e.g. 2.0.0.0]
+ - NET.Framework version
 
 **Additional context**
 Add any other context about the problem here.

Changelog.md

@@ -1,3 +1,8 @@
+# 3.0.0.5
+
+- Added
+  - ```New```, ```Hot```, ```Top``` Reddit channel and user download modes
+
 # 3.0.0.4
 
 - Fixed

FAQ.md

@@ -2,6 +2,8 @@
 
 **Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
 
+**Also read [here](README.md) for basic information.**
+
 Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
 
 Any other questions I will keep in this file.
@@ -44,6 +46,30 @@ A: Check your credentials. Both of these sites require cookies. Check your [Twit
 
 ----
 
+#### Q: **I have set credentials but still nothing is downloading**
+
+A: Click the ```Start downloading``` button
+
+----
+
+#### Q: **Where can I find the release?**
+
+A: https://github.com/AAndyProgram/SCrawler/releases/latest
+
+----
+
+#### Q: **How to run the program?**
+
+A: Double-click ```SCrawler.exe```
+
+----
+
+#### Q: **Where to find binaries?**
+
+A: https://github.com/AAndyProgram/SCrawler/releases/latest
+
+----
+
 #### Q: **Does the program remember the last download and check for new posts, downloading only new posts? Or does the program download the entire profile every time?**
 
 A: The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
@@ -64,4 +90,10 @@ A: There is no functionality to remove an individual label. You can open the ```
 
 #### Q: **How to remove a user from the blacklist**
 
-A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist```` button.
+A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist``` button.
+
+----
+
+#### Q: **Can you add a step-by-step guide or video on how to use the program?**
+
+A: **NO**! I will not do it. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.

README.md

@@ -75,7 +75,7 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
 
 # Requirements
 
-- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
+- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
 - Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for Twitter (if you want to download data from Twitter)
 - Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for Instagram (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for saved Instagram posts, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for Stories and Tagged data
 - ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)). **ffmpeg only works with the x64 version of the program.**
@@ -116,7 +116,7 @@ You need to set up authorization for Twitter and Instagram:
 - Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for **Twitter** (if you want to download data from Twitter)
 - Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for **Instagram** (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for **saved Instagram posts**, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for **Stories** and **Tagged data**
 
-Just add a user profile and click the ```Start downloading``` button.
+Just add a user profile and **click the ```Start downloading``` button**.
 
 You can add users by patterns:
 - https://www.instagram.com/SomeUserName

SCrawler/API/Base/UserDataBase.vb

@@ -248,7 +248,7 @@ BlockPictureFolder:
             End If
 BlockPictureScan:
             On Error GoTo BlockDeletePictureFolder
-            Dim NewPicFile As SFile = SFile.GetFiles(MyFile.CutPath, "*.jpg|*.png",,
+            Dim NewPicFile As SFile = SFile.GetFiles(MyFile.CutPath, "*.jpg|*.jpeg|*.png",,
                                                      New ErrorsDescriber(EDP.ReturnValue) With {
                                                          .ReturnValue = New List(Of SFile),
                                                          .ReturnValueExists = True}).FirstOrDefault
@@ -1088,25 +1088,6 @@ BlockNullPicture:
             Else
                 Return IIf(FriendlyName.IsEmptyString, Name, FriendlyName)
             End If
-            'If Settings.ViewModeIsPicture Then
-            '    If IsCollection Then
-            '        Return CollectionName
-            '    Else
-            '        Return IIf(FriendlyName.IsEmptyString, Name, FriendlyName)
-            '    End If
-            'Else
-            '    Dim t$ = String.Empty
-            '    If Temporary Then
-            '        t = " (T)"
-            '    ElseIf Favorite Then
-            '        t = " (F)"
-            '    End If
-            '    If IsCollection Then
-            '        Return $"Collection [{CollectionName}]{t}"
-            '    Else
-            '        Return $"[{Site}]{t} {IIf(FriendlyName.IsEmptyString, Name, FriendlyName)}"
-            '    End If
-            'End If
         End Function
 #Region "Buttons actions"
         Private Sub BTT_CONTEXT_DOWN_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_DOWN.Click

SCrawler/API/Instagram/UserData.vb

@@ -138,7 +138,7 @@ Namespace API.Instagram
         End Function
         Private Sub ReconfigureAwaiter()
             If WaitNotificationMode = WNM.SkipTemp Then WaitNotificationMode = WNM.Notify
-            If Caught429 Then Caught429 = False ': RequestsCount = 0
+            If Caught429 Then Caught429 = False
             ProgressTempSet = False
         End Sub
         Private Sub NextRequest(ByVal StartWait As Boolean)

SCrawler/API/Reddit/Channel.vb

@@ -12,9 +12,12 @@ Imports PersonalUtilities.Functions.XML
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin.Hosts
 Imports System.Threading
+Imports SCrawler.API.Reddit.RedditViewExchange
+Imports View = SCrawler.API.Reddit.IRedditView.View
+Imports Period = SCrawler.API.Reddit.IRedditView.Period
 Namespace API.Reddit
     Friend Class Channel : Implements ICollection(Of UserPost), IEquatable(Of Channel), IComparable(Of Channel),
-            IRangeSwitcherContainer(Of UserPost), ILoaderSaver, IMyEnumerator(Of UserPost), IChannelLimits, IDisposable
+            IRangeSwitcherContainer(Of UserPost), ILoaderSaver, IMyEnumerator(Of UserPost), IChannelLimits, IRedditView, IDisposable
 #Region "XML Nodes' Names"
         Private Const Name_Name As String = "Name"
         Private Const Name_ID As String = "ID"
@@ -35,6 +38,7 @@ Namespace API.Reddit
         End Property
         Friend ReadOnly Property PostsLatest As List(Of UserPost)
         Friend ReadOnly Property Posts As List(Of UserPost)
+        Friend ReadOnly Property PostsNames As List(Of String)
         Friend ReadOnly Property PostsAll As List(Of UserPost)
             Get
                 Return ListAddList(Nothing, Posts).ListAddList(PostsLatest).ListSort
@@ -57,6 +61,14 @@ Namespace API.Reddit
                 Return $"{ChannelsCollection.ChannelsPath.PathWithSeparator}{ID}.xml"
             End Get
         End Property
+        Private ReadOnly Property FilePosts As SFile
+            Get
+                Dim f As SFile = File
+                f.Name &= "_Posts"
+                f.Extension = "txt"
+                Return f
+            End Get
+        End Property
         Friend ReadOnly Property CachePath As SFile
             Get
                 Return $"{ChannelsCollection.ChannelsPathCache.PathWithSeparator}{ID}\"
@@ -73,6 +85,14 @@ Namespace API.Reddit
             End Get
         End Property
         Private ReadOnly Property Range As RangeSwitcher(Of UserPost)
+        Friend Property ViewMode As View = View.New Implements IRedditView.ViewMode
+        Friend Property ViewPeriod As Period = Period.All Implements IRedditView.ViewPeriod
+        Friend Sub SetView(ByVal Options As IRedditView) Implements IRedditView.SetView
+            If Not Options Is Nothing Then
+                ViewMode = Options.ViewMode
+                ViewPeriod = Options.ViewPeriod
+            End If
+        End Sub
 #Region "Statistics support"
         Private ReadOnly CountOfAddedUsers As List(Of Integer)
         Private ReadOnly CountOfLoadedPostsPerSession As List(Of Integer)
@@ -118,16 +138,20 @@ Namespace API.Reddit
         Private _DownloadLimitCount As Integer? = Nothing
         Friend Property DownloadLimitCount As Integer? Implements IChannelLimits.DownloadLimitCount
             Get
-                If AutoGetLimits Then
-                    If LatestParsedDate.HasValue OrElse Not DownloadLimitPost.IsEmptyString Then
-                        Return Nothing
-                    ElseIf _DownloadLimitCount.HasValue Then
-                        Return _DownloadLimitCount
+                If Not ViewMode = View.New And AutoGetLimits Then
+                    Return _DownloadLimitCount
+                Else
+                    If AutoGetLimits Then
+                        If LatestParsedDate.HasValue OrElse Not DownloadLimitPost.IsEmptyString Then
+                            Return Nothing
+                        ElseIf _DownloadLimitCount.HasValue Then
+                            Return _DownloadLimitCount
+                        Else
+                            Return DefaultDownloadLimitCount
+                        End If
                     Else
-                        Return DefaultDownloadLimitCount
+                        Return _DownloadLimitCount
                     End If
-                Else
-                    Return _DownloadLimitCount
                 End If
             End Get
             Set(ByVal NewLimit As Integer?)
@@ -137,11 +161,15 @@ Namespace API.Reddit
         Private _DownloadLimitPost As String = String.Empty
         Friend Property DownloadLimitPost As String Implements IChannelLimits.DownloadLimitPost
             Get
-                Dim PID$ = ListAddList(Nothing, Posts, LAP.NotContainsOnly).ListAddList(PostsLatest, LAP.NotContainsOnly).ListSort.FirstOrDefault.ID
-                If AutoGetLimits And Not PID.IsEmptyString Then
-                    Return PID
-                Else
+                If Not ViewMode = View.New And AutoGetLimits Then
                     Return _DownloadLimitPost
+                Else
+                    Dim PID$ = ListAddList(Nothing, Posts, LAP.NotContainsOnly).ListAddList(PostsLatest, LAP.NotContainsOnly).ListSort.FirstOrDefault.ID
+                    If AutoGetLimits And Not PID.IsEmptyString Then
+                        Return PID
+                    Else
+                        Return _DownloadLimitPost
+                    End If
                 End If
             End Get
             Set(ByVal NewLimit As String)
@@ -151,10 +179,14 @@ Namespace API.Reddit
         Private _DownloadLimitDate As Date? = Nothing
         Friend Property DownloadLimitDate As Date? Implements IChannelLimits.DownloadLimitDate
             Get
-                If AutoGetLimits And LatestParsedDate.HasValue Then
-                    Return LatestParsedDate
-                Else
+                If Not ViewMode = View.New And AutoGetLimits Then
                     Return _DownloadLimitDate
+                Else
+                    If AutoGetLimits And LatestParsedDate.HasValue Then
+                        Return LatestParsedDate
+                    Else
+                        Return _DownloadLimitDate
+                    End If
                 End If
             End Get
             Set(ByVal NewLimit As Date?)
@@ -174,13 +206,19 @@ Namespace API.Reddit
                 DownloadLimitDate = .DownloadLimitDate
                 AutoGetLimits = .AutoGetLimits
             End With
+            If Not ViewMode = View.New And AutoGetLimits Then
+                DownloadLimitDate = Nothing
+                DownloadLimitCount = Nothing
+                DownloadLimitPost = String.Empty
+            End If
         End Sub
         Friend Property AutoGetLimits As Boolean = True Implements IChannelLimits.AutoGetLimits
 #End Region
         Friend ReadOnly Property HOST As SettingsHost
         Friend Sub New()
             Posts = New List(Of UserPost)
             PostsLatest = New List(Of UserPost)
+            PostsNames = New List(Of String)
             Range = New RangeSwitcher(Of UserPost)(Me)
             CountOfAddedUsers = New List(Of Integer)
             CountOfLoadedPostsPerSession = New List(Of Integer)
@@ -203,6 +241,7 @@ Namespace API.Reddit
         End Function
         Friend Sub Delete()
             File.Delete(, SFODelete.DeleteToRecycleBin)
+            FilePosts.Delete(, SFODelete.DeleteToRecycleBin)
         End Sub
         Friend Sub DownloadData(ByVal Token As CancellationToken, Optional ByVal SkipExists As Boolean = True,
                                 Optional ByVal p As MyProgress = Nothing)
@@ -214,10 +253,13 @@ Namespace API.Reddit
                     .SkipExistsUsers = SkipExists,
                     .ChannelInfo = Me
                 }
-                    d.SetEnvironment(HOST, CUser, False)
-                    d.RemoveUpdateHandlers()
-                    d.SetLimit(Me)
-                    d.DownloadData(Token)
+                    With d
+                        .SetEnvironment(HOST, CUser, False)
+                        .RemoveUpdateHandlers()
+                        .SetLimit(Me)
+                        .SetView(Me)
+                        .DownloadData(Token)
+                    End With
                     Dim b% = Posts.Count
                     Posts.ListAddList(d.GetNewChannelPosts(), LAP.NotContainsOnly)
                     If Posts.Count - b > 0 Then CountOfLoadedPostsPerSession.Add(Posts.Count - b)
@@ -298,6 +340,9 @@ Namespace API.Reddit
                         Dim lc As New ListAddParams(LAP.ClearBeforeAdd)
                         Name = x.Value(Name_Name)
                         ID = x.Value(Name_ID)
+                        ViewMode = x.Value(Name_ViewMode).FromXML(Of Integer)(CInt(View.[New]))
+                        ViewPeriod = x.Value(Name_ViewPeriod).FromXML(Of Integer)(CInt(Period.All))
+                        If FilePosts.Exists Then PostsNames.ListAddList(FilePosts.GetText.StringToList(Of String)("|"), LNC)
                         LatestParsedDate = AConvert(Of Date)(x.Value(Name_Date), XMLDateProvider, Nothing)
                         CountOfAddedUsers.ListAddList(x.Value(Name_UsersAdded).StringToList(Of Integer)("|"), lc)
                         CountOfLoadedPostsPerSession.ListAddList(x.Value(Name_PostsDownloaded).StringToList(Of Integer)("|"), lc)
@@ -317,19 +362,27 @@ Namespace API.Reddit
         Friend Overloads Function Save(Optional ByVal f As SFile = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Boolean Implements ILoaderSaver.Save
             Dim XMLDateProvider As New ADateTime(ADateTime.Formats.BaseDateTime)
             UpdateUsersStats()
+            If Not ViewMode = View.New Then
+                Dim l As New List(Of String)
+                If Posts.Count > 0 Or PostsLatest.Count > 0 Then l.ListAddList((From p In PostsAll Where Not p.ID.IsEmptyString Select p.ID), LAP.NotContainsOnly)
+                l.ListAddList(PostsNames, LAP.NotContainsOnly)
+                If l.Count > 0 Then TextSaver.SaveTextToFile(l.ListToString(, "|"), FilePosts, True,, EDP.SendInLog)
+            End If
             Using x As New XmlFile With {.AllowSameNames = True, .Name = "Channel"}
                 x.Add(Name_Name, Name)
                 x.Add(Name_ID, ID)
+                x.Add(Name_ViewMode, CInt(ViewMode))
+                x.Add(Name_ViewPeriod, CInt(ViewPeriod))
+                x.Add(Name_UsersAdded, CountOfAddedUsers.ListToString(, "|"))
+                x.Add(Name_PostsDownloaded, CountOfLoadedPostsPerSession.ListToString(, "|"))
+                x.Add(Name_UsersExistent, ChannelExistentUserNames.ListToString(, "|"))
                 If Posts.Count > 0 Or PostsLatest.Count > 0 Then
                     Dim tmpPostList As List(Of UserPost) = Nothing
                     tmpPostList.ListAddList(Posts).ListAddList(PostsLatest)
                     tmpPostList.Sort()
                     LatestParsedDate = tmpPostList.FirstOrDefault(Function(pd) pd.Date.HasValue).Date
                     x.Add(Name_Date, AConvert(Of String)(LatestParsedDate, XMLDateProvider, String.Empty))
                     x.Add(Name_PostsNode, String.Empty)
-                    x.Add(Name_UsersAdded, CountOfAddedUsers.ListToString(, "|"))
-                    x.Add(Name_PostsDownloaded, CountOfLoadedPostsPerSession.ListToString(, "|"))
-                    x.Add(Name_UsersExistent, ChannelExistentUserNames.ListToString(, "|"))
                     With x(Name_PostsNode)
                         tmpPostList.Take(200).ToList.ForEach(Sub(p) .Add(New EContainer("Post",
                                                                                   String.Empty,
@@ -354,6 +407,7 @@ Namespace API.Reddit
                 If disposing Then
                     Posts.Clear()
                     PostsLatest.Clear()
+                    PostsNames.Clear()
                     CountOfAddedUsers.Clear()
                     CountOfLoadedPostsPerSession.Clear()
                     Range.Dispose()