2024.7.24.0

YT YouTubeSettings: add 'DefaultVideoConvertNonAVC' property; fix 'OpenFolderInOtherProgram' property serialize and reset; add 'MusicPlaylistCreate_CreationMode' property Update the 'CleanFileName' function to remove line breaks Add the ability to convert non-avc codecs to avc Add 'M3U8CreationMode' enum SCrawler API.OnlyFans: fix incorrect delimiter (rules parsing) API.Threads: add saved posts downloading Feed: add the hotkeys 'Esc' and 'Ctrl+W' to close the form; add the ability to search for missing files in special feeds Scheduler: add the ability to execute a script after the scheduler plan is executed Settings: add enable/disable the use of the 'Esc' to close the feed; add 'AutomationScript' and 'AutomationScript_ExcludeManual' properties MainFrame: add the hotkey 'Ctrl+F' to show the feed; change the hotkey from 'Ctrl+F' to 'Alt+F' to show the search form
AAndyProgram · Jul 24, 2024 · 0b0933b · 0b0933b
1 parent 3ce9c55
commit 0b0933b
Show file tree

Hide file tree

Showing 25 changed files with 721 additions and 338 deletions.
diff --git a/Changelog.md b/Changelog.md
@@ -1,3 +1,26 @@
+# 2024.7.24.0
+
+*2024-07-24*
+
+- Added
+ - YouTube (standalone app)
+ - ability to convert non-`AVC` codecs (eg `VP9`) to `AVC` (`Settings` - `Defaults Video` - `Convert non-AVC codecs to AVC`)
+ - add the ability to set the playlist creation mode: absolute links, relative links, or both (`Settings` - `Music` - `Create M3U8: creation mode`)
+ - Threads: **saved posts downloading**
+ - Feed
+ - hotkeys `Esc` and `Ctrl+W` to close the form
+ - the ability to search for missing files in *special feeds*
+ - Scheduler: the ability to execute a script after the scheduler plan is executed *(`Settings` - `Behavior`)*
+ - Main window:
+ - added hotkey `Ctrl+F` to show the feed
+ - changed the hotkey from `Ctrl+F` to `Alt+F` to show the search form
+- Updated
+ - yt-dlp up to version **2024.07.16**
+- Fixed
+ - YouTube (standalone app): video files with line breaks in the name do not download correctly
+ - OnlyFans: rules parsing bug
+ - Minor bugs
+
 # 2024.6.25.0
 
 *2024-06-25*

diff --git a/ProgramScreenshots/SettingsGlobalBehavior.png b/ProgramScreenshots/SettingsGlobalBehavior.png
diff --git a/ProgramScreenshots/SettingsGlobalFeed.png b/ProgramScreenshots/SettingsGlobalFeed.png
diff --git a/README.md b/README.md
@@ -33,16 +33,16 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
 ![YouTube application](ProgramScreenshots/AppYouTube.png)
 
 # What can program do:
-- Download pictures and videos from users' profiles and subreddits:
+- Download pictures and videos from user profiles:
  - YouTube videos, shorts, community feeds, users, artists, playlists, music, tracks;
  - Reddit images, galleries of images, videos, saved posts;
- - Redgifs videos (https://www.redgifs.com/);
+ - Redgifs images and videos (https://www.redgifs.com/);
  - Twitter images and videos, saved (bookmarked) posts, likes, communities;
  - OnlyFans images and videos, saved (bookmarked) posts, stories;
  - JustForFans images and videos, saved (bookmarked) posts;
  - Mastodon images and videos, saved (bookmarked) posts;
  - Instagram images and videos, tagged posts, stories, saved posts;
- - Threads images and videos;
+ - Threads images and videos, saved posts;
  - Facebook images and videos, stories, saved posts;
  - TikTok videos;
  - Pinterest boards, users, saved posts;
@@ -57,7 +57,7 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
 - Download [saved posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
 - Add users from parsed channel
 - **Advanced user management**
-- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
+- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every `X` minutes)
 - **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files and subscriptions posts)
 - Multiple accounts support
 - Labeling users
@@ -87,7 +87,7 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
 - TikTok
 - RedGifs
 - Pinterest
-- Imgur
+- Imgur *(out of support)*
 - Gfycat
 - LPSG
 - **PornHub**
@@ -109,7 +109,7 @@ First, the program downloads the full profile. After the program downloads only
 
 # Requirements
 
-- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
+- **Windows 10, 11** with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
 - **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
 # Guide
@@ -157,7 +157,7 @@ First, the program downloads the full profile. After the program downloads only
 
 **Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
 
-**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
+**Don't put program in the `Program Files` system folder (this is portable program and program settings are stored in the program folder)**
 
 **I highly doubt you can run SCrawler on Linux or Mac. SCrawler is a program that is heavily dependent on Windows.**
 
@@ -183,7 +183,7 @@ The program has an intuitive interface.
 
 [![How to configure](https://img.youtube.com/vi/XDn7zG4I700/0.jpg)](https://www.youtube.com/watch?v=XDn7zG4I700)
 
-Just add a user profile and **click the ```Download``` button**.
+Just add a user profile and **click the `Download` button**.
 
 ```mermaid
 stateDiagram

diff --git a/SCrawler.YouTube/Base/Structures.vb b/SCrawler.YouTube/Base/Structures.vb
@@ -84,6 +84,11 @@ Namespace API.YouTube.Base
  Before = 1
  After = 2
  End Enum
+ Public Enum M3U8CreationMode As Integer
+ Relative = 0
+ Absolute = 1
+ Both = 2
+ End Enum
  Public Structure MediaObject : Implements IIndexable, IComparable(Of MediaObject)
  Public Type As Plugin.UserMediaTypes
  Public ID As String

diff --git a/SCrawler.YouTube/Base/YouTubeSettings.vb b/SCrawler.YouTube/Base/YouTubeSettings.vb
@@ -131,22 +131,28 @@ Namespace API.YouTube.Base
  <Browsable(True), GridVisible(False), Category("EnvironmentFolder"), DisplayName("Open folders in another program"), DefaultValue(False)>
  Private Property IDownloaderSettings_OpenFolderInOtherProgram As Boolean Implements IDownloaderSettings.OpenFolderInOtherProgram
  Get
- Return OpenFolderInOtherProgram.Use
+ Return OpenFolderInOtherProgram.Attribute.ValueTemp
  End Get
  Set(ByVal use As Boolean)
- OpenFolderInOtherProgram.Use = use
+ OpenFolderInOtherProgram.Attribute.ValueTemp = use
  End Set
  End Property
+ Private Function ShouldSerializeIDownloaderSettings_OpenFolderInOtherProgram() As Boolean
+ Return DirectCast(OpenFolderInOtherProgram.Attribute, IGridValue).ShouldSerializeValue
+ End Function
  <Browsable(True), GridVisible(False), Category("EnvironmentFolder"), DisplayName("Open folders in another program (command)"),
  Description("The command to open a folder."), DefaultValue("")>
  Private Property IDownloaderSettings_OpenFolderInOtherProgram_Command As String Implements IDownloaderSettings.OpenFolderInOtherProgram_Command
  Get
- Return OpenFolderInOtherProgram
+ Return OpenFolderInOtherProgram.ValueTemp
  End Get
  Set(ByVal command As String)
- OpenFolderInOtherProgram.Value = command
+ OpenFolderInOtherProgram.ValueTemp = command
  End Set
  End Property
+ Private Function ShouldSerializeIDownloaderSettings_OpenFolderInOtherProgram_Command() As Boolean
+ Return DirectCast(OpenFolderInOtherProgram, IGridValue).ShouldSerializeValue
+ End Function
  <Browsable(True), GridVisible(False), XMLVN({"Environment"}, True), Category("Environment"), DisplayName("Check new version at start")>
  Friend ReadOnly Property CheckUpdatesAtStart As XMLValue(Of Boolean)
 #End Region
@@ -305,6 +311,9 @@ Namespace API.YouTube.Base
  <Browsable(True), GridVisible, XMLVN({"DefaultsVideo"}, 1080), Category("Defaults Video"), DisplayName("Default definition"),
  Description("The default maximum video resolution. -1 for max definition")>
  Public ReadOnly Property DefaultVideoDefinition As XMLValue(Of Integer)
+ <Browsable(True), GridVisible, XMLVN({"DefaultsVideo"}), Category("Defaults Video"), DisplayName("Convert non-AVC codecs to AVC"),
+ Description("Convert non-AVC codecs (eg 'VP9') to AVC. Not recommended due to high CPU usage!")>
+ Public ReadOnly Property DefaultVideoConvertNonAVC As XMLValue(Of Boolean)
  <Browsable(True), GridVisible, XMLVN({"DefaultsVideo"}, False), Category("Defaults Video"), DisplayName("Embed thumbnail (video)"),
  Description("Embed thumbnail in the video as cover art. Default: true.")>
  Public ReadOnly Property DefaultVideoEmbedThumbnail As XMLValue(Of Boolean)
@@ -432,6 +441,9 @@ Namespace API.YouTube.Base
  <Browsable(True), GridVisible, XMLVN({"Playlists"}), Category("Music"), DisplayName("M3U8 Append file number"),
  Description("Add file number to file name. Default: false.")>
  Public ReadOnly Property MusicPlaylistCreate_M3U8_AppendNumber As XMLValue(Of Boolean)
+ <Browsable(True), GridVisible, XMLVN({"Playlists"}, M3U8CreationMode.Relative), Category("Music"), DisplayName("Create M3U8: creation mode"),
+ Description("Set the playlist creation mode: absolute links, relative links, or both. If 'Both' is selected, two playlists will be created. Default: 'Relative'.")>
+ Public ReadOnly Property MusicPlaylistCreate_CreationMode As XMLValue(Of M3U8CreationMode)
 #End Region
 #End Region
 #Region "Defaults Subtitles"

diff --git a/SCrawler.YouTube/Declarations.vb b/SCrawler.YouTube/Declarations.vb
@@ -66,6 +66,7 @@ Namespace API.YouTube
  If Not f.IsEmptyString And Not f.Name.IsEmptyString Then
  Dim ff As SFile = f
  ff.Name = ff.Name.StringRemoveWinForbiddenSymbols.StringTrim
+ ff.Name = ff.Name.StringReplaceSymbols({vbLf, vbCr, vbCrLf}, String.Empty, EDP.ReturnValue)
  ff.Name = ff.Name.StringTrimEnd(".")
  If Not ff.Name.IsEmptyString And Not MyYouTubeSettings.FileRemoveCharacters.IsEmptyString Then _
  ff.Name = ff.Name.StringReplaceSymbols(MyYouTubeSettings.FileRemoveCharacters.Value.AsList.ListCast(Of String).ToArray, String.Empty, EDP.ReturnValue)

diff --git a/SCrawler.YouTube/My Project/AssemblyInfo.vb b/SCrawler.YouTube/My Project/AssemblyInfo.vb
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.6.25.0")>
-<Assembly: AssemblyFileVersion("2024.6.25.0")>
+<Assembly: AssemblyVersion("2024.7.24.0")>
+<Assembly: AssemblyFileVersion("2024.7.24.0")>
 <Assembly: NeutralResourcesLanguage("en")>
diff --git a/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb b/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb
@@ -899,21 +899,21 @@ Namespace API.YouTube.Objects
  Return Nothing
  End Try
  End Function
- Private Function GetPlaylistRow(ByVal Element As YouTubeMediaContainerBase, Optional ByVal __file As SFile = Nothing) As String
+ Private Function GetPlaylistRow(ByVal Element As YouTubeMediaContainerBase, Optional ByVal __file As SFile = Nothing,
+ Optional ByVal Mode As M3U8CreationMode = M3U8CreationMode.Absolute) As String
  Const m3u8DataRow$ = "#EXTINF:{0},{1}" & vbCrLf & "{2}"
  With Element
  Dim f As SFile = __file.IfNullOrEmpty(.File)
+ Dim __f$ = SymbolsConverter.ASCII.EncodeSymbolsOnly(If(Mode = M3U8CreationMode.Absolute, f.ToString, f.File))
+ If Mode = M3U8CreationMode.Absolute Then __f = $"file:///{__f}"
  Dim fName$ = .Title.IfNullOrEmpty(f.Name)
  If MyYouTubeSettings.MusicPlaylistCreate_M3U8_AppendNumber And .PlaylistIndex > 0 Then fName = $"{ .PlaylistIndex}. {fName}"
  If Not .UserTitle.IsEmptyString Then
  fName = $"{ .UserTitle} - {fName}"
  If MyYouTubeSettings.MusicPlaylistCreate_M3U8_AppendArtist Then fName = $"{ .UserTitle} - {fName}"
  End If
  If MyYouTubeSettings.MusicPlaylistCreate_M3U8_AppendExt Then fName &= $".{f.Extension}"
- Return String.Format(m3u8DataRow,
- CInt(.Duration.TotalSeconds),
- fName,
- $"file:///{SymbolsConverter.ASCII.EncodeSymbolsOnly(f)}")
+ Return String.Format(m3u8DataRow, CInt(.Duration.TotalSeconds), fName, __f)
  End With
  End Function
  Private ReadOnly DownloadProgressPattern As RParams = RParams.DMS("\[download\]\s*([\d\.,]+)", 1, EDP.ReturnValue)
@@ -954,23 +954,41 @@ Namespace API.YouTube.Objects
  Dim t As TextSaver = Nothing
  Try
  Dim f As SFile
- If MyYouTubeSettings.MusicPlaylistCreate_M3U8 Then
- t = New TextSaver
- t.AppendLine("#EXTM3U")
- Elements.ForEach(Sub(e) t.AppendLine(GetPlaylistRow(e)))
- f = $"{Elements(0).File.PathWithSeparator}Playlist.m3u8"
- t.SaveAs(f, EDP.SendToLog)
- If f.Exists Then AddFile(f)
- t.Dispose()
- End If
- If MyYouTubeSettings.MusicPlaylistCreate_M3U Then
- t = New TextSaver
- Elements.ForEach(Sub(e) t.AppendLine(e.File))
- f = $"{Elements(0).File.PathWithSeparator}Playlist.m3u"
- t.SaveAs(f, EDP.SendToLog)
- If f.Exists Then AddFile(f)
- t.Dispose()
- End If
+ Dim arr As M3U8CreationMode() = If(MyYouTubeSettings.MusicPlaylistCreate_CreationMode.Value = M3U8CreationMode.Both,
+ {M3U8CreationMode.Relative, M3U8CreationMode.Absolute},
+ {MyYouTubeSettings.MusicPlaylistCreate_CreationMode.Value})
+ Dim postfix$
+ Dim added As Boolean
+ Dim checkFile As Func(Of IYouTubeMediaContainer, Boolean) = Function(ByVal e As IYouTubeMediaContainer) As Boolean
+ If e.File.Exists Then
+ added = True
+ Return True
+ Else
+ Return False
+ End If
+ End Function
+ For Each cm As M3U8CreationMode In arr
+ If arr.Length > 1 AndAlso cm = M3U8CreationMode.Absolute Then postfix = "Abs" Else postfix = String.Empty
+ added = False
+ If MyYouTubeSettings.MusicPlaylistCreate_M3U8 Then
+ t = New TextSaver
+ t.AppendLine("#EXTM3U")
+ Elements.ForEach(Sub(e) If checkFile(e) Then t.AppendLine(GetPlaylistRow(e,, cm)))
+ f = $"{Elements(0).File.PathWithSeparator}Playlist{postfix}.m3u8"
+ If added Then t.SaveAs(f, EDP.SendToLog)
+ If f.Exists Then AddFile(f)
+ t.Dispose()
+ End If
+ added = False
+ If MyYouTubeSettings.MusicPlaylistCreate_M3U Then
+ t = New TextSaver
+ Elements.ForEach(Sub(e) If checkFile(e) Then t.AppendLine(If(cm = M3U8CreationMode.Relative, e.File.File, e.File.ToString)))
+ f = $"{Elements(0).File.PathWithSeparator}Playlist{postfix}.m3u"
+ If added Then t.SaveAs(f, EDP.SendToLog)
+ If f.Exists Then AddFile(f)
+ t.Dispose()
+ End If
+ Next
  Catch ex As Exception
  ErrorsDescriber.Execute(EDP.SendToLog, ex, "[YouTubeMediaContainerBase.Download.CreatePlaylist]")
  End Try
@@ -1354,15 +1372,29 @@ Namespace API.YouTube.Objects
  'Delete unrequsted files
  If tempFilesList.Count > 0 Then tempFilesList.ForEach(Sub(tfr) If Not tfr.Requested Then tfr.File.Delete(,, EDP.None)) : tempFilesList.Clear()
 
- 'Update video FPS
- If SelectedVideoIndex >= 0 AndAlso OutputVideoFPS > 0 AndAlso SelectedVideo.Bitrate <> OutputVideoFPS Then
- f = File
- f.Name &= "tmp00"
- .Execute($"ffmpeg -i ""{File}"" -filter:v fps={OutputVideoFPS.ToString.Replace(",", ".")} -c:a copy ""{f}""")
- If f.Exists Then
- File.Delete()
- SFile.Rename(f, File,, EDP.LogMessageValue)
- End If
+ If SelectedVideoIndex >= 0 Then
+ Dim reencodeFile As Action(Of String) =
+ Sub(ByVal ffmpegCommand As String)
+ f = File
+ f.Name &= "tmp00"
+ .Execute(String.Format(ffmpegCommand, File.ToString, f.ToString))
+ If f.Exists Then
+ If f.Size > 0 Then
+ File.Delete()
+ SFile.Rename(f, File,, EDP.LogMessageValue)
+ Else
+ f.Delete(, SFODelete.DeletePermanently, EDP.None)
+ End If
+ End If
+ End Sub
+ 'Change video codec to AVC
+ If MyYouTubeSettings.DefaultVideoConvertNonAVC.Value AndAlso
+ Not SelectedVideo.Codec.IsEmptyString AndAlso Not SelectedVideo.Codec.Trim.ToLower.StartsWith("avc") Then _
+ reencodeFile("ffmpeg -i ""{0}"" -c:a copy -c:v libx264 ""{1}""")
+
+ 'Update video FPS
+ If OutputVideoFPS > 0 AndAlso SelectedVideo.Bitrate <> OutputVideoFPS Then _
+ reencodeFile("ffmpeg -i ""{0}"" -filter:v fps=" & OutputVideoFPS.ToString.Replace(", ", ".") & " -c:a copy ""{1}""")
  End If
  End If
  End If

diff --git a/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb b/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.6.25.0")>
-<Assembly: AssemblyFileVersion("2024.6.25.0")>
+<Assembly: AssemblyVersion("2024.7.24.0")>
+<Assembly: AssemblyFileVersion("2024.7.24.0")>
 <Assembly: NeutralResourcesLanguage("en")>
diff --git a/SCrawler/API/Instagram/SiteSettings.vb b/SCrawler/API/Instagram/SiteSettings.vb
@@ -443,11 +443,12 @@ Namespace API.Instagram
  UserRegex = RParams.DMS(String.Format(UserRegexDefaultPattern, "instagram.com/"), 1)
  ImageVideoContains = "instagram.com"
  End Sub
+ Private Const SettingsVersionCurrent As Integer = 2
  Friend Overrides Sub EndInit()
  Try : MyLastRequests.Add(LastDownloadDate.Value, LastRequestsCount.Value) : Catch : End Try
  If Not CBool(HH_IG_WWW_CLAIM_USE.Value) Then Responser.Headers.Remove(Header_IG_WWW_CLAIM)
- If CInt(SettingsVersion.Value) < 2 Then
- SettingsVersion.Value = 2
+ If CInt(SettingsVersion.Value) < SettingsVersionCurrent Then
+ SettingsVersion.Value = SettingsVersionCurrent
  HH_IG_WWW_CLAIM_UPDATE_INTERVAL.Value = 120
  HH_IG_WWW_CLAIM_ALWAYS_ZERO.Value = False
  HH_IG_WWW_CLAIM_RESET_EACH_SESSION.Value = True