Merge pull request #3 from Vollinger0/more_features

new shared data tool overrides for url generation and auto-editing

.github/workflows/build.yml → .github/workflows/build_release_from_tag.yml

@@ -42,6 +42,14 @@ jobs:
         run: |
           gh release create v${{ github.ref_name }} dist/esm-${{ github.ref_name }}.zip
 
+      - name: Commit version changes to main branch
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          git add pyproject.toml
+          git commit -m "Update version number for next development iteration"
+          git push origin main
+
         # uses: softprops/action-gh-release@v1
         # with:
         #   files: dist/esm-${{ github.ref_name }}.zip

esm-custom-config.yaml

@@ -45,6 +45,9 @@ deletes:
 
 downloadtool:
   useSharedDataURLFeature: True
+  #customExternalHostNameAndPort: 'https://my-server.com:12345'
+  #customSharedDataURL: 'https://my-server.com:54321/SharedData.zip'
+  #autoEditDedicatedYaml: False
 
 galaxy:
   territories: #faction Territory Definitions (look up their definitions in the file "$scenario/Content/Configuration/GalaxyConfig.ecf" of your scenario)

esm-default-config.example.yaml

@@ -73,7 +73,10 @@ downloadtool:   # configuration for the shared data download tool
   maxGlobalBandwith: 50000000                                 # max bandwith to use for the downloads globally in bytes, e.g. 50 MB/s
   maxClientBandwith: 30000000                                 # max bandwith to use for the download per client in bytes, e.g. 30 MB/s
   rateLimit: 10 per minute                                    # rate limit of max allowed requests per ip address per time unit, e.g. '10 per minute' or '10 per hour'
+  customExternalHostNameAndPort: ''                           # if set, this will be used as the host instead of the automatically generated host-part of the url. must be something like: 'https://my-server.com:12345'. The path/name of the files will be appended.
   useSharedDataURLFeature: false                              # if true, a zip for the SharedDataURL feature will be created, served and the dedicated yaml will be automatically edited.
+  autoEditDedicatedYaml: true                                 # set to false if you do not want the dedicated yaml to be edited automatically
+  customSharedDataURL: ''                                     # if set, this will be used as the shared data url instead of the automatically generated one. Make sure it is correct!
   autoZipName: SharedData.zip                                 # The filename of the zip file for the auto download of the SharedDataURL feature postfixed with _yyyymmdd_hhmmss so the client recognizes this as a new file on recreation.
   useCustomCacheFolderName: false                             # if true, the custom folder name will be used, if false, the folder name will be generated with following pattern '{gamename}_{serverip}_{uniquegameid}'
   customCacheFolderName: DediGame_127.0.0.1_123456789         # name of the folder included in the zip file, which will look something like 'DediGame_127.0.0.1_12346789', depending on gamename, server ip and unique game id

readme_shareddata.md

@@ -26,6 +26,10 @@ The shared-data-tool will serve another zip file of the shared data with a chang
 ### If you use this feature, make sure to have the shared-data-tool server started when serving the game and make sure to restart the game server whenever you started or stopped the data-tool.**
 ESM will check this for you on server start and will **ABORT** the server start if there is a shared data url configured that is **not** reachable to avoid having an invalid configuration.
 
+You can disable the automatic configuration in the dedicated yaml by setting `autoEditDedicatedYaml` to false in the config.
+You can override the automatic hostip and port generation by setting your own `customExternalHostNameAndPort`, which should look something like: 'https://my-server.com:12345'
+You can override the generation of the whole url and have esm set your own custom url by setting `customSharedDataURL`, which should look something like: 'https://my-server.com:54321/SharedData.zip'
+
 Since the webserver will run on the same server as the game and probably be publicly available, it has a sophisticated configuration to limit the bandwith/connection aswell as the global bandwith used. It also includes several security measures like a rate limiter and an internal whitelist for paths.
 If your server connection supports e.g. 100 MB/s, you can limit the webserver to not use more than e.g. 50MB/s, to make sure the running gameserver network throughput is not affected and the game doesn't lag out the players due to the downloads. If you so desire, you can also limit the bandwith per connection, to make sure that nobody can occupy the whole bandwith. Although this shouldn't take more than 10 seconds, since shared data can't possibly be bigger than 500 MB (current scenario size limit). You can also rate-limit the amount of requests per minute per IP, to avoid simple DoS-attacks (default: 10/m). Check the `esm-default-config.example.yaml` for all configuration options, especially configure the port that is publicly available for your server, since the game clients of your players will need to connect to that.
 

src/esm/ConfigModels.py

@@ -142,8 +142,11 @@ class DownloadToolConfig(BaseModel):
     maxGlobalBandwith: int = Field(50*1000*1000, description="max bandwith to use for the downloads globally in bytes, e.g. 50 MB/s")
     maxClientBandwith: int = Field(30*1000*1000, description="max bandwith to use for the download per client in bytes, e.g. 30 MB/s")
     rateLimit: str = Field("10 per minute", description="rate limit of max allowed requests per ip address per time unit, e.g. '10 per minute' or '10 per hour'")
+    customExternalHostNameAndPort: str = Field("", description="if set, this will be used as the host instead of the automatically generated host-part of the url. must be something like: 'https://my-server.com:12345'. The path/name of the files will be appended.")
 
     useSharedDataURLFeature: bool = Field(False, description="if true, a zip for the SharedDataURL feature will be created, served and the dedicated yaml will be automatically edited.")
+    autoEditDedicatedYaml: bool = Field(True, description="set to false if you do not want the dedicated yaml to be edited automatically")
+    customSharedDataURL: str = Field("", description="if set, this will be used as the shared data url instead of the automatically generated one. Make sure it is correct!")
     autoZipName: str = Field("SharedData.zip", description="The filename of the zip file for the auto download of the SharedDataURL feature postfixed with _yyyymmdd_hhmmss so the client recognizes this as a new file on recreation.")
 
     useCustomCacheFolderName: bool = Field(False, description="if true, the custom folder name will be used, if false, the folder name will be generated with following pattern '{gamename}_{serverip}_{uniquegameid}'")

src/esm/EsmSharedDataServer.py

@@ -55,7 +55,7 @@ def resume(self):
         just resume the shared data server, do not recreate the files nor change configuration
         """
         zipFiles = self.findZipFiles()
-        if len(zipFiles) != 2:
+        if len(zipFiles) < 1 or len(zipFiles) > 2:
             raise RequirementsNotFulfilledError(f"Expected 2 zip files in the wwwroot folder, found {len(zipFiles)}. Aborting resume since files might be missing, start the server without the resume option to regenerate the data.")
         self.startServer(zipFiles)
 
@@ -68,8 +68,8 @@ def startServer(self, zipFiles: List[ZipFile]):
         log.info(f"Server configured to allow max {humanize.naturalsize(self.config.downloadtool.maxGlobalBandwith, gnu=False)}/s in total network bandwidth.")
         log.info(f"Server configured to allow max {humanize.naturalsize(self.config.downloadtool.maxClientBandwith, gnu=False)}/s network bandwith per connection.")
 
-        myHostIp = Tools.getOwnIp(self.config)
-        servingUrlRoot = f"http://{myHostIp}:{self.config.downloadtool.serverPort}"
+        servingUrlRoot = self.getServingUrlRoot()
+        log.debug(f"Server root url is set to: '{servingUrlRoot}'")
 
         manualZipFile = Tools.findZipFileByName(zipFiles, startsWith=self.config.downloadtool.manualZipName)
         if manualZipFile:
@@ -79,18 +79,22 @@ def startServer(self, zipFiles: List[ZipFile]):
         if self.config.downloadtool.useSharedDataURLFeature:
             autoZipFile = Tools.findZipFileByName(zipFiles, startsWith=self.config.downloadtool.autoZipName.split(".")[0])
 
-            sharedDataUrl = f"{servingUrlRoot}/{autoZipFile.name}"
+            sharedDataUrl = self.getSharedDataURL(servingUrlRoot, autoZipFile)
 
             log.info(f"Shared data zip file for server is at: '{sharedDataUrl}'")
-
-            self.configService.backupDedicatedYaml()
-            # actually alter the dedicated.yaml, changing or adding the shareddataurl to what we just created
-            sharedDataUrl = f"_{sharedDataUrl}"
-            self.configService.changeSharedDataUrl(sharedDataUrl)
-
-            # check if the configuration of the dedicated yaml (we will not make any changes to it) has the auto zip url configured properly
-            self.checkDedicatedYamlHasAutoZipUrl(sharedDataUrl)
-            log.warn(f"The dedicated yaml has been updated to point to the shared data tool, make sure to restart the server for it to take effect!")
+            log.warn(f"Using this SharedDataURL feature is dangerous! Make sure the URL above will be reachable for all your players, or it might break the game! Read the readme_shareddata.md for more information.")
+
+            if self.config.downloadtool.autoEditDedicatedYaml:
+                self.configService.backupDedicatedYaml()
+                # actually alter the dedicated.yaml, changing or adding the shareddataurl to what we just created
+                sharedDataUrl = f"_{sharedDataUrl}"
+                self.configService.changeSharedDataUrl(sharedDataUrl)
+
+                # check if the configuration of the dedicated yaml (we will not make any changes to it) has the auto zip url configured properly
+                self.checkDedicatedYamlHasAutoZipUrl(sharedDataUrl)
+                log.warn(f"The dedicated yaml has been updated to point to the shared data tool, make sure to restart the server for it to take effect!")
+            else:
+                log.warn(f"You turned off the autoEditDedicatedYaml feature. The dedicated yaml will NOT be updated automatically, make sure it the correct url! Otherwise it might break the game!")
 
         log.info(f"Starting download server for {len(zipFiles)} zip files (excluding default assets).")
         def NoOp(*args):
@@ -104,10 +108,24 @@ def NoOp(*args):
         finally:
             log.info(f"SharedData server stopped serving. Total downloads: {EsmHttpThrottledHandler.globalZipDownloads}")
 
-            if self.config.downloadtool.useSharedDataURLFeature:
+            if self.config.downloadtool.useSharedDataURLFeature and self.config.downloadtool.autoEditDedicatedYaml:
                 self.configService.rollbackDedicatedYaml()
                 log.warn(f"The dedicated yaml has been rolled back to its original state, make sure to restart the server for it to take effect!")
-
+
+    def getSharedDataURL(self, servingUrlRoot, autoZipFile: ZipFile):
+        if len(self.config.downloadtool.customSharedDataURL) > 1:
+            log.warn(f"Server configured to use a custom shared data url: '{self.config.downloadtool.customSharedDataURL}', make sure it is reachable for all your players, or it might break the game!")
+            return self.config.downloadtool.customSharedDataURL
+        else:
+            return f"{servingUrlRoot}/{autoZipFile.name}"
+
+    def getServingUrlRoot(self):
+        if len(self.config.downloadtool.customExternalHostNameAndPort) > 1:
+            log.debug(f"Server configured to use a custom external host name and port: '{self.config.downloadtool.customExternalHostNameAndPort}'")
+            return self.config.downloadtool.customExternalHostNameAndPort
+        else:
+            myHostIp = Tools.getOwnIp(self.config)
+            return f"http://{myHostIp}:{self.config.downloadtool.serverPort}"
 
     def prepareZipFiles(self) -> List[ZipFile]:
         """

test/test-generated-config.yaml

@@ -73,7 +73,10 @@ downloadtool:   # configuration for the shared data download tool
   maxGlobalBandwith: 50000000                                 # max bandwith to use for the downloads globally in bytes, e.g. 50 MB/s
   maxClientBandwith: 30000000                                 # max bandwith to use for the download per client in bytes, e.g. 30 MB/s
   rateLimit: 10 per minute                                    # rate limit of max allowed requests per ip address per time unit, e.g. '10 per minute' or '10 per hour'
+  customExternalHostNameAndPort: ''                           # if set, this will be used as the host instead of the automatically generated host-part of the url. must be something like: 'https://my-server.com:12345'. The path/name of the files will be appended.
   useSharedDataURLFeature: false                              # if true, a zip for the SharedDataURL feature will be created, served and the dedicated yaml will be automatically edited.
+  autoEditDedicatedYaml: true                                 # set to false if you do not want the dedicated yaml to be edited automatically
+  customSharedDataURL: ''                                     # if set, this will be used as the shared data url instead of the automatically generated one. Make sure it is correct!
   autoZipName: SharedData.zip                                 # The filename of the zip file for the auto download of the SharedDataURL feature postfixed with _yyyymmdd_hhmmss so the client recognizes this as a new file on recreation.
   useCustomCacheFolderName: false                             # if true, the custom folder name will be used, if false, the folder name will be generated with following pattern '{gamename}_{serverip}_{uniquegameid}'
   customCacheFolderName: DediGame_127.0.0.1_123456789         # name of the folder included in the zip file, which will look something like 'DediGame_127.0.0.1_12346789', depending on gamename, server ip and unique game id

todos.md

@@ -37,6 +37,8 @@
 - [ ] add a AI-powered chatbot, chatgpt-like that roleplays as a spacefaring hamster that reacts to player chat?
 
 ## done
+- [x] add option to disable auto-editing of dedicated yaml in shared data tool
+- [x] add option to override the host and port when aut-editing the dedicated yaml in the shared data tool
 - [x] resume mode / create mode for the shared data server to be able to control if we actually want to recreate the zip.
 - [-] find out if there is *any* information the client requests on the shared data server to see how we can implement it => nothing. its useless.
 - [x] maybe the shared data tool could remove the shareddataurl property when it is shut down to make sure the server doesn't use it when its off.