Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Config option to configure the https adaptive window size for requests #529

Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/config/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ pub struct Config {
pub logging: bool,
/// It stores the option to whether enable or disable debug mode.
pub debug: bool,
/// It toggles whether to use adaptive HTTP windows
pub adaptive_window: bool,
/// It stores all the engine names that were enabled by the user.
pub upstream_search_engines: HashMap<String, bool>,
/// It stores the time (secs) which controls the server request timeout.
Expand Down Expand Up @@ -68,6 +70,7 @@ impl Config {

let debug: bool = globals.get::<_, bool>("debug")?;
let logging: bool = globals.get::<_, bool>("logging")?;
let adaptive_window: bool = globals.get::<_, bool>("adaptive_window")?;

if !logging_initialized {
set_logging_level(debug, logging);
Expand Down Expand Up @@ -125,6 +128,7 @@ impl Config {
},
logging,
debug,
adaptive_window,
upstream_search_engines: globals
.get::<_, HashMap<String, bool>>("upstream_search_engines")?,
request_timeout: globals.get::<_, u8>("request_timeout")?,
Expand Down
2 changes: 2 additions & 0 deletions src/results/aggregator.rs
neon-mmd marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@
/// Returns an error a reqwest and scraping selector errors if any error occurs in the results
/// function in either `searx` or `duckduckgo` or both otherwise returns a `SearchResults struct`
/// containing appropriate values.
pub async fn aggregate(

Check warning on line 66 in src/results/aggregator.rs

View workflow job for this annotation

GitHub Actions / Rust project

this function has too many arguments (8/7)
query: &str,
page: u32,
random_delay: bool,
Expand All @@ -71,13 +71,15 @@
upstream_search_engines: &[EngineHandler],
request_timeout: u8,
safe_search: u8,
adaptive_window: bool,
) -> Result<SearchResults, Box<dyn std::error::Error>> {
let client = CLIENT.get_or_init(|| {
ClientBuilder::new()
.timeout(Duration::from_secs(request_timeout as u64)) // Add timeout to request to avoid DDOSing the server
.https_only(true)
.gzip(true)
.brotli(true)
.http2_adaptive_window(adaptive_window)
.build()
.unwrap()
});
Expand Down
1 change: 1 addition & 0 deletions src/server/routes/search.rs
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ async fn results(
.collect::<Vec<EngineHandler>>(),
config.request_timeout,
safe_search_level,
config.adaptive_window,
)
.await?
}
Expand Down
2 changes: 2 additions & 0 deletions websurfx/config.lua
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ rate_limiter = {
number_of_requests = 20, -- The number of request that are allowed within a provided time limit.
time_limit = 3, -- The time limit in which the quantity of requests that should be accepted.
}
-- Set whether the server will use an adaptive/dynamic HTTPS window size, see https://httpwg.org/specs/rfc9113.html#fc-principles
https_adaptive_window_size = false

-- ### Search ###
-- Filter results based on different levels. The levels provided are:
Expand Down
Loading