Fix response body (based on #84) #326

g00g1 · 2024-06-18T21:06:23Z

This PR is an attempt to revive previously developed fix for #41. It seems like its' developer abandoned PR, therefore I would like to try my best to finish it.

I have updated the code base to a current master, therefore it is possible to continue discussion regarding this patch.

Dr-Lazarus-V2 · 2024-10-25T01:30:11Z

Is there any update on when this PR will be reviewed/merged? This solution also solves another issue, wherein if the response size is large, even if the blocking rule is triggered, the response still goes through to the client.

airween · 2024-10-31T16:50:48Z

Hi @g00g1, @Dr-Lazarus-V2,

although all tests were passed, it would be nice to explain the exact problem. @Dr-Lazarus-V2 mentioned this PR in his issue, but unfortunately I wasn't able to reproduce the problem - and then I can't try the fix.

Also this comment mentions that the requested feature is not available yet - this PR adds it?

g00g1 · 2024-10-31T16:54:32Z

@airween how did you try to reproduce it? Have you seen my comment here: #41 (comment) ?

Dr-Lazarus-V2 · 2024-11-05T01:04:18Z

Hey @airween, yes I believe this PR resolves the issue. As the response body is fully examined before the response is sent out.

airween · 2024-11-05T09:53:59Z

@g00g1: I saw your comment, but I'm afraid it's not enough to reproduce the issue. Could you share some more details, eg. (relevant) configuration, and a curl request?

@Dr-Lazarus-V2: I'm sure the PR solves the problem, but first I would like to see the problem itself 😃

airween · 2024-11-07T17:59:50Z

Hi guys,

finally I had some time to reproduce the issue. Coreruleset issue #3277 helped me.

I tried your patch, seems like it solves the problem - but unfortunately the audit log messages key is empty, even though the triggered rule is in the error.log. The other parts are there in audit.log: transaction, response and producer. The response shows the status was 403.

Could you take a look this issue?

g00g1 · 2024-11-07T18:16:52Z

I am still not very familiar with modsecurity code, it might take some time before I figure out how this should work properly and what is lacking to fill messages key currently, unfortunately I couldn't figure out the reason fast.

If anyone wished to help or guide me regarding this, I will be very grateful.

Am I right that this pull request somehow impacts the codepath taken at https://github.com/owasp-modsecurity/ModSecurity/blob/4a720004ddf37963ad63deb1f1a2fdd8113f7d41/src/rule_with_actions.cc#L495 ?

airween · 2024-11-07T18:21:07Z

Sure, I'm happy to help you. You can join to OWASP's Slack server, and choose #project-modsecurity channel. You can reach me there, feel free to ask.

g00g1 · 2024-11-07T18:24:33Z

@airween thanks, but I don't have an account there nor email at owasp.org domain.

airween · 2024-11-07T19:21:51Z

@airween thanks, but I don't have an account there nor email at owasp.org domain.

As I know you don't need any owasp.org email address. But if you send me your email me in private, I can send you an invitation.

airween · 2024-11-07T22:01:54Z

Or you can use this link.

Dr-Lazarus-V2 · 2024-11-12T01:24:35Z

Hey, if it's all right, I can also have a look at the PR and try to debug what the issue might be

Dr-Lazarus-V2 · 2024-11-12T06:57:50Z

I have noticed a weird behaviour:

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST /api/v1/*/* HTTP/1.1" (Variable: REQUEST_LINE)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920160) Executing operator "Rx" with param "^\d+$" against REQUEST_HEADERS:Content-Length.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "150" (Variable: REQUEST_HEADERS:Content-Length)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920170) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920171) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920180) Executing operator "Within" with param "HTTP/2 HTTP/2.0 HTTP/3 HTTP/3.0" against REQUEST_PROTOCOL.

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "GET /api/v1/*/* HTTP/1.1" (Variable: REQUEST_LINE)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920160) Executing operator "Rx" with param "^\d+$" against REQUEST_HEADERS:Content-Length.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "150" (Variable: REQUEST_HEADERS:Content-Length)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920170) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "GET" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars updated.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 1.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Executing chained rule.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 0) Executing operator "Rx" with param "^0?$" against

Within a single transaction (identified by a unique ID), the same rule is triggered twice, but with inconsistent results: once with the HTTP method identified as POST and then as GET. This means that during the same request, ModSecurity is evaluating the method as both POST and GET at different points. This unusual behavior may indicate an issue with how ModSecurity is processing the request, possibly due to a configuration quirk or rule interpretation that causes the request method to appear inconsistent during evaluation.

Dr-Lazarus-V2 · 2024-11-12T07:06:06Z

As an addition to the previously mentioned setup, I created the following custom rule to block responses containing the word "leak":

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
    "id:187, \
    phase:4,\
    deny,\
    status:403,\
    log,\
    t:none,\
    msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

However, at Debug Level 9, when this rule is triggered and blocks the response, the transaction appears to restart unexpectedly. Below are the logs capturing this issue:

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] (Rule: 187) Executing operator "Rx" with param "(?i)(\n|\''|\:|\W*)leak" against RESPONSE_BODY.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Target value: "{"id":"chatcmpl-fd274e7b","created":1731393695,"model":"meta.llama3-8b-instruct- (792 characters omitted)" (Variable: RESPONSE_BODY)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Matched vars updated.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Rule returned 1.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Saving msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Running action: status
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Running action: log
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Saving transaction to logs
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Running (disruptive) action: deny.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [8] Running action deny
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [8] Skipping this phase as this request was already intercepted.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Initializing transaction
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Transaction context created.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Starting phase CONNECTION. (SecRules 0)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [9] This phase consists of 29 rule(s).
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Starting phase URI. (SecRules 0 + 1/2)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Starting phase REQUEST_HEADERS. (SecRules 1)

airween · 2024-11-13T20:49:00Z

Hey, if it's all right, I can also have a look at the PR and try to debug what the issue might be

Hi @Dr-Lazarus-V2, this would be awesome! Thanks!

airween · 2024-11-13T20:59:54Z

Within a single transaction (identified by a unique ID), the same rule is triggered twice, but with inconsistent results: once with the HTTP method identified as POST and then as GET.

Could you show your request with that I can reproduce this? Also please provide the used CRS version too.

Dr-Lazarus-V2 · 2024-11-14T01:21:11Z

Hey @airween, the CRS Version I used is v4.7.0. In the application I am trying to protect, I had an endpoint which returns a JSON response with the word leak. I had used this custom rule to detect it:

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
    "id:187, \
    phase:4,\
    deny,\
    status:403,\
    log,\
    t:none,\
    msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

If you'd like to reproduce this, I think you would need a simple POST endpoint in the Web Application which is protected by the WAF. The POST endpoint needs to return a JSON response with the word "leak". If you perform a curl to this endpoint with this custom rule present in your application. You will notice that the transaction terminates and restarts when the custom rule is triggered (evaluates to true and implements action deny).

Here is my modsecurity configuration for reference:

SecRuleEngine On

SecRequestBodyAccess on
SecRule REQUEST_HEADERS:Content-Type "^(?:application(?:/soap\+|/)|text/)xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "^application/json" \
     "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
SecRequestBodyLimit 10485760
SecRequestBodyNoFilesLimit 1048576
SecRequestBodyLimitAction ProcessPartial
SecRequestBodyJsonDepthLimit 512
SecArgumentsLimit 1000

SecRule &ARGS "@ge 1000" \
"id:'200007', phase:2,t:none,log,deny,status:400,msg:'Failed to fully parse request body due to large argument count',severity:2"

SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"

SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000
SecRule TX:/^MSC_/ "!@streq 0" \
        "id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"

SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml application/json
SecResponseBodyLimit 537600
SecResponseBodyLimitAction Reject

SecTmpDir /tmp/
SecDataDir /tmp/

SecAuditEngine On
SecAuditLogType Serial
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLog /var/log/modsec_logs/modsec_audit_waf.theviscousweb.com.log
SecAuditLogFormat JSON
SecAuditLogParts ABFHZ

SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
SecStatusEngine Off

SecGeoLookupDb /var/GeoLite2-Country/GeoLite2-Country.mmdb

SecDebugLog /var/log/modsec_logs/modsec_debug.log
SecDebugLogLevel 9

Dr-Lazarus-V2 · 2024-11-14T03:41:43Z

@airween, I have tested and realized the following:

I have setup an endpoint: "/healthz" which returns "leak!"

Bug Case:

        location /healthz {
            add_header Content-Type text/plain;
            return 200 'leak!';
        }

Now if I perform the following curl command:

 curl --location --request POST 'http://waf.theviscousweb.com/healthz' \
--data ''

It seems to return the following log:

{
    "transaction": {
        "client_ip": "10.103.253.191",
        "time_stamp": "Thu Nov 14 03:29:54 2024",
        "server_id": "88a6b3d6a5b5bc6b73061e112bad4a1309e1d4b2",
        "client_port": 36616,
        "host_ip": "192.168.32.3",
        "host_port": 8080,
        "unique_id": "bdcd2324b71e0518896ea843b61e488b",
        "request": {
            "method": "GET",
            "http_version": 1.1,
            "uri": "/healthz",
            "headers": {
                "Host": "waf.theviscousweb.com",
                "User-Agent": "curl/7.81.0",
                "Accept": "*/*",
                "Content-Length": "0",
                "Content-Type": "application/x-www-form-urlencoded"
            }
        },
        "response": {
            "http_code": 403,
            "headers": {
                "Server": "nginx/1.27.2",
                "Date": "Thu, 14 Nov 2024 03:29:54 GMT",
                "Content-Type": "text/html",
                "Connection": "keep-alive",
                "ETag": "\"66dfdff2-93d6\""
            }
        },
        "producer": {
            "modsecurity": "ModSecurity v3.0.13 (Linux)",
            "connector": "ModSecurity-nginx v1.0.3",
            "secrules_engine": "Enabled",
            "components": [
                "OWASP_CRS/4.7.0\""
            ]
        },
        "messages": [
            {
                "message": "GET Request Detected",
                "details": {
                    "match": "Matched \"Operator `StrEq' with parameter `GET' against variable `REQUEST_METHOD' (Value: `GET' )",
                    "reference": "v0,3",
                    "ruleId": "191",
                    "file": "/var/opt/modsecurity.d/custom-ruleset/Data-Leak/Data-Leak-RULES.conf",
                    "lineNumber": "18",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Inbound Score: 0 Outbound Score: 0",
                "details": {
                    "match": "",
                    "reference": "",
                    "ruleId": "980145",
                    "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
                    "lineNumber": "40",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            }
        ]
    }
}

You can see that even though the curl command is POST, in the logs method is GET. Now some further pointers, this only occurs if the following custom rule is triggered.

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
    "id:187, \
    phase:4,\
    deny,\
    status:403,\
    log,\
    t:none,\
    msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

Bug Free Case:

If I modify the /healthz endpoint to return something else:

            location /healthz {
            add_header Content-Type text/plain;
            return 200 'test!';
        }

It does not cause any issues and the request is logged as a POST.

        {
    "transaction": {
        "client_ip": "10.103.253.191",
        "time_stamp": "Thu Nov 14 03:36:23 2024",
        "server_id": "88a6b3d6a5b5bc6b73061e112bad4a1309e1d4b2",
        "client_port": 43622,
        "host_ip": "192.168.32.3",
        "host_port": 8080,
        "unique_id": "13fdd4d10d426b9e653cc22a833bd3df",
        "request": {
            "method": "POST",
            "http_version": 1.1,
            "uri": "/healthz",
            "headers": {
                "Host": "waf.theviscousweb.com",
                "User-Agent": "curl/7.81.0",
                "Accept": "*/*",
                "Content-Length": "0",
                "Content-Type": "application/x-www-form-urlencoded"
            }
        },
        "response": {
            "http_code": 200,
            "headers": {
                "Server": "nginx/1.27.2",
                "Date": "Thu, 14 Nov 2024 03:36:23 GMT",
                "Content-Length": "5",
                "Content-Type": "application/octet-stream",
                "Content-Type": "text/plain",
                "Connection": "keep-alive"
            }
        },
        "producer": {
            "modsecurity": "ModSecurity v3.0.13 (Linux)",
            "connector": "ModSecurity-nginx v1.0.3",
            "secrules_engine": "Enabled",
            "components": [
                "OWASP_CRS/4.7.0\""
            ]
        },
        "messages": [
            {
                "message": "Inbound Score: 0 Outbound Score: 0",
                "details": {
                    "match": "",
                    "reference": "",
                    "ruleId": "980145",
                    "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
                    "lineNumber": "40",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            }
        ]
    }
}

Hence I see that when a rule is triggered in the response body, somehow the REQUEST_METHOD changes from POST to GET.

Dr-Lazarus-V2 · 2024-11-14T05:43:37Z

Hello, everyone. Apologies for the confusion regarding the recent PR. I want to clarify that the bug I observed – where malicious POST requests are being logged as GET requests – appears to be a redirection issue, which affects the logging. I had a redirect in my nginx code. This specific issue is not introduced by the recent code changes, as I found it to be present in the current active version of the ModSecurity-Nginx connector. This bug has already been raised and is still open in: #182.
@airween , I attempted to reproduce the core issue concerning the unavailability of messages in the audit logs, and I was not able to reproduce it. I can successfully view the messages in the audit logs.

Here is my audit logs under:

{
    "transaction": {
        "client_ip": "10.103.253.191",
        "time_stamp": "Thu Nov 14 06:18:37 2024",
        "server_id": "135e057d99ba46945f45e685b2bae7652ad9cc73",
        "client_port": 54524,
        "host_ip": "192.168.32.3",
        "host_port": 8080,
        "unique_id": "349dd1ee0e6e742bfc604e2732173015",
        "request": {
            "method": "POST",
            "http_version": 1.1,
            "uri": "/",
            "headers": {
                "Host": "waf.theviscousweb.com",
                "User-Agent": "curl/7.81.0",
                "Accept": "*/*",
                "Content-Type": "application/x-www-form-urlencoded",
                "Content-Length": "89"
            }
        },
        "response": {
            "http_code": 403,
            "headers": {
                "Server": "nginx/1.27.2",
                "Date": "Thu, 14 Nov 2024 06:18:37 GMT",
                "Content-Length": "153",
                "Content-Type": "text/html",
                "Connection": "keep-alive"
            }
        },
        "producer": {
            "modsecurity": "ModSecurity v3.0.13 (Linux)",
            "connector": "ModSecurity-nginx v1.0.3",
            "secrules_engine": "Enabled",
            "components": [
                "OWASP_CRS/4.7.0\""
            ]
        },
        "messages": [
            {
                "message": "OS File Access Attempt",
                "details": {
                    "match": "Matched \"Operator `PmFromFile' with parameter `lfi-os-files.data' against variable `ARGS:cmd' (Value: `;cat /etc/passwd;' )",
                    "reference": "o6,10v220,17t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin",
                    "ruleId": "930120",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
                    "lineNumber": "97",
                    "data": "Matched Data: etc/passwd found within ARGS:cmd: ;cat /etc/passwd;",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "platform-multi",
                        "attack-lfi",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/255/153/126",
                        "PCI/6.5.4"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Remote Command Execution: Unix Shell Code Found",
                "details": {
                    "match": "Matched \"Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:cmd' (Value: `;cat /etc/passwd;' )",
                    "reference": "o5,10v220,17t:cmdLine,t:normalizePath",
                    "ruleId": "932160",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf",
                    "lineNumber": "596",
                    "data": "Matched Data: etc/passwd found within ARGS:cmd:  cat/etc/passwd ",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-shell",
                        "platform-unix",
                        "attack-rce",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/248/88",
                        "PCI/6.5.2"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "XSS Attack Detected via libinjection",
                "details": {
                    "match": "detected XSS using libinjection.",
                    "reference": "v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
                    "ruleId": "941100",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
                    "lineNumber": "82",
                    "data": "Matched Data: XSS data found within ARGS:password: <script>alert('XSS')</script>",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "platform-multi",
                        "attack-xss",
                        "xss-perf-disable",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/242"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "XSS Filter - Category 1: Script Tag Vector",
                "details": {
                    "match": "Matched \"Operator `Rx' with parameter `(?i)<script[^>]*>[\\s\\S]*?' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
                    "reference": "o0,8v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
                    "ruleId": "941110",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
                    "lineNumber": "108",
                    "data": "Matched Data: <script> found within ARGS:password: <script>alert('XSS')</script>",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "platform-multi",
                        "attack-xss",
                        "xss-perf-disable",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/242"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "NoScript XSS InjectionChecker: HTML Injection",
                "details": {
                    "match": "Matched \"Operator `Rx' with parameter `(?i)<[^0-9<>A-Z_a-z]*(?:[^\\s\\x0b\\\"'<>]*:)?[^0-9<>A-Z_a-z]*[^0-9A-Z_a-z]*?(?:s[^0-9A-Z_a-z]*?(?:c[^0-9A-Z_a-z]*?r[^0-9A-Z_a-z]*?i[^0-9A-Z_a-z]*?p[^0-9A-Z_a-z]*?t|t[^0-9A-Z_a-z]*?y[^0-9A-Z_a-z]*?l[^0-9A (4378 characters omitted)' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
                    "reference": "o0,7v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
                    "ruleId": "941160",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
                    "lineNumber": "200",
                    "data": "Matched Data: <script found within ARGS:password: <script>alert('XSS')</script>",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "platform-multi",
                        "attack-xss",
                        "xss-perf-disable",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/242"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Javascript method detected",
                "details": {
                    "match": "Matched \"Operator `Rx' with parameter `(?i)\\b(?:eval|set(?:timeout|interval)|new[\\s\\x0b]+Function|a(?:lert|tob)|btoa|prompt|confirm)[\\s\\x0b]*\\(' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
                    "reference": "o8,6v186,29t:htmlEntityDecode,t:jsDecode",
                    "ruleId": "941390",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
                    "lineNumber": "713",
                    "data": "Matched Data: alert( found within ARGS:password: <script>alert('XSS')</script>",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "attack-xss",
                        "xss-perf-disable",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/242"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "SQL Injection Attack Detected via libinjection",
                "details": {
                    "match": "detected SQLi using libinjection.",
                    "reference": "v157,19",
                    "ruleId": "942100",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf",
                    "lineNumber": "46",
                    "data": "Matched Data: s&sos found within ARGS:username: admin' OR '1'='1;--",
                    "severity": "2",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "application-multi",
                        "language-multi",
                        "platform-multi",
                        "attack-sqli",
                        "paranoia-level/1",
                        "OWASP_CRS",
                        "capec/1000/152/248/66",
                        "PCI/6.5.2"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Inbound Anomaly Score Exceeded (Total Score: 35)",
                "details": {
                    "match": "Matched \"Operator `Ge' with parameter `20' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `35' )",
                    "reference": "",
                    "ruleId": "949110",
                    "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf",
                    "lineNumber": "222",
                    "data": "",
                    "severity": "0",
                    "ver": "OWASP_CRS/4.7.0",
                    "rev": "",
                    "tags": [
                        "anomaly-evaluation",
                        "OWASP_CRS"
                    ],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Inbound Score: 35 Outbound Score: 0",
                "details": {
                    "match": "",
                    "reference": "",
                    "ruleId": "980145",
                    "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
                    "lineNumber": "40",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            }
        ]
    }
}

Here is the curl command:

curl -X POST http://waf.theviscousweb.com \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin' OR '1'='1;--&password=<script>alert('XSS')</script>&cmd=;cat **/etc/passwd;"

Dr-Lazarus-V2 · 2024-11-14T06:38:17Z

@airween, I have also tested it with the custom rule mentioned in: #3277. I was not able to reproduce the bug raised earlier.

Custom Rule:

SecRule RESPONSE_BODY "@rx ^}" "id:911,phase:4, deny, status:403, capture,log, t:none, msg:'multiline active'"

I can see the message key & value in the audit logs

{
    "transaction": {
        "client_ip": "10.103.253.240",
        "time_stamp": "Thu Nov 14 06:33:40 2024",
        "server_id": "135e057d99ba46945f45e685b2bae7652ad9cc73",
        "client_port": 53601,
        "host_ip": "192.168.32.3",
        "host_port": 8080,
        "unique_id": "bdcc99b802f390fe8f0f0d7119e3a0d5",
        "request": {
            "method": "POST",
            "http_version": 1.1,
            "uri": "/check",
            "headers": {
                "Content-Type": "application/json",
                "User-Agent": "PostmanRuntime/7.42.0",
                "Postman-Token": "90021de8-7a7e-4a2d-a78b-1339ea7a1061",
                "Accept": "*/*",
                "Host": "weborion.licensing.portal",
                "Accept-Encoding": "gzip, deflate, br",
                "Connection": "keep-alive",
                "Content-Length": "21"
            }
        },
        "response": {
            "http_code": 403,
            "headers": {
                "Server": "nginx/1.27.2",
                "Date": "Thu, 14 Nov 2024 06:33:40 GMT",
                "Content-Length": "11",
                "Content-Type": "text/plain; charset=utf-8",
                "Connection": "keep-alive"
            }
        },
        "producer": {
            "modsecurity": "ModSecurity v3.0.13 (Linux)",
            "connector": "ModSecurity-nginx v1.0.3",
            "secrules_engine": "Enabled",
            "components": [
                "OWASP_CRS/4.7.0\""
            ]
        },
        "messages": [
            {
                "message": "multiline active",
                "details": {
                    "match": "Matched \"Operator `Rx' with parameter `^}' against variable `RESPONSE_BODY' (Value: `} triggered' )",
                    "reference": "o0,1v259,11",
                    "ruleId": "189",
                    "file": "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf",
                    "lineNumber": "2",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Inbound Score: 0 Outbound Score: 0",
                "details": {
                    "match": "",
                    "reference": "",
                    "ruleId": "980145",
                    "file": "/var/opt/modsecurity.d/owasp-crs/weborion.licensing.portal.crs_setup.conf",
                    "lineNumber": "40",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            }
        ]
    }
}

Dr-Lazarus-V2 · 2024-11-14T06:58:04Z

Outside of that, I do not see any critical bugs in this PR and it looks stable to me.

airween · 2024-11-14T15:22:12Z

Hi @Dr-Lazarus-V2,

thanks for the very detailed explanation.

I tried - I think - the same config as you have:

I replaced my modsecurity.conf with yours, changed only the logs' paths
I use CRS 4.9-dev, but added your rule with id:187 into EXCLUSION-RULES-BEFORE-CRS.conf
created a new server with hostname waf.test
used the same curl request that you gave here

But I still get empty messages key in the audit.log:

{
  "transaction": {
    "client_ip": "127.0.0.1",
    "time_stamp": "Thu Nov 14 16:05:58 2024",
    "server_id": "0d4d434e96996020298a3e094b066f66e6634d3b",
    "client_port": 53650,
    "host_ip": "127.0.0.1",
    "host_port": 8080,
    "unique_id": "173159675881.617821",
    "request": {
      "method": "GET",
      "http_version": 1.1,
      "uri": "/healthz",
      "headers": {
        "Host": "waf.test:8080",
        "User-Agent": "curl/8.10.1",
        "Accept": "*/*",
        "Content-Length": "0",
        "Content-Type": "application/x-www-form-urlencoded"
      }
    },
    "response": {
      "http_code": 403,
      "headers": {
        "Server": "nginx/1.26.0",
        "Date": "Thu, 14 Nov 2024 15:05:58 GMT",
        "Content-Length": "101",
        "Content-Type": "text/html",
        "Last-Modified": "Thu, 17 Oct 2024 11:36:28 GMT",
        "Connection": "keep-alive",
        "ETag": "\"6710f6bc-65\""
      }
    },
    "producer": {
      "modsecurity": "ModSecurity v3.0.13 (Linux)",
      "connector": "ModSecurity-nginx v1.0.3",
      "secrules_engine": "Enabled",
      "components": [
        "OWASP_CRS/4.9.0-dev\""
      ]
    },
    "messages": []
  }
}

In the error.log I get this:

2024/11/14 16:05:58 [error] 43687#43687: *1 [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)leak' against variable `RESPONSE_BODY' (Value: `leak!' ) [file "/home/airween/src/coreruleset/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf"] [line "205"] [id "187"] [rev ""] [msg " 'UNAUTHORIZED RESPONSE DATA ACCESS'"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "127.0.0.1"] [uri "/healthz"] [unique_id "173159675897.213267"] [ref "o0,4o0,0v145,5"], client: 127.0.0.1, server: waf.test, request: "POST /healthz HTTP/1.1", host: "waf.test:8080"

I continue the investigation.

airween · 2024-11-14T15:28:01Z

@airween , I attempted to reproduce the core issue concerning the unavailability of messages in the audit logs, and I was not able to reproduce it. I can successfully view the messages in the audit logs.

With the given curl request I also see the messages in audit.log - but that's okay, probably because the intervention happens in phase:2, not in phase:4. So it's okay.

The question is to me why can't I see the messages in audit.log if the intervention happens in phase:4 (with this patch, of course).

Dr-Lazarus-V2 · 2024-11-15T01:59:55Z

That's interesting. In my system, I use the OWASP Modsecurity-CRS Docker Image and I modify the nginx connector in the Dockerfile. I change the following lines in this file nginx/Dockerfile:

Old Line
RUN set -eux; \ git clone -b master --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git; \

New Line
RUN set -eux; \ git clone -b fix_response_body_v1.0.3 --depth 1 https://github.com/g00g1/ModSecurity-nginx.git; \

I've set up this configuration as my WAF, and based on your audit logs, it’s evident that our configurations are largely similar, except for the Nginx server version. I'm running version 1.27.2 on my end, which I don’t believe should cause any issues.

I have used the same rule:

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
    "id:10, \
    phase:4,\
    deny,\
    status:403,\
    log,\
    t:none,\
    msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

Here is the audit log:

{
    "transaction": {
        "client_ip": "10.103.253.240",
        "time_stamp": "Fri Nov 15 01:43:06 2024",
        "server_id": "6eaa3f3dd6bdf64b45eb7031a0aeddb38c480a84",
        "client_port": 62070,
        "host_ip": "192.168.32.3",
        "host_port": 8080,
        "unique_id": "000f40896fa2781e7e9aba6f05b81d09",
        "request": {
            "method": "GET",
            "http_version": 1.1,
            "uri": "/check",
            "headers": {
                "User-Agent": "PostmanRuntime/7.42.0",
                "Postman-Token": "2a09f719-9cf5-48f7-ba1c-52e5e1e89572",
                "Accept": "*/*",
                "Host": "weborion.licensing.portal",
                "Accept-Encoding": "gzip, deflate, br",
                "Connection": "keep-alive"
            }
        },
        "response": {
            "http_code": 403,
            "headers": {
                "Server": "nginx/1.27.2",
                "Date": "Fri, 15 Nov 2024 01:43:06 GMT",
                "Content-Length": "4",
                "Content-Type": "text/plain; charset=utf-8",
                "Connection": "keep-alive"
            }
        },
        "producer": {
            "modsecurity": "ModSecurity v3.0.13 (Linux)",
            "connector": "ModSecurity-nginx v1.0.3",
            "secrules_engine": "Enabled",
            "components": [
                "OWASP_CRS/4.7.0\""
            ]
        },
        "messages": [
            {
                "message": " 'UNAUTHORIZED RESPONSE DATA ACCESS'",
                "details": {
                    "match": "Matched \"Operator `Rx' with parameter `(?i)(\\n|\\''|\\:|\\W*)leak' against variable `RESPONSE_BODY' (Value: `leak' )",
                    "reference": "o0,4o0,0v208,4",
                    "ruleId": "10",
                    "file": "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf",
                    "lineNumber": "2",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            },
            {
                "message": "Inbound Score: 0 Outbound Score: 0",
                "details": {
                    "match": "",
                    "reference": "",
                    "ruleId": "980145",
                    "file": "/var/opt/modsecurity.d/owasp-crs/weborion.licensing.portal.crs_setup.conf",
                    "lineNumber": "40",
                    "data": "",
                    "severity": "0",
                    "ver": "",
                    "rev": "",
                    "tags": [],
                    "maturity": "0",
                    "accuracy": "0"
                }
            }
        ]
    }
}

Here is the error log:

2024/11/15 01:30:46 [error] 90#90: *5 [client 10.103.253.240] ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)leak' against variable `RESPONSE_BODY' (Value: `leak' ) [file "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf"] [line "2"] [id "10"] [rev ""] [msg " 'UNAUTHORIZED RESPONSE DATA ACCESS'"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "192.168.32.3"] [uri "/check"] [unique_id "4bd7fa0fac010feeb629092cd4c416d4"] [ref "o0,4o0,0v259,4"] while sending to client, client: 10.103.253.240, server: weborion.licensing.portal, request: "POST /check HTTP/1.1", upstream: "http://10.103.253.191:5000/check", host: "weborion.licensing.portal"

Modsecurity-CRS Docker Image
I have also attached the docker image as a tar file, if it is of any assistance. Here is the modified docker file to build the image. You can reference it from here, https://github.com/coreruleset/modsecurity-crs-docker/blob/main/nginx/Dockerfile. I have made the line change for the modsecurity-nginx connector. Here:

ARG NGINX_VERSION="n/a"

FROM nginxinc/nginx-unprivileged:${NGINX_VERSION} AS build

ARG MODSEC3_VERSION="n/a"
ARG LMDB_VERSION="n/a"
ARG LUA_VERSION="n/a"

USER root

# Note: libpcre3-dev (PCRE 1) is required by the build description,
# even though the build will use PCRE2.
RUN set -eux; \
    echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections; \
    apt-get update -qq; \
    LD_LIBRARY_PATH="" apt-get install -y -qq --no-install-recommends --no-install-suggests \
    automake \
    cmake \
    doxygen \
    g++ \
    git \
    libcurl4-gnutls-dev \
    libfuzzy-dev \
    liblua${LUA_VERSION}-dev \
    libpcre3-dev \
    libpcre2-dev \
    libtool \
    libxml2-dev \
    libmaxminddb-dev \
    libyajl-dev \
    make \
    patch \
    pkg-config \
    ruby \
    zlib1g-dev; \
    apt-get clean; \
    rm -rf /var/lib/apt/lists/*

WORKDIR /sources

RUN set -eux; \
    git clone https://github.com/LMDB/lmdb --branch LMDB_${LMDB_VERSION} --depth 1; \
    make -C lmdb/libraries/liblmdb install; \
    strip /usr/local/lib/liblmdb*.so*

RUN set -eux; \
    git clone https://github.com/owasp-modsecurity/ModSecurity --branch "v${MODSEC3_VERSION}" --depth 1 --recursive; \
    cd ModSecurity; \
    ARCH=$(gcc -print-multiarch); \
    sed -ie "s/i386-linux-gnu/${ARCH}/g" build/ssdeep.m4; \
    sed -ie "s/i386-linux-gnu/${ARCH}/g" build/pcre2.m4; \
    ./build.sh; \
    ./configure --with-yajl --with-ssdeep --with-pcre2 --with-maxmind --enable-silent-rules; \
    make install; \
    strip /usr/local/modsecurity/lib/lib*.so*

# We use master
RUN set -eux; \
    git clone -b fix_response_body_v1.0.3 --depth 1 https://github.com/g00g1/ModSecurity-nginx.git; \
    curl -sSL https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o nginx-${NGINX_VERSION}.tar.gz; \
    tar -xzf nginx-${NGINX_VERSION}.tar.gz; \
    cd ./nginx-${NGINX_VERSION}; \
    ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx; \
    make modules; \
    strip objs/ngx_http_modsecurity_module.so; \
    cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/; \
    mkdir /etc/modsecurity.d; \
    curl -sSL https://raw.githubusercontent.com/owasp-modsecurity/ModSecurity/v3/master/unicode.mapping \
    -o /etc/modsecurity.d/unicode.mapping

# Generate/Download Diffie-Hellman parameter files
RUN set -eux; \
    mkdir -p /usr/share/TLS; \
    curl -sSL https://ssl-config.mozilla.org/ffdhe2048.txt -o /usr/share/TLS/dhparam-2048.pem; \
    curl -sSL https://ssl-config.mozilla.org/ffdhe4096.txt -o /usr/share/TLS/dhparam-4096.pem

FROM nginxinc/nginx-unprivileged:${NGINX_VERSION} AS crs_release

ARG CRS_RELEASE

USER root

# hadolint ignore=DL3008,SC2016
RUN set -eux; \
    apt-get update; \
    apt-get -y install --no-install-recommends \
    ca-certificates \
    curl \
    gnupg; \
    mkdir /opt/owasp-crs; \
    chown nginx:nginx /opt/owasp-crs

USER nginx

WORKDIR /sources

RUN curl -sSL https://github.com/coreruleset/coreruleset/releases/download/v${CRS_RELEASE}/coreruleset-${CRS_RELEASE}-minimal.tar.gz -o v${CRS_RELEASE}-minimal.tar.gz; \
    curl -sSL https://github.com/coreruleset/coreruleset/releases/download/v${CRS_RELEASE}/coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc -o coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc; \
    gpg --fetch-key https://coreruleset.org/security.asc; \
    gpg --verify coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc v${CRS_RELEASE}-minimal.tar.gz; \
    tar -zxf v${CRS_RELEASE}-minimal.tar.gz --strip-components=1 -C /opt/owasp-crs; \
    rm -f v${CRS_RELEASE}-minimal.tar.gz coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc; \
    mv -v /opt/owasp-crs/crs-setup.conf.example /opt/owasp-crs/crs-setup.conf

FROM nginxinc/nginx-unprivileged:${NGINX_VERSION}

ARG MODSEC3_VERSION
ARG LMDB_VERSION
ARG LUA_VERSION
ARG LUA_MODULES

LABEL maintainer="Felipe Zipitria <felipe.zipitria@owasp.org>"

ENV \
    ACCESSLOG=/var/log/nginx/access.log \
    BACKEND=http://localhost:80 \
    DNS_SERVER= \
    ERRORLOG=/var/log/nginx/error.log \
    KEEPALIVE_TIMEOUT=60s \
    LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib \
    LOGLEVEL=warn \
    METRICS_ALLOW_FROM='127.0.0.0/24' \
    METRICS_DENY_FROM='all' \
    METRICSLOG=/dev/null \
    MODSEC_ARGUMENT_SEPARATOR="&" \
    MODSEC_ARGUMENTS_LIMIT=1000 \
    MODSEC_AUDIT_ENGINE="RelevantOnly" \
    MODSEC_AUDIT_LOG=/dev/stdout \
    MODSEC_AUDIT_LOG_FORMAT=JSON \
    MODSEC_AUDIT_LOG_PARTS='ABIJDEFHZ' \
    MODSEC_AUDIT_LOG_RELEVANT_STATUS="^(?:5|4(?!04))" \
    MODSEC_AUDIT_LOG_TYPE=Serial \
    MODSEC_COOKIE_FORMAT=0 \
    MODSEC_AUDIT_STORAGE_DIR=/var/log/modsecurity/audit/ \
    MODSEC_DATA_DIR=/tmp/modsecurity/data \
    MODSEC_DEBUG_LOG=/dev/null \
    MODSEC_DEBUG_LOGLEVEL=0 \
    MODSEC_DEFAULT_PHASE1_ACTION="phase:1,pass,log,tag:'\${MODSEC_TAG}'" \
    MODSEC_DEFAULT_PHASE2_ACTION="phase:2,pass,log,tag:'\${MODSEC_TAG}'" \
    MODSEC_DISABLE_BACKEND_COMPRESSION="Off" \
    MODSEC_PCRE_MATCH_LIMIT=100000 \
    MODSEC_PCRE_MATCH_LIMIT_RECURSION=100000 \
    MODSEC_REQ_BODY_ACCESS=on \
    MODSEC_REQ_BODY_JSON_DEPTH_LIMIT=512 \
    MODSEC_REQ_BODY_LIMIT=13107200 \
    MODSEC_REQ_BODY_LIMIT_ACTION="Reject" \
    MODSEC_REQ_BODY_NOFILES_LIMIT=131072 \
    MODSEC_RESP_BODY_ACCESS=on \
    MODSEC_RESP_BODY_LIMIT=1048576 \
    MODSEC_RESP_BODY_LIMIT_ACTION="ProcessPartial" \
    MODSEC_RESP_BODY_MIMETYPE="text/plain text/html text/xml" \
    MODSEC_RULE_ENGINE=on \
    MODSEC_STATUS_ENGINE="Off" \
    MODSEC_TAG=modsecurity \
    MODSEC_TMP_DIR=/tmp/modsecurity/tmp \
    MODSEC_TMP_SAVE_UPLOADED_FILES="on" \
    MODSEC_UNICODE_MAPPING=20127 \
    MODSEC_UPLOAD_DIR=/tmp/modsecurity/upload \
    MODSEC_UPLOAD_FILE_MODE=0600 \
    MODSEC_UPLOAD_KEEP_FILES=Off \
    NGINX_ALWAYS_TLS_REDIRECT=off \
    NGINX_ENVSUBST_OUTPUT_DIR=/etc/nginx \
    PORT=8080 \
    PROXY_SSL_CERT=/etc/nginx/conf/proxy.crt \
    PROXY_SSL_CERT_KEY=/etc/nginx/conf/proxy.key \
    PROXY_SSL_CIPHERS="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
    PROXY_SSL=off \
    PROXY_SSL_PROTOCOLS="TLSv1.2 TLSv1.3" \
    PROXY_SSL_VERIFY_DEPTH=1 \
    PROXY_SSL_VERIFY=off \
    PROXY_TIMEOUT=60s \
    REAL_IP_HEADER="X-REAL-IP" \
    REAL_IP_PROXY_HEADER="X-REAL-IP" \
    REAL_IP_RECURSIVE="on" \
    SERVER_NAME=localhost \
    SERVER_TOKENS=off \
    SET_REAL_IP_FROM="127.0.0.1" \
    SSL_CERT=/etc/nginx/conf/server.crt \
    SSL_CERT_KEY=/etc/nginx/conf/server.key \
    SSL_CIPHERS="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
    SSL_DH_BITS=2048 \
    SSL_OCSP_STAPLING=on \
    SSL_PORT=8443 \
    SSL_PREFER_CIPHERS=off \
    SSL_PROTOCOLS="TLSv1.2 TLSv1.3" \
    SSL_VERIFY_DEPTH=1 \
    SSL_VERIFY=off \
    WORKER_CONNECTIONS=1024 \
    # CRS specific variables
    PARANOIA=1 \
    ANOMALY_INBOUND=5 \
    ANOMALY_OUTBOUND=4 \
    BLOCKING_PARANOIA=1

COPY --from=build /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/
COPY --from=build /etc/nginx/modules/ngx_http_modsecurity_module.so /etc/nginx/modules/ngx_http_modsecurity_module.so
COPY --from=build /usr/local/lib/liblmdb.so /usr/local/lib/
COPY --from=build /usr/share/TLS/dhparam-* /etc/ssl/certs/
COPY --from=build /etc/modsecurity.d/unicode.mapping /etc/modsecurity.d/unicode.mapping
COPY --from=crs_release /opt/owasp-crs /opt/owasp-crs
COPY src/etc/modsecurity.d/modsecurity.conf /etc/nginx/templates/modsecurity.d/modsecurity.conf.template
COPY src/etc/modsecurity.d/modsecurity-override.conf /etc/nginx/templates/modsecurity.d/modsecurity-override.conf.template
COPY src/etc/modsecurity.d/setup.conf /etc/nginx/templates/modsecurity.d/setup.conf.template
COPY nginx/docker-entrypoint.d/*.sh /docker-entrypoint.d/
COPY src/opt/modsecurity/activate-plugins.sh /docker-entrypoint.d/94-activate-plugins.sh
COPY src/opt/modsecurity/activate-rules.sh /docker-entrypoint.d/95-activate-rules.sh
# We use the templating mechanism from the nginx image here.
COPY nginx/templates /etc/nginx/templates/
COPY src/bin/* /usr/local/bin/

USER root

RUN set -eux; \
    echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections; \
    apt-get update -qq; \
    LD_LIBRARY_PATH="" apt-get install -y -qq --no-install-recommends --no-install-suggests \
    ca-certificates \
    curl \
    libcurl4-gnutls-dev \
    libfuzzy2 \
    liblua${LUA_VERSION} \
    ${LUA_MODULES} \
    libxml2 \
    libyajl2 \
    libmaxminddb-dev \
    moreutils; \
    rm -rf /var/lib/apt/lists/*; \
    apt-get clean; \
    mkdir /etc/nginx/ssl; \
    # Comment out the SecDisableBackendCompression option since it is not supported in V3
    sed -i 's/^\(SecDisableBackendCompression .*\)/# \1/' /etc/nginx/templates/modsecurity.d/modsecurity-override.conf.template; \
    ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so.3.0; \
    ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so.3; \
    ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so; \
    ln -sv /opt/owasp-crs /etc/modsecurity.d/; \
    chown nginx:nginx /opt/owasp-crs /etc/modsecurity.d

USER nginx

RUN mkdir -p /tmp/modsecurity/data; \
    mkdir -p /tmp/modsecurity/upload; \
    mkdir -p /tmp/modsecurity/tmp

HEALTHCHECK CMD /usr/local/bin/healthcheck

airween · 2024-11-15T07:23:13Z

Hi @Dr-Lazarus-V2,

thanks again for this very detailed report.

I try to review that soon and reconstruct the expected behavior.

airween · 2024-11-17T20:26:40Z

Guys,

seems like I found the root cause of the issue why isn't there any lines under messages key.

If I have a custom configuration like this:

    error_page 403 /403.html;

    location /403.html {
        root /usr/share/nginx/html;
        internal;
    }

then (I think) this block starts a new transaction, which clears all collected messages inside of the transaction. In the second transaction there is no rule match so the messages in audit log will be empty.

Could you confirm this?

Dr-Lazarus-V2 · 2024-11-18T01:08:59Z

Yes, I can confirm this. With the following configuration, I also noticed that in the second transaction (after the new transaction) there is no rule match and the audit logs are empty. I believe it is linked to this comment: #326 (comment)

Dr-Lazarus-V2 · 2024-11-18T01:29:57Z

I believe this issue relates to the previously raised bug in #182, as the problem appears to be recurring here. Since the rule is in Phase 4, there is no body due to the error redirect, which seems to prevent the rule from being triggered. Based on this, I do not believe this PR introduces the bug. As metioned in #182, this seems to be a potential workaround:

Hi,
@martinhsv thanks for providing those workarounds as I am running into the same issue.

I think I found one more way that might help.

It is to use a named location for the error page instead of a URL, like so:

error_page 403 @error;
location @error {
   ...
}
Then NGINX won't change the request method to GET and full audit log will appear with correct messages in part H.

If there is no need to change URI and method during internal redirection it is possible to pass error processing into a named location: [...]
(http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page)

EDIT: I just noticed the reference to https://github.com/owasp-modsecurity/ModSecurity-nginx/issues/152#issuecomment-593259386 which seems to use the same solution (although in a little more complicated way)

However from a deeper investigation, it is clear that the this is not a full workaround but a temporary one. There does seem to be a PR relating to it: #204

airween · 2024-11-18T08:21:34Z

I believe this issue relates to the previously raised bug in #182, as the problem appears to be recurring here. Since the rule is in Phase 4, there is no body due to the error redirect, which seems to prevent the rule from being triggered. Based on this, I do not believe this PR introduces the bug. As metioned in #182, this seems to be a potential workaround:
Hi,
@martinhsv thanks for providing those workarounds as I am running into the same issue.

I think I found one more way that might help.

It is to use a named location for the error page instead of a URL, like so:

error_page 403 @error;
location @error {
   ...
}
Then NGINX won't change the request method to GET and full audit log will appear with correct messages in part H.

If there is no need to change URI and method during internal redirection it is possible to pass error processing into a named location: [...]
(http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page)

EDIT: I just noticed the reference to https://github.com/owasp-modsecurity/ModSecurity-nginx/issues/152#issuecomment-593259386 which seems to use the same solution (although in a little more complicated way)
However from a deeper investigation, it is clear that the this is not a full workaround but a temporary one. There does seem to be a PR relating to it: #204

Yes, I saw this and other redirect related fixes. Now I try to make a new PR which merges all of these.

Dr-Lazarus-V2 · 2024-11-18T08:32:40Z

Is there anything I can do to help?

airween · 2024-11-18T08:45:36Z

Is there anything I can do to help?

If you have any C experiences, you can try to pick up any of related PR from the pending ones and try to apply those on this one. Or later (if I can progress with the patch) you can help to test that.

airween · 2024-11-19T14:10:06Z

I tried to add these modifications.

The good news is that there is no new transaction, I see only one transaction in debug.log.

The bad news that there is no audit.log at all. (Audit.log had created but it's empty.)

airween · 2024-11-19T15:22:31Z

Okay, picked up the modifications from #273 (that's an old PR and it's not able to build) the audit.log is generated, no new transaction started and this patch also works as well.

PR #204 is not necessary.

I'm going to add a new PR soon.

Dr-Lazarus-V2 · 2024-11-21T01:21:51Z

Hey was caught up in other work, I will have a look at it now!

Fix owasp-modsecurity#41

62639fa

Dr-Lazarus-V2 mentioned this pull request Oct 25, 2024

Custom Response Body Rule Not Blocking for Responses Larger Than 1KB owasp-modsecurity/ModSecurity#3282

Open

airween mentioned this pull request Nov 19, 2024

fix: response body (based on #326) #334

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix response body (based on #84) #326

Fix response body (based on #84) #326

g00g1 commented Jun 18, 2024

Dr-Lazarus-V2 commented Oct 25, 2024

airween commented Oct 31, 2024

g00g1 commented Oct 31, 2024

Dr-Lazarus-V2 commented Nov 5, 2024

airween commented Nov 5, 2024

airween commented Nov 7, 2024

g00g1 commented Nov 7, 2024 •

edited

Loading

airween commented Nov 7, 2024

g00g1 commented Nov 7, 2024

airween commented Nov 7, 2024

airween commented Nov 7, 2024

Dr-Lazarus-V2 commented Nov 12, 2024

Dr-Lazarus-V2 commented Nov 12, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 12, 2024 •

edited

Loading

airween commented Nov 13, 2024

airween commented Nov 13, 2024

Dr-Lazarus-V2 commented Nov 14, 2024

Dr-Lazarus-V2 commented Nov 14, 2024

Dr-Lazarus-V2 commented Nov 14, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 14, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 14, 2024

airween commented Nov 14, 2024

airween commented Nov 14, 2024

Dr-Lazarus-V2 commented Nov 15, 2024

airween commented Nov 15, 2024

airween commented Nov 17, 2024

Dr-Lazarus-V2 commented Nov 18, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 18, 2024 •

edited

Loading

airween commented Nov 18, 2024

Dr-Lazarus-V2 commented Nov 18, 2024

airween commented Nov 18, 2024

airween commented Nov 19, 2024

airween commented Nov 19, 2024

Dr-Lazarus-V2 commented Nov 21, 2024

Fix response body (based on #84) #326

Are you sure you want to change the base?

Fix response body (based on #84) #326

Conversation

g00g1 commented Jun 18, 2024

Dr-Lazarus-V2 commented Oct 25, 2024

airween commented Oct 31, 2024

g00g1 commented Oct 31, 2024

Dr-Lazarus-V2 commented Nov 5, 2024

airween commented Nov 5, 2024

airween commented Nov 7, 2024

g00g1 commented Nov 7, 2024 • edited Loading

airween commented Nov 7, 2024

g00g1 commented Nov 7, 2024

airween commented Nov 7, 2024

airween commented Nov 7, 2024

Dr-Lazarus-V2 commented Nov 12, 2024

Dr-Lazarus-V2 commented Nov 12, 2024 • edited Loading

Dr-Lazarus-V2 commented Nov 12, 2024 • edited Loading

airween commented Nov 13, 2024

airween commented Nov 13, 2024

Dr-Lazarus-V2 commented Nov 14, 2024

Dr-Lazarus-V2 commented Nov 14, 2024

Bug Case:

Bug Free Case:

Dr-Lazarus-V2 commented Nov 14, 2024 • edited Loading

Dr-Lazarus-V2 commented Nov 14, 2024 • edited Loading

Dr-Lazarus-V2 commented Nov 14, 2024

airween commented Nov 14, 2024

airween commented Nov 14, 2024

Dr-Lazarus-V2 commented Nov 15, 2024

airween commented Nov 15, 2024

airween commented Nov 17, 2024

Dr-Lazarus-V2 commented Nov 18, 2024 • edited Loading

Dr-Lazarus-V2 commented Nov 18, 2024 • edited Loading

airween commented Nov 18, 2024

Dr-Lazarus-V2 commented Nov 18, 2024

airween commented Nov 18, 2024

airween commented Nov 19, 2024

airween commented Nov 19, 2024

Dr-Lazarus-V2 commented Nov 21, 2024

g00g1 commented Nov 7, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 12, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 12, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 14, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 14, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 18, 2024 •

edited

Loading

Dr-Lazarus-V2 commented Nov 18, 2024 •

edited

Loading