diff --git a/public/_redirects b/public/_redirects index 24c1aa0a5ee72e..3db8d70a1602dd 100644 --- a/public/_redirects +++ b/public/_redirects @@ -1605,6 +1605,7 @@ /cloudflare-one/identity/idp-integration/saml-okta/ /cloudflare-one/identity/idp-integration/okta-saml/ 301 /cloudflare-one/identity/idp-integration/workspace-one/ /cloudflare-one/identity/devices/service-providers/workspace-one/ 301 /cloudflare-one/identity/login-page/ /cloudflare-one/applications/login-page/ 301 +/cloudflare-one/insights/logs/logpush/rdata/ /cloudflare-one/insights/logs/logpush/#parse-logpush-logs 301 /cloudflare-one/applications/custom-pages/ /cloudflare-one/applications/ 301 /cloudflare-one/identity/service-auth/service-tokens/ /cloudflare-one/identity/service-tokens/ 301 /cloudflare-one/identity/users/validating-json/ /cloudflare-one/identity/authorization-cookie/validating-json/ 301 diff --git a/src/content/docs/cloudflare-one/insights/logs/gateway-logs/index.mdx b/src/content/docs/cloudflare-one/insights/logs/gateway-logs/index.mdx index 3d43889effb5b6..e51343f4fb3e29 100644 --- a/src/content/docs/cloudflare-one/insights/logs/gateway-logs/index.mdx +++ b/src/content/docs/cloudflare-one/insights/logs/gateway-logs/index.mdx @@ -5,7 +5,7 @@ sidebar: order: 3 --- -import { Render } from "~/components"; +import { Render, GlossaryTooltip } from "~/components"; :::note[Private source IP substitution] @@ -33,28 +33,25 @@ These settings will only apply to logs displayed in Zero Trust. Logpush data is | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **DNS** | Name of the domain that was queried. | +| **Query name** | Name of the domain that was queried. | +| **Query ID** | UUID of the query assigned by Cloudflare. | | **Email** | Email address of the user who registered the WARP client where traffic originated from. If a non-identity on-ramp (such as a [proxy endpoint](/cloudflare-one/connections/connect-devices/agentless/pac-files/)) or machine-level authentication (such as a [service token](/cloudflare-one/identity/service-tokens/)) was used, this value will be `non_identity@.cloudflareaccess.com`. | | **Action** | The [Action](/cloudflare-one/policies/gateway/dns-policies/#actions) Gateway applied to the query (such as Allow or Block). | | **Time** | Date and time of the DNS query. | -| **Resolver Decision** | The reason why Gateway applied a particular **Action** to the request. Refer to the [list of resolver decisions](#resolver-decisions). | +| **Resolver decision** | The reason why Gateway applied a particular **Action** to the request. Refer to the [list of resolver decisions](#resolver-decisions). | +| **Resolved IPs** | Resolved IP addresses in the response. | +| **CNAMEs** | `CNAME` records in the query. | -#### Matched policies - -| Field | Description | -| ---------------------- | ---------------------------------------------------- | -| **Policy Name** | Name of the matched policy (if there is one). | -| **Policy ID** | ID of the matched policy (if there is one). | -| **Policy Description** | Description of the matched policy (if there is one). | - -#### Custom resolver +#### Configuration information -| Field | Description | -| -------------------------- | ----------------------------------------------------------- | -| **Address** | Address of your custom resolver. | -| **Policy** | Name of the matched resolver policy. | -| **Response** | Status of the custom resolver response. | -| **Time (in milliseconds)** | Duration of time it took for the custom resolver to respond | +| Field | Description | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| **DNS location** | [User-configured location](/cloudflare-one/connections/connect-devices/agentless/dns/locations/) from where the DNS query was made. | +| **Policy name** | Name of the matched policy. | +| **Policy ID** | ID of the matched policy. | +| **Policy description** | Description of the matched policy. | +| **DoH subdomain** | DoH subdomain of the DNS location. | +| **Protocol** | Protocol that was used to make the DNS query (such as `https`). | #### Identities @@ -62,28 +59,52 @@ These settings will only apply to logs displayed in Zero Trust. Logpush data is | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Email** | Email address of the user who registered the WARP client where traffic originated from. | | **User ID** | UUID of the user. Each unique email address in your organization will have a UUID associated with it. | -| **Device Name** | Display name of the device returned by the operating system to the WARP client. Typically this is the hostname of a device. Not all devices will have a device name. Device names are not guaranteed to be unique. | +| **Device name** | Display name of the device returned by the operating system to the WARP client. Typically this is the hostname of a device. Not all devices will have a device name. Device names are not guaranteed to be unique. | | **Device ID** | UUID of the device connected with the WARP client. Each unique device in your organization will have a UUID associated with it each time the device is registered for a particular email. The same physical device may have multiple UUIDs associated with it. | | **Last authenticated** | Date and time the user last authenticated their Zero Trust session. | #### DNS query details -| Field | Description | -| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| **Query Type** | Type of [DNS query](https://en.wikipedia.org/wiki/List_of_DNS_record_types). | -| **Query Category** | [Content categories](/cloudflare-one/policies/gateway/domain-categories/) that the domain belongs to. | -| **Matched Categories** | Name of the Gateway policy category that match the domain. | -| **Matched Indicator Feed Name** | Name of the indicator feeds that matched a Gateway policy (if any). | -| **Query Indicator Feed Name** | Name of the indicator feeds that a matched domain or IP belongs to (if any). | -| **Source IP** | Public source IP address of the DNS query. | -| **Source IP Country** | Country code of the DNS query. | -| **Source Internal IP** | Private IP address assigned by the user's local network (if any). | -| **Resolver IP** | Public IP address of the DNS resolver. | -| **Resolved IPs** | Resolved IP addresses in the response (if any). | -| **Port** | Port that was used to make the DNS query. | -| **Protocol** | Protocol that was used to make the DNS query (such as `https`). | -| **DNS Location** | [User-configured location](/cloudflare-one/connections/connect-devices/agentless/dns/locations/) from where the DNS query was made. | -| **Location ID** | ID of the DNS location where the query originated. | +| Field | Description | +| ------------------------------------------ | ----------------------------------------------------------------------------------------------------- | +| **Query ID** | UUID of the query assigned by Cloudflare. | +| **Query type** | Type of [DNS query](https://en.wikipedia.org/wiki/List_of_DNS_record_types). | +| **Initial query domain categories** | [Content categories](/cloudflare-one/policies/gateway/domain-categories/) that the domain belongs to. | +| **Matched categories** | Name of the Gateway policy category that match the domain. | +| **Matched indicator feed names** | Name of the indicator feeds that matched a Gateway policy. | +| **Query indicator feed names** | Name of the indicator feeds that a matched domain or IP belongs to. | +| **Resolved continent IP geolocation** | Continent code of the resolved IP address. | +| **Resolved country IP geolocation** | Country code of the resolved IP address. | +| **DoT subdomain** | DoT subdomain of the DNS location. | +| **Source IP** | Public source IP address of the DNS query. | +| **Source IP continent** | Continent code of the source IP address. | +| **Source IP country** | Country code of the source IP address. | +| **Source internal IP** | Private IP address assigned by the user's local network. | +| **Application name** | Name of the application that matched the domain. | +| **Resolver IP** | Public IP address of the DNS resolver. | +| **Port** | Port that was used to make the DNS query. | +| **Location ID** | ID of the DNS location where the query originated. | +| **Scheduling - Time zone** | Time zone of the DNS query source. | +| **Scheduling - Time zone inferred method** | Method used to determine the DNS query source's time zone. | + +#### DNS response details + +| Field | Description | +| ------------------------------- | ---------------------------------------------------------------------------------------- | +| **Resolved CNAME categories** | Content categories associated with the resolved `CNAME` records in the response. | +| **Resolved IP categories** | Content categories associated with the resolved IPs in the response. | +| **Resolved IPs** | Resolved IPs in the response. | +| **Authoritative nameserver IP** | IP address of the authoritative nameserver answering the DNS query. | +| **EDE errors** | [Extended DNS error codes](https://www.rfc-editor.org/rfc/rfc8914.html) in the response. | + +#### Custom resolver + +| Field | Description | +| -------------------------- | ------------------------------------------------------------ | +| **Address** | Address of your custom resolver. | +| **Policy** | Name of the matched resolver policy. | +| **Response** | Status of the custom resolver response. | +| **Time (in milliseconds)** | Duration of time it took for the custom resolver to respond. | ### Resolver decisions @@ -113,7 +134,7 @@ Gateway will only log failed connections in [network session logs](/logs/referen | Field | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Source IP** | IP address of the user sending the packet. | -| **Source Internal IP** | Private IP address assigned by the user's local network. | +| **Source internal IP** | Private IP address assigned by the user's local network. | | **Destination IP** | IP address of the packet's target. | | **Action** | The Gateway [Action](/cloudflare-one/policies/gateway/dns-policies/#actions) taken based on the first rule that matched (such as Allow or Block). | | **Session ID** | ID of the unique session. | @@ -123,9 +144,9 @@ Gateway will only log failed connections in [network session logs](/logs/referen | Field | Description | | ---------------------- | ----------------------------------------------------- | -| **Policy Name** | Name of the matched policy (if there is one). | +| **Policy name** | Name of the matched policy. | | **Policy ID** | ID of the policy enforcing the decision Gateway made. | -| **Policy Description** | Description of the matched policy (if there is one). | +| **Policy description** | Description of the matched policy. | #### Identities @@ -133,26 +154,26 @@ Gateway will only log failed connections in [network session logs](/logs/referen | ---------------------- | ----------------------------------------------------------------------------------- | | **Email** | Email address of the user sending the packet. This is generated by the WARP client. | | **User ID** | ID of the user sending the packet. This is generated by the WARP client. | -| **Device Name** | Name of the device that sent the packet. | +| **Device name** | Name of the device that sent the packet. | | **Device ID** | ID of the device that sent the packet. This is generated by the WARP client. | -| **Last Authenticated** | Date and time the user last authenticated with Zero Trust. | +| **Last authenticated** | Date and time the user last authenticated with Zero Trust. | #### Network query details | Field | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Source IP** | IP address of the user sending the packet. | -| **Source Port** | Source port number for the packet. | -| **Source Country** | Country code for the packet source. | +| **Source port** | Source port number for the packet. | +| **Source country** | Country code for the packet source. | | **Destination IP** | IP address of the packet's target. | -| **Destination Port** | Destination port number for the packet. | -| **Destination Country** | Destination port number for the packet. | +| **Destination port** | Destination port number for the packet. | +| **Destination country** | Destination port number for the packet. | | **Protocol** | Protocol over which the packet was sent. | -| **Detected Protocol** | The detected [network protocol](/cloudflare-one/policies/gateway/network-policies/protocol-detection/). | +| **Detected protocol** | The detected [network protocol](/cloudflare-one/policies/gateway/network-policies/protocol-detection/). | | **SNI** | Host whose Server Name Indication (SNI) header Gateway will filter traffic against. | -| **Virtual Network** | [Virtual network](/cloudflare-one/connections/connect-networks/private-net/cloudflared/tunnel-virtual-networks/) that the client is connected to. | +| **Virtual network** | [Virtual network](/cloudflare-one/connections/connect-networks/private-net/cloudflared/tunnel-virtual-networks/) that the client is connected to. | | **Category details** | Category or categories associated with the packet. | -| **Proxy PAC Endpoint** | [PAC file proxy endpoint](/cloudflare-one/connections/connect-devices/agentless/pac-files/) Gateway forwarded traffic to, if applicable. | +| **Proxy PAC endpoint** | [PAC file proxy endpoint](/cloudflare-one/connections/connect-devices/agentless/pac-files/) Gateway forwarded traffic to, if applicable. | ## HTTP logs @@ -173,20 +194,20 @@ When an HTTP request results in an error, Gateway logs the first 512 bytes of th | **Action** | The Gateway [Action](/cloudflare-one/policies/gateway/dns-policies/#actions) taken based on the first rule that matched (such as Allow or Block). | | **Request ID** | Unique ID of the request. | | **Time** | Date and time of the HTTP request. | -| **Source Internal IP** | Private IP address assigned by the user's local network. | -| **User Agent** | User agent header sent in the request by the originating device. | +| **Source internal IP** | Private IP address assigned by the user's local network. | +| **User agent** | User agent header sent in the request by the originating device. | | **Policy details** | Policy corresponding to the decision Gateway made based on the traffic criteria of the request. | -| **DLP profiles** | Name of the matched [DLP profile](/cloudflare-one/policies/data-loss-prevention/dlp-profiles/) (if there is one). | -| **DLP profile entries** | Name of the matched entry within the DLP profile (if there is one). | +| **DLP profiles** | Name of the matched [DLP profile](/cloudflare-one/policies/data-loss-prevention/dlp-profiles/). | +| **DLP profile entries** | Name of the matched entry within the DLP profile. | | **Uploaded/downloaded file** | | #### Matched policies -| Field | Description | -| ---------------------- | ---------------------------------------------------- | -| **Policy Name** | Name of the matched policy (if there is one). | -| **Policy ID** | ID of the matched policy (if there is one). | -| **Policy Description** | Description of the matched policy (if there is one). | +| Field | Description | +| ---------------------- | ---------------------------------- | +| **Policy name** | Name of the matched policy. | +| **Policy ID** | ID of the matched policy. | +| **Policy description** | Description of the matched policy. | #### Identities @@ -194,25 +215,25 @@ When an HTTP request results in an error, Gateway logs the first 512 bytes of th | ---------------------- | -------------------------------------------------------------------------------------------------------------------- | | **Email** | Email address of the user who made the HTTP request. This is generated by the WARP client. | | **User ID** | ID of the user who made the request. This is generated by the WARP client. | -| **Device Name** | Name of the device that made the request. | +| **Device name** | Name of the device that made the request. | | **Device ID** | ID of the device that made the request. This is generated by the WARP client on the device that created the request. | -| **Last Authenticated** | Date and time the user last authenticated with Zero Trust. | +| **Last authenticated** | Date and time the user last authenticated with Zero Trust. | #### HTTP query details | Field | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------------- | -| **HTTP Version** | HTTP version of the origin that Gateway connected to on behalf of the user. | -| **HTTP Method** | HTTP method used for the request (such as `GET` or `POST`). | -| **HTTP Status Code** | [HTTP status code](/support/troubleshooting/http-status-codes/http-status-codes/) returned in the response. | +| **HTTP version** | HTTP version of the origin that Gateway connected to on behalf of the user. | +| **HTTP method** | HTTP method used for the request (such as `GET` or `POST`). | +| **HTTP status code** | [HTTP status code](/support/troubleshooting/http-status-codes/http-status-codes/) returned in the response. | | **URL** | Full URL of the HTTP request. | | **Referer** | Referer request header containing the address of the page making the request. | | **Source IP** | Public source IP address of the HTTP request. | -| **Source Port** | Port that was used to make the HTTP request. | -| **Source IP Country** | Country code of the HTTP request. | +| **Source port** | Port that was used to make the HTTP request. | +| **Source IP country** | Country code of the HTTP request. | | **Destination IP** | Public IP address of the destination requested. | -| **Destination Port** | Port of the destination requested. | -| **Destination IP Country** | Country code of the destination requested. | +| **Destination port** | Port of the destination requested. | +| **Destination IP country** | Country code of the destination requested. | | **Blocked file reason** | Reason why the file was blocked if a file transfer occurred or was attempted. | | **Category details** | Category the blocked file belongs to. | diff --git a/src/content/docs/cloudflare-one/insights/logs/logpush/index.mdx b/src/content/docs/cloudflare-one/insights/logs/logpush.mdx similarity index 54% rename from src/content/docs/cloudflare-one/insights/logs/logpush/index.mdx rename to src/content/docs/cloudflare-one/insights/logs/logpush.mdx index f1325faf75861b..703aab06af4138 100644 --- a/src/content/docs/cloudflare-one/insights/logs/logpush/index.mdx +++ b/src/content/docs/cloudflare-one/insights/logs/logpush.mdx @@ -6,9 +6,7 @@ sidebar: --- :::note - Only available on Enterprise plans. - ::: With Cloudflare's [Logpush](/logs/about/) service, you can configure the automatic export of Zero Trust logs to third-party storage destinations or to security information and event management (SIEM) tools. Once exported, your team can analyze and audit the data as needed. @@ -16,9 +14,7 @@ With Cloudflare's [Logpush](/logs/about/) service, you can configure the automat ## Export Zero Trust logs with Logpush :::caution[Dashboard limitation] - -Zero Trust does not support configuring [Cloudflare R2](/logs/get-started/enable-destinations/r2/) as a Logpush destination via the dashboard. To use R2 as a destination for Zero Trust logs, configure your Logpush jobs [via the API](/logs/get-started/enable-destinations/r2/#manage-via-api). - +Zero Trust does not support configuring [Cloudflare R2](/logs/get-started/enable-destinations/r2/) as a Logpush destination in the dashboard. To use R2 as a destination for Zero Trust logs, configure your Logpush jobs [with the API](/logs/get-started/enable-destinations/r2/#manage-via-api). ::: To enable Logpush for Zero Trust logs: @@ -54,4 +50,61 @@ Refer to the Logpush documentation for a list of available fields. ## Parse Logpush logs -For more information on parsing DNS logs, refer to [RData](/cloudflare-one/insights/logs/logpush/rdata/). +Cloudflare Gateway logs DNS query information in [resource record format](https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.3), a Base64-encoded binary format. The following resource record fields are available for each query: + +- Query name +- Query type +- Query class +- Response TTL +- Response data + +To parse resource record logs from Logpush, run the following Python script with your desired samples: + +```python +import dnslib +import base64 + + +# The samples from your Logpush output +samples = [ + {"type":"1","data":"BnJlZGRpdANjb20AAAEAAQAAALwABJdlwYw="}, + {"type":"5","data":"BnNlY3VyZQV3bHhycwNjb20AAAUAAQAADggAIgZzZWN1cmUEYmFzZQV3bHhycwNjb20GYWthZG5zA25ldAA="}, + {"type":"28","data":"Bmdvb2dsZQNjb20AABwAAQAAAGkAECYH+LBAIxAJAAAAAAAAAGU="}] + + +# Parse the Logpush RData.data field into Resource Records +# See section "4.1.3. Resource record format" of https://www.ietf.org/rfc/rfc1035.txt +# Includes Query Name, Query Type, Query Class, Response TTL, Response Data +for sample in samples: + decoded = base64.b64decode(sample["data"]) + buffer = dnslib.DNSBuffer(decoded) + r = dnslib.RR.parse(buffer) + print("== Print the full Resource Record ==") + print(r) + print("== Print individual components of the Resource Record ==") + query_name = r.rname + query_type = r.rtype + query_class = r.rclass + response_ttl = r.ttl + response_data = r.rdata + print(f"query name: {query_name} | query type: {query_type} | query class: {query_class} | ttl: {response_ttl} | rdata: {response_data}\n") +``` + +The script will print a list of your samples. For example: + +```txt +== Print the full Resource Record == +reddit.com. 188 IN A 151.101.193.140 +== Print individual components of the Resource Record == +query name: reddit.com. | query type: 1 | query class: 1 | ttl: 188 | rdata: 151.101.193.140 + +== Print the full Resource Record == +secure.wlxrs.com. 3592 IN CNAME secure.base.wlxrs.com.akadns.net. +== Print individual components of the Resource Record == +query name: secure.wlxrs.com. | query type: 5 | query class: 1 | ttl: 3592 | rdata: secure.base.wlxrs.com.akadns.net. + +== Print the full Resource Record == +google.com. 105 IN AAAA 2607:f8b0:4023:1009::65 +== Print individual components of the Resource Record == +query name: google.com. | query type: 28 | query class: 1 | ttl: 105 | rdata: 2607:f8b0:4023:1009::65 +``` diff --git a/src/content/docs/cloudflare-one/insights/logs/logpush/rdata.mdx b/src/content/docs/cloudflare-one/insights/logs/logpush/rdata.mdx deleted file mode 100644 index 6c1221e29c5997..00000000000000 --- a/src/content/docs/cloudflare-one/insights/logs/logpush/rdata.mdx +++ /dev/null @@ -1,68 +0,0 @@ ---- -pcx_content_type: concept -title: RData -sidebar: - order: 1 - ---- - -Cloudflare Gateway logs DNS query information in [RData](https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.3), a Base64-encoded binary format. The following resource record fields are available for each query: - -* Query name -* Query type -* Query class -* Response TTL -* Response data - -## Parse RData - -To parse RData logs from Logpush, run the following Python script with your desired samples: - -```python -import dnslib -import base64 - - -# The samples from your Logpush output -samples = [ - {"type":"1","data":"BnJlZGRpdANjb20AAAEAAQAAALwABJdlwYw="}, - {"type":"5","data":"BnNlY3VyZQV3bHhycwNjb20AAAUAAQAADggAIgZzZWN1cmUEYmFzZQV3bHhycwNjb20GYWthZG5zA25ldAA="}, - {"type":"28","data":"Bmdvb2dsZQNjb20AABwAAQAAAGkAECYH+LBAIxAJAAAAAAAAAGU="}] - - -# Parse the Logpush RData.data field into Resource Records -# See section "4.1.3. Resource record format" of https://www.ietf.org/rfc/rfc1035.txt -# Includes Query Name, Query Type, Query Class, Response TTL, Response Data -for sample in samples: - decoded = base64.b64decode(sample["data"]) - buffer = dnslib.DNSBuffer(decoded) - r = dnslib.RR.parse(buffer) - print("== Print the full Resource Record ==") - print(r) - print("== Print individual components of the Resource Record ==") - query_name = r.rname - query_type = r.rtype - query_class = r.rclass - response_ttl = r.ttl - response_data = r.rdata - print(f"query name: {query_name} | query type: {query_type} | query class: {query_class} | ttl: {response_ttl} | rdata: {response_data}\n") -``` - -The script will print a list of your samples. For example: - -```txt -== Print the full Resource Record == -reddit.com. 188 IN A 151.101.193.140 -== Print individual components of the Resource Record == -query name: reddit.com. | query type: 1 | query class: 1 | ttl: 188 | rdata: 151.101.193.140 - -== Print the full Resource Record == -secure.wlxrs.com. 3592 IN CNAME secure.base.wlxrs.com.akadns.net. -== Print individual components of the Resource Record == -query name: secure.wlxrs.com. | query type: 5 | query class: 1 | ttl: 3592 | rdata: secure.base.wlxrs.com.akadns.net. - -== Print the full Resource Record == -google.com. 105 IN AAAA 2607:f8b0:4023:1009::65 -== Print individual components of the Resource Record == -query name: google.com. | query type: 28 | query class: 1 | ttl: 105 | rdata: 2607:f8b0:4023:1009::65 -```