Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New automatic user event collection #4674

Merged
merged 93 commits into from
Dec 16, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
023828c
lint
simon-id Sep 11, 2024
98090fb
update config
simon-id Sep 11, 2024
5cc7274
add new RC capability
simon-id Sep 11, 2024
3343878
Merge branch 'master' into new_user_collection
simon-id Sep 11, 2024
4e127de
add RC handler
simon-id Sep 12, 2024
b4daf8f
fix require path
simon-id Sep 12, 2024
5f85b8e
add setCollectionMode()
simon-id Sep 12, 2024
1b074ad
use setCollectionMode() in appsec index
simon-id Sep 12, 2024
2da5bef
Merge branch 'master' into new_user_collection
simon-id Sep 17, 2024
5105006
cleanup
simon-id Oct 20, 2024
65c680b
update typings
simon-id Oct 20, 2024
23f965d
cleanup
simon-id Oct 20, 2024
b5b336a
RC config update
simon-id Oct 20, 2024
53a7791
push everything
simon-id Oct 30, 2024
2b7d2e6
simplify passport strategies instrumentation
simon-id Oct 30, 2024
24c2828
fixes
simon-id Oct 30, 2024
cdb89f6
simplify code
simon-id Oct 30, 2024
32edfae
cleanup
simon-id Oct 30, 2024
b5aa0d4
delete passport.js
simon-id Oct 30, 2024
1f30a3c
Update packages/datadog-instrumentations/src/passport-utils.js
simon-id Oct 30, 2024
9fee2bc
update verify subscriber
simon-id Oct 30, 2024
be3115a
rollback config changes
simon-id Oct 30, 2024
38b4419
fix config
simon-id Oct 30, 2024
40c9df1
add blocking for passport strategies
simon-id Oct 30, 2024
e34bb72
update typings and docs
simon-id Oct 30, 2024
d255702
update appsec index
simon-id Oct 30, 2024
b6d4584
update RC
simon-id Oct 30, 2024
75782f9
Merge branch 'master' into new_user_collection
simon-id Oct 30, 2024
4ff8379
push some stuff
simon-id Oct 30, 2024
4a566a8
cleanup
simon-id Oct 30, 2024
882bc8b
cleanup
simon-id Oct 30, 2024
d4345ef
add new usr.login waf address
simon-id Oct 31, 2024
be4a06c
commit some stuff
simon-id Oct 31, 2024
cc86e3a
pass abort controller for blocking
simon-id Nov 1, 2024
ce604c1
change config default
simon-id Nov 15, 2024
56c30e8
handle duplicate RC confs for auto_user_instrum.mode
simon-id Nov 15, 2024
1d3773a
refactor sdk/track_event.js to only be used by the SDK
simon-id Nov 15, 2024
91a59db
remove some comments
simon-id Nov 19, 2024
faea3aa
pass login to WAF in SDK login success event
simon-id Nov 19, 2024
fe70c84
add comments
simon-id Nov 19, 2024
e32f78e
add framework name to passport strategy instrum
simon-id Nov 20, 2024
806ab23
add framework name and waf handleResults() to onPassportVerify()
simon-id Nov 20, 2024
69be031
Merge branch 'master' into new_user_collection
simon-id Nov 20, 2024
1e1bb47
finally commit trackLogin()
simon-id Nov 21, 2024
8eac055
fix tag override condition
simon-id Nov 21, 2024
a444cfe
add telemetry function
simon-id Nov 21, 2024
187b42f
Merge branch 'master' into new_user_collection
simon-id Nov 21, 2024
72a4a8b
Merge branch 'master' into new_user_collection
simon-id Nov 22, 2024
9fa94f6
move user_tracking into a file instead of a subfolder
simon-id Nov 22, 2024
d51e434
Merge branch 'master' into new_user_collection
simon-id Nov 22, 2024
7e3374d
Use addresses enum instead of hardcoded business logic events
simon-id Nov 25, 2024
c099a2a
Update default value of collection mode in typings
simon-id Nov 25, 2024
96cb722
Add string check in getUserId()
simon-id Nov 25, 2024
0a347e8
Merge branch 'master' into new_user_collection
simon-id Nov 26, 2024
85ad6f2
fix existing RC tests
simon-id Nov 26, 2024
70dcbab
add tests for RC collection mode
simon-id Nov 26, 2024
ff72271
fix env var ordering
simon-id Nov 26, 2024
7338bd6
cleanup track_event.spec.js
simon-id Nov 29, 2024
b71ed06
do not export or test trackEvent()
simon-id Nov 29, 2024
166bc23
update test, cleanup, and add missing coverage
simon-id Nov 29, 2024
a93b79a
change ordering of code to match tests
simon-id Nov 29, 2024
a83c4fb
delete passport-utils tests because it's useless
simon-id Nov 29, 2024
6cf7ef2
fix test to correctly use passReqToCallback
simon-id Nov 29, 2024
29c0d62
update passport-local tests
simon-id Nov 29, 2024
afafb8a
update tests for passpot-http
simon-id Dec 2, 2024
956cb9b
allow empty login strings
simon-id Dec 2, 2024
62ae42b
update TS tests
simon-id Dec 2, 2024
375f688
update telemetry tests
simon-id Dec 2, 2024
5a2b1cd
move res definition to add res inside req
simon-id Dec 5, 2024
4a5701a
update appsec/index.js tests
simon-id Dec 5, 2024
13c66a7
update config tests
simon-id Dec 5, 2024
950610b
it's spelled `length`
simon-id Dec 5, 2024
e809b50
Merge branch 'master' into new_user_collection
simon-id Dec 5, 2024
fd15e9d
Remove newline from password in http basic error test
simon-id Dec 8, 2024
e9ceb27
add ASM prefix to logs
simon-id Dec 8, 2024
3728bce
Update packages/dd-trace/src/appsec/remote_config/index.js
simon-id Dec 8, 2024
36d3a1d
Merge branch 'master' into new_user_collection
simon-id Dec 8, 2024
5fb5c81
Update packages/dd-trace/src/appsec/telemetry.js
simon-id Dec 8, 2024
8fe1c2e
fix wrong function name
simon-id Dec 8, 2024
98cae84
rename `incrementMissingUserLogin` to `incrementMissingUserLoginMetric`
simon-id Dec 8, 2024
baa29a6
add error log when RC config is rejected
simon-id Dec 8, 2024
d882589
fix inconsequential typo
simon-id Dec 8, 2024
2562a48
lint
simon-id Dec 8, 2024
7d5ef29
add user_tracking tests
simon-id Dec 11, 2024
3325149
delete old passport tests
simon-id Dec 11, 2024
1effb29
Merge branch 'master' into new_user_collection
simon-id Dec 11, 2024
18659f1
Update packages/dd-trace/test/appsec/user_tracking.spec.js
simon-id Dec 11, 2024
134c561
lint
simon-id Dec 12, 2024
1df40e2
add SDK tag for login alias
simon-id Dec 12, 2024
5fe9404
Merge branch 'master' into new_user_collection
simon-id Dec 12, 2024
172da58
lint again omg
simon-id Dec 12, 2024
b1ba64a
Merge branch 'master' into new_user_collection
simon-id Dec 13, 2024
a752423
Update packages/dd-trace/src/appsec/sdk/track_event.js
simon-id Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -651,7 +651,7 @@ declare namespace tracer {
* On extended mode, no redaction will take place.
* @default 'safe'
*/
mode?: 'safe' | 'extended' | 'disabled'
mode?: 'anonymous' | 'identification' | 'safe' | 'extended' | 'disabled'
},
/**
* Configuration for Api Security sampling
Expand Down
16 changes: 2 additions & 14 deletions packages/datadog-instrumentations/src/passport-http.js
Original file line number Diff line number Diff line change
@@ -1,22 +1,10 @@
'use strict'

const shimmer = require('../../datadog-shimmer')
const { addHook } = require('./helpers/instrument')
const { wrapVerify } = require('./passport-utils')
const { strategyHook } = require('./passport-utils')

addHook({
name: 'passport-http',
file: 'lib/passport-http/strategies/basic.js',
versions: ['>=0.3.0']
}, BasicStrategy => {
return shimmer.wrapFunction(BasicStrategy, BasicStrategy => function () {
const type = 'http'

if (typeof arguments[0] === 'function') {
arguments[0] = wrapVerify(arguments[0], false, type)
} else {
arguments[1] = wrapVerify(arguments[1], (arguments[0] && arguments[0].passReqToCallback), type)
}
return BasicStrategy.apply(this, arguments)
})
})
}, strategyHook)
16 changes: 2 additions & 14 deletions packages/datadog-instrumentations/src/passport-local.js
Original file line number Diff line number Diff line change
@@ -1,22 +1,10 @@
'use strict'

const shimmer = require('../../datadog-shimmer')
const { addHook } = require('./helpers/instrument')
const { wrapVerify } = require('./passport-utils')
const { strategyHook } = require('./passport-utils')

addHook({
name: 'passport-local',
file: 'lib/strategy.js',
versions: ['>=1.0.0']
}, Strategy => {
return shimmer.wrapFunction(Strategy, Strategy => function () {
const type = 'local'

if (typeof arguments[0] === 'function') {
arguments[0] = wrapVerify(arguments[0], false, type)
} else {
arguments[1] = wrapVerify(arguments[1], (arguments[0] && arguments[0].passReqToCallback), type)
}
return Strategy.apply(this, arguments)
})
})
}, strategyHook)
57 changes: 38 additions & 19 deletions packages/datadog-instrumentations/src/passport-utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,52 @@ const { channel } = require('./helpers/instrument')

const passportVerifyChannel = channel('datadog:passport:verify:finish')

function wrapVerifiedAndPublish (username, password, verified, type) {
if (!passportVerifyChannel.hasSubscribers) {
return verified
}
function wrapVerifiedAndPublish (username, verified) {
return shimmer.wrapFunction(verified, function wrapVerify (verified) {
return function wrappedVerified (err, user) {
// if there is an error, it's neither an auth success nor a failure
if (!err) {
passportVerifyChannel.publish({ login: username, user, success: !!user })
}

// eslint-disable-next-line n/handle-callback-err
return shimmer.wrapFunction(verified, verified => function (err, user, info) {
const credentials = { type, username }
passportVerifyChannel.publish({ credentials, user })
return verified.apply(this, arguments)
return verified.apply(this, arguments)
}
})
}

function wrapVerify (verify, passReq, type) {
if (passReq) {
return function (req, username, password, verified) {
arguments[3] = wrapVerifiedAndPublish(username, password, verified, type)
return verify.apply(this, arguments)
function wrapVerify (verify) {
return function wrappedVerify (req, username, password, verified) {
simon-id marked this conversation as resolved.
Show resolved Hide resolved
if (passportVerifyChannel.hasSubscribers) {
// replace the callback with our own wrapper to get the result
// if we ever need the type of strategy, we can get it from this.name
if (this._passReqToCallback) {
arguments[3] = wrapVerifiedAndPublish(username, verified)
} else {
arguments[2] = wrapVerifiedAndPublish(req, password) // shifted args
simon-id marked this conversation as resolved.
Show resolved Hide resolved
}
}
} else {
return function (username, password, verified) {
arguments[2] = wrapVerifiedAndPublish(username, password, verified, type)
return verify.apply(this, arguments)

return verify.apply(this, arguments)
}
}

function wrapStrategy (Strategy) {
return function wrappedStrategy () {
// verify function can be either the first or second argument
if (typeof arguments[0] === 'function') {
arguments[0] = wrapVerify(arguments[0])
} else {
arguments[1] = wrapVerify(arguments[1])
}

return Strategy.apply(this, arguments)
}
}

function strategyHook (Strategy) {
return shimmer.wrapFunction(Strategy, wrapStrategy)
}

module.exports = {
wrapVerify
strategyHook
}
5 changes: 3 additions & 2 deletions packages/dd-trace/src/appsec/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ const web = require('../plugins/util/web')
const { extractIp } = require('../plugins/util/ip_extractor')
const { HTTP_CLIENT_IP } = require('../../../../ext/tags')
const { isBlocked, block, setTemplates, getBlockingAction } = require('./blocking')
const { passportTrackEvent } = require('./passport')
const { setCollectionMode, passportTrackEvent } = require('./passport')
const { storage } = require('../../../datadog-core')
const graphql = require('./graphql')
const rasp = require('./rasp')
Expand Down Expand Up @@ -66,6 +66,7 @@ function enable (_config) {
responseSetHeader.subscribe(onResponseSetHeader)

if (_config.appsec.eventTracking.enabled) {
setCollectionMode(_config.appsec.eventTracking.mode)
passportVerify.subscribe(onPassportVerify)
}

Expand Down Expand Up @@ -221,7 +222,7 @@ function onPassportVerify ({ credentials, user }) {
return
}

passportTrackEvent(credentials, user, rootSpan, config.appsec.eventTracking.mode)
passportTrackEvent(credentials, user, rootSpan)
}

const responseAnalyzedSet = new WeakSet()
Expand Down
110 changes: 0 additions & 110 deletions packages/dd-trace/src/appsec/passport.js

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,6 @@ module.exports = {
APM_TRACING_ENABLED: 1n << 19n,
ASM_RASP_SQLI: 1n << 21n,
ASM_RASP_SSRF: 1n << 23n,
APM_TRACING_SAMPLE_RULES: 1n << 29n
APM_TRACING_SAMPLE_RULES: 1n << 29n,
ASM_AUTO_USER_INSTRUM_MODE: 1n << 31n
}
13 changes: 13 additions & 0 deletions packages/dd-trace/src/appsec/remote_config/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ const Activation = require('../activation')
const RemoteConfigManager = require('./manager')
const RemoteConfigCapabilities = require('./capabilities')
const apiSecuritySampler = require('../api_security_sampler')
const { setCollectionMode } = require('../passport')

let rc

Expand All @@ -28,6 +29,10 @@ function enable (config, appsec) {
rc.updateCapabilities(RemoteConfigCapabilities.ASM_API_SECURITY_SAMPLE_RATE, true)
}

if (config.appsec.eventTracking?.enabled) {
rc.updateCapabilities(RemoteConfigCapabilities.ASM_AUTO_USER_INSTRUM_MODE, true)
}

rc.setProductHandler('ASM_FEATURES', (action, rcConfig) => {
if (!rcConfig) return

Expand All @@ -36,6 +41,14 @@ function enable (config, appsec) {
}

apiSecuritySampler.setRequestSampling(rcConfig.api_security?.request_sample_rate)

if (config.appsec.eventTracking?.enabled && typeof rcConfig.auto_user_instrum?.mode === 'string') {
if (action === 'apply' || action === 'modify') {
setCollectionMode(rcConfig.auto_user_instrum.mode)
} else {
setCollectionMode(config.appsec.eventTracking.mode)
}
}
})
}

Expand Down
11 changes: 11 additions & 0 deletions packages/dd-trace/src/appsec/sdk/set_user.js
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,18 @@ function setUser (tracer, user) {
return
}

// must get user ID with USER_ID_FIELDS

// _dd.appsec.user.collection_mode: collectionMode // sdk/ident/anon

setUserTags(user, rootSpan)

/*
User IDs generated through the SDK must now be provided to libddwaf as a persistent addresses.
If the user monitoring SDK has already resulted in a call to libddwaf before any automated instrumentation or collection method has been executed, no extra call should be made.
If the automated instrumentation or collection method has resulted in a call to libddwaf before the user monitoring SDK has been executed, a second call must be performed with the user ID obtained through the SDK.
*/
// will the second call trigger tho ? make some edge case tests
}

module.exports = {
Expand Down
2 changes: 1 addition & 1 deletion packages/dd-trace/src/appsec/sdk/track_event.js
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ function trackEvent (eventName, fields, sdkMethodName, rootSpan, mode) {
tags[`_dd.appsec.events.${eventName}.sdk`] = 'true'
}

if (mode === 'safe' || mode === 'extended') {
if (mode && mode !== 'sdk') {
tags[`_dd.appsec.events.${eventName}.auto.mode`] = mode
}

Expand Down
22 changes: 22 additions & 0 deletions packages/dd-trace/src/appsec/user_tracking/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
'use strict'

let collectionMode

function setCollectionMode (mode) {
switch (mode) {
case 'ident':
case 'identification':
collectionMode = 'ident'
break
case 'anon':
case 'anonymization':
collectionMode = 'anon'
break
default:
collectionMode = null // disabled
}
}

module.exports = {
setCollectionMode
}
Loading