v2.37.0 proposal #3268

nsavoire · 2023-06-21T08:45:03Z

Features

iast: Unvalidated redirect analyzer (Unvalidated redirect analyzer #3204)
tracer: Tedious - service naming (Tedious - service naming #3061)
tracer: MySQL databases - service naming (MySQL databases - service naming #3057)
iast: Taint cookies and headers (Taint cookies and headers #3232)
iast: No HttpOnly vulnerability detection (No HttpOnly vulnerability detection #3228)
iast: No SameSite cookie vulnerability detection (No SameSite cookie vulnerability detection #3246)
tracer: add external log writer (add external log writer #3201)
tracer: Auto-instrument @opentelemetry/sdk-trace-node (Auto-instrument @opentelemetry/sdk-trace-node #3248)
tracer: add support for global fetch (add support for global fetch #3258)

Improvements

tracer: make tracer config available to plugins (make tracer config available to plugins #3235)
iast: Add _dd.iast.enabled=1 metric out of request vulnerabilities tags (Add _dd.iast.enabled=1 metric out of request vulnerabilities tags #3231)
ci-visibility: Better git commands ([ci-visibility] Better git commands #3236)
tracer: Add test in shimmer wrap to preserve function name (Add test in shimmer wrap to preserve function name #3237)
tracer: add environment variable to disable instrumentations completely (add environment variable to disable instrumentations completely #3234)
profiling: Add debug log listing found source maps ([profiling] Add debug log listing found source maps #3242)

Bug Fixes

ci-visibility: Fix agentless exporter test ([ci-visibility] Fix agentless exporter test #3241)
ci-visibility: Fix windows tracing test ([ci-visibility] Fix windows tracing test #3243)
tracer: fix grpc custom errors not being reported (fix grpc custom errors not being reported #3230)
tracer: Disable metrics.spec.js tests for windows (Disable metrics.spec.js tests (part of tracing tests) for windows #3250)
ci-visibility: Use correct repository URL for git metadata upload ([ci-visibility] Use correct repository URL for git metadata upload #3253)
ci-visibility: Fix random cypress integration tests timeouts ([ci-visibility] Fix random cypress integration tests timeouts #3255)
iast: Check store has value before use it (Check store has value before use it #3257)
tracer: Fix setup in integration tests (Fix setup in integration tests #3254)

* add migration guide for 3.x to 4.x * Update MIGRATING.md Co-authored-by: Juan Antonio Fernández de Alba <[email protected]> * Update MIGRATING.md Co-authored-by: Juan Antonio Fernández de Alba <[email protected]> * Update README.md Co-authored-by: Ugaitz Urien <[email protected]> --------- Co-authored-by: Juan Antonio Fernández de Alba <[email protected]> Co-authored-by: Ugaitz Urien <[email protected]>

* update readme to include v4 * use version module instead of semver to match library version

* Unvalidated redirect analyzer * Ignore tainteds from Referer header

) * Add _dd.iast.enabled=1 in out of request vulnerabilities tags * Rename constants.js to tags.js

* add v0 naming to tedious (mssql) * add v1 naming to tedious (mssql) * switch to a mostly working test sqlserver The standard mssql server image does not work on ARM [1]. Instead, we use `azure-sql-edge` [2], which provides a sufficient subset of mssql server API to test most of our integration. Unfortunately, this image does not support stored procedures [3], so tests related to these will still fail locally. [1] microsoft/mssql-docker#668 [2] https://hub.docker.com/_/microsoft-azure-sql-edge [3] https://learn.microsoft.com/en-us/azure/azure-sql-edge/features#unsupported-features

* add v0 naming to mysql integrations * add v1 to mysql integrations * add tests on v1 for mysql databases --------- Co-authored-by: Thomas Hunter II <[email protected]>

* Taint cookies and headers * Bump minimum node version for v4 on cookie plugin test * Add test with latest node version for cookie plugin test * Provide iastContext from index when tainting headers * Add test for cookie tainting in taint tracking plugin * Remove iast transaction after taint tracking plugin tests to avoid hiting setMaxTransactions in tests * Add test for taintObject with taintingKeys flag * Address header tainting test for keys shorter than 10 chars * Upgrade native-iast-taint-tracking to v1.5.0 * Rewrite expect in taint tracking plugin test * Fix tag requiring in IAST index

--------- Co-authored-by: Thomas Hunter II <[email protected]>

)

* Add sources tests * styles

…3253)

* appsec-iast benchmark tests * writing fixes Co-authored-by: simon-id <[email protected]> * small fixes --------- Co-authored-by: simon-id <[email protected]>

* Reduce request per iteration in IAST benchmarks * Reduce a bit more * 30 iterations 200 requests

github-actions · 2023-06-21T08:46:07Z

Overall package size

Self size: 4.4 MB
Deduped: 71.1 MB
No deduping: 71.15 MB

Dependency sizes

name	version	self size	total size
@datadog/pprof	2.2.3	14.25 MB	15.13 MB
@datadog/native-iast-taint-tracking	1.5.0	14.86 MB	14.86 MB
@datadog/native-appsec	3.2.0	13.38 MB	13.39 MB
@datadog/native-metrics	1.6.0	7.88 MB	7.89 MB
protobufjs	7.2.3	2.77 MB	6.65 MB
@types/node	18.11.19	3.58 MB	3.58 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
@opentelemetry/core	1.3.1	784.66 kB	1.37 MB
@opentelemetry/api	1.4.1	780.32 kB	780.32 kB
opentracing	0.14.7	194.81 kB	194.81 kB
lru-cache	7.18.3	133.92 kB	133.92 kB
semver	7.5.3	93.39 kB	123.79 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
ignore	5.2.4	51.22 kB	51.22 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
node-abort-controller	3.1.1	16.89 kB	16.89 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-06-21T08:47:19Z

Codecov Report

Merging #3268 (79e63a9) into v2.x (fbc2899) will decrease coverage by 7.17%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             v2.x    #3268      +/-   ##
==========================================
- Coverage   93.51%   86.34%   -7.17%     
==========================================
  Files         222      196      -26     
  Lines        8829     7698    -1131     
  Branches        0       33      +33     
==========================================
- Hits         8256     6647    -1609     
- Misses        573     1051     +478

Impacted Files	Coverage Δ
integration-tests/ci-visibility/test/sum.js	`69.19% <ø> (ø)`
packages/datadog-instrumentations/src/mocha.js	`0.00% <0.00%> (-98.64%)`	⬇️
packages/datadog-plugin-fastify/src/index.js	`100.00% <ø> (ø)`

... and 311 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-06-21T13:18:34Z

Benchmarks

Comparing candidate commit 79e63a9 in PR branch v2.37.0-proposal with baseline commit 4a427a2 in branch v2.x.

Found 0 performance improvements and 40 performance regressions! Performance is the same for 560 metrics, 48 unstable metrics.

scenario:log-without-log-14

🟥 execution_time [+11.777ms; +14.904ms] or [+5.365%; +6.790%]

scenario:plugin-bluebird-with-tracer-14

🟥 cpu_user_time [+15.989ms; +22.015ms] or [+5.425%; +7.470%]
🟥 execution_time [+18.246ms; +20.167ms] or [+5.553%; +6.138%]

scenario:log-without-log-16

🟥 execution_time [+13.132ms; +14.686ms] or [+5.579%; +6.239%]

scenario:log-without-log-18

🟥 cpu_user_time [+25.921ms; +34.495ms] or [+12.275%; +16.335%]
🟥 execution_time [+29.044ms; +31.910ms] or [+11.948%; +13.127%]
🟥 instructions [+44; +44] or [+7.826%; +7.891%]
🟥 max_rss_usage [+3.131KB; +3.284KB] or [+5.611%; +5.885%]

scenario:log-skip-log-18

🟥 cpu_user_time [+11.933ms; +20.984ms] or [+5.591%; +9.832%]
🟥 execution_time [+18.691ms; +19.786ms] or [+7.660%; +8.109%]
🟥 instructions [+43; +44] or [+7.565%; +7.672%]
🟥 max_rss_usage [+3.006KB; +3.219KB] or [+5.390%; +5.772%]

scenario:log-with-error-18

🟥 cpu_user_time [+11.029ms; +19.478ms] or [+5.200%; +9.184%]
🟥 execution_time [+16.861ms; +19.849ms] or [+6.892%; +8.113%]
🟥 instructions [+44; +44] or [+7.606%; +7.709%]
🟥 max_rss_usage [+3.084KB; +3.308KB] or [+5.530%; +5.932%]

scenario:log-with-debug-18

🟥 cpu_user_time [+13.849ms; +21.894ms] or [+6.441%; +10.183%]
🟥 execution_time [+16.190ms; +19.755ms] or [+6.608%; +8.064%]
🟥 instructions [+43; +44] or [+7.551%; +7.658%]
🟥 max_rss_usage [+2.955KB; +3.175KB] or [+5.296%; +5.691%]

scenario:appsec-control-with-attacks-18

🟥 cpu_user_time [+23.231ms; +29.595ms] or [+8.198%; +10.444%]
🟥 execution_time [+28.889ms; +37.540ms] or [+8.355%; +10.857%]
🟥 instructions [+42; +43] or [+5.741%; +5.874%]
🟥 max_rss_usage [+4.007KB; +4.135KB] or [+6.821%; +7.039%]

scenario:appsec-control-18

🟥 cpu_user_time [+20.693ms; +27.257ms] or [+6.759%; +8.903%]
🟥 execution_time [+31.012ms; +40.534ms] or [+8.487%; +11.093%]
🟥 instructions [+42; +43] or [+5.714%; +5.845%]
🟥 max_rss_usage [+3.840KB; +4.009KB] or [+6.526%; +6.813%]

scenario:appsec-appsec-enabled-18

🟥 cpu_user_time [+18.066ms; +25.285ms] or [+5.177%; +7.245%]
🟥 execution_time [+33.523ms; +44.108ms] or [+8.321%; +10.948%]

scenario:appsec-appsec-enabled-with-attacks-18

🟥 cpu_user_time [+22.309ms; +29.819ms] or [+6.126%; +8.188%]
🟥 execution_time [+32.144ms; +41.612ms] or [+7.683%; +9.946%]

scenario:plugin-graphql-with-depth-and-collapse-on-18

🟥 max_rss_usage [+90.223KB; +107.333KB] or [+10.760%; +12.800%]

scenario:plugin-graphql-with-depth-on-max-18

🟥 max_rss_usage [+80.852KB; +108.532KB] or [+9.569%; +12.844%]

scenario:plugin-graphql-with-depth-off-18

🟥 max_rss_usage [+73.470KB; +105.090KB] or [+8.784%; +12.564%]

scenario:plugin-bluebird-with-tracer-18

🟥 execution_time [+18.397ms; +20.657ms] or [+5.631%; +6.323%]
🟥 instructions [+41; +42] or [+5.301%; +5.433%]

scenario:net-with-tracer-18

🟥 cpu_user_time [+15.861ms; +22.384ms] or [+5.466%; +7.714%]
🟥 instructions [+42; +42] or [+5.602%; +5.651%]
🟥 max_rss_usage [+3.035KB; +3.097KB] or [+5.161%; +5.268%]

rochdev · 2023-06-22T13:50:19Z

packages/datadog-plugin-fetch/test/index.spec.js

+        getPort().then(port => {
+          agent
+            .use(traces => {
+              expect(traces[0][0]).to.have.property('service', 'test-http-client')


This is different between version. Can the change be done in a backward-compatible way to avoid merge conflicts?

I added #3282 to the release which remove the difference between branches.

For each new push on master, publish a new injection image with tag "dev". We need to tag master branch with `dev` tag to trigger the `deploy_to_docker_registries` job in gitlab CI.

….js 12 (#3277)

…'ll throw whatever is thrown by the handler

* Add DD_PROFILING_DEBUG_SOURCE_MAPS option DD_PROFILING_DEBUG_SOURCE_MAPS env variable enables printing of detailed diagnostics concerning source maps. Pass logger to profiler module to enable logging. * Bump profiler version to 2.2.3

rochdev and others added 26 commits June 21, 2023 10:27

update readme for v4 and remove semver for version feature flags (#3148)

abed941

* update readme to include v4 * use version module instead of semver to match library version

Unvalidated redirect analyzer (#3204)

a7302a4

* Unvalidated redirect analyzer * Ignore tainteds from Referer header

make tracer config available to plugins (#3235)

18d8196

Add _dd.iast.enabled=1 metric out of request vulnerabilities tags (#3231

295abde

) * Add _dd.iast.enabled=1 in out of request vulnerabilities tags * Rename constants.js to tags.js

[ci-visibility] Better git commands (#3236)

5b1597d

[ci-visibility] Fix agentless exporter test (#3241)

ff5f68c

Add test in shimmer wrap to preserve function name (#3237)

3203197

MySQL databases - service naming (#3057)

f6f1092

* add v0 naming to mysql integrations * add v1 to mysql integrations * add tests on v1 for mysql databases --------- Co-authored-by: Thomas Hunter II <[email protected]>

[ci-visibility] Fix windows tracing test (#3243)

0b689b3

fix grpc custom errors not being reported (#3230)

6043a56

No HttpOnly vulnerability detection (#3228)

2665f5a

No SameSite cookie vulnerability detection (#3246)

d97d20d

add external log writer (#3201)

55a8431

--------- Co-authored-by: Thomas Hunter II <[email protected]>

Fix unvalidated redirects (#3252)

f3745d4

add environment variable to disable instrumentations completely (#3234)

43d20de

Disable metrics.spec.js tests (part of tracing tests) for windows (#3250

f773c06

)

Add sources tests (#3249)

11d16d7

* Add sources tests * styles

[ci-visibility] Use correct repository URL for git metadata upload (#…

d218c8b

…3253)

Add IAST benchmark tests (#3193)

8874cfc

* appsec-iast benchmark tests * writing fixes Co-authored-by: simon-id <[email protected]> * small fixes --------- Co-authored-by: simon-id <[email protected]>

[ci-visibility] Fix random cypress integration tests timeouts (#3255)

5be73e8

Reduce request per iteration in IAST benchmarks (#3260)

9391690

* Reduce request per iteration in IAST benchmarks * Reduce a bit more * 30 iterations 200 requests

Check store has value before use it (#3257)

809e3c3

Fix setup in integration tests (#3254)

d31f080

nsavoire added the semver-minor label Jun 21, 2023

uurien previously approved these changes Jun 22, 2023

View reviewed changes

rochdev requested changes Jun 22, 2023

View reviewed changes

nsavoire and others added 13 commits June 23, 2023 01:32

Fix typo --ignore-engine => --ignore-engines (#3270)

fb30452

Auto-instrument @opentelemetry/sdk-trace-node (#3248)

4fb7acb

Publish "dev" injection image (#3276)

3a2be24

For each new push on master, publish a new injection image with tag "dev". We need to tag master branch with `dev` tag to trigger the `deploy_to_docker_registries` job in gitlab CI.

Skip OTel auto-instrumentation test as sdk-node does not support Node…

112c50d

….js 12 (#3277)

add support for global fetch (#3258)

8f752c3

fix: Lambda handler must be awaited

d313532

feat: Move handler call until after context patch. Remove catch as we…

f155427

…'ll throw whatever is thrown by the handler

feat: Simplify patch, remove .then in favor of async/await

12920f5

fix: lint

fd9cb44

feat: Add unit test

6f24411

feat: Use then instead of async/await

f905de6

Unify test code between release branches (#3282)

59a6e2f

upgrade semver to fix audit lint errors (#3285)

889999d

nsavoire dismissed stale reviews from uurien and juan-fernandez via 68db403 June 22, 2023 23:39

nsavoire force-pushed the v2.37.0-proposal branch from bb2ea4e to 68db403 Compare June 22, 2023 23:39

uurien previously approved these changes Jun 23, 2023

View reviewed changes

nsavoire requested a review from rochdev June 23, 2023 12:44

rochdev previously approved these changes Jun 23, 2023

View reviewed changes

nsavoire added 2 commits June 23, 2023 14:48

Bump profiler version to 2.2.3 (#3286)

6afceb7

* Add DD_PROFILING_DEBUG_SOURCE_MAPS option DD_PROFILING_DEBUG_SOURCE_MAPS env variable enables printing of detailed diagnostics concerning source maps. Pass logger to profiler module to enable logging. * Bump profiler version to 2.2.3

v2.37.0

79e63a9

nsavoire dismissed stale reviews from rochdev and uurien via 79e63a9 June 23, 2023 12:48

nsavoire force-pushed the v2.37.0-proposal branch from 68db403 to 79e63a9 Compare June 23, 2023 12:48

astuyve approved these changes Jun 23, 2023

View reviewed changes

uurien approved these changes Jun 23, 2023

View reviewed changes

tlhunter merged commit 85a8949 into v2.x Jun 23, 2023

nsavoire deleted the v2.37.0-proposal branch July 17, 2023 08:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2.37.0 proposal #3268

v2.37.0 proposal #3268

nsavoire commented Jun 21, 2023 •

edited

Loading

github-actions bot commented Jun 21, 2023 •

edited

Loading

codecov bot commented Jun 21, 2023 •

edited

Loading

pr-commenter bot commented Jun 21, 2023 •

edited

Loading

rochdev Jun 22, 2023

nsavoire Jun 23, 2023

v2.37.0 proposal #3268

v2.37.0 proposal #3268

Conversation

nsavoire commented Jun 21, 2023 • edited Loading

Features

Improvements

Bug Fixes

github-actions bot commented Jun 21, 2023 • edited Loading

Overall package size

Dependency sizes

codecov bot commented Jun 21, 2023 • edited Loading

Codecov Report

pr-commenter bot commented Jun 21, 2023 • edited Loading

Benchmarks

scenario:log-without-log-14

scenario:plugin-bluebird-with-tracer-14

scenario:log-without-log-16

scenario:log-without-log-18

scenario:log-skip-log-18

scenario:log-with-error-18

scenario:log-with-debug-18

scenario:appsec-control-with-attacks-18

scenario:appsec-control-18

scenario:appsec-appsec-enabled-18

scenario:appsec-appsec-enabled-with-attacks-18

scenario:plugin-graphql-with-depth-and-collapse-on-18

scenario:plugin-graphql-with-depth-on-max-18

scenario:plugin-graphql-with-depth-off-18

scenario:plugin-bluebird-with-tracer-18

scenario:net-with-tracer-18

rochdev Jun 22, 2023

Choose a reason for hiding this comment

nsavoire Jun 23, 2023

Choose a reason for hiding this comment

nsavoire commented Jun 21, 2023 •

edited

Loading

github-actions bot commented Jun 21, 2023 •

edited

Loading

codecov bot commented Jun 21, 2023 •

edited

Loading

pr-commenter bot commented Jun 21, 2023 •

edited

Loading