feat: add client side logging with slf4j #3403

zhumin8 · 2024-11-27T22:08:22Z

Changes to introduce client logging debug feature to Gax. Guide on usage will be added separately to README file.

Brings in slf4j-api as optional dependency to gax-java
LoggingUtils, LogData and LoggerProvider are public because access are needed from gax-grpc and gax-httpjson packages
- LoggingUtils handles logic to enable/disable from env var, and contains shared utility methods for record data for logging and record logs (abstracting away any need for SLF4J classes).
- Slf4jUtils any logic interacting with SLF4J classes.
- LoggerProvider provides the SLF4J Logger. This is so that Logger interceptor classes do not declare Logger directly.
This feature is guarded by env var GOOGLE_SDK_JAVA_LOGGING, only turned on if true.
By default it is off, user app should act as before (usual tests behaves as usual)

Tests added:

Unit tests in Gax that need either GOOGLE_SDK_JAVA_LOGGING unset, or do not depend on env var
LoggingEnabledTest: test for logger correctly setups when GOOGLE_SDK_JAVA_LOGGING = true. This is added to existing envVarTest profile
Showcase tests:
- ITLoggingDisabled: test no logging event should record when env var is turned off.
- ITLogging: test logging event with slf4j2.x+logback
- ITLogging1x: test logging event with slf4j1.x+logback
  These tests may not compile depending on the logging dependency brought in. Set up profiles to control test compile and run:
  - Slf4j2_logback, slf4j1_logback, disabledLogging: brings in logging dependencies and setup compile config exclusions
  - Do not include it/logging folder for compile by default, or enable-golden-tests. native is also excluded for now.

Notable changes since last reviewed:

Changes related to get slf4j-api deps as optional
Logics to switch logging behavior based on slf4j major version
Moved helper methods into Gax
Revamped test setup
use Protobuf utils to serialize message

Context: go/java-client-logging-design

TODO:

add rules for renovate bot to bypass some versions for testing

…g needs more work.

…not enabled.

gax-java/gax/src/main/java/com/google/api/gax/logging/LoggingUtils.java

gax-java/gax-grpc/src/main/java/com/google/api/gax/grpc/GrpcLoggingInterceptor.java

gax-java/gax/src/main/java/com/google/api/gax/logging/JsonContextMapHandler.java

gax-java/gax-grpc/src/main/java/com/google/api/gax/grpc/GrpcLoggingInterceptor.java

…ngUtils.

…g entries, add request id to track.

… not done yet.)

…ing interceptors.

gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/HttpJsonLoggingInterceptor.java

lqiu96 · 2025-02-18T17:07:43Z

gax-java/gax/src/main/java/com/google/api/gax/logging/LogData.java

+
+@InternalApi
+@AutoValue
+public abstract class LogData {


Do you think it would make sense to have something like LogRequestData (or RequestLogData or whatever name) that extends from LogData and the equivalent for Response? Request and Response classes would have fields specific them that aren't shared from LogData

Or is the request/ response data for an RPC supposed to be tied to a single LogData object?

Currently both request and respond logging extracts data from this shared LogData. The main benefit is less overhead in object creation, and both request/respond logging can access shared info(rpc name, service name).

We can create separate structure for request/respond data potentially, but I don't see it necessary at this point as there is no conflict/overrides between request and response data. Considering that we also want to limit classes exposed as public.

Ok, I think I'm fine with this now, but I am just curious how this is going to work with Streaming calls. We can figure that out as part of the follow up tasks.

gax-java/gax/src/main/java/com/google/api/gax/logging/Slf4jUtils.java

zhumin8 · 2025-02-20T21:08:00Z

gax-java/gax/src/main/java/com/google/api/gax/logging/LoggingUtils.java

+import java.util.Map;
+
+@InternalApi
+public class LoggingUtils {


From offline discussion, need to improve readability on these logging classes.
Ideas for renaming:
LoggingUtils -> SdkLogger
Slf4jLoggingHelpers -> Slf4jLogger

refactor ideas: create a SDKLogger for public access methods, hide all others. Classes that needs logger should initialize a SDKLogger, and call methods within the SDKLogger. This should also avoid most of the static calls.

Added as a task to migration ticket

lqiu96 · 2025-02-21T19:17:03Z

gax-java/gax/src/main/java/com/google/api/gax/logging/LoggingUtils.java

+    try {
+      action.run();
+    } catch (Throwable t) {
+      // let logging exceptions fail silently


I wonder if it would make since to log this error at the lowest levels (maybe TRACE?) about any logging exceptions?

It would be like using logging to debug logging errors... Which honestly doesn't sound like something people would be doing. Just some thoughts, probably doesn't make any sense.

lqiu96 · 2025-02-21T19:17:48Z

gax-java/gax/src/main/java/com/google/api/gax/logging/Slf4jLoggingHelpers.java

+  }
+
+  private static void addIfNotEmpty(Consumer<String> setter, String value) {
+    if (value != null && !value.isEmpty()) {


nit: Guava's Strings.isEmptyorNull

lqiu96

LGTM. I think we have a few follow up tasks and clean up that we can tackle in future PRs.

sonarqubecloud · 2025-02-21T21:07:37Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
60 New issues
0 Accepted issues

Measures
0 Security Hotspots
80.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-02-21T21:11:19Z

Quality Gate failed for 'java_showcase_integration_tests'

Failed conditions
30.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@timestamp

This Pr contains changes for adding client logging capability to auth. see [go/java-client-logging-design](http://goto.google.com/java-client-logging-design) Some logging setups are mirror of same setting in Gax ([pr 3403](googleapis/sdk-platform-java#3403)) Changes includes: - add slf4j as optional dependency, enable logging only when GOOGLE_SDK_JAVA_LOGGING=true - user app should add SLF4J + binding dependencies accordingly. For SLF4J 1x, we record extra info with MDC; for SLF4J2x, we record extra info with KeyValuePairs. More user guide to be added via README (see [draft](https://docs.google.com/document/d/1g8HJCstkEEZZc73gFlQO8uPGs07SkWKkr7NG-Yg4bvM/edit?tab=t.0#heading=h.9t58aw3up7to)). - If env var not true, or no binding present, default to no-op. - Add log for request and response made to auth endpoints for UserCredentials, ServiceAccountCredentials, ComputeEngineCredentials, ImpersonatedCredentials. For token values included, added hash in log. Note this PR mainly focuses on adding logging setups, logging statements can be added incrementally later if necessary. Here is an example logging output for access token request from UserCredentials when DEBUG level is allowed. (TO UPDATE) ``` {"@timestamp":"2024-12-04T21:10:48.596382834-05:00","@Version":"1","message":"Sending auth request to refresh access token.","logger_name":"com.google.auth.oauth2.UserCredentials","thread_name":"main","severity":"DEBUG","level_value":10000,"request.method":"POST","request.headers":{"x-goog-api-client":"gl-java/19.0.1 auth/1.32.2-SNAPSHOT cred-type/u","accept-encoding":["gzip"]},"request.url":"https://foo.com/bar","request.payload":{"refresh_token":"bae0258be92ea1d1e14f984507bee05ff4502a29104f4101d22a4a88706b0fc0","grant_type":"refresh_token","client_secret":"2d3c802ef65d75e88b098792e2268cd3f55a08bcbb8c8c4672f1195d1951d4b5","client_id":"ya29.1.AADtN_UtlxN3PuGAxrN2XQnZTVRvDyVWnYq4I6dws"}} {"@timestamp":"2024-12-04T21:10:48.624914522-05:00","@Version":"1","message":"Received auth respond for refresh access token.","logger_name":"com.google.auth.oauth2.UserCredentials","thread_name":"main","severity":"INFO","level_value":20000,"response.status":"200","response.headers":"{}"} {"@timestamp":"2024-12-04T21:10:48.627483862-05:00","@Version":"1","message":"Auth response payload.","logger_name":"com.google.auth.oauth2.UserCredentials","thread_name":"main","severity":"DEBUG","level_value":10000,"access_token":"1/MkS*****KPY2","refresh_token":"1/Tl6*****yGWY","token_type":"Bearer","expires_in":"3600"} ``` Testing setup - Currently added flavor of tests: - unit tests with no extra dependency - unit tests depending on either binding be present, or logback implementation to capture logging for test - test for log behaviors for various requests where logs are added. (see LoggingTest) Remaining issue with test scenarios: 1. when no binding present: this is tested via regular tests 2. when env var is T and binding present: tested via test-logging (profile with logback deps) 3. no SLF4J present: same logic as 1, it should be fine with coverage of 1. This is hard to setup this because SLF4J is needed at compile. 4. SLF4J 1x + binding: Hard to setup because SLF4J 2x is needed at compile

🤖 I have created a release *beep* *boop* --- <details><summary>2.54.0</summary> ## [2.54.0](v2.53.0...v2.54.0) (2025-02-25) ### Features * add client side logging with slf4j ([#3403](#3403)) ([fe002fa](fe002fa)) ### Bug Fixes * S2A gRPC flow creates ComputeEngineCredentials via newBuilder. ([#3651](#3651)) ([29c061e](29c061e)) ### Dependencies * update dependency ch.qos.logback:logback-core to v1.3.15 [security] ([#3654](#3654)) ([093d867](093d867)) * update google api dependencies ([#3631](#3631)) ([48db2a1](48db2a1)) * update google auth library dependencies to v1.33.1 ([#3656](#3656)) ([f7877a5](f7877a5)) * update google http client dependencies to v1.46.3 ([#3657](#3657)) ([9d5b3b5](9d5b3b5)) * update grpc to 1.70.0 ([#3641](#3641)) ([ad26cf9](ad26cf9)) * update grpc to 1.70.0 (missed update) ([#3658](#3658)) ([6ca0599](6ca0599)) * Update opentelemetry-semconv to v1.29.0-alpha ([#3635](#3635)) ([49ac09d](49ac09d)) ### Documentation * update showcase readme ([#3659](#3659)) ([0ddf073](0ddf073)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Joe Wang <[email protected]>

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.53.0` -> `2.54.0` | | [com.google.cloud:google-cloud-core-http](https://github.com/googleapis/sdk-platform-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.52.0` -> `2.53.0` | | [com.google.cloud:google-cloud-core](https://github.com/googleapis/sdk-platform-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.52.0` -> `2.53.0` | | [com.google.api:gax](https://github.com/googleapis/sdk-platform-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.62.0` -> `2.63.0` | | [com.autonomousapps.dependency-analysis](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin) | plugin | misk/gradle/libs.versions.toml | gradle | minor | `2.10.1` -> `2.11.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.37` -> `2.30.38` | --- ### Release Notes <details> <summary>googleapis/sdk-platform-java (com.google.api.grpc:proto-google-common-protos)</summary> ### [`v2.54.0`](https://github.com/googleapis/sdk-platform-java/blob/HEAD/CHANGELOG.md#2540-2025-02-25) ##### Features - add client side logging with slf4j ([#3403](googleapis/sdk-platform-java#3403)) ([fe002fa](googleapis/sdk-platform-java@fe002fa)) ##### Bug Fixes - S2A gRPC flow creates ComputeEngineCredentials via newBuilder. ([#3651](googleapis/sdk-platform-java#3651)) ([29c061e](googleapis/sdk-platform-java@29c061e)) ##### Dependencies - update dependency ch.qos.logback:logback-core to v1.3.15 \[security] ([#3654](googleapis/sdk-platform-java#3654)) ([093d867](googleapis/sdk-platform-java@093d867)) - update google api dependencies ([#3631](googleapis/sdk-platform-java#3631)) ([48db2a1](googleapis/sdk-platform-java@48db2a1)) - update google auth library dependencies to v1.33.1 ([#3656](googleapis/sdk-platform-java#3656)) ([f7877a5](googleapis/sdk-platform-java@f7877a5)) - update google http client dependencies to v1.46.3 ([#3657](googleapis/sdk-platform-java#3657)) ([9d5b3b5](googleapis/sdk-platform-java@9d5b3b5)) - update grpc to 1.70.0 ([#3641](googleapis/sdk-platform-java#3641)) ([ad26cf9](googleapis/sdk-platform-java@ad26cf9)) - update grpc to 1.70.0 (missed update) ([#3658](googleapis/sdk-platform-java#3658)) ([6ca0599](googleapis/sdk-platform-java@6ca0599)) - Update opentelemetry-semconv to v1.29.0-alpha ([#3635](googleapis/sdk-platform-java#3635)) ([49ac09d](googleapis/sdk-platform-java@49ac09d)) ##### Documentation - update showcase readme ([#3659](googleapis/sdk-platform-java#3659)) ([0ddf073](googleapis/sdk-platform-java@0ddf073)) </details> <details> <summary>autonomousapps/dependency-analysis-android-gradle-plugin (com.autonomousapps.dependency-analysis)</summary> ### [`v2.11.0`](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/blob/HEAD/CHANGELOG.md#Version-2110) - \[Feat]: new task (`:computeAllDependencies`) for producing a version catalog file containing all dependencies. - \[Fix]: kotlin-reflect must be on the compile classpath. - \[Test]: Update AGP 8.9.0 stable. - \[Test]: conditional signing. </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: c7d38837c081de3bba13d5d7f2d42008ba98f440

zhumin8 added 9 commits November 19, 2024 11:16

poc: add log in grpc interceptor.

59b5d73

try out basics with httpjson interceptor.

d34ec88

implement conditional logging, use JUL when no binding. log formattin…

b6417ae

…g needs more work.

temp manual testing with showcase.

0c169ad

simplify logic for conditional logger.

9ced175

implement logging for basic case.

681f7d5

add a JsonContextMapHandler to format in JUL case.

5c01ef2

fix response payload for streaming. Add if guards to bypass log when …

6605ef3

…not enabled.

when logging disabled, assign NOPLogger directly.

ebbae15

product-auto-label bot added the size: l Pull request size is large. label Nov 27, 2024

some temp settings for testing with showcase.

543298d

blakeli0 reviewed Dec 3, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/logging/LoggingUtils.java Outdated Show resolved Hide resolved

blakeli0 reviewed Dec 3, 2024

View reviewed changes

gax-java/gax-grpc/src/main/java/com/google/api/gax/grpc/GrpcLoggingInterceptor.java Outdated Show resolved Hide resolved

blakeli0 reviewed Dec 3, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/logging/JsonContextMapHandler.java Outdated Show resolved Hide resolved

blakeli0 reviewed Dec 3, 2024

View reviewed changes

gax-java/gax-grpc/src/main/java/com/google/api/gax/grpc/GrpcLoggingInterceptor.java Outdated Show resolved Hide resolved

zhumin8 added 11 commits December 5, 2024 16:25

remove JUL wrapper for now. refactor getLogger(). Add tests for Loggi…

aac66f2

…ngUtils.

minor clean up.:

487614c

fix test after adding interceptor for logging for http.

cc3465b

cleanup and fix response header.

211a3d1

lint

2970210

lint: license.

32f8b7c

refactor GrpcLoggingInterceptor for readability, intro LogData.

245c14d

minor cleanups.

8358497

GrpcLoggingInterceptor fix for thread safe, separate out debug loggin…

97087b2

…g entries, add request id to track.

minor changes + add showcase it test.(need to run with env var, setup…

727a3e9

… not done yet.)

refactor HttpJsonLoggingInterceptor. remove public modifier from logg…

23bc111

…ing interceptors.

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Dec 6, 2024

zhumin8 added 2 commits December 7, 2024 21:41

add GrpcLoggingInterceptorTest and test interceptor structure.

bbe0700

fix status code '0'.

3d7c7f9

lqiu96 reviewed Feb 18, 2025

View reviewed changes

gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/HttpJsonLoggingInterceptor.java Outdated Show resolved Hide resolved

lqiu96 reviewed Feb 18, 2025

View reviewed changes

zhumin8 added 2 commits February 18, 2025 14:21

add final to class, add comments.

091ab38

revert final change.

b3b328e

lqiu96 reviewed Feb 18, 2025

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/logging/Slf4jUtils.java Outdated Show resolved Hide resolved

lqiu96 reviewed Feb 18, 2025

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/logging/Slf4jUtils.java Outdated Show resolved Hide resolved

zhumin8 added 3 commits February 18, 2025 15:59

minor refactor: split internal Slf4jUtils and Slf4jLoggingHelpers.

6aece18

check type before casting to Message

c3db3e7

fix: rm grpc from log message.

a975fd5

This was referenced Feb 19, 2025

Add showcase testing for logging on stream cases #3645

Open

verify debug logging works as expected for streaming cases #3646

Closed

migrate existing logged warnings to slf4j logs #3647

Open

zhumin8 added 2 commits February 20, 2025 15:08

continue: rm grpc from log message.

b1af879

Merge branch 'main' into client-log

157921b

zhumin8 commented Feb 20, 2025

View reviewed changes

zhumin8 mentioned this pull request Feb 20, 2025

Logging: capture and re-throw more clear error when sfl4j not in classpath but logging enabled. #3649

Open

blakeli0 approved these changes Feb 21, 2025

View reviewed changes

lqiu96 reviewed Feb 21, 2025

View reviewed changes

lqiu96 approved these changes Feb 21, 2025

View reviewed changes

zhumin8 added 2 commits February 21, 2025 15:26

minor change: use guava isNullOrEmpty.

89da1b4

fix typo.

fad1865

zhumin8 merged commit fe002fa into main Feb 24, 2025
51 of 53 checks passed

zhumin8 deleted the client-log branch February 24, 2025 17:03

release-please bot mentioned this pull request Feb 24, 2025

chore(main): release 2.54.0 #3636

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add client side logging with slf4j #3403

feat: add client side logging with slf4j #3403

zhumin8 commented Nov 27, 2024 •

edited

Loading

lqiu96 Feb 18, 2025 •

edited

Loading

zhumin8 Feb 18, 2025

lqiu96 Feb 21, 2025

zhumin8 Feb 20, 2025 •

edited

Loading

lqiu96 Feb 21, 2025

lqiu96 Feb 21, 2025

lqiu96 left a comment

sonarqubecloud bot commented Feb 21, 2025

sonarqubecloud bot commented Feb 21, 2025

feat: add client side logging with slf4j #3403

feat: add client side logging with slf4j #3403

Conversation

zhumin8 commented Nov 27, 2024 • edited Loading

lqiu96 Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

zhumin8 Feb 18, 2025

Choose a reason for hiding this comment

lqiu96 Feb 21, 2025

Choose a reason for hiding this comment

zhumin8 Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

lqiu96 Feb 21, 2025

Choose a reason for hiding this comment

lqiu96 Feb 21, 2025

Choose a reason for hiding this comment

lqiu96 left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 21, 2025

Quality Gate passed for 'gapic-generator-java-root'

sonarqubecloud bot commented Feb 21, 2025

Quality Gate failed for 'java_showcase_integration_tests'

zhumin8 commented Nov 27, 2024 •

edited

Loading

lqiu96 Feb 18, 2025 •

edited

Loading

zhumin8 Feb 20, 2025 •

edited

Loading