Add LLMObs integration tests for Java by ygree · Pull Request #6364 · DataDog/system-tests

ygree · 2026-02-20T23:58:31Z

Motivation

Add support for system LLMObs tests for Java OpenAI SDK, covering chat completions, tool calls, responses API, and embedding interactions.
Add support for parametric LLMObs manual API tests for Java.

Changes

Add a new Java test app (openai_app) built with Gradle/Javalin that implements OpenAI proxy endpoints for
testing
Add openai-java.Dockerfile to build and run the Java test app with dd-java-agent
Bump dd-apm-test-agent from v1.38.0 to v1.42.0
Add support for /evp_proxy/v4 endpoint in llmobs_requests(); add gzip decompression and MessagePack
decoding for Java tracer payloads

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-02-20T23:59:04Z

CODEOWNERS have been resolved as:

tests/integration_frameworks/utils/vcr-cassettes/openai/test_chat_completion_stream_False_openai_chat_completions_post_424f6218.json  @DataDog/system-tests-core
tests/integration_frameworks/utils/vcr-cassettes/openai/test_chat_completion_tool_call_stream_False_openai_chat_completions_post_ec6364ef.json  @DataDog/system-tests-core
tests/integration_frameworks/utils/vcr-cassettes/openai/test_embedding_openai_embeddings_post_e8a65512.json  @DataDog/system-tests-core
tests/integration_frameworks/utils/vcr-cassettes/openai/test_responses_create_stream_False_openai_responses_post_661cf6eb.json  @DataDog/system-tests-core
utils/build/docker/java/openai-java.Dockerfile                          @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/.gitignore                           @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/build.gradle                         @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradle/wrapper/gradle-wrapper.jar    @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradle/wrapper/gradle-wrapper.properties  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradlew                              @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/settings.gradle                      @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/src/main/java/SingleFileServer.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/system_tests_library_version.sh      @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/controller/LlmObsController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
tests/integration_frameworks/llm/openai/test_openai_llmobs.py           @DataDog/ml-observability
tests/integration_frameworks/llm/utils.py                               @DataDog/ml-observability
utils/_context/_scenarios/integration_frameworks.py                     @DataDog/system-tests-core
utils/_context/_scenarios/parametric.py                                 @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core

datadog-official · 2026-02-21T00:09:39Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 42 Tests failed

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_mix_snapshot[uds] from system_tests_suite

(Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7fdf27e2b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7fdf27e2b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7fdf19ca1040>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7fdf19ca1040>

    def test_mix_snapshot(self):
>       self._assert()

...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[poc] from system_tests_suite

(Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f43aba2b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f43aba2b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f439d96c320>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f439d96c320>

    def test_log_method_snapshot(self):
>       self._assert()

...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[spring-boot-openliberty]

from system_tests_suite

(Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f270a42b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f270a42b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f27002b3cb0>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f27002b3cb0>

    def test_log_method_snapshot(self):
>       self._assert()

...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a8b160e | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

output_tokens n/a thus not set java uses base64 encoding format instead of float

ygree · 2026-03-17T22:10:26Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d0c54e45c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifests/java.yml

tests/integration_frameworks/llm/openai/test_openai_llmobs.py

…ther libraries

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0289d40807

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

utils/build/docker/java/openai-java.Dockerfile

utils/build/docker/java/openai_app/src/main/java/SingleFileServer.java

tests/integration_frameworks/llm/openai/test_openai_llmobs.py

cbeauchesne

You need to add the new weblog in integration_frameworks_weblogs here : utils/scripts/ci_orchestrators/workflow_data.py

- Add openai-java@4.29.0 to integration_frameworks weblogs map - Enables CI to run integration tests for Java OpenAI SDK Rationale: New openai-java weblog was added to the repo and needs to be registered in CI orchestration This commit made by [/dd:git:commit:quick](https://github.com/DataDog/claude-marketplace/tree/main/dd/commands/git/commit/quick.md)

cbeauchesne

CI is failing. Feel free to ping me on slack if you need some help.

ygree · 2026-03-24T17:55:52Z

CI is failing. Feel free to ping me on slack if you need some help.

Two tests were expected to fail because the corresponding Java changes haven't been released yet. I changed the manifest so that they would be expected to fail until the next minor version.

ygree added 2 commits February 20, 2026 12:51

Add openai java test app

f9fe66a

Generate missing cassettes. openai-java generates requests differently.

8829033

ygree self-assigned this Feb 20, 2026

ygree force-pushed the ygree/openai-java-tests branch 2 times, most recently from 335d036 to 5e40f8a Compare February 21, 2026 00:03

Add /evp_proxy/v4 support for openai-java tests

53695b1

ygree force-pushed the ygree/openai-java-tests branch from 5e40f8a to 53695b1 Compare February 21, 2026 00:35

ygree added 3 commits February 27, 2026 13:07

Fix TestOpenAiEmbeddingInteractions for java.

91edef8

output_tokens n/a thus not set java uses base64 encoding format instead of float

set env and version in openai-java test app

eda8ace

Fix openai_app

d26893b

ygree mentioned this pull request Mar 6, 2026

fix(llmobs): openai-java payload mapping for responses, tool metadata, and prompt tracking DataDog/dd-trace-java#10644

Open

ygree changed the title ~~openai-java tests~~ Add LLMObs integration tests for Java OpenAI SDK Mar 6, 2026

ygree added 4 commits March 17, 2026 11:50

Merge branch 'main' into ygree/openai-java-tests

c9c6534

Update docker image

5765ba8

detect the latest openai version by default

fec0549

java: implement LLMObs parametric tests

5d0c54e

chatgpt-codex-connector bot reviewed Mar 17, 2026

View reviewed changes

manifests/java.yml Show resolved Hide resolved

tests/integration_frameworks/llm/openai/test_openai_llmobs.py Outdated Show resolved Hide resolved

ygree added 2 commits March 17, 2026 15:45

manifests: mark Test_Dataset as incomplete_test_app for Java

dc6c188

fix: set FLOAT encoding format in Java embedding call to align with o…

1201239

…ther libraries

ygree changed the title ~~Add LLMObs integration tests for Java OpenAI SDK~~ Add LLMObs integration tests for Java SDK Mar 18, 2026

ygree changed the title ~~Add LLMObs integration tests for Java SDK~~ Add LLMObs integration tests for Java Mar 18, 2026

fix format

0289d40

ygree marked this pull request as ready for review March 18, 2026 02:12

ygree requested review from a team as code owners March 18, 2026 02:12

ygree requested review from daniel-romano-DD and manuel-alvarez-alvarez March 18, 2026 02:12

chatgpt-codex-connector bot reviewed Mar 18, 2026

View reviewed changes

utils/build/docker/java/openai-java.Dockerfile Outdated Show resolved Hide resolved

utils/build/docker/java/openai_app/src/main/java/SingleFileServer.java Outdated Show resolved Hide resolved

Kyle-Verhoog approved these changes Mar 18, 2026

View reviewed changes

tests/integration_frameworks/llm/openai/test_openai_llmobs.py Show resolved Hide resolved

ygree added 2 commits March 18, 2026 13:05

remove gradle init

06e1c30

fix: handle missing ml_app in evaluation submission

31eeed7

cbeauchesne requested changes Mar 19, 2026

View reviewed changes

ygree added 2 commits March 19, 2026 09:50

fix(openai-java): use multi-stage build with runtime image and app.sh

ea45820

ygree changed the title ~~Add LLMObs integration tests for Java~~ Add LLMObs integration tests for Java [java@ygree/llmobs-systest-fixes] Mar 19, 2026

ygree requested a review from cbeauchesne March 19, 2026 23:16

ygree changed the title ~~Add LLMObs integration tests for Java [java@ygree/llmobs-systest-fixes]~~ Add LLMObs integration tests for Java [java@661ea70f3f531dc34e6edb4ed185cb8d509187d7] Mar 20, 2026

cbeauchesne requested changes Mar 23, 2026

View reviewed changes

Enable llm/openai java tests starting v1.61.0

9604517

ygree requested a review from cbeauchesne March 24, 2026 17:56

ygree changed the title ~~Add LLMObs integration tests for Java [java@661ea70f3f531dc34e6edb4ed185cb8d509187d7]~~ Add LLMObs integration tests for Java Mar 24, 2026

Merge branch 'main' into ygree/openai-java-tests

a8b160e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LLMObs integration tests for Java#6364

Add LLMObs integration tests for Java#6364
ygree wants to merge 19 commits intomainfrom
ygree/openai-java-tests

ygree commented Feb 20, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 20, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 21, 2026 •

edited by datadog-datadog-prod-us1 bot

Loading

Uh oh!

ygree commented Mar 17, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne left a comment

Uh oh!

cbeauchesne left a comment

Uh oh!

ygree commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ygree commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official bot commented Feb 21, 2026 • edited by datadog-datadog-prod-us1 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

ygree commented Mar 17, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

ygree commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ygree commented Feb 20, 2026 •

edited

Loading

github-actions bot commented Feb 20, 2026 •

edited

Loading

datadog-official bot commented Feb 21, 2026 •

edited by datadog-datadog-prod-us1 bot

Loading