Refactor/remove obp special endpoints#2816
Open
hongwei1 wants to merge 10 commits into
Open
Conversation
…ve http4s Replace LiftRules.statelessDispatch registration of OBPAPIDynamicEndpoint with a dedicated in-process Lift adapter (Http4sDynamicEndpoint) wired into Http4sApp.baseServices, positioned ahead of the Lift bridge. Covers both runtime pieces: - Piece B (proxy): ImplementationsDynamicEndpoint.dynamicEndpoint, matched by DynamicReq.unapply and proxied to a backend connector / obp_mock. - Piece C (runtime-compiled): DynamicEndpoints.dynamicEndpoint, serving practise / dynamic-resource-doc endpoints compiled from user Scala via DynamicUtil.compileScalaCode[OBPEndpoint]. Piece C compiled artifacts are hardwired to Lift types (PartialFunction[Req, CallContext => Box[JsonResponse]]) and cannot be natively rewritten. The adapter runs the exact wrapped form Lift held in statelessDispatch — routes.map(apiPrefix andThen buildOAuthHandler) — inside S.init, preserving failIfBadAuthorizationHeader / failIfBadJSON semantics and endpoint metrics unchanged. Changes: - Http4sDynamicEndpoint.scala (new): in-process Lift adapter. Buffers body, builds Lift Req via buildLiftReq, enters S.init, collectFirst over wrappedRoutes, converts Box[LiftResponse] to http4s Response. Catches JsonResponseException (eager failIfBadAuthorizationHeader) and ContinuationException (async Lift). No withBusinessDBTransaction wrap (dynamic-endpoint wrote on autocommit connections via the bridge too). - Http4sLiftWebBridge.scala: promote buildLiftReq, liftResponseToHttp4s, resolveContinuation from private to public so Http4sDynamicEndpoint (different package) can reuse them. - OBPRestHelper.scala: extract buildOAuthHandler from oauthServe. Returns the identical PartialFunction[Req, () => Box[LiftResponse]] without registering into statelessDispatch, so the adapter can construct the exact wrapped form in-process. - Http4sApp.scala: add dynamicEndpointRoutes val (gate on ApiVersion.dynamic-endpoint) and wire into baseServices orElse chain after dynamicEntityRoutes, before Http4sLiftWebBridge. - APIUtil.scala: comment out LiftRules.statelessDispatch.append for ApiVersion.dynamic-endpoint; keep empty case label so it does not fall through to the ScannedApiVersion branch. - OBPAPIDynamicEndpoint.scala: comment out statelessDispatch self- registration and OPTIONS serve (CORS now handled globally by Http4sApp.corsHandler). Keep object / version / allResourceDocs / routes intact (adapter source list + resource-docs aggregation). Verified: 45 / 45 dynamic regression tests pass on JDK 11 (DynamicEndpointsTest 30, DynamicUtilTest 9, DynamicResourceDocTest 3, DynamicMessageDocTest 2, DynamicIntegrationTest 1).
Stage 1 of removing the Lift adapter from the dynamic-endpoint dispatch: the proxy path (DynamicReq-matched requests proxied to a backend connector or obp_mock) is now served by a native http4s handler instead of building a Lift Req via buildLiftReq and running inside S.init. - DynamicEndpointHelper: extract the framework-neutral core of DynamicReq.unapply into DynamicReq.resolveProxyTarget(method, partPath, query, body). The Lift unapply now delegates to it after its content-type/prefix gate; the native dispatcher calls the same method, so both build the identical proxy 9-tuple from the same DB lookup (dynamicEndpointInfos / findDynamicEndpoint). - APIMethodsDynamicEndpoint: extract proxyHandle(...) -> Future[(JValue, Int)], the framework-neutral proxy logic (before/after authenticate interceptors, authentication, entitlement check, dynamic-entity mapping branch or mock/connector proxy). The before interceptor is reduced to (message, code) via JsonResponseExtractor + booleanToFuture (mirroring the existing after interceptor and Http4sDynamicEntity) instead of returning a Lift JsonResponse directly. The Lift dynamicEndpoint handler now delegates to proxyHandle. - Http4sDynamicEndpoint: add native proxy(req) — builds CallContext via Http4sCallContextBuilder, matches via resolveProxyTarget, runs proxyHandle and renders the connector/mock status code through the new EndpointHelpers.executeFutureWithStatus. Tried ahead of the Lift adapter; a non-match falls through to the adapter, which still serves Piece C (runtime-compiled endpoints). Proxy writes stay on auto-commit (no withBusinessDBTransaction), matching the prior bridge/adapter behaviour. - Http4sSupport: add EndpointHelpers.executeFutureWithStatus for rendering a (result, statusCode) pair with a dynamic HTTP status + metric + error handling. The mock-response thread-local (MockResponseHolder) is read synchronously by the connector at Future-construction time, so wrapping the connector call in MockResponseHolder.init inside proxyHandle preserves behaviour on the cats-effect thread pool. Verified on JDK 11: 154 / 154 pass across DynamicEndpointsTest (proxy E2E), DynamicEndpointHelperTest, ForceError/JsonSchema/AuthenticationType validation (interceptor regression), DynamicResourceDocTest, DynamicMessageDocTest, DynamicIntegrationTest, DynamicUtilTest.
…e http4s Stage 2 (final) of removing the Lift adapter from the dynamic-endpoint dispatch. The runtime-compiled / practise endpoints are now served natively; Http4sDynamicEndpoint no longer uses buildLiftReq / liftResponseToHttp4s / S.init / statelessSession at all. The dynamic-code authoring contract is redefined from Lift to native http4s: process(callContext, request: net.liftweb.http.Req, pathParams): Box[JsonResponse] becomes process(callContext, request: org.http4s.Request[IO], pathParams): IO[Response[IO]] Bodies read the request payload from callContext.httpBody, return errors via the new errorResponse(msg, code) helper (replacing Full(errorJsonResponse(...))), and may still yield an OBPReturnType which an injected implicit converts to IO[Response[IO]] (status from CallContext.httpCode). BREAKING: existing DB-stored methodBody rows written against the Lift contract no longer compile; DynamicResourceDocsEndpointGroup now isolates a failing row (log + skip) instead of crashing the group/boot, with a message to re-author against the native contract. Changes: - APIUtil: add type OBPEndpointIO = PartialFunction[Request[IO], CallContext => IO[Response[IO]]] (distinct from the Lift OBPEndpoint, which is shared by every static endpoint and unchanged); add ResourceDoc.dynamicHttp4sFunction: Option[OBPEndpointIO] = None to carry the compiled native handler; add ResourceDoc.matchesPartPath (public form of the wrappedWithAuthCheck URL match) and ResourceDoc.authCheckIO (native mirror of wrappedWithAuthCheck's auth/obp-id/bank/roles/account/view/counterparty chain, reusing the same predicates + *Fun). - DynamicCompileEndpoint: process returns IO[Response[IO]]; endpoint is OBPEndpointIO; add the OBPReturnType[T] => IO[Response[IO]] implicit and errorResponse helper; run via runInSandboxIO. - DynamicUtil.Sandbox: add runInSandboxIO — builds the body's IO under the privileged context (synchronous construction restricted, matching the Lift path) and evaluates it outside, and recovers a NonLocalReturnControl so a `return errorResponse(...)` in template code yields its response instead of a 500. - DynamicEndpoints: native code-generation template (OBPEndpointIO, http4s imports, pathParams from request.uri segments); EndpointGroup drops the Lift endpoints/wrapEndpoint; findEndpoint now takes Request[IO] and returns the matched ResourceDoc; the Lift dynamicEndpoint is removed. - DynamicResourceDocsEndpointGroup / PractiseEndpointGroup: carry the compiled handler in dynamicHttp4sFunction with partialFunction = dynamicEndpointStub; per-row try/skip for legacy rows. - PractiseEndpoint / ExampleValue.dynamicResourceDocMethodBodyExample: rewritten to the native contract (the body operators copy from). - Http4sDynamicEndpoint: native pieceC dispatch (findEndpoint -> authCheckIO -> compiled handler in sandbox -> IO[Response]); the entire Lift adapter is deleted. Entry is proxy.orElse(pieceC). - OBPAPIDynamicEndpoint.routes: drop the removed Lift Piece C entry. - DynamicResourceDocTest: add native-execution E2E scenarios (practise endpoint anonymous; create-and-call a runtime-compiled doc — happy path, and no-body 400 exercising the NonLocalReturn recovery). These prove the doc RUNS, not just compiles. - test props (build_pull_request.yml + test.default.props.template): grant the standard dynamic_code_sandbox_permissions (reflection / getenv) so dynamic bodies can execute under the sandbox in tests, matching default.props / production.default.props. Verified on JDK 11: 156 / 156 pass across DynamicResourceDocTest (incl. the 2 new E2E), DynamicEndpointsTest, DynamicEndpointHelperTest, DynamicMessageDocTest, DynamicIntegrationTest, DynamicUtilTest, and ForceError / JsonSchema / AuthenticationType interceptor regression.
… gate The native dynamic-endpoint proxy carried an isJsonRequest gate (introduced with the Piece B migration) that only matched requests whose Content-Type or Accept literally contained "json". The Lift DynamicReq extractor it replaced gated on testResponse_?, which treats a wildcard Accept (and an absent Accept) as JSON-acceptable — so it matched the OBP test client's GET proxy calls (Accept */*, Content-Type text/plain). The literal check rejected those GETs, so a created dynamic endpoint called via GET fell through to the Lift bridge and returned 404 (caught by RateLimitingTest's Dynamic Endpoint scenario in the full suite). Remove the gate entirely: the native dispatch has no XML alternative, and resolveProxyTarget already returns None for any path that is not a registered dynamic-endpoint (falling through to Piece C / the chain), so the gate is unnecessary. POST proxy calls (JSON body) and GET proxy calls now both match. Verified on JDK 11: RateLimitingTest, DynamicEndpointsTest, DynamicResourceDocTest all pass (44/44).
Adds a DynamicResourceDocTest scenario that creates a runtime-compiled dynamic-resource-doc gated by a (system-level) dynamic role and asserts the native auth chain enforces it: 401 without authentication, 403 when authenticated but missing the role, 200 once the role is granted. This was the only branch of ResourceDoc.authCheckIO (the native mirror of wrappedWithAuthCheck introduced in the Piece C migration) not previously exercised — the existing native-execution scenarios only covered the no-role/anonymous path. Verified on JDK 11: DynamicResourceDocTest 6/6 pass. Note: the proxy entity-mapping branch (isDynamicEntityResponse, in proxyHandle) is intentionally not given a new HTTP E2E here — it is verbatim-relocated Lift code (no logic change in the migration), already has an isDynamicEntityResponse unit test (DynamicEndpointHelperTest) plus mock-branch HTTP coverage (DynamicEndpointsTest / RateLimitingTest), and the existing example fixtures (swagger host=obp_mock, mapping referencing unrelated entities) are not aligned for a clean end-to-end call; a bespoke fixture would be brittle for little gain.
Adds a regression safety net ahead of refactoring the DynamicMessageDoc runtime mechanism. Two new scenarios in DynamicMessageDocTest: - 401: the management endpoints (POST/GET/GET-all/PUT/DELETE on /management/dynamic-message-docs) reject unauthenticated requests with 401. (Previously only the metadata CRUD and role-403 paths were covered.) - Runtime invoke chain: store a DynamicMessageDoc (Scala methodBody) via DynamicMessageDocProvider.create, then call DynamicConnector.invoke and assert the stored body is compiled and run, returning the expected object. This covers the full DB-stored-methodBody -> invoke -> getFunction -> getByProcess -> createFunction (DynamicUtil.compileScalaCode) -> execute path; InternalConnectorTest only exercised createFunction+executeFunction in isolation, bypassing the DB and invoke/getFunction. Connector methods do not run inside the security sandbox, so no sandbox-permission setup is needed; the Scala methodBody is compiled at runtime, which requires JDK 11. Test-only; no main code changed. The DynamicMessageDoc management endpoints are already native http4s and the runtime path uses no Lift web layer, so this is a coverage safety net rather than a migration. Verified on JDK 11: DynamicMessageDocTest 4/4, InternalConnectorTest, DynamicUtilTest pass.
Cleanup after the dynamic-endpoint/entity native migration. No behaviour change;
removes dead Lift web types that no longer participate in dispatch.
Part A — shared (DynamicUtil sandbox):
- Drop the two NonLocalReturnControl[JsonResponse] catch clauses in
Sandbox.createSandbox.runInSandbox (now a plain AccessController.doPrivileged) and
the `import net.liftweb.http.JsonResponse`. The only caller is runInSandboxIO,
whose forceBodyIO already recovers a NonLocalReturnControl before it reaches
runInSandbox; connector methods (DynamicMessageDoc) never run inside the sandbox.
Part B — dynamic-endpoint dead Lift refs:
- ResourceDocsAPIMethods: add `case dynamic-endpoint => resourceDocs` to
activeResourceDocs (mirrors dynamic-entity), so the dynamic-endpoint resource docs
are returned unfiltered instead of being filtered by Lift route class. This must
precede removing the routes entry, otherwise the proxy docs would be filtered out.
- APIMethodsDynamicEndpoint: remove the dead Lift `dynamicEndpoint: OBPEndpoint`
(matched by DynamicReq.unapply) — dispatch is fully native via proxyHandle. Drop the
now-unused DynamicReq and net.liftweb.http.{JsonResponse, Req} imports.
- DynamicEndpointHelper: remove the dead `DynamicReq.unapply(r: Req)` extractor (its
only consumer was the removed dynamicEndpoint); keep resolveProxyTarget. DynamicReq
no longer extends JsonTest with JsonBody. Drop the net.liftweb.http.Req import.
- OBPAPIDynamicEndpoint: routes reduced to List(dynamicEndpointStub); drop the
net.liftweb.http.{LiftResponse, PlainTextResponse} import (commented-out CORS only).
- Tests: DynamicendPointsTest / ForceErrorValidationTest referenced the removed
dynamicEndpoint via nameOf for a Tag; kept the tag name as the literal "dynamicEndpoint"
(same convention already used there for the migrated genericEndpoint).
The RestHelper mixins on DynamicEndpointHelper / APIMethodsDynamicEndpoint are kept
(DynamicEndpointHelper overrides RestHelper's `formats`); removing them is higher-risk
and out of scope for this cleanup.
Verified on JDK 11: 105/105 across DynamicEndpointsTest, DynamicResourceDocTest,
DynamicEndpointHelperTest, DynamicMessageDocTest, DynamicUtilTest, InternalConnectorTest,
ForceErrorValidationTest. Full run_all_tests.sh in progress.
The develop/container CI (build_container.yml) generates test.default.props from
scratch via echo lines and was missing dynamic_code_sandbox_permissions — only
build_pull_request.yml had it. Without those permissions the dynamic-code security
sandbox denies getenv (connector metric prop reads), reflection and
NetPermission("specifyStreamHandler"), so DynamicResourceDocTest's three
native-execution scenarios (practise endpoint, create+call a runtime-compiled doc,
role-gated doc) fail with AccessControlException in shard 1 (v4 only).
Add the same permission list used by build_pull_request.yml / default.props /
production.default.props so dynamic resource-doc bodies can execute under the sandbox
in this workflow too. CI-only change.
…ponse format
Add Http4sBGv13PIIS (1 endpoint: POST /funds-confirmations) and Http4sBGv13 aggregator
wired into Http4sApp ahead of the Lift bridge.
Fix ErrorResponseConverter to emit ErrorMessagesBG (tppMessages) format for all
Berlin Group paths, mirroring APIUtil.failedJsonResponse's URL-prefix check. Without
this, BG v1.3 tests that assert tppMessages structure receive the standard OBP
{code,message} format and fail with "head of empty list".
…ttp4s Also fixes ResourceDocMatcher.apiPrefixPattern to handle Berlin Group paths (/berlin-group/v1.3/...) in addition to OBP-standard paths (/obp/vX.X.X/...). Without this fix the middleware could not strip the BG prefix, segment counts mismatched, findResourceDoc returned None, and auth was bypassed.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.