diff --git a/README.md b/README.md index ea4e4ea0d1..0c8a9514ed 100644 --- a/README.md +++ b/README.md @@ -608,6 +608,176 @@ There are 3 API endpoints related to webhooks: --- +## Email Delivery + +OBP-API sends emails for several reasons: signup confirmation, email-address +validation, password reset, user invitations, role-grant notifications, SCA +(Strong Customer Authentication) challenges over email, and uncaught-exception +alerts to admins. All of these go through a single Jakarta Mail wrapper +(`code.api.util.CommonsEmailWrapper`). + +### SMTP configuration + +Set these props (defaults shown): + +```props +mail.smtp.host=localhost +mail.smtp.port=1025 +mail.smtp.user= +mail.smtp.password= +mail.smtp.starttls.enable=false +mail.smtp.ssl.enable=false +mail.smtp.ssl.protocols=TLSv1.2 +``` + +For local development, [MailHog](https://github.com/mailhog/MailHog) or +[Mailpit](https://github.com/axllent/mailpit) on port 1025 lets you capture +outbound mail without configuring a real SMTP server. + +### Portal URL (required for validation and reset links) + +Signup-validation and password-reset emails embed a link built from +`portal_external_url`. If this prop is not set, the signup flow silently skips +the validation email — the user gets a 201 response but no mail. Set it: + +```props +portal_external_url=https://portal.yourdomain.com +``` + +OBP-API logs a multi-line WARN block at startup if this prop is missing (or +blank). It also surfaces on the `/status` page (see below). + +### Sender (From) addresses + +Different email types read different sender props. The most important ones: + +```props +# Used by signup, email validation, password reset (AuthUser.emailFrom) +# Default: noreply@example.com — most SMTP servers will reject this because of +# SPF/DKIM/anti-spoof, so change it before going live. +mail.users.userinfo.sender.address=noreply@yourdomain.com + +# Used by role-grant notifications (no default — required for those emails) +mail.api.consumer.registered.sender.address=noreply@yourdomain.com + +# Used by uncaught-exception alerts +mail.exception.sender.address=alerts@yourdomain.com +mail.exception.registered.notification.addresses=ops@yourdomain.com,oncall@yourdomain.com +``` + +### Test mode (no SMTP needed) + +Set `mail.test.mode=true` to log every would-be email at INFO level instead of +sending it over SMTP. Useful in CI and for local development without a mail +catcher. When this is on, no SMTP connection is attempted. + +### Startup configuration check + +On boot, OBP-API logs a WARN block if either: + +- `portal_external_url` is unset or blank, or +- `mail.users.userinfo.sender.address` is still the default `noreply@example.com`. + +Both conditions cause validation / password-reset emails to be silently skipped +or rejected downstream. The WARN block names the prop and points the operator +at `POST /obp/v7.0.0/management/self-test-emails` for end-to-end verification. + +### Status page (`/status`) + +`GET /status` (HTML at the URL, JSON when `Accept: application/json`) includes +two email-related rows under an "Email" section: + +- **`config`** — `ok` if `portal_external_url` is set and the sender address is + non-default; `warn` (with a one-line reason) otherwise. +- **`smtp`** — `ok` if a TCP connect to `mail.smtp.host:mail.smtp.port` succeeds + AND the server returns a `220` greeting within 2 s; `fail` otherwise (with + the exception class + message, e.g. `ConnectException: Connection refused`). + +The SMTP probe result is cached for 60 s so frequent `/status` pollers +(Prometheus blackbox, k8s probes) don't open a TCP connection on every request. +Neither row flips the overall readiness flag — email is a soft dependency, so a +broken SMTP won't make K8s kill the pod. + +The probe only reads the SMTP greeting banner. It does not exercise STARTTLS or +AUTH, so a server that requires TLS + credentials and would reject a real send +can still show `smtp: ok` here. The self-test endpoint exercises that full path. + +### Self-test endpoint + +There is a v7.0.0 admin endpoint to verify SMTP delivery end-to-end: + +``` +POST /obp/v7.0.0/management/self-test-emails +``` + +- Role required: `CanCreateTestEmail` +- Recipient: always the authenticated user's own email address (no `to` + parameter — eliminates "spam anyone else" as a DoS surface) +- From address: same as signup / password-reset emails (`AuthUser.emailFrom` → + prop `mail.users.userinfo.sender.address`) +- The body of the email includes the resolved `portal_external_url` so the + admin can confirm visually what users will see in real validation / reset + emails. +- Returns 201 with `{ to, from, subject, message_id }` on success. + +If the email cannot be sent, returns 500 with the most specific OBP error code +for the underlying cause. The exception chain (class name + message) is always +appended after `Detail:` so the operator can diagnose without server logs: + +| Failure | Status | Code | +|---|---|---| +| Caller has no email address | 400 | `OBP-30339 UserEmailAddressMissing` | +| `portal_external_url` unset | 500 | `OBP-10056 IncompleteServerConfiguration` | +| `mail.users.userinfo.sender.address` is default | 500 | `OBP-10056 IncompleteServerConfiguration` | +| SMTP rejected credentials | 500 | `OBP-30341 SmtpAuthenticationFailed` | +| TCP connect / host unreachable / timeout / DNS fail | 500 | `OBP-30342 SmtpConnectionFailed` | +| TLS / SSL handshake fail | 500 | `OBP-30343 SmtpTlsHandshakeFailed` | +| Recipient / From / message rejected by server | 500 | `OBP-30344 SmtpRecipientRejected` | +| Other Jakarta Mail protocol error | 500 | `OBP-30345 SmtpProtocolError` | +| Truly unknown | 500 | `OBP-30340 EmailSendingFailed` (fallback) | + +Use this from APIManager (or a `curl` with appropriate auth headers) to confirm +that signup / password-reset emails will be deliverable on this instance, +without needing to create a real account or trigger a real reset. + +### Logging + +Every successful send is logged at INFO level by `CommonsEmailWrapper`: + +``` +sendTextEmail says: sent to=alice@example.com subject='OBP test email from ...' messageId=<...@smtp...> +sendHtmlEmail says: sent to=alice@example.com subject='Sign up confirmation' messageId=<...@smtp...> +``` + +Failures are logged at ERROR level with the exception stack trace. The signup +and anonymous password-reset flows currently treat send failures as +fire-and-forget (the user request still succeeds with a 201), so the log line +is the authoritative record that an email did or did not leave the JVM. The +self-test endpoint above is the operator's complement for confirming the SMTP +path works at all. + +### SMTP-level debugging + +For diagnosing SMTP handshake failures, authentication issues, or message +rejections, set: + +```props +mail.debug=true +``` + +This enables Jakarta Mail's debug stream, which writes the entire SMTP protocol +conversation — EHLO, STARTTLS, AUTH, MAIL FROM, RCPT TO, DATA, every server +response, and the full message body — to `System.out`. + +**Security warning:** With debug on, the `AUTH LOGIN` exchange includes the +base64-encoded SMTP username and password, and the message body includes any +password-reset links, validation JWT tokens, and SCA OTP codes in plain text. +Anyone with stdout access (operators, log aggregators, kubectl logs, CI +artifacts) can read these. Use `mail.debug=true` only on developer laptops +pointed at a local mail catcher — never on a shared or production environment. + +--- + ## OpenID Connect **Note:** OpenID Connect authentication is supported for API authentication. Portal login functionality has been moved to the separate [OBP-Portal](https://github.com/OpenBankProject/OBP-Portal) project. diff --git a/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala b/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala index 398014cfd0..6b23db9b4e 100644 --- a/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala +++ b/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala @@ -338,6 +338,8 @@ class Boot extends MdcLoggable { warnAboutSuperAdminUsers() + warnAboutEmailDeliveryConfiguration() + createBootstrapOidcOperatorUser() createBootstrapOidcOperatorConsumer() @@ -942,6 +944,41 @@ class Boot extends MdcLoggable { } } + /** + * Warn at startup about email-delivery configuration that would silently break + * signup-validation and password-reset flows. Both flows embed a link built from + * `portal_external_url`; if the prop is missing the signup endpoint skips the + * send with no log line and the user gets a 201. Many SMTP servers reject the + * default `noreply@example.com` From address, in which case Transport.send + * succeeds at the boundary but mail is dropped downstream. + */ + private def warnAboutEmailDeliveryConfiguration(): Unit = { + val portalUrl = APIUtil.getPropsValue("portal_external_url") + val senderAddress = APIUtil.getPropsValue("mail.users.userinfo.sender.address", "noreply@example.com") + val portalMissing = portalUrl.isEmpty || portalUrl.exists(_.trim.isEmpty) + val senderIsDefault = senderAddress == "noreply@example.com" + if (portalMissing || senderIsDefault) { + logger.warn("========================================================================") + logger.warn("WARNING: Email delivery configuration is incomplete.") + logger.warn("") + if (portalMissing) { + logger.warn(" portal_external_url is not set.") + logger.warn(" Signup-validation and password-reset emails are silently skipped") + logger.warn(" when this prop is missing. Users will complete signup with a 201") + logger.warn(" response but never receive a validation email.") + } + if (senderIsDefault) { + logger.warn(" mail.users.userinfo.sender.address is still the default") + logger.warn(" 'noreply@example.com'. Most SMTP servers reject this From address") + logger.warn(" (SPF/DKIM/anti-spoof). SMTP send may succeed but mail will be") + logger.warn(" dropped downstream.") + } + logger.warn("") + logger.warn("Verify end-to-end delivery with POST /obp/v7.0.0/management/self-test-emails.") + logger.warn("========================================================================") + } + } + /** * Bootstrap OIDC Operator User * Given the following credentials, OBP will create a user *if it does not exist already*. diff --git a/obp-api/src/main/scala/code/api/util/ApiRole.scala b/obp-api/src/main/scala/code/api/util/ApiRole.scala index 7d57ea37fe..d09f02849a 100644 --- a/obp-api/src/main/scala/code/api/util/ApiRole.scala +++ b/obp-api/src/main/scala/code/api/util/ApiRole.scala @@ -288,6 +288,9 @@ object ApiRole extends MdcLoggable{ case class CanGetOidcClient(requiresBankId: Boolean = false) extends ApiRole lazy val canGetOidcClient = CanGetOidcClient() + case class CanCreateTestEmail(requiresBankId: Boolean = false) extends ApiRole + lazy val canCreateTestEmail = CanCreateTestEmail() + case class CanCreateTransactionType(requiresBankId: Boolean = true) extends ApiRole lazy val canCreateTransactionType = CanCreateTransactionType() diff --git a/obp-api/src/main/scala/code/api/util/ApiTag.scala b/obp-api/src/main/scala/code/api/util/ApiTag.scala index 56d35db260..4f98f938de 100644 --- a/obp-api/src/main/scala/code/api/util/ApiTag.scala +++ b/obp-api/src/main/scala/code/api/util/ApiTag.scala @@ -104,6 +104,7 @@ object ApiTag { val apiTagCounterpartyLimits = ResourceDocTag("Counterparty-Limits") val apiTagDevOps = ResourceDocTag("DevOps") val apiTagSystem = ResourceDocTag("System") + val apiTagEmail = ResourceDocTag("Email") val apiTagCache = ResourceDocTag("Cache") val apiTagLogCache = ResourceDocTag("Log-Cache") val apiTagTrading = ResourceDocTag("Trading") diff --git a/obp-api/src/main/scala/code/api/util/CommonsEmailWrapper.scala b/obp-api/src/main/scala/code/api/util/CommonsEmailWrapper.scala index 20fa029330..e34688a7d3 100644 --- a/obp-api/src/main/scala/code/api/util/CommonsEmailWrapper.scala +++ b/obp-api/src/main/scala/code/api/util/CommonsEmailWrapper.scala @@ -121,6 +121,7 @@ object CommonsEmailWrapper extends MdcLoggable { setCommonHeaders(message, content) message.setText(content.textContent.getOrElse(""), "UTF-8") Transport.send(message) + logger.info(s"sendTextEmail says: sent to=${content.to.mkString(",")} subject='${content.subject}' messageId=${message.getMessageID}") Full(message.getMessageID) } catch { case e: Exception => @@ -129,6 +130,31 @@ object CommonsEmailWrapper extends MdcLoggable { } } + // Variant that preserves the exception so callers can classify SMTP failures + // (auth / connect / TLS / recipient rejection) instead of returning a generic + // EmailSendingFailed. Used by the self-test endpoint; existing fire-and-forget + // callers (signup, password reset) keep using sendTextEmail above. + def sendTextEmailEither(content: EmailContent): Either[Throwable, String] = { + if (isTestMode) Right("test-mode-message-id-" + System.currentTimeMillis()) + else sendTextEmailEither(getDefaultEmailConfig(), content) + } + + def sendTextEmailEither(config: EmailConfig, content: EmailContent): Either[Throwable, String] = { + try { + val session = createSession(config) + val message = new MimeMessage(session) + setCommonHeaders(message, content) + message.setText(content.textContent.getOrElse(""), "UTF-8") + Transport.send(message) + logger.info(s"sendTextEmailEither says: sent to=${content.to.mkString(",")} subject='${content.subject}' messageId=${message.getMessageID}") + Right(message.getMessageID) + } catch { + case e: Throwable => + logger.error(s"sendTextEmailEither says: failed to send text email: ${e.getMessage}", e) + Left(e) + } + } + def sendHtmlEmail(config: EmailConfig, content: EmailContent): Box[String] = { try { logger.debug(s"Sending HTML email from ${content.from} to ${content.to.mkString(", ")}") @@ -150,6 +176,7 @@ object CommonsEmailWrapper extends MdcLoggable { } message.setContent(multipart) Transport.send(message) + logger.info(s"sendHtmlEmail says: sent to=${content.to.mkString(",")} subject='${content.subject}' messageId=${message.getMessageID}") Full(message.getMessageID) } catch { case e: Exception => @@ -196,6 +223,7 @@ object CommonsEmailWrapper extends MdcLoggable { } message.setContent(multipart) Transport.send(message) + logger.info(s"sendEmailWithAttachments says: sent to=${content.to.mkString(",")} subject='${content.subject}' messageId=${message.getMessageID} attachments=${content.attachments.length}") Full(message.getMessageID) } catch { case e: Exception => diff --git a/obp-api/src/main/scala/code/api/util/ErrorMessages.scala b/obp-api/src/main/scala/code/api/util/ErrorMessages.scala index c9b4c33cac..f5e8e0f416 100644 --- a/obp-api/src/main/scala/code/api/util/ErrorMessages.scala +++ b/obp-api/src/main/scala/code/api/util/ErrorMessages.scala @@ -677,6 +677,14 @@ object ErrorMessages { val InvalidViewPermissionName = "OBP-30337: The view permission name does not exist in OBP." val DeleteViewPermissionError = "OBP-30338: Could not delete the View Permission." + val UserEmailAddressMissing = "OBP-30339: User does not have an email address set. Cannot send test email." + val EmailSendingFailed = "OBP-30340: Failed to send email. Check the server logs and SMTP configuration." + val SmtpAuthenticationFailed = "OBP-30341: SMTP authentication failed. Check mail.smtp.user and mail.smtp.password." + val SmtpConnectionFailed = "OBP-30342: Could not connect to SMTP server. Check mail.smtp.host, mail.smtp.port and network reachability." + val SmtpTlsHandshakeFailed = "OBP-30343: TLS handshake with SMTP server failed. Check mail.smtp.starttls.enable, mail.smtp.ssl.enable and mail.smtp.ssl.protocols." + val SmtpRecipientRejected = "OBP-30344: SMTP server rejected the recipient or message. The From address may be unauthorised, the recipient may be invalid, or the message may have failed policy/anti-spam checks." + val SmtpProtocolError = "OBP-30345: SMTP protocol error from the mail server." + // Branch related messages val BranchesNotFoundLicense = "OBP-32001: No branches available. License may not be set." val BranchesNotFound = "OBP-32002: No branches available." diff --git a/obp-api/src/main/scala/code/api/util/http4s/StatusPage.scala b/obp-api/src/main/scala/code/api/util/http4s/StatusPage.scala index 0f5c8037b7..af28d2a8f0 100644 --- a/obp-api/src/main/scala/code/api/util/http4s/StatusPage.scala +++ b/obp-api/src/main/scala/code/api/util/http4s/StatusPage.scala @@ -4,7 +4,7 @@ import java.lang.management.ManagementFactory import cats.effect.IO import code.api.cache.Redis -import code.api.util.DoobieUtil +import code.api.util.{APIUtil, DoobieUtil} import code.util.Helper.MdcLoggable import doobie._ import doobie.implicits._ @@ -39,10 +39,78 @@ object StatusPage extends MdcLoggable { } } - private case class Checks(database: String, redis: String) { + private case class EmailChecks( + config: String, // "ok" / "warn" + configDetail: String, // human-readable reason when "warn", empty otherwise + smtp: String, // "ok" / "fail" + smtpDetail: String // host:port that was probed, plus error message on fail + ) + + private case class Checks(database: String, redis: String, email: EmailChecks) { + // Email is a soft dependency — config/smtp warnings do NOT flip overall to 503. def allOk: Boolean = database == "ok" && redis == "ok" } + // SMTP probe is the only expensive bit. Cache it for 60s so frequent /status + // pollers (Prometheus blackbox, k8s probes) don't hammer the mail server with + // a TCP connect on every request. Config checks are essentially free so we + // recompute them each time. + private val SmtpCacheTtlMillis = 60000L + private val smtpCache: java.util.concurrent.atomic.AtomicReference[(Long, String, String)] = + new java.util.concurrent.atomic.AtomicReference((0L, "", "")) + + private def probeSmtp(): (String, String) = { + val host = APIUtil.getPropsValue("mail.smtp.host", "localhost") + val port = APIUtil.getPropsAsIntValue("mail.smtp.port", 25) + val target = s"$host:$port" + try { + val socket = new java.net.Socket() + try { + socket.connect(new java.net.InetSocketAddress(host, port), 2000) + socket.setSoTimeout(2000) + val in = new java.io.BufferedReader(new java.io.InputStreamReader(socket.getInputStream)) + val banner = Option(in.readLine()).getOrElse("") + if (banner.startsWith("220")) "ok" -> s"$target (banner: ${banner.take(80)})" + else "fail" -> s"$target — unexpected greeting: ${banner.take(80)}" + } finally socket.close() + } catch { + case e: Throwable => + logger.warn(s"StatusPage says: smtp check to $target failed: ${e.getMessage}") + "fail" -> s"$target — ${e.getClass.getSimpleName}: ${e.getMessage}" + } + } + + private def cachedSmtpCheck(): (String, String) = { + val now = System.currentTimeMillis() + val cached = smtpCache.get() + if (now - cached._1 < SmtpCacheTtlMillis && cached._2.nonEmpty) { + (cached._2, cached._3) + } else { + val (status, detail) = probeSmtp() + smtpCache.set((now, status, detail)) + (status, detail) + } + } + + private def runEmailChecks: EmailChecks = { + val portalUrl = APIUtil.getPropsValue("portal_external_url") + val sender = APIUtil.getPropsValue("mail.users.userinfo.sender.address", "noreply@example.com") + val portalMissing = portalUrl.isEmpty || portalUrl.exists(_.trim.isEmpty) + val senderIsDefault = sender == "noreply@example.com" + val (configStatus, configDetail) = + if (portalMissing && senderIsDefault) + "warn" -> "portal_external_url not set; mail.users.userinfo.sender.address is default 'noreply@example.com'" + else if (portalMissing) + "warn" -> "portal_external_url not set — validation/reset emails will be silently skipped" + else if (senderIsDefault) + "warn" -> "mail.users.userinfo.sender.address is default 'noreply@example.com' — most SMTP servers will reject it" + else + "ok" -> "" + + val (smtpStatus, smtpDetail) = cachedSmtpCheck() + EmailChecks(configStatus, configDetail, smtpStatus, smtpDetail) + } + private def runChecks: IO[Checks] = IO { val db = try { DoobieUtil.runQuery(sql"SELECT 1".query[Int].unique) @@ -59,7 +127,12 @@ object StatusPage extends MdcLoggable { logger.warn(s"StatusPage says: redis check failed: ${e.getMessage}") "fail" } - Checks(db, redis) + val email = try runEmailChecks catch { + case e: Throwable => + logger.warn(s"StatusPage says: email checks errored: ${e.getMessage}") + EmailChecks("warn", s"email check error: ${e.getMessage}", "fail", "") + } + Checks(db, redis, email) } val routes: HttpRoutes[IO] = HttpRoutes.of[IO] { @@ -76,6 +149,9 @@ object StatusPage extends MdcLoggable { Ok("""{"status":"ok"}""").map(_.withContentType(`Content-Type`(MediaType.application.json, Charset.`UTF-8`))) } + private def jsonEscape(s: String): String = + s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n").replace("\r", "\\r").replace("\t", "\\t") + private def jsonResponse(checks: Checks): IO[Response[IO]] = { val status = if (checks.allOk) "ok" else "degraded" val json = @@ -86,19 +162,34 @@ object StatusPage extends MdcLoggable { | "uptime_seconds": $uptimeSeconds, | "checks": { | "database": "${checks.database}", - | "redis": "${checks.redis}" + | "redis": "${checks.redis}", + | "email": { + | "config": "${checks.email.config}", + | "config_detail": "${jsonEscape(checks.email.configDetail)}", + | "smtp": "${checks.email.smtp}", + | "smtp_detail": "${jsonEscape(checks.email.smtpDetail)}" + | } | } |}""".stripMargin Ok(json).map(_.withContentType(`Content-Type`(MediaType.application.json, Charset.`UTF-8`))) } + private def htmlEscape(s: String): String = + s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\"", """) + private def htmlResponse(checks: Checks): IO[Response[IO]] = { val overall = if (checks.allOk) "ok" else "degraded" val overallColor = if (checks.allOk) "#2e7d32" else "#c62828" def badge(v: String): String = { - val color = if (v == "ok") "#2e7d32" else "#c62828" + val color = v match { + case "ok" => "#2e7d32" + case "warn" => "#ef6c00" + case _ => "#c62828" + } s"""$v""" } + def detailCell(detail: String): String = + if (detail.isEmpty) "" else s"""
| config | ${badge(checks.email.config)}${detailCell(checks.email.configDetail)} |
|---|---|
| smtp | ${badge(checks.email.smtp)}${detailCell(checks.email.smtpDetail)} |