# Initiative Brief: Complete the OutShine Notification Runtime and External Event Ingestion ## Desired outcome Complete the existing OutShine notification subsystem so that notification-worthy events reliably flow from creation through routing and delivery, and expose a safe service-to-service ingestion contract that other Xyence services can use. This initiative should **finish and harden the notification architecture already present in OutShine rather than replace it**. The immediate cross-service consumer is `Xyence/orchestrator#355`, which needs to raise operator-facing notifications for events such as provider usage limits, unrecoverable work-item failures, human-required merge conflicts, and stalled runs. ## Current state - OutShine already contains a substantial notification foundation: - durable `NotificationEvent` records; - declarative notification routes; - delivery records and retries; - correlation-key deduplication; - payload redaction; - message templates; - email-provider abstraction; - rate-limiting seams; - administrative APIs for inspecting events, routes, deliveries, and templates; - Console UI for operational notification management; - notification emission from existing domains such as Work Items. The architecture is directionally correct: **event → route resolution → delivery → provider** OutShine should remain the canonical owner of this pipeline. However, the subsystem is incomplete in several important ways. Event creation does not itself complete notification delivery — `create_notification_event()` persists a durable event, but routing and delivery happen separately through `process_notification_event()`. Several internal callers appear to emit events without clearly ensuring they are subsequently processed. An external ingestion endpoint that merely calls `create_notification_event()` would therefore accept an event successfully while potentially leaving it indefinitely in `pending`. There is no service-to-service notification ingestion API — Current notification APIs are primarily administrative/read/operational APIs. External services cannot safely raise a platform notification through OutShine. This is the direct blocker for `Xyence/orchestrator#355`. Notification preferences appear only partially implemented — The schema and documentation describe per-user notification preferences, but the current routing path does not appear to consistently enforce them. The initiative should reconcile the implementation with the intended contract rather than leaving a feature that exists in schema and documentation but not reliably in behavior. Multi-channel support is mostly a future seam — The data model anticipates channels beyond email, but email is currently the only meaningful outbound provider. Unsupported channels are intentionally suppressed. This initiative should preserve that extensibility without expanding scope into implementing SMS, chat, push, or other channels. ## Rejected / deferred alternatives - replacing the notification architecture; - implementing a notification subsystem inside Orchestrator; - SMS delivery; - chat delivery; - mobile push; - browser push; - generalized webhook delivery unless already present and trivially supported; - introducing Kafka, SQS, Redis queues, Celery, or another worker architecture solely for notifications; - building a full consumer-facing notification inbox; - broad UI redesign of the Console Notifications area; - notification of every routine platform event; - workflow authoring redesign. ## Repository scope - In Scope - canonical raise-and-process notification service operation; - service-authenticated external event ingestion; - mandatory/strong correlation-key deduplication for service producers; - redaction and payload validation; - synchronous route/delivery processing; - existing producer audit and migration; - Work Item notification-path verification/fix; - notification preference completion or explicit reconciliation; - route/target contract reconciliation; - operational logging and tests; - documentation of the external producer contract; - support necessary for `Xyence/orchestrator#355`. ## Expected behavior - [ ] OutShine has one canonical service operation for raising a notification-worthy event through persistence, routing, and supported delivery. - [ ] Existing code can still create a durable event without processing only where that lower-level behavior is explicitly required. - [ ] `POST /api/notifications/events` or the equivalent service API accepts authenticated events from trusted Xyence services. - [ ] Service-ingested events require or strongly enforce deterministic correlation-key deduplication. - [ ] The ingestion path validates payload size/shape and redacts prohibited sensitive data before persistence. - [ ] Successfully ingested events are processed through routing and delivery rather than being left indefinitely `pending`. - [ ] Email remains the supported outbound channel; unsupported channels remain explicit, observable suppressed seams. - [ ] Notification provider failure preserves the durable event and delivery failure state. - [ ] Producer operations remain unaffected by notification failure. - [ ] Existing OutShine notification-event producers are inventoried. - [ ] Existing callers that intend actual notification delivery use the canonical processed path. - [ ] Work Item notifications are verified to reach the routing/delivery pipeline. - [ ] `NotificationPreference` behavior is either fully enforced according to a documented contract or explicitly removed from claims of current runtime behavior. - [ ] Route target types are consistent across schema, models, service validation/resolution, APIs, and documentation. - [ ] Console operators can inspect accepted events, routing outcomes, suppressed deliveries, failed deliveries, and retries. - [ ] Tests cover event creation, processing, dedupe, redaction, route resolution, preference behavior, provider failure, retry, service authentication, malformed payloads, abuse/flood protection, and fail-open producer behavior. - [ ] Documentation defines the service producer contract and includes a safe example payload. - [ ] The resulting API satisfies the OutShine dependency in `Xyence/orchestrator#355`.