#Customer Responsibilities for Operating a Resilient Application on OSC new

#Purpose and scope

This document defines the minimum customer responsibilities and recommended engineering practices customers must follow when designing, deploying, and operating resilient production and business-critical applications on Open Sovereign Cloud (OSC). It applies to workloads running on OSC Kubernetes clusters and using OSC networking and storage services.

OSC provides resilient platform building blocks, including multiple Availability Zones (AZs), managed Kubernetes capabilities, storage services, and network services. End-to-end application resilience remains a shared responsibility. Unless explicitly included in a managed service agreement, the customer is responsible for:

• Application architecture and workload design
• Capacity management
• Backup and restore
• Disaster recovery
• Data lifecycle and housekeeping
• Application-level security

Most obligations in this document are not unique to OSC. They reflect standard cloud-native and Kubernetes engineering responsibilities that customers must meet on any managed Kubernetes platform when they operate a resilient application. OSC-specific requirements mainly arise from the applicable OSC service description, supported-version matrix, upgrade windows, and OSC network or storage service characteristics.

#Key message

1. Most of these obligations are standard Kubernetes and cloud-native operating requirements, not OSC-specific exceptions.
2. Design the workload so that it can survive the failure of a full AZ - not only for normal operation asuming components never fail or restart.
3. Keep workloads, manifests, and integrations within the versions and API levels supported by the applicable OSC service description. Unsupported components become both stability and security risks.
4. Keep enough spare capacity to absorb failover and maintenance.
5. Expand and clean up storage before it becomes critical.
6. Minimize hard runtime dependencies on OSC/Kubernetes APIs and shared services.
7. Plan network CIDRs and NAT behavior up front so connectivity, peering, VPNs, and egress scale do not fail by design.
8. Prove resilience through regular testing, restore exercises and documented runbooks.

#Shared responsibility model

#Mandatory customer responsibilities

#Design for failure domains

• Critical production applications MUST be designed so that loss of one AZ does not create a full service outage.
• Customers MUST distribute replicas, traffic, and data across failure domains using replicas, topology spread constraints, pod anti-affinity, PodDisruptionBudgets, and multi-AZ replication where applicable.
• Customers MUST define degraded-mode behavior for AZ loss, node loss, and temporary network interruptions.
• Production and non-production MUST NOT share the same hardware pool.
• For clusters that are expected to survive AZ loss while maintaining spreading flexibility, a practical starting point is 6 worker nodes (2 per AZ). Smaller footprints require explicit review because they often cannot maintain spread, disruption budgets, and headroom simultaneously.
• Applications MUST tolerate at least 60 seconds of transient network interruption or cluster networking restart (for example Cilium restart or reconvergence) without uncontrolled cascading failure.

#Capacity and utilization management

• Customers MUST maintain a rolling capacity plan for compute, storage, and critical stateful components. Forecasts SHOULD cover at least the next 90 days and be reviewed regularly for production workloads.
• Capacity MUST be modeled against failure scenarios, not only against normal-state averages. The relevant metric is requested or reserved capacity plus expected failover load.
• Customers MUST maintain sufficient spare capacity to absorb the loss of the largest failure domain and at least one additional node (N+1) where relevant.
• All production workloads MUST define CPU and memory requests. Limits MUST be defined or explicitly justified based on workload behavior and performance testing.
• Customers MUST configure priority classes for business-critical workloads.
• Memory overcommit MUST NOT be used for critical production components. CPU overcommit above 4:1 is not acceptable; jitter-sensitive or demanding workloads SHOULD remain at or below 2:1.
• Prefer small and medium instances and horizontal scale over very large single nodes or single large VMs.
• Relying on ad hoc capacity procurement during an incident is NOT an acceptable resilience strategy.

#Storage capacity, performance, and expansion

• Customers MUST monitor storage used capacity, growth rate, latency, IOPS, queue depth, replication lag where applicable, and the load generated by backups and housekeeping jobs.
• Customers MUST implement housekeeping, retention, archival, and cleanup for databases, filesystems, and object storage. Keep primary production datasets as small as practical.
• Customers MUST initiate expansion or cleanup before storage becomes constrained. Waiting until a volume or bucket is almost full is not acceptable operational practice.
• Customers MUST correct workload patterns that create persistent maximum storage load or inefficient I/O, such as small-block single-thread I/O when the application could use caching, larger blocks, or parallel I/O.
• For high-volume temporary or session-like data, customers SHOULD use caches or memory-optimized stores where compatible with the application design instead of pushing avoidable load into primary transactional databases.
• Object storage used for backup MUST have retention and cleanup policies. Object storage is not a substitute for high-IO transactional storage and should not be used for very large populations of tiny files.
• If the customer deploys unmanaged RWX/shared storage or a customer-owned Ceph solution inside the cluster, the customer owns the design, operation, and support of that solution.

#Reduce dependencies on APIs and shared services

• Customers MUST minimize hard runtime dependency on OSC APIs, the Kubernetes API, DNS, identity providers, and other shared control-plane services.
• No critical business transaction path SHOULD require synchronous access to the Kubernetes API unless that dependency is explicitly justified, tested, and protected by fallback behavior.
• Health probes MUST target the local application endpoint and MUST NOT depend on DNS, public FQDNs, or other application components. Probe configuration must reflect measured startup and failure behavior.
• API clients MUST use connection reuse, exponential backoff with jitter, sensible circuit breakers, and realistic timeouts. Very small 1-5 second timeouts are not acceptable for critical platform interactions; a default around 60 seconds is a better starting point unless testing justifies something else.
• Large data objects and high-churn application state MUST NOT be stored in Kubernetes API or etcd. If the application requires etcd-like semantics or high request volume, the customer SHOULD run a dedicated application-owned data store inside the cluster.
• Customers SHOULD avoid broad cross-cluster peering or tightly coupled network meshes. Prefer namespace isolation and expose only the few endpoints that truly require cross-cluster reachability.

#Platform lifecycle and deprecated version compliance

• Customers MUST comply with supported versions, upgrade windows, and deprecation rules. Unsupported components become both stability and security risks.
• Workloads MUST be kept compatible with supported Kubernetes versions, supported Garden Linux images, supported add-ons, and supported API versions.
• Customers MUST remediate use of deprecated or removed Kubernetes APIs, controllers, or integration patterns before the applicable platform upgrade deadline.
• Customer-managed components inside the cluster - such as operators, ingress controllers, service meshes, CSI drivers, admission webhooks, sidecars, or application SDKs - remain the customer's responsibility unless expressly covered by a managed service agreement.

#Network architecture and IP address planning

• Customers MUST own end-to-end IP planning for cluster node CIDRs, pod and service networks where relevant, remote on-premises networks, partner networks, and other cloud environments that may later be connected through VPNs, peerings, or transit routing.
• Address ranges that are intended to communicate with each other MUST NOT overlap. Overlapping CIDRs can break routing, peering, or VPN connectivity and often require disruptive remediation or avoidable NAT workarounds.
• Customers MUST validate IP overlap not only against today's connected networks but also against reasonably planned future connectivity, additional clusters, and disaster recovery environments.
• When cross-network connectivity is business-critical, customers SHOULD maintain an IP address management process and an approved addressing plan before provisioning new clusters or remote connections.

#Network sizing and NAT gateway planning

• Customers MUST size node networks for both dimensions of the problem: enough addresses for projected peak node growth and enough NAT port capacity for the expected egress pattern, especially persistent connections to the same destination endpoints.
• Customers MUST understand that broader node networks provide more IP addresses but fewer NAT ports per network interface, while narrower node networks provide more NAT ports per interface but less room for node growth. This trade-off must be evaluated during design, not after saturation occurs.
• Customers MUST NOT rely on default NAT gateway settings. NAT gateway sizing, ports per interface, and network sizes MUST be validated against the expected number of nodes, load balancers, connection fan-out, and long-lived outbound connections.
• If the application depends on high outbound connection counts or many persistent connections to a small set of external endpoints, customers MUST explicitly review NAT gateway port sizing as part of capacity management.

#Kubernetes workload hardening

• Use startup, readiness, and liveness probes that reflect actual application health and are tuned based on measured startup time and recovery characteristics.
• Use replicas, PodDisruptionBudgets, topology spread, anti-affinity, and autoscaling where appropriate for the workload and business target.
• Critical components MUST NOT be singletons unless the business owner explicitly accepts the resulting single point of failure.
• Customers MUST validate that upgrades or restarts of pods, nodes, and network components do not trigger chain reactions across the application landscape.
• Dependencies to identity providers, external databases, object storage, or external APIs MUST be explicitly documented and protected with retries, caching, queues, or degraded-mode behavior.

#Data protection, backup, and disaster recovery

• Customers MUST ensure compliance with relevant laws and regulations, such as data sovereignty and privacy. Check if data is personal or regulated; ensure allowed processing; sign data-processing agreements; and follow industry rules. Sensitive data must be protected through encryption and access controls.
• Customers MUST keep at least one backup copy outside the primary OSC storage domain and preferably outside the primary OSC environment for business-critical services.
• Restore testing MUST be performed regularly. The default expectation is quarterly restore validation and annual end-to-end disaster recovery rehearsal, plus additional validation after major architecture changes.
• Backup strategy SHOULD combine full and incremental backups where supported and should be aligned with business RPO and RTO targets.
• Backups MUST be encrypted, monitored, and covered by alerting and runbooks.
• Replication does NOT replace backup.
• Backup frequency MUST be balanced against storage load and operational impact; more backups are not automatically better if they jeopardize production stability.
• The customer owns and operates its Business Continuity Management framework and Business Continuity Plan, including disaster recovery planning, backup and restore procedures, recovery testing, escalation, and decision‑making authority; OSC does not assume responsibility for customer-specific business continuity management.

#Security

• Security is not separate from resilience. Security failures frequently result in service outages, forced shutdowns, or data loss. Customers MUST design, deploy, and operate workloads so that security incidents can be prevented, detected, contained, and recovered without destabilizing the platform or other tenants.
• Customers own and protect all their tenant credentials (passwords, keys, tokens, certificates). Least‑privilege access, regular rotation, and immediate revocation after suspected compromise are mandatory.
• Customers are responsible for application‑level security controls, including authentication, authorization, and endpoint exposure.
• Customers MUST ensure workloads do not generate abusive or amplified traffic. Misconfigurations that could destabilize the platform or impact others are the customer’s responsibility. OSC may immediately deactivate workloads that threaten platform stability.
• Customers MUST comply with supported versions, upgrade windows, and deprecation rules. Unsupported components become both stability and security risks.

#Testing, evidence, and operational governance

• Before go-live, customers MUST complete end-to-end resilience tests aligned with OSC failure scenarios, including AZ failure survival, node maintenance, transient networking interruption, backup restore, version-upgrade compatibility, IP overlap validation for required connectivity paths, and dependency outage handling.
• Customers MUST maintain runbooks, escalation contacts, ownership information, and a clear responsibility and communication matrix.
• Exceptions to these requirements MUST be documented, risk-assessed, and approved by the customer service owner and the OSC counterpart before go-live.
• Evidence of compliance SHOULD be retained and updated after major changes, scale events, or architecture changes.

#Operational thresholds and trigger points

#Go-live quality gates

#Exceptions and risk acceptance

Any deviation from these requirements must be documented, risk-assessed, and approved by the customer service owner and the OSC counterpart before go-live. Accepted exceptions should define compensating controls, expected business impact, and a review date. Exceptions should not be treated as permanent defaults.