Disaster Recovery Policy
On this page
This policy defines Tanqory’s disaster recovery (DR) framework and procedures for restoring services after major disruptions, ensuring safe and timely recovery in line with law, contract, and international standards. All personnel and critical vendors must read this with the Business Continuity Plan (BCP), Incident Response Plan (IRP), Backup Policy, Network Security Policy, and Third-Party Risk Management (TPRM). If documents conflict, the more specific/recent document or stricter law prevails.
1) Purpose and Scope
Objectives: minimize downtime, limit data loss, meet contracted RTO/RPO, and produce auditable evidence. Scope: all cloud environments (Production/Staging/Sandbox), networks, applications, databases, storage, CI/CD, work devices used for operations, and vendor-operated services. Applies to all data classifications, especially personal/payment data with statutory rules. Triggers include natural disasters, power/data center failures, human error, cyberattacks causing loss of service, and critical third-party outages.
2) Governance, Roles, and Decision Authority
- Disaster Recovery Manager (typically CISO or delegate) owns the policy and may activate DR/failover per runbook criteria.
- Command structure aligned to BCP/IRP: Incident Commander declares and leads; Service Owners/Engineering lead technical recovery; Infra/DB Leads handle storage/databases; Communications Lead manages customer/partner comms; Legal/DPO handles legal/regulatory notifications; Vendor Manager coordinates critical suppliers.
- All decisions and timestamps must be logged (audit trail) for post-event review and improvement.
3) Service Tiers, RTO/RPO
Prioritize recovery via tiering with documented RTO/RPO per runbook/CMDB:
- Tier 1 (Critical): Payments/Checkout/Auth/Risk — target RTO ≤ 15 minutes, near-zero RPO (sync/near-sync replication).
- Tier 2 (High): Merchant Dashboard/Order Management — RTO ≤ 1 hour, RPO ≤ 5 minutes.
- Tier 3 (Medium): Notifications/Support/Logs — RTO ≤ 4 hours, RPO ≤ 1 hour.
- Tier 4 (Low): Analytics/Internal Tools — RTO ≤ 24 hours, RPO ≤ 4 hours.
Tier changes require DR Manager approval and CMDB updates.
4) DR Architecture and Multi-Region Strategy
- Multi-region design using Active/Passive or Active/Active as appropriate. Primary region must honor data residency (e.g., SG or EU); at least one DR region maintains encrypted, tested replicas.
- Global load balancer/traffic manager with health checks and automated failover.
- Classify sites as Hot/Warm/Cold with capacity ramp-up times defined.
- All data in transit/at rest encrypted; adherence to Encryption/Network Security Policy.
- Respect contractual data residency during failover; document allowable regions in runbooks.
5) Backup and Replication
- Meet RPO via continuous/streaming or near-sync replication for transactions/configuration; snapshots/incrementals (AES-256) for secondary data in permitted regions only.
- Secrets/keys backed up separately with regional separation; test key recovery on the same cadence as data restores.
- Verify integrity via hash/checksum; perform restore tests per Appendix A cadence.
- No unencrypted backups; use secure channels only. Retention/rotation per Backup Policy.
6) Declaration Criteria, Failover, and Failback
- Activate DR when: region/DC outage, SLA breach for critical services, security events requiring isolation, or executive/legal mandate.
- DR runbook steps: assess data/state; select failover target (Hot/Warm); shift traffic via DNS/LB; validate integrity/functionality; communicate stakeholders.
- Failback only after primary is secure and tested; manage data divergence risk with clear cutover window and rollback steps.
- Record MTTD/MTTFo/MTTR and achieved RTO/RPO for every event.
7) Communications and Stakeholder Notifications
- Use durable channels (email, dashboard, status page) and logged emergency paths.
- Notify customers/partners when service is down or data is affected per contract: what happened, actions taken, customer actions, recovery ETA.
- If personal data is involved, coordinate with DPO/Legal per IRP and law (e.g., GDPR 72h, PDPA 3 days).
- Notify executives/board per P1/P2 thresholds; log messages and timestamps.
8) Contingency Plans for Core Systems and Vendors
- Maintain system-specific runbooks (Payment Gateway, DNS, CDN, Cloud, DB, Queue, CI/CD) with emergency contacts, console/break-glass access (MFA, logging), and temporary workarounds.
- If a critical vendor fails, invoke assessed alternates per TPRM; enforce vendor SLAs and escalation paths.
- Document dependency mappings and minimum viable service (MVS) modes if alternates are degraded.
9) Testing, Exercises, and Verification
- Cadence: tabletop at least twice yearly; simulation/partial failover at least annually; full/regional failover at least annually for Tiers 1–2.
- Regular restore tests; compare actual vs target RTO/RPO.
- Each test needs an approver, issues/technical debt logged, action items with owners/due dates; rerun if targets are missed.
- Include comms drills, DNS/LB cutover tests, backup key recovery, and data integrity checks.
10) Metrics, Logging, and Continual Improvement
- Track: achieved RTO/RPO, Mean Time to Failover (MTTFo), Mean Time to Recover (MTTR), test success rate, post-test defects, DR events per year.
- Quarterly reporting to leadership; annual BCP/DR review.
- Each event requires Incident/DR report, post-incident review, and decision evidence.
- Retain records ≥3–5 years or longer per contract/law.
11) Change Management and Exceptions
- DR architecture/tier/RTO/RPO/region changes must follow Change Management and be approved by DR Manager/exec, with impact assessment (data residency/contract).
- Exceptions (e.g., weaker region/protocol) need justification, sunset date, and compensating controls; review quarterly.
12) Legal and Standards Alignment
Aligned with ISO 22301 (BCM), ISO/IEC 27001/27002 Annex A (continuity), NIST SP 800-34, SOC 2 (CC7), PCI DSS 4.0 (Req.12), GDPR/PDPA. Add mappings/runbooks for sectoral/local rules as applicable (e.g., MAS TRM, DORA/NIS2, HIPAA/FedRAMP, FISC).
Appendix A: Tier, RTO/RPO, and Testing Cadence (Example)
- Tier 1 (Critical): Payments/Checkout/Auth/Risk — RTO ≤ 15 mins; near-zero RPO; full failover ≥1/year; tabletop twice/year; restore tests monthly.
- Tier 2 (High): Merchant Dashboard/Order Management — RTO ≤ 1 hr; RPO ≤ 5 mins; simulation/partial ≥1/year; tabletop twice/year; restore tests monthly.
- Tier 3 (Medium): Notifications/Support/Logs — RTO ≤ 4 hrs; RPO ≤ 1 hr; tabletop ≥1/year; restore tests quarterly.
- Tier 4 (Low): Analytics/Internal Tools — RTO ≤ 24 hrs; RPO ≤ 4 hrs; tabletop ≥1/year; restore tests semiannual.
- Log results, approvers, and action items when targets are missed.
Appendix B: Contacts and Communication (Example)
- DR/Incident Hotline (On-call): [insert live number/channel]
- Security/Incident: security@tanqory.com / incident@tanqory.com
- DPO/Legal: dpo@tanqory.com / legal@tanqory.com
- Status page (if applicable): [URL]
- Critical vendors (Cloud/DNS/CDN/Payment): emergency contacts and SLAs per contract (attach internal table).
- Use recorded channels; avoid personal/unlogged channels except designated emergency phone lines.
- Data residency per customer cohort:
- SG/ASEAN: primary SG; failover only to contract-allowed regions (e.g., SG → SG/JP).
- EU/EEA/UK: primary/DR within EU/UK; no failover outside if contractually restricted.
- US (no special restriction): primary/DR in US unless contract says otherwise.
- Do not violate residency; if cross-region failover is unavoidable, obtain written consent where legally permissible.
Appendix C: Glossary
- BCP: Business Continuity Plan
- DR: Disaster Recovery
- RTO/RPO: Recovery Time/Point Objective
- MTTFo: Mean Time to Failover
- MTTR: Mean Time to Recover
- H/A: High Availability
- DNS: Domain Name System
- WAF: Web Application Firewall
- CDN: Content Delivery Network
- KMS: Key Management Service
- SCC: Standard Contractual Clauses
- SLA: Service Level Agreement
- P1/P2/P3/P4: Severity levels (Critical/High/Medium/Low)
- Tabletop/Simulation/Full Failover: Exercise types by rigor