Added Geo Disaster Recovery documentation.

doc/gitlab-geo/README.md

@@ -12,6 +12,7 @@ locations as a read-only fully operational version.
     - [Database Replication](./database.md)
     - [Configuration](./configuration.md)
 - [Current limitations](#current-limitations)
+- [Disaster Recovery](./disaster-recovery.md)
 - [Frequently Asked Questions](#frequently-asked-questions)
     - [Can I use Geo in a disaster recovery situation?](#can-i-use-geo-in-a-disaster-recovery-situation)
     - [What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)
@@ -77,6 +78,9 @@ There are limitations to what we replicate (see Current limitations).
 In an extreme data-loss situation you can make a secondary Geo into your
 primary, but this is not officially supported yet.
 
+If you still want to proceed, see our step-by-step instructions on how to 
+manually [promote a secondary node](./disaster-recovery.md) into primary. 
+
 ### What data is replicated to a secondary node?
 
 We currently replicate project repositories and the whole database. This

doc/gitlab-geo/disaster-recovery.md

+# GitLab Geo Disaster Recovery
+
+> **Note:**
+This is not officially supported yet, please don't use as your only 
+Disaster Recovery strategy as you may loose data.
+
+GitLab Geo replicates your database and your Git repositories. We will
+support and replicate more data in the future, that will enable you to
+fail-over with minimal effort, in a disaster situation. 
+
+See [current limitations](./README.md#current-limitations) 
+for more information.
+
+
+## Promoting a secondary node
+
+We don't provide yet an automated way to promote a node and do fail-over,
+but you can do it manually if you have `root` access to the machine.
+
+You must make the changes in the exact specific order:
+
+1. Take down your primary node (or make sure it will not go up during this 
+   process or you may loose data)
+2. Wait for any database replication to finish
+3. Promote the Postgres in your secondary node as primary
+4. Log-in to your secondary node with a user with `sudo` permission
+5. Open the interactive rails console: `sudo gitlab-rails console` and execute:
+    1. `Gitlab::Geo.primary_node.update(primary: false)`
+    2. List your secondary nodes and note down the id of the one you want to 
+       promote: `Gitlab::Geo.secondary_nodes`
+    3. To promote a node with id `2` execute: 
+       `GeoNode.find(2).update(primary: true)`
+    4. To exit the interactive console, type: `exit`
+6. Rsync everything in `/var/opt/gitlab/gitlab-rails/uploads` and 
+   `/var/opt/gitlab/gitlab-rails/shared` from your old node to the new one.