Merge pull request #6813 from Ocelot-Social-Community/6812-script-auto-backups-for-kubernetes-servers

chore(other): add deployment script 'clusters.cron-backups.sh'

.gitignore

@@ -8,6 +8,7 @@ yarn-error.log*
 .yarn-integrity
 .eslintcache
 kubeconfig.yaml
+backup-cron-job.log
 
 node_modules/
 cypress/videos

deployment/.env.dist

@@ -1 +1,7 @@
-CONFIGURATION=example
+# branding folder used for "docker compose up" run in deployment folder
+CONFIGURATION=stage.ocelot.social
+
+# used in "scripts/clusters.backup-multiple-servers.sh"
+BACKUP_CONFIGURATIONS="stage.ocelot.social stage.wir.social"
+# if '<= 0' no backups will be deleted
+BACKUP_SAVED_BACKUPS_NUMBER=7

deployment/configurations/stage.ocelot.social

@@ -1 +1 @@
-Subproject commit a930f11d8f2d0c11136ff0bf9188aaab9cabc5d1
+Subproject commit fdc2e52fa444b300e1c4736600bc0e9ae3314222

deployment/deployment.md

@@ -22,15 +22,15 @@ After the first deployment of the new network on your server, the database is in
 
 ***ATTENTION:*** When you are logged in for the first time, please change your (the admin's) e-mail to an existing one and change your password to a secure one !!!
 
-## Use the Scripts
+## Using the Scripts
 
-To use all the scripts you have to set the variable `CONFIGURATION` in your terminal by entering:
+To use most of the scripts you have to set the variable `CONFIGURATION` in your terminal by entering:
 
 ```bash
 # in deployment folder
 
 # set configuration name to folder name in 'configurations' folder (network name)
-$ export CONFIGURATION=`<your-configuration-name>`
+$ export CONFIGURATION=<your-configuration-name>
 # to check this
 $ echo $CONFIGURATION
 ```
@@ -64,9 +64,13 @@ $ scripts/cluster.maintenance.sh on
 $ scripts/cluster.maintenance.sh off
 ```
 
-### Backup Script
+### Backup Scripts
 
-To save a locale backup of the database and uploaded images:
+Save backups.
+
+#### Single Backup
+
+To save a local backup of the database and uploaded images:
 
 ```bash
 # in deployment folder
@@ -76,3 +80,58 @@ $ scripts/cluster.backup.sh
 ```
 
 The backup will be saved into your network folders `backup` folder in a new folder with the date and time.
+
+#### Multiple Networks Backup
+
+In order to save several network backups locally, you must define the configuration names of all networks in `.env`. The template for this is `deployment/.env.dist`:
+
+```bash
+# in the deployment folders '.env' set as example
+BACKUP_CONFIGURATIONS="stage.ocelot.social stage.wir.social"
+BACKUP_SAVED_BACKUPS_NUMBER=7
+```
+
+If `BACKUP_SAVED_BACKUPS_NUMBER <= 0` then no backups will be deleted.
+
+To actually save all the backups run:
+
+```bash
+# in deployment folder
+
+# save all backups listed in 'BACKUP_CONFIGURATIONS'
+# delete all backups older then the 'BACKUP_SAVED_BACKUPS_NUMBER' newest ones
+$ scripts/clusters.backup-multiple-servers.sh
+```
+
+The backups will be saved into your networks folders `backup` folder in a new folder with the date and time.
+
+#### Automated Backups
+
+⚠️ *Attention: Please check carefully whether really the oldest backups have been deleted. As shells on different systems behave differently with regard to the commands used in this script.*
+
+Install automated backups by a [cron job](https://en.wikipedia.org/wiki/Cron).
+Be aware of having the bash shell installed to run the script.
+The environment variables for the automated backups are described above.
+
+Installing a cron job by editing the cron table file:
+
+```bash
+# edit cron job table
+$ crontab -e
+```
+
+In the editor add the line:
+
+```bash
+# in cron job table file
+
+# set a cron job every night at 04am server time
+# min   hour    day     month   weekday command
+00      04      *       *       *       /root/Ocelot-Social/deployment/scripts/clusters.backup-multiple-servers.sh >> /root/Ocelot-Social/deployment/backup-cron-job.log
+```
+
+This way the terminal output is written into a log file named `backup-cron-job.log` located in the deployment folder.
+
+Be aware that the server datetime can differ from your local time.
+Especially by the change between summer and winter time, because servers usually have UTC.
+Find out the actual difference by running the command `date` on your server.

deployment/docker-compose.yml

@@ -168,6 +168,9 @@ services:
       # Uncomment following line for Neo4j Enterprise version instead of Community version
       # TODO: clarify if that is the only thing needed to unlock the Enterprise version
       # - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
+      # set the name of the database to be used
+      # - NEO4J_dbms_default__database=graph.db
+      # - NEO4J_dbms_default__database=neo4j
       # TODO: Remove the playground from production
     # bring the database in offline mode to export or load dumps
     # command: ["tail", "-f", "/dev/null"]

deployment/scripts/cluster.backup.sh

@@ -1,34 +1,44 @@
 #!/bin/bash
 
+# time stamp
+printf "Backup started at:\n  "
+date
+
 # base setup
 SCRIPT_PATH=$(realpath $0)
 SCRIPT_DIR=$(dirname $SCRIPT_PATH)
 
 # check CONFIGURATION
-if [ -z ${CONFIGURATION} ]; then
-  echo "You must provide a `CONFIGURATION` via environment variable"
+if [[ -z "$CONFIGURATION" ]]; then
+  echo "!!! You must provide a CONFIGURATION via environment variable !!!"
   exit 1
 fi
 
+printf "  Cluster:  %s\n" $CONFIGURATION
+
 # configuration
 KUBECONFIG=${KUBECONFIG:-${SCRIPT_DIR}/../configurations/${CONFIGURATION}/kubeconfig.yaml}
 BACKUP_DATE=$(date "+%F_%H-%M-%S")
 BACKUP_FOLDER=${BACKUP_FOLDER:-${SCRIPT_DIR}/../configurations/${CONFIGURATION}/backup/${BACKUP_DATE}}
 
-# create backup fodler
+printf "Backup folder name:  %s\n" $BACKUP_DATE
+# create backup folder
 mkdir -p ${BACKUP_FOLDER}
 
 # cluster maintenance mode on && Neo4j maintenance mode on
 ${SCRIPT_DIR}/cluster.neo4j.sh maintenance on
 
 # database backup
+echo "Dumping database ..."
 kubectl --kubeconfig=${KUBECONFIG} -n default exec -it \
     $(kubectl --kubeconfig=${KUBECONFIG} -n default get pods | grep ocelot-neo4j | awk '{ print $1 }') \
-    -- neo4j-admin dump --database=graph.db --to=/var/lib/neo4j/$BACKUP_DATE-neo4j-dump
+    -- neo4j-admin dump --to=/var/lib/neo4j/$BACKUP_DATE-neo4j-dump
 # copy neo4j backup to local drive
+echo "Copying database to local file system ..."
 kubectl --kubeconfig=${KUBECONFIG} cp \
     default/$(kubectl --kubeconfig=${KUBECONFIG} -n default get pods | grep ocelot-neo4j |awk '{ print $1 }'):/var/lib/neo4j/$BACKUP_DATE-neo4j-dump $BACKUP_FOLDER/neo4j-dump
 # copy image data
+echo "Copying public uploads to local file system ..."
 kubectl --kubeconfig=${KUBECONFIG} cp \
     default/$(kubectl --kubeconfig=${KUBECONFIG} -n default get pods | grep ocelot-backend |awk '{ print $1 }'):/app/public/uploads $BACKUP_FOLDER/public-uploads
 

deployment/scripts/cluster.maintenance.sh

@@ -5,7 +5,7 @@ SCRIPT_PATH=$(realpath $0)
 SCRIPT_DIR=$(dirname $SCRIPT_PATH)
 
 # check CONFIGURATION
-if [ -z ${CONFIGURATION} ]; then
+if [[ -z "$CONFIGURATION" ]]; then
   echo "You must provide a `CONFIGURATION` via environment variable"
   exit 1
 fi
@@ -16,9 +16,11 @@ KUBECONFIG=${KUBECONFIG:-${SCRIPT_DIR}/../configurations/${CONFIGURATION}/kubeco
 
 case $1 in
     on)
+        echo "Network maintenance:  on"
         kubectl --kubeconfig=${KUBECONFIG} patch ingress ingress-ocelot-webapp --type merge --patch-file ${SCRIPT_DIR}/../src/kubernetes/patches/patch.ingress.maintenance.on.yaml
     ;;
     off)
+        echo "Network maintenance:  off"
         kubectl --kubeconfig=${KUBECONFIG} patch ingress ingress-ocelot-webapp --type merge --patch-file ${SCRIPT_DIR}/../src/kubernetes/patches/patch.ingress.maintenance.off.yaml
     ;;
     *)

deployment/scripts/cluster.neo4j.sh

@@ -5,7 +5,7 @@ SCRIPT_PATH=$(realpath $0)
 SCRIPT_DIR=$(dirname $SCRIPT_PATH)
 
 # check CONFIGURATION
-if [ -z ${CONFIGURATION} ]; then
+if [[ -z "$CONFIGURATION" ]]; then
   echo "You must provide a `CONFIGURATION` via environment variable"
   exit 1
 fi
@@ -21,20 +21,24 @@ case $1 in
                 ${SCRIPT_DIR}/cluster.maintenance.sh on
 
                 # set Neo4j in offline mode (maintenance)
+                echo "Neo4j maintenance:  on"
                 kubectl --kubeconfig=${KUBECONFIG} get deployment ocelot-neo4j -o json \
                     | jq '.spec.template.spec.containers[] += {"command": ["tail", "-f", "/dev/null"]}' \
                     | kubectl --kubeconfig=${KUBECONFIG} apply -f -
 
                 # wait for the container to restart
+                echo "Wait 60s ..."
                 sleep 60
             ;;
             off)
                 # set Neo4j in online mode
+                echo "Neo4j maintenance:  off"
                 kubectl --kubeconfig=${KUBECONFIG} get deployment ocelot-neo4j -o json \
                     | jq 'del(.spec.template.spec.containers[].command)' \
                     | kubectl --kubeconfig=${KUBECONFIG} apply -f -
 
                 # wait for the container to restart
+                echo "Wait 60s ..."
                 sleep 60
 
                 # maintenance mode off

deployment/scripts/clusters.backup-multiple-servers.sh

@@ -0,0 +1,91 @@
+#!/bin/bash
+
+# time stamp
+printf "\n\nMultiple backups started at:\n  "
+date
+
+# base setup
+SCRIPT_PATH=$(realpath $0)
+SCRIPT_DIR=$(dirname $SCRIPT_PATH)
+
+# save old CONFIGURATION for later reset
+export SAVE_CONFIGURATION=$CONFIGURATION
+
+# export all variables in "../.env"
+set -a            
+source ${SCRIPT_DIR}/../.env
+set +a
+
+# check BACKUP_CONFIGURATIONS
+if [[ -z "$BACKUP_CONFIGURATIONS" ]]; then
+  #%! echo "You must provide a BACKUP_CONFIGURATIONS via environment variable"
+  printf "!!! You must provide a BACKUP_CONFIGURATIONS via environment variable !!!\n"
+  exit 1
+fi
+
+# check BACKUP_SAVED_BACKUPS_NUMBER
+if [[ -z ${BACKUP_SAVED_BACKUPS_NUMBER} ]]; then
+  #%! echo "You must provide a BACKUP_SAVED_BACKUPS_NUMBER via environment variable"
+  printf "!!! You must provide a BACKUP_SAVED_BACKUPS_NUMBER via environment variable !!!\n"
+  exit 1
+fi
+
+# convert configurations to array
+IFS=' ' read -a CONFIGURATIONS_ARRAY <<< "$BACKUP_CONFIGURATIONS"
+
+# display the clusters
+printf "Backup the clusters:\n"
+for i in "${CONFIGURATIONS_ARRAY[@]}"
+do
+  echo "  $i"
+done
+
+# deleting backups?
+if (( BACKUP_SAVED_BACKUPS_NUMBER >= 1 )); then
+  printf "Keep the last %d backups for all networks.\n" $BACKUP_SAVED_BACKUPS_NUMBER
+else
+  echo "!!! ATTENTION: No backups are deleted !!!"
+fi
+
+echo "Cancel by ^C. You have 15 seconds"
+# wait for the admin to react
+sleep 15
+
+printf "\n"
+
+for i in "${CONFIGURATIONS_ARRAY[@]}"
+do
+  export CONFIGURATION=$i
+  # individual cluster backup
+  ${SCRIPT_DIR}/cluster.backup.sh
+
+  # deleting backups?
+  if (( BACKUP_SAVED_BACKUPS_NUMBER >= 1 )); then
+    # delete all oldest backups, but leave the last BACKUP_SAVED_BACKUPS_NUMBER
+
+    keep=$BACKUP_SAVED_BACKUPS_NUMBER
+    path="$SCRIPT_DIR/../configurations/$CONFIGURATION/backup/"
+
+    cd $path
+
+    printf "In\n  '$path'\n  remove:\n"
+    while [ `ls -1  |  wc -l` -gt $keep ]; do
+      oldest=`ls -c1  |  sort -n  |  head -1`
+      printf "  %s\n" $oldest
+      rm -rf $oldest
+    done
+
+    printf "Keep the last %d backups:\n" $BACKUP_SAVED_BACKUPS_NUMBER
+    ls -c1  |  sort -n  |  awk '{print "  " $0}'
+
+    cd $SCRIPT_DIR
+  else
+    echo "!!! ATTENTION: No backups are deleted !!!"
+  fi
+
+  printf "\n"
+done
+
+# reset CONFIGURATION to old
+export CONFIGURATION=$SAVE_CONFIGURATION
+echo "Reset to CONFIGURATION=$CONFIGURATION"

deployment/src/kubernetes/Backup.md

@@ -137,7 +137,7 @@ $ kubectl -n default exec -it $(kubectl -n default get pods | grep ocelot-neo4j
 # bash: enter bash of Neo4j
 $ kubectl -n default exec -it $(kubectl -n default get pods | grep ocelot-neo4j | awk '{ print $1 }') -- bash
 # generate Dump
-neo4j% neo4j-admin dump --database=graph.db --to=/var/lib/neo4j/$(date +%F)-neo4j-dump
+neo4j% neo4j-admin dump --to=/var/lib/neo4j/$(date +%F)-neo4j-dump
 # exit bash
 neo4j% exit
 
@@ -145,6 +145,9 @@ neo4j% exit
 $ kubectl -n default exec -it $(kubectl -n default get pods | grep ocelot-neo4j | awk '{ print $1 }') -- ls
 ```
 
+If you need a specific database name, add the option `--database=<name>` to the command `neo4j-admin dump`.
+To find out the default database name, see the [Neo4j readme](https://github.com/Ocelot-Social-Community/Ocelot-Social/blob/master/neo4j/README.md).
+
 Lets copy the dump backup
 
 ```bash

deployment/src/kubernetes/templates/neo4j/ConfigMap.yml

@@ -20,4 +20,5 @@ data:
   NEO4J_dbms_security_procedures_unrestricted: "{{ .Values.NEO4J.DBMS_SECURITY_PROCEDURES_UNRESTRICTED }}"
   NEO4J_dbms_allow__format__migration: "true"
   NEO4J_dbms_allow__upgrade: "true"
-  NEO4J_apoc_import_file_enabled: "{{ .Values.NEO4J.APOC_IMPORT_FILE_ENABLED }}" 
+  NEO4J_dbms_default__database: "{{ .Values.NEO4J.DBMS_DEFAULT_DATABASE }}"
+  NEO4J_apoc_import_file_enabled: "{{ .Values.NEO4J.APOC_IMPORT_FILE_ENABLED }}"

deployment/src/old/volumes/neo4j-offline-backup/README.md

@@ -79,8 +79,8 @@ $ kubectl -n ocelot-social get pods
 $ kubectl cp ./neo4j-backup human-connection/<POD-ID>:/root/
 $ kubectl -n ocelot-social exec -it <POD-ID> bash
 # Once you're in the pod restore the backup and overwrite the default database
-# called `graph.db` with `--force`.
-# This will delete all existing data in database `graph.db`!
+# called `neo4j` with `--force`.
+# This will delete all existing data in database `neo4j`!
 > neo4j-admin load --from=/root/neo4j-backup --force
 > exit
 ```

docker-compose.yml

@@ -114,9 +114,12 @@ services:
       - NEO4J_dbms_security_procedures_unrestricted=algo.*,apoc.*
       - NEO4J_dbms_allow__format__migration=true
       - NEO4J_dbms_allow__upgrade=true
-      # Uncomment following line for Neo4j Enterprise version instead of Community version
+      # uncomment following line for Neo4j Enterprise version instead of Community version
       # TODO: clarify if that is the only thing needed to unlock the Enterprise version
       # - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
+      # set the name of the database to be used
+      # - NEO4J_dbms_default__database=graph.db
+      # - NEO4J_dbms_default__database=neo4j
       # TODO: Remove the playground from production
     # bring the database in offline mode to export or load dumps
     # command: ["tail", "-f", "/dev/null"]

neo4j/README.md

@@ -1,4 +1,4 @@
-# Neo4J
+# Neo4j 4.4
 
 Human Connection is a social network. Using a graph based database which can
 model nodes and edges natively - a network - feels like an obvious choice. We
@@ -16,8 +16,7 @@ docker-compose up
 ```
 
 You can access Neo4J through [http://localhost:7474/](http://localhost:7474/)
-for an interactive cypher shell and a visualization of the graph.
-
+for an interactive Cypher shell and a visualization of the graph.
 
 ## Installation Without Docker
 
@@ -51,7 +50,7 @@ in `backend/.env`.
 
 Start Neo4J and confirm the database is running at [http://localhost:7474](http://localhost:7474).
 
-## Operations on Neo4j
+## Operations on Neo4j 4.4
 
 ### Docker or Docker Compose
 
@@ -68,13 +67,15 @@ To create a dump in Neo4j running in a Docker container:
 # connect to the Docker containers Neo4j terminal
 $ docker exec -it neo4j bash
 # generate Dump
-neo4j% neo4j-admin dump --database=graph.db --to=/var/lib/neo4j/$(date +%F)-neo4j-dump
+neo4j% neo4j-admin dump --to=/var/lib/neo4j/$(date +%F)-neo4j-dump
 # exit bash
 neo4j% exit
 # copy the dump out of the running Docker container
 $ docker cp <docker-image-name('neo4j')>:/var/lib/neo4j/neo4j-dump <local-folder-path>/$(date +%F)-neo4j-dump
 ```
 
+If you need a specific database name, add the option `--database=<name>` to the command `neo4j-admin dump`.
+
 ### Import Neo4j Dump
 
 To import a dump into Neo4j running in a Docker container:
@@ -88,11 +89,14 @@ $ docker cp <local-folder-path>/neo4j-dump <docker-image-name('neo4j')>:/var/lib
 # connect to the Docker containers Neo4j terminal
 $ docker exec -it neo4j bash
 # to load the dump into the database we need the following command in this terminal
-neo4j% neo4j-admin load --database=neo4j --from /var/lib/neo4j/$(date +%F)-neo4j-dump --force
+neo4j% neo4j-admin load --from /var/lib/neo4j/$(date +%F)-neo4j-dump --force
 # leave the terminal by entering
 neo4j% exit
 ```
 
+If you need a specific database name, add the option `--database=<name>` to the command `neo4j-admin load`.
+To find out the default database name, see below.
+
 ## Commands
 
 Here we describe some rarely used Cypher commands for Neo4j that are needed from time to time:
@@ -143,7 +147,7 @@ $ kubectl -n default exec -it $(kubectl -n default get pods | grep ocelot-backen
 ***Cypher commands to show indexes and constraints***
 
 ```bash
-# in browser command line or cypher shell
+# in browser command line or Cypher shell
 
 # show all indexes and constraints
 $ :schema
@@ -158,7 +162,7 @@ $ CALL db.constraints();
 ***Cypher commands to create and drop indexes and constraints***
 
 ```bash
-# in browser command line or cypher shell
+# in browser command line or Cypher shell
 
 # create indexes
 $ CALL db.index.fulltext.createNodeIndex("post_fulltext_search",["Post"],["title", "content"]);
@@ -171,3 +175,21 @@ $ DROP CONSTRAINT ON ( image:Image ) ASSERT image.url IS UNIQUE
 # drop all indexes and constraints
 $ CALL apoc.schema.assert({},{},true) YIELD label, key RETURN * ;
 ```
+
+### Database Management Commands
+
+***Cypher commands to manage databases***
+
+```bash
+# in browser command line or Cypher shell
+
+# show the default database
+$ SHOW DEFAULT DATABASE
+# show all databases
+$ SHOW DATABASES
+```
+
+To set the default database by configuration, use `NEO4J_dbms_default__database` as an environment variable when starting Neo4j 4.4, see [Docker specific configuration settings](<https://neo4j.com/docs/operations-manual/4.4/docker/ref-settings/>).
+
+If a database with this name does not exist, an empty database with this name is created and all other databases remain.
+You can switch back to an existing database without damaging it.