ocelot-Social-Community/Ocelot-Social
1
0
Fork 0
mirror of https://github.com/Ocelot-Social-Community/Ocelot-Social.git synced 2025-12-12 23:35:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(backend): clean migrate scripts (#8317)

Browse Source 
* clean migrate scripts

- refactor migrate:init
  - separate admin seed
  - separate categories seed
- rework backend README regarding the database
- remove `db:clean` command as its a duplicate of `db:reset`
- remove `__migrate` helper alias

* renamed clean.ts to reset.ts

* set indices & constrains in init function

* fix comment

* disable migrations touching indices

* remove obsolete comment

* always run init on kubernetes

* reset db with or without migrations

* lint fixes

* Refine 'README.md'

* Refine more 'README.md'

* fix lint

---------

Co-authored-by: Wolfgang Huß <wolle.huss@pjannto.com>
This commit is contained in:
Ulf Gebhardt 2025-04-10 09:52:49 +02:00 committed by GitHub
parent 1b07b06ca7
commit fcc99ab58e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 249 additions and 218 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

188
backend/README.md
View File

@ -6,12 +6,12 @@ Run the following command to install everything through docker.
The installation takes a bit longer on the first pass or on rebuild ...
```bash
```sh
# in main folder
$ docker-compose up
$ docker compose up
# or
# rebuild the containers for a cleanup
$ docker-compose up --build
$ docker compose up --build
```
Wait a little until your backend is up and running at [http://localhost:4000/](http://localhost:4000/).
@ -26,7 +26,7 @@ some known problems with more recent node versions). You can use the
[node version manager](https://github.com/nvm-sh/nvm) `nvm` to switch
between different local Node versions:
```bash
```sh
# install Node
$ cd backend
$ nvm install v20.12.1
@ -35,7 +35,7 @@ $ nvm use v20.12.1
Install node dependencies with [yarn](https://yarnpkg.com/en/):
```bash
```sh
# in main folder
$ cd backend
$ yarn install
@ -47,7 +47,7 @@ $ nvm use && yarn
Copy Environment Variables:
```bash
```sh
# in backend/
$ cp .env.template .env
```
@ -57,14 +57,14 @@ a [local Neo4J](http://localhost:7474) instance is up and running.
Start the backend for development with:
```bash
```sh
# in backend/
$ yarn run dev
```
or start the backend in production environment with:
```bash
```sh
# in backend/
$ yarn run start
```
@ -79,154 +79,120 @@ More details about our GraphQL playground and how to use it with ocelot.social c
![GraphQL Playground](../.gitbook/assets/graphql-playground.png)
### Database Indexes and Constraints
## Database
Database indexes and constraints need to be created and upgraded when the database and the backend are running:
A fresh database needs to be initialized and migrated.
::: tabs
@tab:active Docker
```bash
# in main folder while docker-compose is running
$ docker exec backend yarn run db:migrate init
# only once: init admin user and create indexes and constraints in Neo4j database
# for development
$ docker compose exec backend yarn prod:migrate init
# in production mode use command
$ docker compose exec backend /bin/sh -c "yarn prod:migrate init"
```sh
# in folder backend while database is running
yarn db:migrate init
# for docker environments:
docker exec backend yarn db:migrate init
# for docker production:
docker exec backend yarn prod:migrate init
```
```bash
# in main folder with docker compose running
$ docker exec backend yarn run db:migrate up
```sh
# in backend with database running (In docker or local)
yarn db:migrate up
# for docker development:
docker exec backend yarn db:migrate up
# for docker production
docker exec backend yarn prod:migrate up
```
@tab Without Docker
### Optional Data
```bash
# in folder backend/ while database is running
# make sure your database is running on http://localhost:7474/browser/
yarn run db:migrate init
You can seed some optional data into the database.
To create the default admin <admin@example.org> with password `1234` use:
```sh
# in backend with database running (In docker or local)
yarn db:data:admin
```
```bash
# in backend/ with database running (In docker or local)
yarn run db:migrate up
When using `CATEGORIES_ACTIVE=true` you also want to seed the categories with:
```sh
# in backend with database running (In docker or local)
yarn db:data:categories
```
:::
### Seed Data
#### Seed Database
For a predefined set of test data you can seed the database with:
If you want your backend to return anything else than an empty response, you
need to seed your database:
```sh
# in backend with database running (In docker or local)
yarn db:seed
::: tabs
@tab:active Docker
In another terminal run:
```bash
# in main folder while docker-compose is running
$ docker exec backend yarn run db:seed
# for docker
docker exec backend yarn db:seed
```
To reset the database run:
### Reset Data
```bash
# in main folder while docker-compose is running
$ docker exec backend yarn run db:reset
In order to reset the database you can run:
```sh
# in backend with database running (In docker or local)
yarn db:reset
# or deleting the migrations as well
yarn db:reset:withmigrations
# for docker
docker exec backend yarn db:reset
# or deleting the migrations as well
docker exec backend yarn db:reset:withmigrations
# you could also wipe out your neo4j database and delete all volumes with:
$ docker-compose down -v
# if container is not running, run this command to set up your database indexes and constraints
$ docker exec backend yarn run db:migrate init
# And then upgrade the indexes and const
$ docker exec backend yarn run db:migrate up
docker compose down -v
```
@tab Without Docker
Run:
```bash
# in backend/ while database is running
$ yarn run db:seed
```
To reset the database run:
```bash
# in backend/ while database is running
$ yarn run db:reset
```
:::
> Note: This just deletes the data and not the constraints, hence you do not need to rerun `yarn db:migrate init` or `yarn db:migrate up`.
### Data migrations
Although Neo4J is schema-less,you might find yourself in a situation in which
you have to migrate your data e.g. because your data modeling has changed.
::: tabs
@tab:active Docker
Generate a data migration file:
```bash
# in main folder while docker-compose is running
$ docker-compose exec backend yarn run db:migrate:create your_data_migration
# Edit the file in ./src/db/migrations/
```
To run the migration:
```bash
# in main folder while docker-compose is running
$ docker exec backend yarn run db:migrate up
```
@tab Without Docker
Generate a data migration file:
```bash
# in backend/
```sh
# in backend
$ yarn run db:migrate:create your_data_migration
# Edit the file in ./src/db/migrations/
# for docker
# in main folder while docker compose is running
$ docker compose exec backend yarn run db:migrate:create your_data_migration
# Edit the file in ./src/db/migrations/
```
To run the migration:
```bash
```sh
# in backend/ while database is running
$ yarn run db:migrate up
```
:::
# for docker
# in main folder while docker compose is running
$ docker exec backend yarn run db:migrate up
```
## Testing
**Beware**: We have no multiple database setup at the moment. We clean the
database after each test, running the tests will wipe out all your data!
::: tabs
@tab:active Docker
Run the unit tests:
```bash
# in main folder while docker-compose is running
$ docker exec backend yarn run test
```
@tab Without Docker
Run the unit tests:
```bash
```sh
# in backend/ while database is running
$ yarn run test
```
:::
# for docker
# in main folder while docker compose is running
$ docker exec backend yarn run test
```

13
backend/package.json
View File

@ -8,19 +8,20 @@
"private": false,
"main": "src/index.ts",
"scripts": {
"__migrate": "migrate --compiler 'ts:./src/db/compiler.ts' --migrations-dir ./src/db/migrations",
"prod:migrate": "migrate --migrations-dir ./build/src/db/migrations --store ./build/src/db/migrate/store.js",
"start": "node build/src/",
"build": "tsc && tsc-alias && ./scripts/build.copy.files.sh",
"dev": "nodemon --exec ts-node --require tsconfig-paths/register src/ -e js,ts,gql",
"dev:debug": "nodemon --exec babel-node --inspect=0.0.0.0:9229 src/ -e js,ts,gql",
"lint": "eslint --max-warnings=0 --ext .js,.ts ./src",
"test": "cross-env NODE_ENV=test NODE_OPTIONS=--max-old-space-size=8192 jest --runInBand --coverage --forceExit --detectOpenHandles",
"db:clean": "ts-node --require tsconfig-paths/register src/db/clean.ts",
"db:reset": "yarn run db:clean",
"db:reset": "ts-node --require tsconfig-paths/register src/db/reset.ts",
"db:reset:withmigrations": "ts-node --require tsconfig-paths/register src/db/reset-with-migrations.ts",
"db:seed": "ts-node --require tsconfig-paths/register src/db/seed.ts",
"db:migrate": "yarn run __migrate --store ./src/db/migrate/store.ts",
"db:migrate:create": "yarn run __migrate --template-file ./src/db/migrate/template.ts --date-format 'yyyymmddHHmmss' create"
"db:data:admin": "ts-node --require tsconfig-paths/register src/db/admin.ts",
"db:data:categories": "ts-node --require tsconfig-paths/register src/db/categories.ts",
"db:migrate": "migrate --compiler 'ts:./src/db/compiler.ts' --migrations-dir ./src/db/migrations --store ./src/db/migrate/store.ts",
"db:migrate:create": "migrate --compiler 'ts:./src/db/compiler.ts' --migrations-dir ./src/db/migrations --template-file ./src/db/migrate/template.ts --date-format 'yyyymmddHHmmss' create",
"prod:migrate": "migrate --migrations-dir ./build/src/db/migrations --store ./build/src/db/migrate/store.js"
},
"dependencies": {
"@babel/cli": "~7.27.0",

51
backend/src/db/admin.ts Normal file
View File

@ -0,0 +1,51 @@
import { hashSync } from 'bcryptjs'
import { v4 as uuid } from 'uuid'
import { getDriver } from './neo4j'
const defaultAdmin = {
email: 'admin@example.org',
password: hashSync('1234', 10),
name: 'admin',
id: uuid(),
slug: 'admin',
}
const createDefaultAdminUser = async () => {
const driver = getDriver()
const session = driver.session()
const createAdminTxResultPromise = session.writeTransaction(async (txc) => {
txc.run(
`MERGE (e:EmailAddress {
email: "${defaultAdmin.email}",
createdAt: toString(datetime())
})-[:BELONGS_TO]->(u:User {
name: "${defaultAdmin.name}",
encryptedPassword: "${defaultAdmin.password}",
role: "admin",
id: "${defaultAdmin.id}",
slug: "${defaultAdmin.slug}",
createdAt: toString(datetime()),
allowEmbedIframes: false,
showShoutsPublicly: false,
sendNotificationEmails: true,
deleted: false,
disabled: false
})-[:PRIMARY_EMAIL]->(e)`,
)
})
try {
await createAdminTxResultPromise
console.log('Successfully created default admin user!') // eslint-disable-line no-console
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(error) // eslint-disable-line no-console
} finally {
session.close()
driver.close()
}
}
;(async function () {
await createDefaultAdminUser()
})()

36
backend/src/db/categories.ts Normal file
View File

@ -0,0 +1,36 @@
import { categories } from '@constants/categories'
import { getDriver } from './neo4j'
const createCategories = async () => {
const driver = getDriver()
const session = driver.session()
const createCategoriesTxResultPromise = session.writeTransaction(async (txc) => {
categories.forEach(({ icon, name }, index) => {
const id = `cat${index + 1}`
txc.run(
`MERGE (c:Category {
icon: "${icon}",
slug: "${name}",
name: "${name}",
id: "${id}",
createdAt: toString(datetime())
})`,
)
})
})
try {
await createCategoriesTxResultPromise
console.log('Successfully created categories!') // eslint-disable-line no-console
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(`Error creating categories: ${error}`) // eslint-disable-line no-console
} finally {
session.close()
driver.close()
}
}
;(async function () {
await createCategories()
})()

19
backend/src/db/factories.ts
View File

@ -17,18 +17,19 @@ const uniqueImageUrl = (imageUrl) => {
return newUrl.toString()
}
export const cleanDatabase = async (options: any = {}) => {
const { driver = getDriver() } = options
export const cleanDatabase = async ({ withMigrations } = { withMigrations: false }) => {
const driver = getDriver()
const session = driver.session()
const clean = `
MATCH (everything)
${withMigrations ? '' : "WHERE NOT 'Migration' IN labels(everything)"}
DETACH DELETE everything
`
try {
await session.writeTransaction((transaction) => {
return transaction.run(
`
MATCH (everything)
WHERE NOT 'Migration' IN labels(everything)
DETACH DELETE everything
`,
)
return transaction.run(clean)
})
} finally {
session.close()

119
backend/src/db/migrate/store.ts
View File

@ -1,108 +1,45 @@
import { hashSync } from 'bcryptjs'
import { v4 as uuid } from 'uuid'
import CONFIG from '@config/index'
import { categories } from '@constants/categories'
import { getDriver, getNeode } from '@db/neo4j'
const defaultAdmin = {
email: 'admin@example.org',
password: hashSync('1234', 10),
name: 'admin',
id: uuid(),
slug: 'admin',
}
const createCategories = async (session) => {
const createCategoriesTxResultPromise = session.writeTransaction(async (txc) => {
categories.forEach(({ icon, name }, index) => {
const id = `cat${index + 1}`
txc.run(
`MERGE (c:Category {
icon: "${icon}",
slug: "${name}",
name: "${name}",
id: "${id}",
createdAt: toString(datetime())
})`,
)
})
})
try {
await createCategoriesTxResultPromise
console.log('Successfully created categories!') // eslint-disable-line no-console
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(`Error creating categories: ${error}`) // eslint-disable-line no-console
}
}
const createDefaultAdminUser = async (session) => {
const readTxResultPromise = session.readTransaction(async (txc) => {
const result = await txc.run('MATCH (user:User) RETURN count(user) AS userCount')
return result.records.map((r) => r.get('userCount'))
})
let createAdmin = false
try {
const userCount = parseInt(String(await readTxResultPromise))
if (userCount === 0) createAdmin = true
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(error) // eslint-disable-line no-console
}
if (createAdmin) {
const createAdminTxResultPromise = session.writeTransaction(async (txc) => {
txc.run(
`MERGE (e:EmailAddress {
email: "${defaultAdmin.email}",
createdAt: toString(datetime())
})-[:BELONGS_TO]->(u:User {
name: "${defaultAdmin.name}",
encryptedPassword: "${defaultAdmin.password}",
role: "admin",
id: "${defaultAdmin.id}",
slug: "${defaultAdmin.slug}",
createdAt: toString(datetime()),
allowEmbedIframes: false,
showShoutsPublicly: false,
deleted: false,
disabled: false
})-[:PRIMARY_EMAIL]->(e)`,
)
})
try {
await createAdminTxResultPromise
console.log('Successfully created default admin user!') // eslint-disable-line no-console
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(error) // eslint-disable-line no-console
}
}
}
class Store {
async init(next) {
async init(errFn) {
const neode = getNeode()
const { driver } = neode
const session = driver.session()
await createDefaultAdminUser(session)
if (CONFIG.CATEGORIES_ACTIVE) await createCategories(session)
const writeTxResultPromise = session.writeTransaction(async (txc) => {
await txc.run('CALL apoc.schema.assert({},{},true)') // drop all indices and constraints
const session = neode.driver.session()
const txFreshIndicesConstrains = session.writeTransaction(async (txc) => {
// drop all indices and constraints
await txc.run('CALL apoc.schema.assert({},{},true)')
/*
#############################################
# ADD YOUR CUSTOM INDICES & CONSTRAINS HERE #
#############################################
*/
// Search indexes (also part of migration 20230320130345-fulltext-search-indexes)
await txc.run(
`CALL db.index.fulltext.createNodeIndex("user_fulltext_search",["User"],["name", "slug"])`,
)
await txc.run(
`CALL db.index.fulltext.createNodeIndex("post_fulltext_search",["Post"],["title", "content"])`,
)
await txc.run(`CALL db.index.fulltext.createNodeIndex("tag_fulltext_search",["Tag"],["id"])`) // also part of migration 20200207080200-fulltext_index_for_tags
// Search indexes (also part of migration 20220803060819-create_fulltext_indices_and_unique_keys_for_groups)
await txc.run(`
CALL db.index.fulltext.createNodeIndex("group_fulltext_search",["Group"],["name", "slug", "about", "description"])
`)
})
try {
await writeTxResultPromise
// Due to limitations of neode in combination with the limitations of the community version of neo4j
// we need to have all constraints and indexes defined here. They can not be properly migrated
await txFreshIndicesConstrains
await getNeode().schema.install()
// eslint-disable-next-line no-console
console.log('Successfully created database indices and constraints!')
next()
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (error) {
console.log(error) // eslint-disable-line no-console
next(error, null)
errFn(error)
} finally {
session.close()
driver.close()
neode.driver.close()
}
}

6
backend/src/db/migrations-examples/20200207080200-fulltext_index_for_tags.ts
View File

@ -9,10 +9,13 @@ export async function up(next) {
const transaction = session.beginTransaction()
try {
// We do do this in /src/db/migrate/store.ts
/*
await transaction.run(`
CALL db.index.fulltext.createNodeIndex("tag_fulltext_search",["Tag"],["id"])
`)
await transaction.commit()
*/
next()
} catch (error) {
const { message } = error
@ -39,10 +42,13 @@ export async function down(next) {
try {
// Implement your migration here.
// We do do this in /src/db/migrate/store.ts
/*
await transaction.run(`
CALL db.index.fulltext.drop("tag_fulltext_search")
`)
await transaction.commit()
*/
next()
} catch (error) {
// eslint-disable-next-line no-console

7
backend/src/db/migrations/20220803060819-create_fulltext_indices_and_unique_keys_for_groups.ts
View File

@ -18,9 +18,13 @@ export async function up(next) {
// await transaction.run(`
// CREATE CONSTRAINT ON ( group:Group ) ASSERT group.slug IS UNIQUE
// `)
// We do do this in /src/db/migrate/store.ts
/*
await transaction.run(`
CALL db.index.fulltext.createNodeIndex("group_fulltext_search",["Group"],["name", "slug", "about", "description"])
`)
*/
await transaction.commit()
next()
} catch (error) {
@ -42,6 +46,8 @@ export async function down(next) {
try {
// Implement your migration here.
// We do do this in /src/db/migrate/store.ts
/*
await transaction.run(`
DROP CONSTRAINT ON ( group:Group ) ASSERT group.id IS UNIQUE
`)
@ -52,6 +58,7 @@ export async function down(next) {
CALL db.index.fulltext.drop("group_fulltext_search")
`)
await transaction.commit()
*/
next()
} catch (error) {
// eslint-disable-next-line no-console

6
backend/src/db/migrations/20230320130345-fulltext-search-indexes.ts
View File

@ -8,6 +8,8 @@ export async function up(next) {
const transaction = session.beginTransaction()
try {
// We do do this in /src/db/migrate/store.ts
/*
// Drop indexes if they exist because due to legacy code they might be set already
const indexesResponse = await transaction.run(`CALL db.indexes()`)
const indexes = indexesResponse.records.map((record) => record.get('name'))
@ -31,6 +33,7 @@ export async function up(next) {
`CALL db.index.fulltext.createNodeIndex("tag_fulltext_search",["Tag"],["id"])`,
)
await transaction.commit()
*/
next()
} catch (error) {
// eslint-disable-next-line no-console
@ -50,10 +53,13 @@ export async function down(next) {
const transaction = session.beginTransaction()
try {
// We do do this in /src/db/migrate/store.ts
/*
await transaction.run(`CALL db.index.fulltext.drop("user_fulltext_search")`)
await transaction.run(`CALL db.index.fulltext.drop("post_fulltext_search")`)
await transaction.run(`CALL db.index.fulltext.drop("tag_fulltext_search")`)
await transaction.commit()
*/
next()
} catch (error) {
// eslint-disable-next-line no-console

20
backend/src/db/reset-with-migrations.ts Normal file
View File

@ -0,0 +1,20 @@
/* eslint-disable n/no-process-exit */
import CONFIG from '@config/index'
import { cleanDatabase } from './factories'
if (CONFIG.PRODUCTION && !CONFIG.PRODUCTION_DB_CLEAN_ALLOW) {
throw new Error(`You cannot clean the database in a non-staging and real production environment!`)
}
;(async function () {
try {
await cleanDatabase({ withMigrations: true })
console.log('Successfully deleted all nodes and relations including!') // eslint-disable-line no-console
process.exit(0)
// eslint-disable-next-line no-catch-all/no-catch-all
} catch (err) {
console.log(`Error occurred deleting the nodes and relations (reset the db)\n\n${err}`) // eslint-disable-line no-console
process.exit(1)
}
})()

0
backend/src/db/clean.ts → backend/src/db/reset.ts
View File

2
deployment/helm/charts/ocelot-social/templates/backend/stateful-set.yaml
View File

@ -18,7 +18,7 @@ spec:
- name: {{ .Release.Name }}-backend-migrations
image: "{{ .Values.backend.image.repository }}:{{ .Values.backend.image.tag | default (include "defaultTag" .) }}"
imagePullPolicy: {{ quote .Values.global.image.pullPolicy }}
command: ["/bin/sh", "-c", "yarn prod:migrate up"]
command: ["/bin/sh", "-c", "yarn prod:migrate init && yarn prod:migrate up"]
{{- include "resources" .Values.backend.resources | indent 10 }}
envFrom:
- configMapRef: