Ok, so here are multiple issues:
1. In cypher, `NOT NULL` will return `NULL` not `FALSE`. If we want
`FALSE` to be set in the database import, we should use `COAELESCE`
to find the first not-null value.
See:
https://neo4j.com/docs/cypher-manual/current/syntax/working-with-null/https://markhneedham.com/blog/2017/02/22/neo4j-null-values-even-work/
2. I removed the `disabled` and `deleted` checks on the commented
counter. With `neo4j-graphql-js` it is not possible to filter on the
join models (at least not without a lot of complexity) for disabled or
deleted items. Let's live with the fact that the list of commented posts
will include those posts, where the user has deleted his comment or where
the user's comment was disabled. It's being displayed as "not available"
so I think this is OK for now.
3. De-couple the pagination counters from the "commented", "shouted"
etc. counters. It might be that the list of posts is different for
different users. E.g. if the user has blocked you, the "posts" list
will be empty. The "shouted" or "commented" list will not have the
posts of the author. If you are a moderator, the list will include
disabled posts. So the counters are not in sync with the actual list
coming from the backend. Therefore I implemented "fetch and check if
resultSet < pageSize" instead of a global counter.
@ulfgebhardt: The docs at `man cypher-shell` say that you can pass
`NEO4J_USERNAME` and `NEO4J_PASSWORD` per environment variable. So the
command line arguments are obsolete here.
@ulfgebhardt: I wondered about the list of tags after importing the
legacy db. It seems, each tag has at most 1 contribution. I guess it's
because we create a unique id for each tag, so two tags with the same
`name` e.g. `#hashtag` and `#hashtag` are not de-duplicated.
I'm currently sitting in the train and cannot run the data import myself, could
you double-check?
- include all collections (commented out)
- refactored neo4j import script
- use of .env file for (additional) configurations / configuration overrides
- lots of fiddeling with neo4j cql files and cypher shell
I encourage @ulfgebhardt to run the following command once:
```
SH_USERNAME=ulf SSH_HOST=***** MONGODB_USERNAME='hc-api' MONGODB_PASSWORD=***** MONGODB_DATABASE=hc_api MONGODB_AUTH_DB=admin UPLOADS_DIRECTORY=/data/api/uploads docker-compose -f docker-compose.maintenance.yml up --build
```
Once you're done with everything. You don't have to run docker for
development, but this procedure would ensure docker environment works
as expected.
- include all collections
- refactored mongodb export script
- renamed to export
- use of .env file for (additional) configurations / configuration overrides
After endless try/error I found the way to share volumes between
multiple docker-compose.ymls: You have to place those files in the same
folder. Also the import scripts must be adapted.
@appinteractive thanks for pointing out `split`. You just saved me some
days of work to refactor the import statements to use CSV instead of
JSON files.
@Tirokk when I enter `:schema` in Neo4J web UI, I see the following:
```
:schema
Indexes
ON :Badge(id) ONLINE
ON :Category(id) ONLINE
ON :Comment(id) ONLINE
ON :Post(id) ONLINE
ON :Tag(id) ONLINE
ON :User(id) ONLINE
No constraints
```
So I temporarily removed the unique constraints on `slug` and added
plain indices on `id` for all relevant node types. We cannot omit the
`:Label` unfortunately, neo4j does not allow this. So I had to add all
indices for all known node labels instead.
With indices the import finishes in:
```
Time elapsed: 351 seconds
```
🎉
@appinteractive when I keep the unique indices on slug, I get an error
during import that a node with label `:User` and slug `tobias` already
exists. Ie. we have unqiue constraint violations in our production data.
@mattwr18 @ulfgebhardt @ogerly I started the application on my machine
on the production data and it turns out that the index page
http://localhost:3000/ takes way to long. Visiting my profile page at
http://localhost:3000/profile/5b1693daf850c11207fa6109/robert-schafer
is fine, though. Even pagination works. When I visit a post page with
not too many comments, the application is fast enough, too:
http://localhost:3000/post/5bbf49ebc428ea001c7ca89c/neues-video-format-human-connection-tech-news
We're going into the direction of removing the backend and database
deployments, accessing `/uploads` and `/data` through the maintenance
worker pod and carrying out tasks from there.
