71 Commits

Author SHA1 Message Date
6948666cfe
Merge branch 'master' into 2019/kw22/refactor_backend_structure
# Conflicts:
#	backend/src/middleware/index.js
#	backend/src/schema/resolvers/comments.spec.js
#	backend/src/schema/resolvers/socialMedia.spec.js
2019-06-03 19:48:47 +02:00
Robert Schäfer
df70ae451c Parameterize split size
@ulfgebhardt that was a really good suggestion, thanks!
2019-06-03 13:11:12 +02:00
Robert Schäfer
f9e9aa47b7 Disable collections without import statement
Even if the import statement does literally nothing, this slows down the
import. Let's avoid this until we do sth. with the data.
2019-05-31 20:14:08 +02:00
f78c54a3e8
refactored backend structure 2019-05-31 19:31:39 +02:00
Robert Schäfer
115022fc2e Decrease split size
This might take longer but will require less memory. I guess this might
work even on our kubernetes cluster.
2019-05-31 18:38:58 +02:00
Robert Schäfer
9a9118c721 Merge remote-tracking branch 'origin/master' into 2019/kw22/alpha_data_import_status_schema_split 2019-05-31 16:26:06 +02:00
Robert Schäfer
58add8fc5f Install envsubst in Dockerfile
@ulfgebhardt please setup docker on your machine or a remote machine.

Installing `envsubst` on alpine fails with circular dependencies
(awkward). So this repo here has a solution:
https://github.com/cirocosta/alpine-envsubst/blob/master/Dockerfile#L6
2019-05-31 16:26:00 +02:00
Robert Schäfer
10864e7d18 Remove command line arguments -u and -p
@ulfgebhardt: The docs at `man cypher-shell` say that you can pass
`NEO4J_USERNAME` and `NEO4J_PASSWORD` per environment variable. So the
command line arguments are obsolete here.
2019-05-31 16:26:00 +02:00
37be5481c0
remaining legacy table descriptions and dummy import scripts 2019-05-29 20:43:58 +02:00
355205028f
removed debug output 2019-05-29 20:04:03 +02:00
271af7dde2
keep index file for already imported files, do not reimport already imported file, invites, notifications, emotions and organisations scripts and descriptions 2019-05-29 20:02:30 +02:00
968578f10a
delete cql files 2019-05-29 17:30:46 +02:00
816fbdf7cd
fixed slug required - it was not in Nitro 2019-05-29 16:41:27 +02:00
Robert Schäfer
58224381a8 Add missing environment var to maintenance-worker 2019-05-29 16:12:03 +02:00
Robert Schäfer
a83aad3f60 Fix wrong path in Dockerfile 2019-05-29 16:00:45 +02:00
67595f6400
missing change for windows comment 2019-05-29 15:45:32 +02:00
7e44667dd1
modified .env file for neo4j import since its had problems with file locks on windows (cypher-shell) 2019-05-29 15:45:02 +02:00
6f0447515a
- fixed several errors handling import
- split graphql schema into parts
- which data is imported which is not - a list
2019-05-29 15:37:28 +02:00
Robert Schäfer
e1a113e7e4 Fix wrong mountpath
We're saving the files to /uploads. If the maintenance-worker does not
mount the uploads persistent volume there, we don't get persistent
files.
2019-05-29 15:16:08 +02:00
Robert Schäfer
7cb77c8c8d Increase volume size for uploads 2019-05-29 15:11:05 +02:00
Robert Schäfer
2c8dcaa592 Yet another typo 2019-05-29 15:08:14 +02:00
Robert Schäfer
95fe115198 Fix typo 2019-05-29 15:01:56 +02:00
Robert Schäfer
5622b1aa30 Avoid deletion of entire namespace by accident
Yep, that happened to me. I did

```
kubectl delete -f deployment/human-connection
```

 and I forgot about the namespace in that folder. Now everything is gone.
2019-05-29 14:22:49 +02:00
860a0d41d0
Merge pull request #697 from Human-Connection/2019/kw22/alpha_data_import
🍰 2019/kw22/alpha_data_import
2019-05-29 00:43:10 +02:00
4ccfe3822c
added existing data imports 2019-05-29 00:20:43 +02:00
Robert Schäfer
ab7a4ffc3e
Fix duplicate tags by using the name as the id
@ulfgebhardt: I wondered about the list of tags after importing the
legacy db. It seems, each tag has at most 1 contribution. I guess it's
because we create a unique id for each tag, so two tags with the same
`name` e.g. `#hashtag` and `#hashtag` are not de-duplicated.

I'm currently sitting in the train and cannot run the data import myself, could
you double-check?
2019-05-28 22:41:51 +02:00
014104e055
fixed data import from alpha data:
- include all collections (commented out)
- refactored neo4j import script
- use of .env file for (additional) configurations / configuration overrides
- lots of fiddeling with neo4j cql files and cypher shell
2019-05-28 18:53:05 +02:00
Robert Schäfer
f9ac22e560 Add executable UNIX permissions to export script
I encourage @ulfgebhardt to run the following command once:

```
SH_USERNAME=ulf SSH_HOST=***** MONGODB_USERNAME='hc-api' MONGODB_PASSWORD=***** MONGODB_DATABASE=hc_api MONGODB_AUTH_DB=admin UPLOADS_DIRECTORY=/data/api/uploads docker-compose -f docker-compose.maintenance.yml up --build
```

Once you're done with everything. You don't have to run docker for
development, but this procedure would ensure docker environment works
as expected.
2019-05-28 18:14:21 +02:00
4b5138880d
missing default .env file 2019-05-28 17:07:54 +02:00
301e7fa60c
fixed data export for alpha:
- include all collections
- refactored mongodb export script
- renamed to export
- use of .env file for (additional) configurations / configuration overrides
2019-05-28 17:03:44 +02:00
340a26fc5d
Merge pull request #585 from Human-Connection/doc-update-minikube-setup-documentation
More descriptive minikube setup instructions
2019-05-27 15:20:19 +02:00
Robert Schäfer
29c5c64c4e Fix wrong screenshot in kubernetes docs
I made a mistake. The correct external IP address is visible on your
Digital Ocean dashboard, not the kubernetes dashboard.
2019-05-16 23:26:01 +02:00
Robert Schäfer
5881a7d5df
Merge pull request #573 from Human-Connection/docs-354-kubernetes_cron_job_for_backups
Docs 354 kubernetes cron job for backups
2019-05-15 18:52:53 +02:00
Robert Schäfer
0ef2c26f03 Clean up and add a content warning 2019-05-15 16:25:01 +02:00
Robert Schäfer
1b3ee63565 Document Volume Snapshots 2019-05-15 16:25:01 +02:00
Robert Schäfer
222b153242 Provide volume snapshot according to DO support
Read:
https://github.com/Human-Connection/Human-Connection/pull/573#issuecomment-491358847
2019-05-15 16:25:01 +02:00
Robert Schäfer
018dd89797 Followed docs for heptio-ark and DO plugin v0.11.0 2019-05-15 16:24:05 +02:00
Robert Schäfer
8485b583a4 Update docs, remove redundant instructions 2019-05-15 16:24:05 +02:00
Robert Schäfer
90aae2f872 Add docs how to setup velero schedules 2019-05-15 16:24:05 +02:00
Robert Schäfer
49e6457cfd Setup relevant annotations 2019-05-15 16:24:05 +02:00
Robert Schäfer
e2ecd7904d Write docs for Velero and Disaster Recovery 2019-05-15 16:24:05 +02:00
Robert Schäfer
d70719040e Consequently create another template folder
... in human-connection/ where we store our specific kubernetes
configuraitons.
2019-05-15 14:01:07 +02:00
Robert Schäfer
24fe55c2d5 Streamline https configuration
I added a folder /templates to make it easier to reproduce.
2019-05-15 14:01:07 +02:00
Dominic Motuka
3e2b9666ca
More descriptive minikube setup instructions 2019-05-09 16:27:27 +03:00
Robert Schäfer
dfef37b3f6 Prevent argument list too long error 2019-05-08 00:46:07 +02:00
Robert Schäfer
bcc2c4dbbb Configure scripts and docker-compose.yml
After endless try/error I found the way to share volumes between
multiple docker-compose.ymls: You have to place those files in the same
folder. Also the import scripts must be adapted.
2019-05-07 21:48:09 +02:00
Robert Schäfer
5771efc920 Add binary idle to keep container spinning 2019-05-07 19:17:42 +02:00
Robert Schäfer
7c139bed1a Remove obsolete binary to create RSA keys
We can supply files on kubernetes through secrets
2019-05-07 19:17:21 +02:00
Robert Schäfer
b5d91cffef Implement @appinteractive's suggestions
This:
https://github.com/Human-Connection/Human-Connection/pull/529#discussion_r280065855
2019-05-07 19:15:47 +02:00
Robert Schäfer
497f77ae10 Breakthrough! Use split+indices for performance
@appinteractive thanks for pointing out `split`. You just saved me some
days of work to refactor the import statements to use CSV instead of
JSON files.

@Tirokk when I enter `:schema` in Neo4J web UI, I see the following:
```
:schema
Indexes
   ON :Badge(id) ONLINE
   ON :Category(id) ONLINE
   ON :Comment(id) ONLINE
   ON :Post(id) ONLINE
   ON :Tag(id) ONLINE
   ON :User(id) ONLINE

No constraints
```

So I temporarily removed the unique constraints on `slug` and added
plain indices on `id` for all relevant node types. We cannot omit the
`:Label` unfortunately, neo4j does not allow this. So I had to add all
indices for all known node labels instead.

With indices the import finishes in:
```
Time elapsed: 351 seconds
```
🎉

@appinteractive when I keep the unique indices on slug, I get an error
during import that a node with label `:User` and slug `tobias` already
exists. Ie. we have unqiue constraint violations in our production data.

@mattwr18 @ulfgebhardt @ogerly I started the application on my machine
on the production data and it turns out that the index page
http://localhost:3000/ takes way to long. Visiting my profile page at
http://localhost:3000/profile/5b1693daf850c11207fa6109/robert-schafer
is fine, though. Even pagination works. When I visit a post page with
not too many comments, the application is fast enough, too:
http://localhost:3000/post/5bbf49ebc428ea001c7ca89c/neues-video-format-human-connection-tech-news
2019-05-01 12:25:28 +02:00