diff --git a/docu/Concepts/TechnicalRequirements/Federation.md b/docu/Concepts/TechnicalRequirements/Federation.md index 8b53125cc..3cd1e1ea3 100644 --- a/docu/Concepts/TechnicalRequirements/Federation.md +++ b/docu/Concepts/TechnicalRequirements/Federation.md @@ -27,10 +27,325 @@ The Variant A with an internal server contains the benefit to be as independent The Varaint B with an external server contains the benefit to reduce the implementation efforts and the responsibility for an own ActivitPub-Server. But it will cause an additional dependency to a third party service provider and the growing hosting costs. -## HyperSwarm +## HyperSwarm {HyperSwarm} The decision to switch from ActivityPub to HyperSwarm base on the arguments, that the *hyperswarm/dht* library will satify the most federation requirements out of the box. It is now to design the business requirements of the [gradido community communication](../BusinessRequirements/CommunityVerwaltung.md#UC-createCommunity) in a technical conception. -At first the following diagramm shows the pure logical handshake between an existing community-A and a new created community-B and the data exchange for buildup an authenticated relationship. +The challenge for the decentralized communities of gradido will be *how to become a new community aquainted with an existing community* ? + +To enable such a relationship between an existing community and a new community several stages has to run through: + +1. Federation + * join&connect + * direct exchange +2. Authentication +3. Autorized Communication + +### Overview {Overview} + +At first the following diagramm gives an overview of the three stages and shows the handshake between an existing community-A and a new created community-B including the data exchange for buildup such a federated, authenticated and autorized relationship. ![FederationHyperSwarm.png](./image/FederationHyperSwarm.png) + +### Prerequisits {Prerequisits} + +Before starting in describing the details of the federation handshake, some prerequisits have to be defined. + +#### Database {Database} + +With the federation additional data tables/entities have to be created. + +##### Community-Entity + +Create the new *Community* table to store attributes of the own community. This table is used more like a frame for lists of federated foreign communities, users, AUF- and Welfare-account and the profile data of the own community: + +| Attributes | Type | Description | +| ----------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| id | int | technical unique key of this entity | +| uuid | string | unique key for a community, which will never changed as long as the community exists | +| name | string | name of the community shown on UI e.g for selection of a community | +| description | string | description of the community shown on UI to present more community details | +| ... | | for the near future additional attributes like profile-info, trading-level, set-of-rights,... will follow with the next level of Multi-Community Readyness | + +##### CommunityFederation-Entity + +Create the new *CommunityFederation* table to store at this point of time only the attributes used by the federation handshake: + +| Attributes | Type | Description | +| ---------------- | --------- | ------------------------------------------------------------------------------------------------------ | +| id | int | technical unique key of this entity | +| uuid | string | unique key for a community, which will never changed as long as the community exists | +| foreign | boolean | flag to mark the entry as a foreign or own community entry | +| createdAt | timestamp | the timestamp the community entry was created | +| privateKey | string | the private key of the community for asynchron encryption (only set for the own community) | +| pubKey | string | the public key of the community for asynchron encryption | +| pubKeyVerifiedAt | timestamp | the timestamp the pubkey of this foreign community is verified (for the own community its always null) | +| authenticatedAt | timestamp | the timestamp of the last successfull authentication with this foreign community | +| | | for the near future additional attributes will follow with the next level of Multi-Community Readyness | + + +##### CommunityApiVersion-Entity + +Create the new *CommunityApiVersion* table to support several urls and apiversion of one community at once. It references the table *CommunityFederation* with the foreignkey *communityFederationID* (naming pattern foreignkea = `ID`) for a 1:n relationship + +| Attributes | Type | Description | +| --------------------- | --------- | ------------------------------------------------------------------------------------------------------------------ | +| id | int | technical unique key of this entity | +| communityFederationID | int | the technical foreign key to the community entity | +| url | string | the URL the community will provide its services, could be changed during lifetime of the community | +| apiversion | string | the API version the community will provide its services, will increase with each release | +| validFrom | timestamp | the timestamp as of the url and api are provided by the community | +| verifiedAt | timestamp | the timestamp the url and apiversion of this foreign community is verified (for the own community its always null) | + + + +#### Configuration {Configuration} + +The preparation of a community infrastructure needs some application-outside configuration, which will be read during the start phase of the gradido components. The configuration will be defined in a file based manner like key-value-pairs as properties or in a structured way like xml or json files. + +| Key | Value | Default-Value | Description | +| ----------------------------------------------- | :------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------- | +| stage.name | dev
stage1
stage2
prod | dev | defines the name of the stage this instance will serve | +| stage.host | | | the name of the host or ip this instance will run | +| stage.mode | test
prod | test | the running mode this instance will work | +| federation.communityname | | Gradido-Akademie | the name of this community | +| federation.apiversion | `` | current 1.7 | defines the current api version this instance will provide its services | +| federation.apiversion.`.`url | |
gdd.gradido.net |
defines the url on which this instance of a community will provide its services | +| federation.apiversion.`.`validFrom | | | defines the timestamp the apiversion is or will be valid | +| federation.dhtnode.topic | | dht_gradido_topic | defines the name of the federation topic, which is used to join and connect as federation-channel | +| federation.dhtnode.host | | | defines the host where the DHT-Node is hosted, if outside apollo | +| federation.dhtnode.port | | | defines the port on which the DHT-node will provide its services, if outside apollo | + +#### 1st Start of a community {1st Start of Community} + +The first time a new community infrastructure on a server is started, the start-phase has to check and prepair the own community database for federation. That means the application has to read the configuration and check against the database, if all current configured data is propagated in the database especially in the *CommunityXXX* entities. + +* check if the*Community* table is empty or if an exisiting community entry is not equals the configured values, then update as follows: + + * community.id = next sequence value + * community.uuid = generated UUID (version 4) + * community.name = federation.communityname + * community.description = null +* prepare the *CommunityFederation* table + + * communityFederation.id = next sequence value + * communityFederation.uuid = community.uuid + * communityFederation.foreign = FALSE + * communityFederation.createdAt = NOW + * communityFederation.privateKey = null + * communityFederation.pubKey = null + * communityFederation.pubKeyVerifiedAt = null + * communityFederation.authenticatedAt = null +* prepare the *CommunityApiVersion* table with all configured apiversions: + + * communityApiVersion.id = next sequence value + * communityApiVersion.communityFederationID = communityFederation.id + * communityApiVersion.url = federation.apiversion.``.url + * communityApiVersion.apiversion = federation.apiversion + * communityApiVersion.validFrom = federation.apiversion.``.validFrom + * communityApiVersion.verifiedAt = null + +### Stage1 - Federation {Stage1 - Federation} + +For the 1st stage the *hyperswarm dht library* will be used. It supports an easy way to connect a new community with other existing communities. As shown in the picture above the *hyperswarm dht library* will be part of the component *DHT-Node* separated from the *apollo server* component. The background for this separation is to keep off the federation activity from the business processes or to enable component specific scaling in the future. In consequence for the inter-component communication between *DHT-Node*, *apollo server* and other components like *database* the interface and security has to be defined. + +For the first federation release the DHT-node will be part of the apollo server, but internally designed and implemented as a logical saparated component. + +#### Sequence join&connect {join&connect} + +1. In the own database of community_A the entites *Community*, *CommunityFederation* and *CommunityApiVersion* are initialized +2. When starting the *DHT-Node* of community_A it search per *apollo-ORM* for the own community entry and check on existing keypair *CommunityFederation.pubKey* and *CommunityFederation.privateKey* in the database. If they not exist, the *DHT-Node* generates the keypair *pubkey* and *privatekey* and writes them per *apollo-ORM* in the database +3. For joining with the correct channel of *hyperswarm dht* a topic has to be used. The *DHT-Node* reads the configured value of the property *federation.dhtnode.topic*. +4. with the *CommunityFederation.pubKey* and the *federation.dhtnode.topic* the *DHT-Node* joins the *hyperswarm dht* and listen for other *DHT-nodes* on the topic. +5. As soon as a the *hyperswarm dht* notifies an additional node in the topic the *DHT-node* reads the *pubKey* of this additional node and search it per *apollo-ORM* in the *CommunityFederation* table by filtering with *CommunityFederation.foreign* = TRUE +6. if an entry with the C*ommunityFederation.pubKey* of the additional node still exists and the *CommunityFederation.pubKeyVerifiedAt* is not NULL the *DHT-node* do nothing and join and listen again on the topic +7. if an entry with the *CommunityFederation.pubKey* of the additional node can't be found, the *DHT-Node* starts with the next step *direct exchange* of the federation handshake + +#### Sequence direct exchange {direct exchange} + +1. if the *CommunityFederation.pubKey* of the additional node does not exists in the *CommunityFederation* table the *DHT-node* starts a *direct exchange* with this additional node +2. the *DHT-node* opens a direct connection per hyperswarm with the additional node and exchange respectively the *url* and *apiversion* between each other. + 1. to support the future feature that one community can provide several urls and apiversion at once the exchanged data should be in a format, which can represent structured information, like JSON (or simply CSV - feel free how to implement, but be aware about security aspects to avoid possible attacks during parsing the exchanged data and the used parsers for it) + + ``` + { + "API": + { + "url" : "comB.com", + "version" : "1.0", + "validFrom" : "2020.01.01" + } + "API" : + { + "url" : "comB.com", + "version" : "1.1", + "validFrom" : "2020.04.15" + } + "API" : + { + "url" : "comB.de", + "version" : "2.0", + "validFrom" : "2022.06.01" + } + } + ``` + 2. the *DHT-Node* writes per *apollo-ORM* the received and parsed data from the other node in the database + + 1. For the future an optimization step will be introduced here to avoid possible attacks of a foreign node by polute our database with mass of data. + + 1. before the *apollo-ORM* writes the data in the database, the *apollo-graphQL* invokes for all received urls and apiversions at the foreign node the request https.//`//getPubKey()`. + 2. Normally the foreign node will repsonse in a very short time with its publicKey + 3. if such a request runs in a timeout, the previous exchanged data with the foreign node will be a fake and can be refused without storing in database. + 4. if the response is in time the received publicKey must be equals with the pubKey of the foreign node the DHT-Node gets from hyperswarm dht per topic before + 5. if both keys are the same, the writing of the exchanged data per apollo-ORM can go on. + 6. if both keys will not match the exchanged data during the direct connection will be a fake and can be refused without storing in database. + 2. the *apollo-ORM* write the received data as follow + + * insert in the *CommunityFederation* table for this foreign node: + * communityFederation.id = next sequence value + * communityFederation.uuid = null + * communityFederation.foreign = TRUE + * communityFederation.createdAt = NOW + * communityFederation.privateKey = null + * communityFederation.pubKey = exchangedData.pubKey + * communityFederation.pubKeyVerifiedAt = null + * communityFederation.authenticatedAt = null + * insert in the *CommunityApiVersion* table for all exchangedData APIs: + * communityApiVersion.id = next sequence value + * communityApiVersion.communityFederationID = communityFederation.id + * communityApiVersion.url = exchangedData.API.url + * communityApiVersion.apiversion = exchangedData.API.version + * communityApiVersion.validFrom = exchangedData.API.validFrom + * communityApiVersion.verifiedAt = null + 3. After all received data is stored successfully, the DHT-Node starts the *stage2 - Authentication* of the federation handshake + +### Stage2 - Authentication + +The 2nd stage of federation is called authentication, because during the 1st stage the hyperswarm dht only ensures the knowledge that one node is the owner of its keypairs pubkey and private key. The exchanged data between two nodes during the *direct exchange* on the hyperswarm dht channel must be verified, means ensure if the proclaimed url and apiversion of a node is the correct address to reach the same node outside the hyperswarm infrastructure. + +As mentioned the DHT-node invokes the authentication stage on apollo server with the received data from *direct exchange*. + +#### Sequence + +### Stage3 - Autorized Business Communication + +# Review von Ulf + +## Communication concept + +The communication happens in 4 stages. + +- Stage1: Federation +- Stage2: Direct-Connection +- Stage3: GraphQL-Verification +- Stage4: GraphQL-Content + +### Stage1 - Federation + +Using the hyperswarm dht library we can find eachother easily and exchange a pubKey and data of which we know that the other side owns the private key of. + +``` +ComA ---- announce ----> DHT +ComB <--- listen ------- DHT +``` + +Each peer will know the `pubKey` of the other participants. Furthermore a direct connection is possible. + +``` +ComB ---- connect -----> ComA +ComB ---- data --------> ComA +``` + +### Stage2 - Direct-Connection + +The hyperswarm dht library offers a secure channel based on the exchanged `pubKey` so we do not need to verify things. + +The Idea is now to exchange the GraphQL Endpoints and their corresponding versions API versions in form of json + +``` +{ + "API": { + "1.0": "https://comB.com/api/1.0/", + "1.1": "https://comB.com/api/1.1/", + "2.4": "https://comB.de/api/2.4/" + } +} +``` + +### Stage3 - GraphQL-Verification + +The task of Stage3 is to verify that the collected data through the two Federation Stages are correct, since we did not verify yet that the proclaimed URL is actually the guy we talked to in the federation. Furthermore the sender must be verified to ensure the queried community does not reveal things to a third party not authorized. + +``` +ComA ----- verify -----> ComB +ComA <---- authorize --- ComB +``` + +Assuming this Dataset on ComA after a federation (leaving out multiple API endpoints to simplify things): + +``` +| PubKey | API-Endpoint | PubKey Verified On | +|--------|---------------|--------------------| +| PubA* | ComA.com/api/ | NULL | +| PubB | ComB.com/api/ | NULL | +| PubC | ComB.com/api/ | NULL | + +* = self +``` + +using the GraphQL Endpoint to query things: + +``` +ComA ---- getPubKey ---> ComB.com +ComA <--- PubB --------- ComB.com + +ComA UPDATE database SET pubKeyVerifiedOn = now WHERE API-Endpoint=queryURL AND PubKey=QueryResult +``` + +resulting in: + +``` +| PubKey | API-Endpoint | PubKey Verified On | +|--------|---------------|--------------------| +| PubA* | ComA.com/api/ | 1.1.1970 | +| PubB | ComB.com/api/ | NOW | +| PubC | ComB.com/api/ | NULL | +``` + +Furthermore we use the Header to transport a proof of who the caller is when calling and when answering: + +``` +ComA ---- getPubKey, sign({pubA, crypt(timeToken,pubB)},privA) --> ComB.com +ComB: is pubA known to me? +ComB: is the signature correct? +ComB: can I decrypt payload? +ComB: is timeToken <= 10sec? +ComA <----- PubB, sign({timeToken}, privB) ----------------------- ComB.com +ComA: is timeToken correct? +ComA: is signature correct? +``` + +This process we call authentication and can result in several errors: + +1. Your pubKey was not known to me +2. Your signature is incorrect +3. I cannot decrypt your payload +4. Token Timeout (recoverable) +5. Result token was incorrect +6. Result signature was incorrect + +``` +| PubKey | API-Endpoint | PubKey Verified On | AuthenticationLastSuccess | +|--------|---------------|--------------------|----------------------------| +| PubA* | ComA.com/api/ | 1.1.1970 | 1.1.1970 | +| PubB | ComB.com/api/ | NOW | NOW | +| PubC | ComB.com/api/ | NULL | NULL | +``` + +The next process is the authorization. This happens on every call on the receiver site to determine which call is allowed for the other side. + +``` +ComA ---- getPubKey, sign({pubA, crypt(timeToken,pubB)},privA) --> ComB.com +ComB: did I verify pubA? SELECT PubKeyVerifiedOn FROm database WHERE PubKey = pubA +ComB: is pubA allowed to query this? +``` diff --git a/docu/Concepts/TechnicalRequirements/graphics/FederationHyperSwarm.drawio b/docu/Concepts/TechnicalRequirements/graphics/FederationHyperSwarm.drawio index 6c849775b..6bfce4310 100644 --- a/docu/Concepts/TechnicalRequirements/graphics/FederationHyperSwarm.drawio +++ b/docu/Concepts/TechnicalRequirements/graphics/FederationHyperSwarm.drawio @@ -1,61 +1,61 @@ - + - + - - + + - + - - + + - + - - + + - - + + - - + + - + - + - + - + - + - + - + - + - - + + @@ -65,8 +65,8 @@ - - + + @@ -76,26 +76,26 @@ - - + + - - + + - + - + - + - - + + @@ -105,8 +105,8 @@ - - + + @@ -115,15 +115,15 @@ - + - + - - + + @@ -132,12 +132,12 @@ - + - - + + @@ -145,13 +145,13 @@ - + - + - + @@ -159,39 +159,39 @@ - + - + - + - + - + - - + + - - + + - - + + - + - - + + @@ -199,36 +199,36 @@ - + - + - + - + - + - - + + - - + + @@ -237,43 +237,43 @@ - + - - + + - + - - + + - + - + - + - - + + @@ -282,153 +282,153 @@ - + - - + + - + - + - - + + - - + + - + - - + + - - + + - + - + - - + + - + - - + + - - + + - - + + - - + + - - + + - + - - + + - - + + - + - - + + - - + + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - - + + - - + + - + - - + + - - + + - + - - + + - + - - + + @@ -436,65 +436,65 @@ - - + + - + - + - + - + - - + + - + - + - - + + - + - - + + - + - + - - + + - - + + - + - - + + - - + + @@ -502,130 +502,148 @@ - + - + - + - + - - + + - + - - + + - - + + - - + + - - + + - + - - + + - + - - + + - - + + - + - + - + - - + + - + - + - - + + - + - + - + - + - - + + - + - - + + - + - + - - + + - + - - + + - - + + + + + + + + + + + + + + + + + + + + diff --git a/docu/Concepts/TechnicalRequirements/image/FederationHyperSwarm.png b/docu/Concepts/TechnicalRequirements/image/FederationHyperSwarm.png index 628593c1f..13e1f3d72 100644 Binary files a/docu/Concepts/TechnicalRequirements/image/FederationHyperSwarm.png and b/docu/Concepts/TechnicalRequirements/image/FederationHyperSwarm.png differ