Repository navigation
[Dual controller] Work to be done #560
Description
Feature description
The Dual Controller would need two or more AlbaNodes working together in a master - slave fashion. The goal is to failover to one if the other would become unresponsive or enough OSDs are reporting failures
First iteration
This will only contain happy path changes
The changes to be made:
- Implement a class which can combine AlbaNodes (AlbaNodeCluster)
- Change all AlbaNode related code to work with the potential AlbaNodeCluster relation
- Change the ASD Manager to work actively and passively
- Display these changes in the GUI
- Update ASD manager setup for entering IPMI info
- Implement new calls to shutdown everything related to the OSDs on a specific AlbaNode (through IPMI in the first implementation)
- Stop services
- Unmount
- Implement new calls to start everything related to the passive OSDs on a AlbaNode
- Mount
- Deploy service
- Update OSD within Alba
- Implement a heartbeat which fetches all OSD states (in a locked state to avoid choking Alba) from Alba and when a certain threshold is reached for a specific cluster, create a persistent lock and failover all active drives to the passive ones. (The reason behind the heartbeat is to avoid waiting too long on the ovs-workers to process a task)
- Implement an option to move OSDs back to another AlbaNode (switching master-slave roles for that OSD)
Second iteration
Complete reinstall of a Dual Controller AlbaNode should be possible
- Sync all information from one node of the Controller to the other
- Prevent that an ASD manager is part of 2 Dual controllers
Every step should be registered under it's own ticket. These steps will both include this repo and the alba-asdmanager repo
Activity
wimpers commentedon Feb 6, 2018
The active passive is on ASD level, not on ASD manager level.
JeffreyDevloo commentedon Feb 6, 2018
My intention was to leverage change all calls to support active/passive routes. That is what I meant with that :)
wimpers commentedon Feb 6, 2018
Can we not already have the service deployed but stopped? Any reason why you want to deploy it only now?
wimpers commentedon Feb 6, 2018
https://docs.google.com/document/d/1Jzptv2gkq7xbnStq9r93fHsT8Yfw8qo4_1hYyzfilqc/edit#
JeffreyDevloo commentedon Feb 6, 2018
Deploying the service when the filesystem is not mounted might cause some issues. We can already generate the servicefile though
dejonghb commentedon Feb 6, 2018
IMO fstab should be prepared and entries should have the "noauto" option to prevent automounting at boot.
The mounting itself should be done when starting the service (in that service file or via dependencies in systemd). When mounting fails, the service cannot start.
When stopping the service, the device should be umounted again.
Starting/stopping the service should only be done after a check if it's safe to do so.
JeffreyDevloo commentedon Feb 9, 2018
Estimations
First iteration
This will only contain happy path changes
The changes to be made:
Second iteration
Complete reinstall of a Dual Controller AlbaNode should be possible
JeffreyDevloo commentedon May 4, 2018
Waiting for testing hardware