cifmw_cephadm

Deploys a Ceph cluster on a set of EDPM nodes using cephadm.

The openstack-k8s-operators HCI documentation describes how to run Ceph on EDPM nodes but leaves it to the reader to install Ceph with cephadm. The cifmw_cephadm role and hooks/playbooks/ceph.yml hook playbook may be used to automate the Ceph installation.

Before this role is run the following roles should be run.

cifmw_create_admin: creates a user for cephadm
cifmw_block_device: creates a virtual disk to store data
cifmw_ceph_spec: defines the Ceph cluster layout

After this role is run, the cifmw_ceph_client role can generate a k8s CR which OpenStack can use to connect to the deployed Ceph cluster.

The ceph.yml hook playbook in the hooks/playbooks directory provides a complete working example which does all of the above and has been tested on a three EDPM node deployment from install_yamls.

Privilege escalation

Requires an Ansible user who can become root to install Ceph server.

Parameters

The hooks/playbooks/ceph.yml hook playbook defaults these parameters so that they do not need to be changed for a typical EDPM deployment.

cifmw_cephadm_basedir: (String) Base directory for artifacts and logs. Defaults to cifmw_basedir, which defaults to {{ ansible_user_dir ~ '/ci-framework-data' }}.
cifmw_cephadm_default_container: If this is value is true, then cephadm bootstrap is not passed the --image parameter and whatever default Ceph container defined inside of cephadm is used. Otherwise use cifmw_cephadm_container_ns (e.g. “quay.io/ceph”), cifmw_cephadm_container_image (e.g. “ceph”) and cifmw_cephadm_container_tag (e.g. “v18”).
cifmw_cephadm_spec_ansible_host: the path to the Ceph spec generated by the cifmw_ceph_spec role (e.g. /tmp/ceph_spec.yml).
cifmw_cephadm_bootstrap_conf: the path to the initial Ceph configuration file generated by the cifmw_ceph_spec role (e.g. /tmp/initial_ceph.conf)
cifmw_ceph_client_vars: the path to ceph client variables passed as input to the cifmw_ceph_client role (e.g. /tmp/ceph_client.yml).
cifmw_cephadm_pools: see below
cifmw_cephadm_keys: see below
cifmw_cephadm_certs: The path on the ceph host where TLS/SSL certificates are located. It points to /etc/pki/tls.
cifmw_cephadm_certificate: The SSL/TLS certificate signed by CA which is an optional parameter. If it is provided, ceph dashboard and RGW will be configured for SSL automatically. Certificate should be made available in cifmw_cephadm_certs path only. To enable SSL for dashboard, both cifmw_cephadm_certificate and cifmw_cephadm_key are needed. These certificates can be generated automatically by setting cifmw_cephadm_certificate and cifmw_cephadm_key to the desired path, provided that the cifmw_openshift_kubeconfig is set correctly so that Ansible can request that k8s create a certificate using an existing root CA.
cifmw_cephadm_key: The SSL/TLS certificate key which is an optional parameter. If it is provided, ceph dashboard and rgw will be configured for SSL automatically.
cifmw_cephadm_monitoring_network: the Ceph public_network where the dashboard monitoring stack instances should be bound. The network range is gathered from the cifmw_cephadm_bootstrap_conf file, which represents the initial Ceph configuration file passed at bootstrap time.
cifmw_cephadm_rgw_network: the Ceph public_network where the radosgw instances should be bound. The network range is gathered from the cifmw_cephadm_bootstrap_conf file, which represents the initial Ceph configuration file passed at bootstrap time.
cifmw_cephadm_rgw_vip: the ingress daemon deployed along with radosgw requires a VIP that will be owned by keepalived. This IP address will be used as entry point to reach the radosgw backends through haproxy.
cifmw_cephadm_nfs_vip: the ingress daemon deployed along with the nfs cluster requires a VIP that will be owned by keepalived. This IP address is the same used for rgw unless an override is passed, and it’s used as entry point to reach the ganesha backends through an haproxy instance where proxy-protocol is enabled.
cifmw_cephadm_ns: Name of the OpenStack controlplane namespace used in configuring swift objects.
cifmw_cephadm_config_key_set_ssl_option: Optional colon separated list of SSL context options (default: no_sslv2:sslv3:no_tlsv1:no_tlsv1_1)
cifmw_rgw_ssl_backward_compatibility: This option is true by default because this role is able to manage older Ceph releases (starting from Squid). Set it to false if the target Ceph release is equal to or greater than Tentacle.
cifmw_cephadm_rgw_s3_glance: (Bool) If this is value is true, then cephadm will create glance secrets using the discovered RGW settings
cifmw_cephadm_nfsv3: (Bool) If this value is true, cephadm enables NFSv3 during Ceph NFS cluster creation. This option is required to explicitly enable NFSv3 for Ceph releases starting with Tentacle. For releases prior to Tentacle, this option is not required, as both NFSv3 and NFSv4 are enabled by default.
cifmw_cephadm_auth_allowed_ciphers: (String) When set, runs ceph mon set auth_allowed_ciphers <value> during cluster configuration. Example values are "aes,aes256k" or "aes256k" or "aes". Defaults to "" (unset, no command is run).

Use the cifmw_cephadm_pools list of dictionaries to define pools for Nova (vms), Cinder (volumes), Cinder-backups (backups), and Glance (images).

cifmw_cephadm_pools:
- name: vms
  pg_autoscale_mode: True
  target_size_ratio: 0.3
  application: rbd
- name: volumes
  pg_autoscale_mode: True
  target_size_ratio: 0.3
  application: rbd
- name: backups
  pg_autoscale_mode: True
  target_size_ratio: 0.2
  application: rbd
- name: images
  target_size_ratio: 0.2
  pg_autoscale_mode: True
  application: rbd

Use the cifmw_cephadm_keys list of dictionaries to define a CephX key which OpenStack can use authenticate to Ceph. The cephx_key Ansible module will generate a random value to pass for the key value.

cifmw_cephadm_keys:
  - name: client.openstack
    key: "{{ cephx.key }}"
    mode: '0600'
    caps:
      mgr: allow *
      mon: profile rbd
      osd: profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=images

Examples

See ceph.yml in the hooks/playbooks directory.

Tips for using standalone

Pick the appropriate storage network

In the hooks/playbooks/ceph.yml hook playbook, set the storage_network_range variable.

If network isolation is not being used, then set the storage_network_range variable to 192.168.122.0/24 (the default EDPM IP address range).
If network isolation is used, then as per the openstack-k8s-operators networking documentation, the default storage network is 172.18.0.0/24 and the storage_network_range variable should be set accordingly. As per the openstack-k8s-operators HCI documentation a shortened OpenStackDataPlane services list can be used to configure the storage network before Ceph and OpenStack are deployed.

See the README of the cifmw_ceph_spec role for more details on how the storage_network_range variable is used.

Update the Ansible inventory and environment variables

This example assumes ci-framework and install_yamls git repositories are in in $HOME and that EDPM nodes have been provisioned.

The cifmw_cephadm, cifmw_create_admin, and cifmw_block_device roles need to be able to SSH into all EDPM nodes but the default inventory only has localhost. The devsetup process in install_yamls generates each EDPM node and its IP address sequentially starting at 192.168.122.100. The following command may be used to create an inventory with the group edpm containing N EDPM nodes.

export N=2
echo -e "localhost ansible_connection=local\n[edpm]" > ~/ci-framework/inventory.yml
for I in $(seq 100 $((N+100))); do
  echo 192.168.122.${I} >> ~/ci-framework/inventory.yml
done

install_yamls generates an SSH key install_yamls/out/edpm/ansibleee-ssh-key-id_rsa for root on every EDPM node. Configure the Ansible environment to use this user and key.

export ANSIBLE_REMOTE_USER=cloud-admin
export ANSIBLE_SSH_PRIVATE_KEY=~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa
export ANSIBLE_HOST_KEY_CHECKING=False

Run the Ceph playbook

Direct playbook execution using ansible-playbook

cd ~/ci-framework/
ansible-playbook hooks/playbooks/ceph.yml

Using run_hook role

- name: Deploy ceph
  hosts: localhost
  vars:
    post_ceph:
      - name: Run ceph hook playbook
        type: playbook
        source: ceph.yml
  tasks:
    - name: Run post_ceph hook
      vars:
        step: post_ceph
      ansible.builtin.import_role:
        name: run_hook

Regarding the disks used as OSDs

By default the hooks/playbooks/ceph.yml hook playbook assumes there are no block devices for Ceph to use and calls the cifmw_block_device role to create block devices and has the cifmw_ceph_spec role configure a spec to use the created block devices.

If cifmw_ceph_spec_data_devices is passed to the hooks/playbooks/ceph.yml hook playbook, then the cifmw_block_device role is not called and the spec created by the cifmw_ceph_spec role will use whatever block devices were passed by cifmw_ceph_spec_data_devices. Use of cifmw_ceph_spec_data_devices implies that the block devices are already exist on the nodes to be deployed.

If the ci-framework is run in a reproducer scenario and the following parameter is passed, then the libvirt_manager role will deploy compute nodes with three 30G disks (/dev/vd{b,c,d}).

cifmw_libvirt_manager_configuration_patch_01_add_compute_volumes:
  vms:
    compute:
      extra_disks_num: 3
      extra_disks_size: 30G

Assuming those disks are on the nodes as created by the above, the following parameter may be passed.

cifmw_ceph_spec_data_devices: >-
  data_devices:
    all: true

The above sets the cifmw_ceph_spec_data_devices parameter. The >- is necessary so that the YAML block underneath it is treated as a string. This string is then passed to the cifmw_ceph_spec role which will create a Ceph spec containing the following:

service_type: osd
service_id: default_drive_group
data_devices:
  all: true

The above will result in Ceph using all disks that it judges to be available. Other embedded YAML options may be passed for the cifmw_ceph_spec_data_devices as described in the Advanced OSD Service Specifications.