kolla: libvirtError: internal error: No more available PCI slots
prod-1-k8s-node-7 is currently scheduled on compute-6.bm.cri.epita.fr. Volume attachements for k8s pods fail with (in nova-compute.log on the hypervisor):
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging return f(self, context, *args, **kw)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging return function(self, context, *args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging compute_utils.add_instance_fault_from_exc(context,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self.force_reraise()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise self.value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging return function(self, context, *args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7028, in attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging do_attach_volume(context, instance, driver_bdm)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging return f(*args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7026, in do_attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging bdm.destroy()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self.force_reraise()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise self.value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7023, in do_attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging return self._attach_volume(context, instance, driver_bdm)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7064, in _attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging compute_utils.notify_about_volume_attach_detach(
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self.force_reraise()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise self.value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7042, in _attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging bdm.attach(context, instance, self.volume_api, self.driver,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging ret_val = method(obj, context, *args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self._do_attach(context, instance, volume, volume_api,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self._volume_attach(context, volume, connector, instance,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 606, in _volume_attach
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging volume_api.attachment_delete(context,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self.force_reraise()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise self.value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 595, in _volume_attach
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging virt_driver.attach_volume(
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2083, in attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self._disconnect_volume(context, connection_info, instance,
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self.force_reraise()
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise self.value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2067, in attach_volume
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging guest.attach_device(conf, persistent=True, live=live)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 315, in attach_device
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging self._domain.attachDeviceFlags(device_xml, flags=flags)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging rv = execute(f, *args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging six.reraise(c, e, tb)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging raise value
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging rv = meth(*args, **kwargs)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3/dist-packages/libvirt.py", line 630, in attachDeviceFlags
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging libvirt.libvirtError: internal error: No more available PCI slots For now, I cordoned prod-1-k8s-node-7 until we figure out what happens.