An error occurred while fetching participants.

kolla: libvirtError: internal error: No more available PCI slots

prod-1-k8s-node-7 is currently scheduled on compute-6.bm.cri.epita.fr. Volume attachements for k8s pods fail with (in nova-compute.log on the hypervisor):

2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):                                                                                                                                             
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped                                                                        
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     return f(self, context, *args, **kw)                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function                                                               
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     return function(self, context, *args, **kwargs)                                                                                                                            
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function                                                              
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     compute_utils.add_instance_fault_from_exc(context,                                                                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self.force_reraise()                                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise self.value                                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function                                                              
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     return function(self, context, *args, **kwargs)                                                                                                                            
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7028, in attach_volume                                                                  
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     do_attach_volume(context, instance, driver_bdm)                                                                                                                            
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner                                                                     
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     return f(*args, **kwargs)                                                                                                                                                  
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7026, in do_attach_volume                                                               
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     bdm.destroy()                                                                                                                                                              
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self.force_reraise()                                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise self.value                                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7023, in do_attach_volume                                                               
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     return self._attach_volume(context, instance, driver_bdm)                                                                                                                  
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7064, in _attach_volume                                                                 
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     compute_utils.notify_about_volume_attach_detach(                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self.force_reraise()                                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise self.value                                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7042, in _attach_volume                                                                 
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     bdm.attach(context, instance, self.volume_api, self.driver,                                                                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped                                                                        
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     ret_val = method(obj, context, *args, **kwargs)                                                                                                                            
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach                                                                        
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self._do_attach(context, instance, volume, volume_api,                                                                                                                     
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self._volume_attach(context, volume, connector, instance,                                                                                                                  
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 606, in _volume_attach                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     volume_api.attachment_delete(context,                                                                                                                                      
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self.force_reraise()                                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise self.value                                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 595, in _volume_attach                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     virt_driver.attach_volume(                                                                                                                                                 
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2083, in attach_volume                                                              
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self._disconnect_volume(context, connection_info, instance,                                                                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self.force_reraise()                                                                                                                                                       
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise self.value                                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2067, in attach_volume                                                              
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     guest.attach_device(conf, persistent=True, live=live)                                                                                                                      
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 315, in attach_device                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     self._domain.attachDeviceFlags(device_xml, flags=flags)                                                                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit                                                                                  
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     result = proxy_call(self._autowrap, f, *args, **kwargs)                                                                                                                    
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call                                                                            
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     rv = execute(f, *args, **kwargs)                                                                                                                                           
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute                                                                               
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     six.reraise(c, e, tb)                                                                                                                                                      
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise                                                                                                   
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     raise value                                                                                                                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker                                                                                
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     rv = meth(*args, **kwargs)                                                                                                                                                 
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3/dist-packages/libvirt.py", line 630, in attachDeviceFlags                                                                                             
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)                                                                                         
2021-11-26 01:31:35.288 7 ERROR oslo_messaging.notify.messaging libvirt.libvirtError: internal error: No more available PCI slots

For now, I cordoned prod-1-k8s-node-7 until we figure out what happens.

Edited 3 years ago

Child items

...

Show labels

Activity

Marc Schmitt added IncidentActive incident labels 3 years ago

added IncidentActive incident labels
Marc Schmitt changed the severity to Medium - S3 3 years ago

changed the severity to Medium - S3
Marin Hannache @mareo · 3 years ago

Contributor

Possibly related: https://bugs.launchpad.net/nova/+bug/1836065

Maybe we shoud try setting num_pcie_ports to 28 (max value) in nova configuration.
Marc Schmitt changed title from Cannot attach volumes to prod-1-k8s-node-7 to kolla: libvirtError: internal error: No more available PCI slots 3 years ago

changed title from Cannot attach volumes to prod-1-k8s-node-7 to kolla: libvirtError: internal error: No more available PCI slots
Marc Schmitt @risson · 3 years ago

Author Contributor

This is now also happening on compute-11. I'll try setting num_pcie_ports to 26.
Marc Schmitt changed the severity to High - S2 3 years ago

changed the severity to High - S2
Marc Schmitt mentioned in merge request !331 (merged) 3 years ago

mentioned in merge request !331 (merged)
Marc Schmitt assigned to @risson 3 years ago

assigned to @risson
Marc Schmitt closed via merge request !331 (merged) 3 years ago

closed via merge request !331 (merged)
Marc Schmitt changed the status to Resolved by closing the incident 3 years ago

changed the status to Resolved by closing the incident

Please register or sign in to reply

Severity

High - S2

Status

None

Labels

Incident Active incident

Milestone

None

Due date

None

Confidentiality

Not confidential

0 Participants

Admin message

kolla: libvirtError: internal error: No more available PCI slots

Child items

Linked incidents or issues

Related merge requests

Activity