Cyborg (OpenStack)

sync data to placement raise resource in use exception

Bug #2017513 reported by Wenping Song on 2023-04-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cyborg (OpenStack)	New	Undecided	Unassigned

Bug Description

when we remove the pci device that is used from host, the cyborg-agent will raise 'resource provider in use ' exception.

2023-04-24 14:50:08.267 1 ERROR oslo_service.periodic_task [None req-3493a3b4-7275-46cb-8767-253b414eac60 - - - - -] Error during AgentManager.update_available_resource: cyborg.common.exception_Remote.ResourceProviderInUse_Remote: An unknown exception occurred.
Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
res = self.dispatcher.dispatch(message)

File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
result = func(ctxt, **new_args)

File "/var/lib/kolla/venv/lib/python3.6/site-packages/cyborg/conductor/manager.py", line 86, in report_data
old_driver_device_list, driver_device_list)

File "/var/lib/kolla/venv/lib/python3.6/site-packages/cyborg/conductor/manager.py", line 112, in drv_device_make_diff
self._delete_provider_and_sub_providers(context, rp_uuid)

File "/var/lib/kolla/venv/lib/python3.6/site-packages/cyborg/conductor/manager.py", line 388, in _delete_provider_and_sub_providers
self.placement_client.delete_provider(rp["uuid"])

File "/var/lib/kolla/venv/lib/python3.6/site-packages/cyborg/common/placement_client.py", line 313, in delete_provider
raise exception.ResourceProviderInUse()

cyborg.common.exception.ResourceProviderInUse: An unknown exception occurred.

Revision history for this message

chandan kumar (chkumar246) wrote on 2026-05-06:

Download full text (36.7 KiB)

I am not sure I understand this bug correctly. Are we removing the pci device from cyborg.conf or trying to delete the pci device from device profile?

I am tried to create a reproducer by removing the pci device from the config. Below is the error
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device attribute list
+--------------------------------------+---------------+--------+----------------------------+
| uuid                                 | deployable_id | key    | value                      |
+--------------------------------------+---------------+--------+----------------------------+
| 3ebba534-0aaf-4681-880d-a01a6f001c30 |             2 | rc     | CUSTOM_PCI                 |
| c54284b2-b042-4995-8aa9-8a92fd9782a3 |             2 | trait0 | CUSTOM_PCI_PRODUCT_ID_0010 |
+--------------------------------------+---------------+--------+----------------------------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device list
+--------------------------------------+------+--------+-----------------+--------------------------------------------+
| uuid                                 | type | vendor | hostname        | std_board_info                             |
+--------------------------------------+------+--------+-----------------+--------------------------------------------+
| c81426fe-e130-45f0-bdd5-278776173f12 | GPU  | 1b36   | cyborg-devstack | {"product_id": "0010", "controller": null} |
+--------------------------------------+------+--------+-----------------+--------------------------------------------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device profile create bug2017513 '[{"resources:CUSTOM_PCI": "1", "trait:CUSTOM_PCI_PRODUCT_ID_0010": "required"}]'
+-------------+---------------------------------------------------------------------------------+
| Field       | Value                                                                           |
+-------------+---------------------------------------------------------------------------------+
| created_at  | 2026-05-06 05:26:51+00:00                                                       |
| updated_at  | None                                                                            |
| uuid        | 32a5bf34-a7f9-419d-939e-143eef0e04cd                                            |
| name        | bug2017513                                                                      |
| groups      | [{'resources:CUSTOM_PCI': '1', 'trait:CUSTOM_PCI_PRODUCT_ID_0010': 'required'}] |
| description | None                                                                            |
+-------------+---------------------------------------------------------------------------------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device profile list
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
| uuid                                 | name       | groups                                                                          | description |
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
| 32a5bf34-a7f9-419d-939e-143eef0e04cd | bug2017513 | [{'resources:CUSTOM_PCI': '1', 'trait:CUSTOM_PCI_PRODUCT_ID_0010': 'required'}] | None        |
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack flavor set bug2017513flavor --property="accel:device_profile=bug2017513"
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack flavor show bug2017513flavor
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| description                | None                                 |
| disk                       | 1                                    |
| id                         | 87d7ad65-b095-44a5-a6da-b696d7329839 |
| name                       | bug2017513flavor                     |
| os-flavor-access:is_public | True                                 |
| properties                 | accel:device_profile='bug2017513'    |
| ram                        | 512                                  |
| rxtx_factor                | 1.0                                  |
| swap                       | 0                                    |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+

(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack image list
+--------------------------------------+--------------------------+--------+
| ID                                   | Name                     | Status |
+--------------------------------------+--------------------------+--------+
| fe1b6c0c-8081-46b9-9ce7-2839398de4f6 | cirros-0.6.3-x86_64-disk | active |
+--------------------------------------+--------------------------+--------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack server create --flavor bug2017513flavor --image fe1b6c0c-8081-46b9-9ce7-2839398de4f6 --network private --wait bug2017513server

(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack server list
+--------------------------------------+------------------+--------+----------------------------------------------+--------------------------+------------------+
| ID                                   | Name             | Status | Networks                                     | Image                    | Flavor           |
+--------------------------------------+------------------+--------+----------------------------------------------+--------------------------+------------------+
| f7db88b0-6728-4a27-b134-6506a1664cb5 | bug2017513server | ACTIVE | private=10.0.0.18,                           | cirros-0.6.3-x86_64-disk | bug2017513flavor |
|                                      |                  |        | fd5e:3467:a78c:0:f816:3eff:fec9:5e24         |                          |                  |
+--------------------------------------+------------------+--------+----------------------------------------------+--------------------------+------------------+

(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack resource provider list
+--------------------------------------+------------------------------+------------+--------------------------------------+--------------------------------------+
| uuid                                 | name                         | generation | root_provider_uuid                   | parent_provider_uuid                 |
+--------------------------------------+------------------------------+------------+--------------------------------------+--------------------------------------+
| b02fc995-33b4-4b0b-93b7-041a85eb8b32 | cyborg-devstack              |          3 | b02fc995-33b4-4b0b-93b7-041a85eb8b32 | None                                 |
| a3702bd4-04d2-367d-925a-0db0a336a9f6 | cyborg-devstack_0000:08:01.0 |          6 | b02fc995-33b4-4b0b-93b7-041a85eb8b32 | b02fc995-33b4-4b0b-93b7-041a85eb8b32 |
+--------------------------------------+------------------------------+------------+--------------------------------------+--------------------------------------+

(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack resource provider inventory list a3702bd4-04d2-367d-925a-0db0a336a9f6
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| CUSTOM_PCI     |              1.0 |        1 |        1 |        0 |         1 |     1 |    1 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack resource provider inventory list b02fc995-33b4-4b0b-93b7-041a85eb8b32
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
| VCPU           |              4.0 |        1 |        4 |        0 |         1 |     4 |    1 |
| MEMORY_MB      |              1.0 |        1 |    11957 |      512 |         1 | 11957 |  512 |
| DISK_GB        |              1.0 |        1 |      144 |        0 |         1 |   144 |    1 |
+----------------+------------------+----------+----------+----------+-----------+-------+------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device profile list
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:41: EventletDeprecationWarning: 
Eventlet is deprecated. It is currently being maintained in bugfix mode, and
we strongly recommend against using it for new projects.

If you are already using Eventlet, we recommend migrating to a different
framework.  For more detail see
https://eventlet.readthedocs.io/en/latest/asyncio/migration.html

import eventlet
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
| uuid                                 | name       | groups                                                                          | description |
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
| 32a5bf34-a7f9-419d-939e-143eef0e04cd | bug2017513 | [{'resources:CUSTOM_PCI': '1', 'trait:CUSTOM_PCI_PRODUCT_ID_0010': 'required'}] | None        |
+--------------------------------------+------------+---------------------------------------------------------------------------------+-------------+
(venv) stack@cyborg-devstack:/opt/stack/cyborg$ openstack accelerator device profile delete 32a5bf34-a7f9-419d-939e-143eef0e04cd
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:41: EventletDeprecationWarning: 
Eventlet is deprecated. It is currently being maintained in bugfix mode, and
we strongly recommend against using it for new projects.

If you are already using Eventlet, we recommend migrating to a different
framework.  For more detail see
https://eventlet.readthedocs.io/en/latest/asyncio/migration.html

import eventlet
HttpException: 500: Server Error for url: http://192.168.122.143/accelerator/v2/device_profiles/32a5bf34-a7f9-419d-939e-143eef0e04cd, Remote error: DBReferenceError (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`cyborg`.`extended_accelerator_requests`, CONSTRAINT `extended_accelerator_requests_ibfk_1` FOREIGN KEY (`device_profile_id`) REFERENCES `device_profiles` (`id`) ON DELETE RESTRICT)')
[SQL: DELETE FROM device_profiles WHERE device_profiles.uuid = %(uuid_1)s]
[parameters: {'uuid_1': '32a5bf34-a7f9-419d-939e-143eef0e04cd'}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
['Traceback (most recent call last):\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/cursors.py", line 153, in execute\n    result = self._query(query)\n             ^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/cursors.py", line 322, in _query\n    conn.query(q)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 575, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 826, in _read_query_result\n    result.read()\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 1203, in read\n    first_packet = self.connection._read_packet()\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 782, in _read_packet\n    packet.raise_for_error()\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`cyborg`.`extended_accelerator_requests`, CONSTRAINT `extended_accelerator_requests_ibfk_1` FOREIGN KEY (`device_profile_id`) REFERENCES `device_profiles` (`id`) ON DELETE RESTRICT)')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n    result = func(ctxt, **new_args)\n             ^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 68, in device_profile_delete\n    obj_devprof.destroy(context)\n', '  File "/opt/stack/cyborg/cyborg/objects/device_profile.py", line 115, in destroy\n    self.dbapi.device_profile_delete(context, self.uuid)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/api.py", line 144, in wrapper\n    with excutils.save_and_reraise_exception() as ectxt:\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 271, in __exit__\n    self.force_reraise()\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 233, in force_reraise\n    raise self.value\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/api.py", line 142, in wrapper\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1159, in wrapper\n    return fn(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/cyborg/cyborg/db/sqlalchemy/api.py", line 683, in device_profile_delete\n    count = query.delete()\n            ^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 3217, in delete\n    self.session.execute(\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute\n    return self._execute_internal(\n           ^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal\n    result: Result[Any] = compile_state_cls.orm_execute_statement(\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/orm/bulk_persistence.py", line 2033, in orm_execute_statement\n    return super().orm_execute_statement(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement\n    result = conn.execute(\n             ^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute\n    return meth(\n           ^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection\n    return connection._execute_clauseelement(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement\n    ret = self._execute_context(\n          ^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context\n    return self._exec_single_context(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context\n    self._handle_dbapi_exception(\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2360, in _handle_dbapi_exception\n    raise newraise.with_traceback(exc_info[2]) from e\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/cursors.py", line 153, in execute\n    result = self._query(query)\n             ^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/cursors.py", line 322, in _query\n    conn.query(q)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 575, in query\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 826, in _read_query_result\n    result.read()\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 1203, in read\n    first_packet = self.connection._read_packet()\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/connections.py", line 782, in _read_packet\n    packet.raise_for_error()\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error\n    err.raise_mysql_exception(self._data)\n', '  File "/opt/stack/data/venv/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception\n    raise errorclass(errno, errval)\n', "oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`cyborg`.`extended_accelerator_requests`, CONSTRAINT `extended_accelerator_requests_ibfk_1` FOREIGN KEY (`device_profile_id`) REFERENCES `device_profiles` (`id`) ON DELETE RESTRICT)')\n[SQL: DELETE FROM device_profiles WHERE device_profiles.uuid = %(uuid_1)s]\n[parameters: {'uuid_1': '32a5bf34-a7f9-419d-939e-143eef0e04cd'}]\n(Background on this error at: https://sqlalche.me/e/20/gkpj)\n"].

Remove the pci device from cyborg.conf and reran the cyborg-agent

(venv) stack@cyborg-devstack:/opt/stack/cyborg$ cyborg-agent --config-file=/etc/cyborg/cyborg.conf

026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task [-] Error during AgentManager.update_available_resource: cyborg.common.exception_Remote.ResourceProviderInUse_Remote: Resource provider has allocations.
Traceback (most recent call last):

File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming
    res = self.dispatcher.dispatch(message)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
             ^^^^^^^^^^^^^^^^^^^^^^

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 123, in report_data
    self.drv_device_make_diff(

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 158, in drv_device_make_diff
    self._delete_provider_and_sub_providers(context, rp_uuid)

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 484, in _delete_provider_and_sub_providers
    self.placement_client.delete_provider(rp["uuid"])

File "/opt/stack/cyborg/cyborg/common/placement_client.py", line 378, in delete_provider
    raise exception.ResourceProviderInUse()

cyborg.common.exception.ResourceProviderInUse: Resource provider has allocations.
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task Traceback (most recent call last):
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     task(self, context)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/agent/manager.py", line 190, in update_available_resource
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     self._rt.update_usage(context)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 528, in inner
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/agent/resource_tracker.py", line 78, in update_usage
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     self.conductor_api.report_data(context, self.host, acc_list)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/rpcapi.py", line 55, in report_data
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     cctxt.call(
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py", line 180, in call
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     result = self.transport._send(
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task              ^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/transport.py", line 123, in _send
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     return self._driver.send(target, ctxt, message,
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 800, in send
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     return self._send(target, ctxt, message, wait_for_reply, timeout,
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 792, in _send
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     raise result
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task cyborg.common.exception_Remote.ResourceProviderInUse_Remote: Resource provider has allocations.
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task Traceback (most recent call last):
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     res = self.dispatcher.dispatch(message)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     return self._do_dispatch(endpoint, method, ctxt, args)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     result = func(ctxt, **new_args)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task              ^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 123, in report_data
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     self.drv_device_make_diff(
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 158, in drv_device_make_diff
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     self._delete_provider_and_sub_providers(context, rp_uuid)
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 484, in _delete_provider_and_sub_providers
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     self.placement_client.delete_provider(rp["uuid"])
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/common/placement_client.py", line 378, in delete_provider
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task     raise exception.ResourceProviderInUse()
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task cyborg.common.exception.ResourceProviderInUse: Resource provider has allocations.
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.687 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:42:16.692 117742 DEBUG oslo.service.backend._threading.loopingcall [-] Dynamic interval looping call 'cyborg.agent.manager.AgentManager.periodic_tasks' sleeping for 60.00 seconds _run_loop /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/loopingcall.py:125
2026-05-06 05:43:16.693 117742 DEBUG oslo_service.periodic_task [-] Running periodic task AgentManager.update_available_resource run_periodic_tasks /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/periodic_task.py:210
2026-05-06 05:43:16.693 117742 DEBUG oslo_concurrency.lockutils [-] Acquiring lock "agent_resources" by "cyborg.agent.resource_tracker.ResourceTracker.update_usage" inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506
2026-05-06 05:43:16.693 117742 DEBUG oslo_concurrency.lockutils [-] Lock "agent_resources" acquired by "cyborg.agent.resource_tracker.ResourceTracker.update_usage" :: waited 0.000s inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519
2026-05-06 05:43:17.199 117742 DEBUG oslo_concurrency.lockutils [-] Lock "agent_resources" "released" by "cyborg.agent.resource_tracker.ResourceTracker.update_usage" :: held 0.506s inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task [-] Error during AgentManager.update_available_resource: cyborg.common.exception_Remote.ResourceProviderInUse_Remote: Resource provider has allocations.
Traceback (most recent call last):

File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
             ^^^^^^^^^^^^^^^^^^^^^^

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 123, in report_data
    self.drv_device_make_diff(

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 158, in drv_device_make_diff
    self._delete_provider_and_sub_providers(context, rp_uuid)

File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 484, in _delete_provider_and_sub_providers
    self.placement_client.delete_provider(rp["uuid"])

File "/opt/stack/cyborg/cyborg/common/placement_client.py", line 378, in delete_provider
    raise exception.ResourceProviderInUse()

cyborg.common.exception.ResourceProviderInUse: Resource provider has allocations.
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task Traceback (most recent call last):
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     task(self, context)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/agent/manager.py", line 190, in update_available_resource
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     self._rt.update_usage(context)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 528, in inner
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/agent/resource_tracker.py", line 78, in update_usage
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     self.conductor_api.report_data(context, self.host, acc_list)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/rpcapi.py", line 55, in report_data
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     cctxt.call(
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py", line 180, in call
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     result = self.transport._send(
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task              ^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/transport.py", line 123, in _send
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     return self._driver.send(target, ctxt, message,
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 800, in send
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     return self._send(target, ctxt, message, wait_for_reply, timeout,
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 792, in _send
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     raise result
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task cyborg.common.exception_Remote.ResourceProviderInUse_Remote: Resource provider has allocations.
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task Traceback (most recent call last):
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     res = self.dispatcher.dispatch(message)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     return self._do_dispatch(endpoint, method, ctxt, args)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     result = func(ctxt, **new_args)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task              ^^^^^^^^^^^^^^^^^^^^^^
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 123, in report_data
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     self.drv_device_make_diff(
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 158, in drv_device_make_diff
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     self._delete_provider_and_sub_providers(context, rp_uuid)
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/conductor/manager.py", line 484, in _delete_provider_and_sub_providers
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     self.placement_client.delete_provider(rp["uuid"])
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task   File "/opt/stack/cyborg/cyborg/common/placement_client.py", line 378, in delete_provider
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task     raise exception.ResourceProviderInUse()
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task cyborg.common.exception.ResourceProviderInUse: Resource provider has allocations.
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.200 117742 ERROR oslo_service.periodic_task 
2026-05-06 05:43:17.202 117742 DEBUG oslo.service.backend._threading.loopingcall [-] Dynamic interval looping call 'cyborg.agent.manager.AgentManager.periodic_tasks' sleeping for 60.00 seconds _run_loop /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/loopingcall.py:125

I can see similar traceback but not sure, it is the exact usecase I am trying to reproduce.

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2026-05-06:

just putting this here for future us

15:04 <sean-k-mooney> https://github.com/openstack/nova/commit/26c41eccade6412f61f9a8721d853b545061adcc https://github.com/openstack/nova/commit/284ea72e96604bdf16d1c5c4db47247334841b2f https://github.com/openstack/nova/commit/0208be629c3853863bcd49b8bdbe2b9889b85012 https://github.com/openstack/nova/commit/f37cdf0c4182103ad81dbf39188ff39955da3850
15:04 <sean-k-mooney> those are the nova patches related to the isseu reproted in https://bugs.launchpad.net/openstack-cyborg/+bug/2017513
15:05 <sean-k-mooney> https://bugs.launchpad.net/nova/+bug/1633120 https://bugs.launchpad.net/nova/+bug/1969496 and https://bugs.launchpad.net/nova/+bug/2115905 are the releated nova bugs we had
15:07 <sean-k-mooney> the tl;dr is if a device is refence by a ARQ and that device is not in the whitelist or viaabel on teh host anymore we cannot remove the device form the db or placmeent until that ARQ is deleted and we shoudl not do that automaticlly
15:07 <sean-k-mooney> the admin need to move or delete the vm or readd the device
15:08 <sean-k-mooney> we shoudl complain very very loadly in teh logs when the compute agent start up in a miscondifured state but we dont geenrally want to make that an agent startup failure as that a potical dos vector if we do

the tldr is we shoudl not allwo removing trackign fo a device that is currently allocated even if the driver noloanger allows that device to be manged.

removign it is operator error but it shoudl not cause the agent to fail to start or the sync comamdn to fail.

this will requrie some non triival tought to resolve correctly.

nova's pci tracker had a simialr desgin issue that was fixed but those fixes never made it to cybrogs version

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.