Skip to content

PermissionError: [Errno 13] Permission denied: b'/slurm/scripts/template_info.cache.dat' #3946

Closed
Closed
PermissionError: [Errno 13] Permission denied: b'/slurm/scripts/template_info.cache.dat'#3946
Labels
bugSomething isn't workingfixed; not releasedissues that have been fixed on the develop branch but have not yet been part of a tagged release.stale
@casassg

Description

@casassg
casassg
opened on Apr 15, 2025

Describe the bug

Issue with the use of shelve to store cache of the template information on disk.

according to https://docs.python.org/3/library/shelve.html -> The filename specified is the base filename for the underlying database.

Steps to reproduce

Steps to reproduce the behavior:

  1. Use shelve.py on debian-9-6 image
  2. Check file created is template_info.cache.dat instead of template_info.cache
  3. This makes line

    cluster-toolkit/community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/util.py

    Line 1852 in 5254651

    chown_slurm(self.template_cache_path)
    not do as expected.

Expected behavior

No error when updating cache.

Actual behavior

Cache error pollutes logs a bit.

Version (gcluster --version)

gcluster version v1.48.0

Expanded Blueprint

If applicable, please attach or paste the expanded blueprint. The expanded blueprint can be obtained by running gcluster expand your-blueprint.yaml.

Disregard if the bug occurs when running gcluster expand ... as well.



Output and logs


DEFAULT 2025-04-14T21:51:23Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:51:23 smallslurm-controller collectd[1351]: uc_update: Value too old: name = smallslurm-controller/processes-all/disk_octets; value time = 1744667483.605; last cache update = 1744667483.605;
DEFAULT 2025-04-14T21:52:05Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:05 smallslurm-controller systemd[1]: Starting Slurm cluster management daemon...
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ERROR: failed to reconfigure slurm
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: Traceback (most recent call last):
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/util.py", line 1810, in template_cache
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: yield cache
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/util.py", line 1850, in template_info
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: cache[template_name] = template.to_dict()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/shelve.py", line 125, in __setitem__
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self.dict[key.encode(self.keyencoding)] = f.getvalue()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/dbm/dumb.py", line 203, in __setitem__
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self._addkey(key, self._addval(val))
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/dbm/dumb.py", line 161, in _addval
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: with _io.open(self._datfile, 'rb+') as f:
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: PermissionError: [Errno 13] Permission denied: b'/slurm/scripts/template_info.cache.dat'
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: During handling of the above exception, another exception occurred:
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: Traceback (most recent call last):
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/slurmsync.py", line 578, in main
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: reconfigure_slurm()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~~~~~^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/slurmsync.py", line 399, in reconfigure_slurm
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: conf.gen_controller_configs(lookup())
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/conf.py", line 625, in gen_controller_configs
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: gen_cloud_conf(lkp)
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~~^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/conf.py", line 266, in gen_cloud_conf
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: content = make_cloud_conf(lkp)
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/conf.py", line 253, in make_cloud_conf
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: lines = [
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ...<7 lines>...
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ]
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/conf.py", line 256, in <genexpr>
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: *(nodeset_lines(n, lkp) for n in lkp.cfg.nodeset.values()),
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/conf.py", line 135, in nodeset_lines
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: template_info = lkp.template_info(nodeset.instance_template)
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/util.py", line 1849, in template_info
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: with self.template_cache(writeback=True) as cache:
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/contextlib.py", line 162, in __exit__
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self.gen.throw(value)
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~~~^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/scripts/util.py", line 1812, in template_cache
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: cache.close()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/shelve.py", line 144, in close
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self.sync()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/shelve.py", line 168, in sync
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self[key] = entry
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/shelve.py", line 125, in __setitem__
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self.dict[key.encode(self.keyencoding)] = f.getvalue()
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/dbm/dumb.py", line 203, in __setitem__
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: self._addkey(key, self._addval(val))
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~~~~~^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: File "/slurm/python/lib/python3.13/dbm/dumb.py", line 161, in _addval
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: with _io.open(self._datfile, 'rb+') as f:
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-04-14T21:52:20Z [resource.labels.instanceId: smallslurm-controller] Apr 14 21:52:20 smallslurm-controller slurmsync.py[11455]: PermissionError: [Errno 13] Permission denied: b'/slurm/scripts/template_info.cache.dat'
ERROR 2025-04-14T21:52:20.762Z [resource.labels.instanceId: smallslurm-controller] ERROR: failed to reconfigure slurm




Screenshots


Image
Metadata
Metadata
Assignees
No one assigned
    Labels
    bugSomething isn't workingfixed; not releasedissues that have been fixed on the develop branch but have not yet been part of a tagged release.stale
    Type
    No type
    Fields
    No fields configured for issues without a type.
    Projects
    No projects
    Milestone
    No milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions