Ceph » CephFS

Jos, start by checking if the workload isn't heavy enough to trigger subtree export/import (which would then update the respective perf counters). If that's the case check-counters would trip since it expects there counters to be present in `perf dump`. Also, do the same for other counters that show up in failed runs.

Another failed instance - https://pulpito.ceph.com/vshankar-2024-05-06_12:08:33-fs-wip-vshankar-testing-20240506.064357-testing-default-smithi/7693835/

Venky Shankar wrote in #note-2:

Jos, start by checking if the workload isn't heavy enough to trigger subtree export/import (which would then update the respective perf counters). If that's the case check-counters would trip since it expects there counters to be present in `perf dump`. Also, do the same for other counters that show up in failed runs.

I'll check that. check_counter.py and l_mds_imported/l_mds_exported are there since long time. So wondering what made the workload suddenly isn't heavy enough?

https://pulpito.ceph.com/leonidus-2024-05-28_05:46:16-fs-wip-lusov-testing-20240527.173700-squid-distro-default-smithi/7729471/
https://pulpito.ceph.com/leonidus-2024-06-12_09:41:32-fs-wip-lusov-testing-20240611.123850-squid-distro-default-smithi/7751673

Jos Collin wrote in #note-4:

Venky Shankar wrote in #note-2:

Jos, start by checking if the workload isn't heavy enough to trigger subtree export/import (which would then update the respective perf counters). If that's the case check-counters would trip since it expects there counters to be present in `perf dump`. Also, do the same for other counters that show up in failed runs.

I'll check that. check_counter.py and l_mds_imported/l_mds_exported are there since long time. So wondering what made the workload suddenly isn't heavy enough?

Any update on this @Jos Collin ?

Venky Shankar wrote in #note-7:

Jos Collin wrote in #note-4:

Venky Shankar wrote in #note-2:

Jos, start by checking if the workload isn't heavy enough to trigger subtree export/import (which would then update the respective perf counters). If that's the case check-counters would trip since it expects there counters to be present in `perf dump`. Also, do the same for other counters that show up in failed runs.

I'll check that. check_counter.py and l_mds_imported/l_mds_exported are there since long time. So wondering what made the workload suddenly isn't heavy enough?

Any update on this @Jos Collin ?

Not yet. Will continue working on this soon.

Another instance: https://pulpito.ceph.com/vshankar-2024-07-24_07:34:06-fs-wip-vshankar-testing-20240723.143739-debug-testing-default-smithi/7816146

Jos Collin wrote in #note-11:

This is a duplicate of https://tracker.ceph.com/issues/67360. Closing this as https://tracker.ceph.com/issues/67360 contains better debug info.

@Jos Collin This ticket was marked duplicate but we never stopped seeing this failure in QA runs even thought the PR for #67360 has been merged. Perhaps this was separate issue.

https://pulpito.ceph.com/vshankar-2025-02-27_13:15:04-fs-wip-vshankar-testing-20250227.075314-debug-testing-default-smithi/8158516

@Jos Collin Could you RCA please?

@Venky Shankar
This is caused by qa/suites/fs/workload/ranks/multi/export-check.yaml.
Only in this yaml, the max_mds is not set.

Jos Collin wrote in #note-18:

@Venky Shankar
This is caused by qa/suites/fs/workload/ranks/multi/export-check.yaml.
Only in this yaml, the max_mds is not set.

qa/tasks/check_counter.py should get the counters from all active MDSs. Have you checked why directories aren't exported to other ranks? Maybe it's related to the balancer configuration (random, etc..), which might be causing the directories to not be exported. Note that the default balancer is disabled and we specifically turn on some balance in QA. E.g.: ranks/multi/{balancer/random

Update:

From the logs where mds.imported/mds.exported failed to set, it's run by fs/workload/ranks/multi/balancer/random.yaml in all the failed jobs.