Project

General

Profile

Actions
Copy link

Bug #57655

closed

qa: fs:mixed-clients kernel_untar_build failure

Added by Patrick Donnelly over 3 years ago. Updated 8 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Milind Changire
Category:
Correctness/Safety
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Q/A
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa-failure
Pull request ID:
54414
Tags (freeform):
Merge Commit:
aca254a3ed6e1bfe9b9cc33d57cf68abb23086e4
Fixed In:
v18.0.0-7428-gaca254a3ed6
Released In:
v19.2.0~1202
Upkeep Timestamp:
2025-07-13T00:40:20+00:00

Description

2022-09-12T12:12:00.425 INFO:tasks.workunit.client.1.smithi176.stderr:fs/compat.o: warning: objtool: missing symbol for section .text
2022-09-12T12:12:00.487 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/binfmt_misc.o
2022-09-12T12:12:00.842 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/binfmt_script.o
2022-09-12T12:12:00.980 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/binfmt_elf.o
2022-09-12T12:12:01.273 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/compat_binfmt_elf.o
2022-09-12T12:12:01.278 INFO:tasks.workunit.client.1.smithi176.stdout:  AR      kernel/built-in.a
2022-09-12T12:12:01.714 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/mbcache.o
2022-09-12T12:12:01.739 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/posix_acl.o
2022-09-12T12:12:01.742 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/coredump.o
2022-09-12T12:12:01.777 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/drop_caches.o
2022-09-12T12:12:01.795 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/fhandle.o
2022-09-12T12:12:02.186 INFO:tasks.workunit.client.1.smithi176.stdout:  CC      fs/dcookies.o
2022-09-12T12:12:02.982 INFO:tasks.workunit.client.1.smithi176.stderr:fs/dcookies.o: warning: objtool: missing symbol for section .text
2022-09-12T12:12:02.999 INFO:tasks.workunit.client.1.smithi176.stdout:  AR      fs/built-in.a
2022-09-12T12:12:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 2
2022-09-12T12:12:03.196 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.1...

Seen: /ceph/teuthology-archive/dparmar-2022-09-12_11:38:14-fs:mixed-clients-main-distro-default-smithi/7029223/teuthology.log

and more recently: /ceph/teuthology-archive/pdonnell-2022-09-22_12:22:37-fs-wip-pdonnell-testing-20220920.234701-distro-default-smithi/7041086/teuthology.log

Related issues 3 (0 open — 3 closed)

Copied to CephFS - Backport #63588: pacific: qa: fs:mixed-clients kernel_untar_build failureResolvedMilind ChangireActions
Copied to CephFS - Backport #63590: reef: qa: fs:mixed-clients kernel_untar_build failureResolvedMilind ChangireActions
Copied to CephFS - Backport #66636: quincy: qa: fs:mixed-clients kernel_untar_build failureResolvedMilind ChangireActions
Actions
Copy link
 #1

Updated by Patrick Donnelly over 3 years ago

  • Related to Bug #57280: qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman added
Actions
Copy link
 #2

Updated by Venky Shankar over 3 years ago

  • Related to deleted (Bug #57280: qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman)
Actions
Copy link
 #3

Updated by Venky Shankar over 3 years ago

  • Category set to Correctness/Safety
  • Status changed from New to Triaged
  • Assignee set to Milind Changire
  • Backport set to pacific,quincy
Actions
Copy link
 #7

Updated by Venky Shankar almost 3 years ago

  • Target version changed from v18.0.0 to v19.0.0
  • Backport changed from pacific,quincy to reef,quincy,pacific

- https://pulpito.ceph.com/vshankar-2023-05-12_08:25:27-fs-wip-vshankar-testing-20230509.090020-1-testing-default-smithi/7272427/

Milind, do we know if this (kernel build failure) is a compiler related things or an issue (bug) in cephfs in the IO path?

Actions
Copy link
 #8

Updated by Kotresh Hiremath Ravishankar almost 3 years ago

Milind, PTAL at the discussion in https://tracker.ceph.com/issues/59342 if it helps. I closed that as duplicate of this.

Actions
Copy link
 #11

Updated by Milind Changire almost 3 years ago

kernel_untar_build.sh test passes with latest code (HEAD 17f4abe9c9c) in the main branch
Xiubo pointed that a revert PR helps with that

so we just need to get this backported

Actions
Copy link
 #15

Updated by Venky Shankar over 2 years ago

/a/https://pulpito.ceph.com/vshankar-2023-09-12_06:47:30-fs-wip-vshankar-testing-20230908.065909-testing-default-smithi/7394622/

This time with a slightly different failure (but along the same lines):

2023-09-12T08:12:53.754 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr.o: In function `sr_block_check_events':
2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr.c:(.text+0x1062): undefined reference to `cdrom_check_events'
2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr.o: In function `sr_block_revalidate_disk':
2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr.c:(.text+0x11f2): undefined reference to `cdrom_get_last_written'
2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr_ioctl.o: In function `sr_drive_status':
2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr_ioctl.c:(.text+0x6b0): undefined reference to `cdrom_get_media_event'
2023-09-12T08:12:54.083 INFO:tasks.workunit.client.0.smithi064.stderr:make: *** [Makefile:1077: vmlinux] Error 1
Actions
Copy link
 #17

Updated by Patrick Donnelly over 2 years ago

  • Priority changed from Normal to Immediate
2023-10-26T06:08:42.898 INFO:tasks.workunit.client.1.smithi155.stderr:ld: arch/x86/boot/compressed/pgtable_64.o:(.bss+0x0): multiple definition of `__force_order'; arch/x86/boot/compressed/kaslr_64.o:(.bss+0x0): first defined here
2023-10-26T06:08:42.902 INFO:tasks.workunit.client.1.smithi155.stderr:ld: arch/x86/boot/compressed/head_64.o: warning: relocation in read-only section `.head.text'
2023-10-26T06:08:42.931 INFO:tasks.workunit.client.1.smithi155.stderr:ld: warning: creating DT_TEXTREL in a PIE
2023-10-26T06:08:42.936 INFO:tasks.workunit.client.1.smithi155.stderr:make[2]: *** [arch/x86/boot/compressed/Makefile:118: arch/x86/boot/compressed/vmlinux] Error 1
2023-10-26T06:08:42.937 INFO:tasks.workunit.client.1.smithi155.stderr:make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Actions
Copy link
 #18

Updated by Milind Changire over 2 years ago

Patrick Donnelly wrote:

[...]

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build

e.g. there's no multiple definition of this variable in the testing kernel sources

Actions
Copy link
 #19

Updated by Patrick Donnelly over 2 years ago

Milind Changire wrote:

Patrick Donnelly wrote:

[...]

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build

e.g. there's no multiple definition of this variable in the testing kernel sources

You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.

Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.

Actions
Copy link
 #20

Updated by Milind Changire over 2 years ago

Patrick Donnelly wrote:

Milind Changire wrote:

Patrick Donnelly wrote:

[...]

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build

e.g. there's no multiple definition of this variable in the testing kernel sources

You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.

Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.

centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logs

so this is 100% reproducible for centos9 and ubuntu 22.04 as well

we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 job

Actions
Copy link
 #21

Updated by Venky Shankar over 2 years ago

Milind Changire wrote:

Patrick Donnelly wrote:

Milind Changire wrote:

Patrick Donnelly wrote:

[...]

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build

e.g. there's no multiple definition of this variable in the testing kernel sources

You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.

Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.

centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logs

so this is 100% reproducible for centos9 and ubuntu 22.04 as well

we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 job

Are you saying it was an issue with the kernel tarball the test was fetching and using a latest tarball gives a clean build?

EDIT: Since the current tarball fails build in other distros, then we could update our tests to fetch the latest tarball to be used.

Actions
Copy link
 #22

Updated by Milind Changire over 2 years ago

  • Pull request ID set to 54414

Venky Shankar wrote:

Milind Changire wrote:

Patrick Donnelly wrote:

Milind Changire wrote:

Patrick Donnelly wrote:

[...]

/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log

This one is pretty scary. I'm raising priority.

Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build

e.g. there's no multiple definition of this variable in the testing kernel sources

You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.

Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.

centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logs

so this is 100% reproducible for centos9 and ubuntu 22.04 as well

we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 job

Are you saying it was an issue with the kernel tarball the test was fetching and using a latest tarball gives a clean build?

EDIT: Since the current tarball fails build in other distros, then we could update our tests to fetch the latest tarball to be used.

yes, using a newer tarball fixed the build issue
btw, the old tarball wasn't corrupted, it just a had a 'multiple deinition' of a symbol
I've added the PR number as well

Actions
Copy link
 #25

Updated by Upkeep Bot over 2 years ago

  • Copied to Backport #63588: pacific: qa: fs:mixed-clients kernel_untar_build failure added
Actions
Copy link
 #27

Updated by Upkeep Bot over 2 years ago

  • Copied to Backport #63590: reef: qa: fs:mixed-clients kernel_untar_build failure added
Actions
Copy link
 #29

Updated by Jos Collin over 1 year ago

  • Copied to Backport #66636: quincy: qa: fs:mixed-clients kernel_untar_build failure added
Actions
Copy link
 #30

Updated by Rishabh Dave over 1 year ago

  • Status changed from Pending Backport to Resolved

Marking this as resolved since all backports have been merged.

Actions
Copy link
 #31

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to aca254a3ed6e1bfe9b9cc33d57cf68abb23086e4
  • Fixed In set to v18.0.0-7428-gaca254a3ed6
  • Released In set to v19.2.0~1202
  • Upkeep Timestamp set to 2025-07-13T00:40:20+00:00
Actions
Copy link

Also available in: Atom PDF