Bug Report
What did you do?
I am using the kuttl scorecard image (i.e., quay.io/operator-framework/scorecard-test-kuttl:v2.0.0) and in my config.yaml I have two test suites. When I run the scorecard command using the test selection, the scorecard actually runs all the tests regardless the selection I specify.
operator-sdk scorecard ./bundle --selector='suite=sanity-suite' --namespace=<ns_name> --skip-cleanup --wait-time <timeout> --service-account <SA_name>
This is the config.yaml:
apiVersion: scorecard.operatorframework.io/v1alpha3
kind: Configuration
metadata:
name: config
stages:
- parallel: true
tests:
- entrypoint:
- entrypoint
- 00-ods-ci-settings
- 01-smoke
image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
labels:
cluster-product: ocp
cluster-size: small
phase: msp-main
test: smoke
suite: smoke-suite
storage:
spec:
mountPath: {}
- entrypoint:
- entrypoint
- 00-ods-ci-settings
- 02-sanity
image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
labels:
cluster-product: ocp
cluster-size: small
phase: msp-main
test: sanity
suite: sanity-suite
storage:
spec:
mountPath: {}
What did you expect to see?
I would expect to see only 00-ods-ci-settings and 02-sanity running
What did you see instead? Under which circumstances?
I see instead both the test suites running:
Environment
Operator type:
Kubernetes cluster type:
Openshift 4.10.12
$ operator-sdk version
operator-sdk version: "v1.19.1", commit: "079d8852ce5b42aa5306a1e33f7ca725ec48d0e3"
$ go version (if language is Go)
go version: "go1.17.8"
$ kubectl version
kubectl version: "1.21"
Possible Solution
One possiblesolution would invole replacing lines from 6 to 9 in the entrypoint script with:
if [ $# -eq 0 ]
then
kubectl-kuttl test ${KUTTL_PATH} \
--config=${KUTTL_CONFIG} \
--namespace=${SCORECARD_NAMESPACE} \
--report=JSON --artifacts-dir=/tmp > /tmp/kuttl.stdout 2> /tmp/kuttl.stderr
else
for i in $*; do
kubectl-kuttl test ${KUTTL_PATH} \
--test $i \
--config=${KUTTL_CONFIG} \
--namespace=${SCORECARD_NAMESPACE} \
--report=JSON --artifacts-dir=/tmp > /tmp/kuttl.stdout 2> /tmp/kuttl.stderr
done
fi
One problem to address would be to make the scorecard able to handle more than one kuttl result file
Additional context
Bug Report
What did you do?
I am using the kuttl scorecard image (i.e.,
quay.io/operator-framework/scorecard-test-kuttl:v2.0.0) and in my config.yaml I have two test suites. When I run the scorecard command using the test selection, the scorecard actually runs all the tests regardless the selection I specify.operator-sdk scorecard ./bundle --selector='suite=sanity-suite' --namespace=<ns_name> --skip-cleanup --wait-time <timeout> --service-account <SA_name>This is the config.yaml:
What did you expect to see?
I would expect to see only
00-ods-ci-settingsand02-sanityrunningWhat did you see instead? Under which circumstances?
I see instead both the test suites running:
Environment
Operator type:
Kubernetes cluster type:
Openshift 4.10.12
$ operator-sdk versionoperator-sdk version: "v1.19.1", commit: "079d8852ce5b42aa5306a1e33f7ca725ec48d0e3"
$ go version(if language is Go)go version: "go1.17.8"
$ kubectl versionkubectl version: "1.21"
Possible Solution
One possiblesolution would invole replacing lines from 6 to 9 in the entrypoint script with:
One problem to address would be to make the scorecard able to handle more than one kuttl result file
Additional context