Secure testing gap analysis map

This issue shall map features present in Secure against current testing resources.

Any gaps should arise and be raised/triaged for action accordingly.

For the purposes of this ticket, the following are defined:

Scanners

Scanner Unit test Integration test E2E test
Container Scanning - PROTECT STAGE
Threat Monitoring
Dependency Scanning

Threat Monitoring

Feature Unit test Integration test E2E test
Container Network Policy ✅ cluster application; auto-deploy-app; backend models: network policy cilium network policy; backend controller; backend services; frontend: policy editor, network policy list, network policy drawer, threat monitoring sections

Dependency Scanning

Feature Unit test (see matching language/framework in dir) Integration test (see *-qa jobs in linked ci files) E2E test
Ruby Bundler ✅ link ✅ link
PHP Composer ✅ link ✅ link
C Conan ✅ link ✅ link
C++ Conan ✅ link ✅ link
Go Golang ✅ link ✅ link
Maven ✅ link ✅ link
Gradle ✅ link ✅ link
SBT ✅ link ✅ link
Javascript NPM ✅ link ✅ link
.NET nuget ✅ link ✅ link
Javascript Yarn ✅ link ✅ link
C# nuget ✅ link ✅ link
Python setuptools ❌ ❌
Python pip gitlab-org/quality/testcases#1091 (closed) ✅ link ✅ link
Python pipenv gitlab-org/quality/testcases#1091 (closed) ✅ link ✅ link
Python poetry gitlab-org/gitlab#7006 (closed) ✅ link ❌ gitlab-org/gitlab#299294
Maven (test building java 8 to 14) ❌ ✅ link
Gradle (test building java 8 to 14) ❌ ✅ link
Scala SBT (test building java 8 to 14) ❌ ✅ link
Javascript NPM using retire.js scanner ❌ ✅ link
Ruby Bundler using bundler-audit scanner ❌ ✅ link
Environment Variables gitlab-org/gitlab#294208 (closed) , gitlab-org/gitlab#294207 (closed) ❌ ❌
Omnibus (GitLab only) with gitlab-depscan ✅ link ❌
Offline java projects (mvn, gradle, scala) ❌ ✅ (see offline-FREEZE branch in java-maven, java-gradle, scala-sbt test projects)
Offline python projects (pip, pipenv, setuptools ❌ ✅ (see offline-FREEZE branch in python-pip, python-pipenv test projects)
Offline js project (npm, yarn) ❌ ✅ (see offline-FREEZE branch in js-npm, js-yarn test projects)
Offline ruby project (bundler) ❌ ✅ (see offline-FREEZE branch in rub-bundler test project)
Private mvn repo ❌ ❌
Private pypi repo ❌ ❌
Private npm repo ❌ ❌
Private gem repo ❌ ❌
Remediation ❌ ❌ ✅ link
Enable within the UI ( gitlab-org/gitlab#284108 (closed) ) ❌ gitlab-org/quality/testcases#1666 (closed)
Use a different stage (instead of default "test") ❌

Dependency List

Feature Unit test Integration test E2E test
Project level list ✅ frontend controller parser frontend: ❌ backend: ❌ ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/ee/page/project/secure/dependency_list.rb
Click arrow to view dependencies ✅ frontend no need for backend test frontend: ❌ backend: ❌
Dependency Paths ✅ frontend, frontend, backend frontend: ❌ backend: ❌
Licenses are shown in same list ✅ frontend controller frontend: ❌ backend: ❌
Empty state UI ✅ frontend, frontend controller frontend: ❌ backend: ❌ gitlab-org/quality/testcases#1128 (closed)
Vulnerablities related to dependency ✅ controller parser backend: ❌

SAST

Feature Unit test Integration test E2E test
Project level list
.NET Core
.NET Framework
Apex Salesforce
C/C++
Elixir
Go
Groovy Ant
Groovy Gradle
Groovy Maven
Groovy SBT
Helm Charts
Java Ant
Java Gradle
Java Maven
Java SBT
Java Android
JS
Kotlin Android
K8s manifests
NodeJS
ObjectiveC iOS
PHP
Python PIP
React
Ruby on Rails
Scala Ant
Scala Gradle
Scala Maven
Scala SBT
Swift iOS
Typescript
Tier support (CE/Core vs EE/Ultimate)
Environment variables
Kubesec
Offline
bandit
brakeman
eslint
flawfinder
gosec
kubesec
mobsf
nodejs-scan
phpcs-security-audit
pmd-apex
security-code-scan
Enable via UI ✅ component specs 🔶 weak frontend ✅ gitlab-org/quality/testcases#1667 (closed)
Use a different stage (instead of default "test") ❌
False positives https://gitlab.com/gitlab-org/gitlab/-/issues/292686 ❌ gitlab-org/quality/testcases#2259 (closed)

Secret Detection

Feature Unit test Integration test E2E test
Gitleaks ruleset https://docs.gitlab.com/ee/user/application_security/secret_detection/#supported-secrets
Environment variables including Full History Secret Scan
Custom ruleset
Offline
Use a different stage (instead of default "test") ❌

Auto Remediation

Feature Unit test Integration test E2E test
Create remediation MR ✅ backend; frontend - not applicable ❌ ❌
Show solutions in Vulnerability report Frontend: UI-heavy component, covered by the integration tests; backend ✅ frontend Link ❌ backend ❌
Settings ✅ Frontend: UI-heavy component, covered by the integration tests; backend ✅ frontend Link ❌ backend ❌
Available MRs in Project Security Dashboard header ✅ Frontend: UI-heavy component, covered by the integration tests; backend not applicable ✅ frontend Link ❌ backend ❌

DAST

Feature Unit test Integration test E2E test
Project level View details of a DAST vulnerability
Environment Variables
Full / API / URL scans
DAST on demand scans ✅ Frontend gitlab-org/quality/testcases#1122
DAST profiles ✅ Frontend gitlab-org/quality/testcases#1122
Use a different stage (instead of default "dast") ❌

API Fuzzing

Feature Unit test Integration test E2E test
Fuzzing scan
OpenAPI
HTTP Archive
Postman
Environment Variables
Overrides
Header fuzzing
Turn off a check/assertion
Offline
Vulnerability MR widget ✅ Frontend
Vulnerability Detail Page ✅ Frontend
Pipeline Security Tab ✅ Frontend
Security Dashboard ✅ Frontend
Artifacts Download ✅ Frontend
Artifacts
Use a different stage (instead of default "fuzz") ❌

Coverage fuzz testing

Feature Unit test Integration test E2E test
C/C++
GoLang
Swift
Rust
Java JQF
Java javafuzz
Full fuzz
Regression fuzz
Environment variables
Offline
Continuous Fuzzing
Vulnerability MR widget ✅ Frontend
Vulnerability Detail Page ✅ Frontend
Security Dashboard ✅ Frontend
Pipeline Security Tab ✅ Frontend
Fuzzing artifacts ✅ Frontend
Artifacts
Use a different stage (instead of default "fuzz") ❌

License Compliance

License Compliance

Feature Unit test Integration test E2E test
In MR ✅ frontend specs ✅ backend 🔶 weak frontend ❌ backend ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
In Pipeline ✅ frontend specs (same as MR frontend specs above) backend frontend, backend: ❌ ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb
Approve license frontend spec, frontend spec backend frontend, backend: ❌ ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
Deny license frontend spec, frontend spec backend frontend, backend: ❌ ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/merge_request_license_widget_spec.rb
Javascript Bower no test ✅ link
Javascript NPM no test ✅ link
JavaScript Yarn ✅ ✅ link
Go Godep No test ❌
Go mod No test ✅ link
Java Gradle no test ✅ link
Java Maven ✅ link ✅ link
.NET nuget ✅ link ✅ link
Python Pip ✅ link ✅ link
Ruby Gem ✅ link ✅ link
Experimental supported languages only yarn rust, Pipenv
Environment variables ❌ ❌
Custom Dependencies ❌ ❌
License Approvals within a project frontend specs, frontend specs frontend: ❌
License policy tab gitlab-org/gitlab#219461 (comment 446528615) frontend spec frontend: ❌
Empty state dashboard frontend specs frontend: ❌ ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/license_compliance_spec.rb#L33
Use a different stage (instead of default "test") ❌

Dashboards / UI driven

Feature Unit test Integration test E2E test
Project level View details of a DAST vulnerability
MR View details of a DAST vulnerability ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
MR View details of a Dependency Scanning vulnerability ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
MR View details of a Container Scanning vulnerability ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb
Dismiss a vulnerability ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Dismiss multiple vulnerabilities
Create an issue for a vulnerability (nicole assumes w/out edits) ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Create an issue for a vulnerability WITH EDITS 🚧 gitlab-org/quality/testcases#1140 (closed)
New create a JIRA issue for a vulnerability gitlab-org/quality/testcases#1101 (closed)
Automatic remediation for vulnerabilities - DS ✅ see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#auto-remediation ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/vulnerability_management_spec.rb
Automatic remediation for vulnerabilities - CS ❌
Automatic remediation for vulnerabilities - Manually apply the suggested patch
Create MR from a vulnerability
Create MR from a vulnerability twice, check message
Add related issue for a vulnerability
Remove related issue for a vulnerability
Security approvals in MRs
License Approvals within a project see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#license-compliance
Security report out of date
Security Configuration status
Security Configuration Autodevops
Security Configuration SAST
Security Configuration DAST
Dependency List see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/736#dependency-list ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb#L133
Standalone vulnerabilities on Dependency List page ❌ gitlab-org/quality/testcases#1672 (closed)
Security Dashboard Secrets
Pipeline Security tab Secrets
MR widget Secrets
Pipeline Security Dashboard ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Project Security Dashboard graph gitlab-org/quality/testcases#1092
Project Vulnerability Report ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/project_security_dashboard_spec.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Group Security Dashboard ✅ https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
Security Centre
Export
Vulnerability Report
Project Security Dashboard when pipeline is blocked by a manual step (gitlab-org/gitlab#247490 (closed)) ❌

Other?

Feature

Feature Unit test Integration test E2E test
CI YAML templates present/not?
CI for MRs
Autodevops
Feature flag enable/disable MR security widget frontend specs
Change vulnerability status

Schemas

https://gitlab.com/gitlab-org/security-products/security-report-schemas/

Likely E2E candidates:

Feature Unit test Integration test E2E test
Test against minimal fields returned
Test against all possible fields returned

Dashboard

Feature Unit test Integration test E2E test
Dismiss vulnerability
Create issue
Automatic Remediation

Offline

Feature Unit test Integration test E2E test
Private Maven repo
Fuzzing

Out of Scope (Protect Stage)

Container Scanning

Feature Unit test Integration test E2E test
Environment variables
Vulnerability allowlisting
Offline
Edited by Will Meek