fix(scorecard): dangerous-workflow, signed-releases, pinned-deps#356
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add EU AI Act, Colorado AI Act, and GPAI obligations timeline with AGT coverage mapping. Reference Microsoft Purview DSPM for AI as complementary data governance layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Scorecard API rejects workflows with write permissions at the workflow level. id-token: write and security-events: write must be scoped to the job level only. Restores permissions: read-all at workflow level while keeping job-level write permissions intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ft#324) Add Google-style docstrings with Args, Returns, Raises, Attributes, and Example sections to MCPMessageType, MCPAdapter, and MCPServer classes. Also enhances docstrings for key methods including handle_message, _handle_tools_call, _handle_resources_read, and _map_tool_to_action. Fixes microsoft#316
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
…s (dependency confusion) (microsoft#325) - Replace !pip install agent-os with !pip install -e ../.. in all 6 notebooks; agent-os is not on PyPI and installing it from PyPI is a dependency confusion vector - Replace zendesk-sdk/freshdesk-sdk with zenpy/freshdesk (the real published SDKs) in customer-service/requirements.txt - Remove hashlib-compat from healthcare-hipaa/requirements.txt; hashlib is stdlib and hashlib-compat is not a real PyPI package
…stall agent-os with agent-os-kernel Replace all remaining instances of `pip install agent-os` (unregistered on PyPI) with `pip install agent-os-kernel` (the actual package) across docs, examples, TypeScript extensions, CLI source, tests, and SVG assets. Also fixes `pip install emk` references to point to `agent-os-kernel[full]` since emk is a submodule, not a standalone PyPI package. Completes the fix started in PR microsoft#325 which only covered notebooks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dify 65K→133K, AutoGen 42K→55K, CrewAI 28K→46K, Semantic Kernel 24K→27K, LangGraph 24K→27K, Haystack 22K→24K, Agent Framework 7.6K→8K. Added star counts for OpenAI Agents SDK (20K) and Google ADK (18K). Sorted by stars descending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…it CI - scripts/check_dependency_confusion.py: Pre-commit hook that scans for pip install commands referencing unregistered PyPI packages. Maintains an allowlist of known registered packages. - .github/workflows/weekly-security-audit.yml: Weekly CI job running dependency confusion scan, security skills scan, and weak crypto check. Uploads reports as artifacts with 90-day retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#349) * docs: add testing guide for external testers and customers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add regulatory alignment table and Purview positioning to README Add EU AI Act, Colorado AI Act, and GPAI obligations timeline with AGT coverage mapping. Reference Microsoft Purview DSPM for AI as complementary data governance layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): restore read-all at workflow level for Scorecard verification The Scorecard API rejects workflows with write permissions at the workflow level. id-token: write and security-events: write must be scoped to the job level only. Restores permissions: read-all at workflow level while keeping job-level write permissions intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add comprehensive docstrings to mcp_adapter.py classes (microsoft#324) Add Google-style docstrings with Args, Returns, Raises, Attributes, and Example sections to MCPMessageType, MCPAdapter, and MCPServer classes. Also enhances docstrings for key methods including handle_message, _handle_tools_call, _handle_resources_read, and _map_tool_to_action. Fixes microsoft#316 * ci: add markdown link checker workflow (microsoft#323) Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * feat: add policy evaluation heatmap to SRE dashboard (microsoft#309) (microsoft#326) * fix: remove unregistered PyPI packages from notebooks and requirements (dependency confusion) (microsoft#325) - Replace !pip install agent-os with !pip install -e ../.. in all 6 notebooks; agent-os is not on PyPI and installing it from PyPI is a dependency confusion vector - Replace zendesk-sdk/freshdesk-sdk with zenpy/freshdesk (the real published SDKs) in customer-service/requirements.txt - Remove hashlib-compat from healthcare-hipaa/requirements.txt; hashlib is stdlib and hashlib-compat is not a real PyPI package * fix(security): complete dependency confusion fix — replace all pip install agent-os with agent-os-kernel Replace all remaining instances of `pip install agent-os` (unregistered on PyPI) with `pip install agent-os-kernel` (the actual package) across docs, examples, TypeScript extensions, CLI source, tests, and SVG assets. Also fixes `pip install emk` references to point to `agent-os-kernel[full]` since emk is a submodule, not a standalone PyPI package. Completes the fix started in PR microsoft#325 which only covered notebooks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update framework star counts to current values Dify 65K→133K, AutoGen 42K→55K, CrewAI 28K→46K, Semantic Kernel 24K→27K, LangGraph 24K→27K, Haystack 22K→24K, Agent Framework 7.6K→8K. Added star counts for OpenAI Agents SDK (20K) and Google ADK (18K). Sorted by stars descending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Parsa Faraji Alamouti <165321600+parsa-faraji@users.noreply.github.com> Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com> Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Zeel Desai <72783325+zeel2104@users.noreply.github.com> Co-authored-by: Xavier Garceau-Aranda <xavier.garceau-aranda@posteo.net>
- agentmesh → agentmesh-platform (5 files) - agentmesh-governance → agent-governance-toolkit - agent-os-observability → agent-os-kernel[observability] - hashlib.md5 → hashlib.sha256 (3 files) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ore, pin CI deps Dangerous-Workflow: Remove ref: base.sha from pull_request_target workflows — default checkout IS the base branch, explicit ref triggers Scorecard false positive. Security preserved via persist-credentials: false + MSRC comments. Signed-Releases: Add sigstore/gh-action-sigstore-python to publish workflow for Python package signing alongside existing attest-build-provenance. Pinned-Dependencies: Pin pip install versions in CI workflows: - ci.yml: pytest==8.4.1, pytest-asyncio==0.26.0 - benchmarks.yml: add --no-cache-dir - security-scan.yml: pyyaml==6.0.2 - weekly-security-audit.yml: pyyaml==6.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🤖 AI Agent: code-reviewer
Review Summary
This PR addresses three key areas for improving the repository's security posture and compliance with the OpenSSF Scorecard checks: Dangerous Workflow, Signed Releases, and Pinned Dependencies. The changes are well-documented and aim to mitigate potential security risks, such as untrusted code execution in GitHub Actions and dependency drift. However, there are some areas that require further attention to ensure robustness and avoid potential issues.
Key Issues and Recommendations
🔴 CRITICAL: Missing Verification of Sigstore Signatures
The addition of Sigstore for signing release artifacts is a positive step toward supply chain security. However, there is no evidence in the PR that the signatures are being verified during the release process or by downstream consumers. Without verification, the signing process does not provide meaningful security guarantees.
Recommendation:
- Add a step to verify Sigstore signatures during the release process or provide documentation for downstream consumers on how to verify the signatures.
🔴 CRITICAL: Lack of Integrity Verification for pip install
While pinning dependencies improves reproducibility, the pip install commands in the workflows do not include hash-based integrity checks (e.g., --require-hashes). This leaves the pipeline vulnerable to supply chain attacks, such as dependency confusion or compromised PyPI packages.
Recommendation:
- Use
pip install --require-hasheswith arequirements.txtfile that includes hashes for all dependencies. This ensures that only the expected versions of dependencies are installed.
🔴 CRITICAL: Potential Sandbox Escape in benchmarks.yml
The benchmarks.yml workflow installs dependencies using pip install -e ".[dev]". This allows the execution of arbitrary code during the installation process if the repository contains malicious code in its setup.py or pyproject.toml.
Recommendation:
- Use a virtual environment or containerized environment for running benchmarks to isolate the execution environment.
- Consider using a read-only filesystem for the container to prevent unintended modifications.
🟡 WARNING: Potential Breaking Changes in Dependency Versions
The PR pins pytest to version 8.4.1 and pytest-asyncio to version 0.26.0. While this improves reproducibility, it may cause issues for users who rely on newer versions of these dependencies.
Recommendation:
- Clearly document the rationale for pinning these versions in the
CHANGELOG.mdand/orREADME.md. - Consider using semantic versioning ranges (e.g.,
pytest>=8.4,<9.0) to allow for minor updates while avoiding breaking changes.
🟡 WARNING: Backward Compatibility for pyyaml Version Pinning
The pyyaml dependency is pinned to version 6.0.2. This may break compatibility for users who rely on other versions of pyyaml.
Recommendation:
- Evaluate whether pinning
pyyamlto a specific version is strictly necessary. If it is, document the reason for this decision in theCHANGELOG.mdand/orREADME.md.
💡 SUGGESTION: Use actions/checkout@v4 Instead of Commit Hash
The workflows use a specific commit hash (11bd71901bbe5b1630ceea73d27597364c9af683) for the actions/checkout action. While this ensures stability, it may lead to missing out on important updates or security patches.
Recommendation:
- Use the latest stable version tag (e.g.,
actions/checkout@v4) to balance stability and security.
💡 SUGGESTION: Add Tests for Policy Engine Changes
While this PR does not directly modify the policy engine, it is critical to ensure that the changes to CI/CD workflows do not inadvertently affect the correctness of the policy engine.
Recommendation:
- Add automated tests to validate the policy engine's behavior under different scenarios, especially for edge cases.
💡 SUGGESTION: Document MSRC Case Reference
The comments referencing "MSRC Case 111178" are helpful for context but may not be accessible to all contributors.
Recommendation:
- Provide a brief explanation or link to public documentation (if available) about the MSRC case to help contributors understand the rationale behind these changes.
Final Assessment
The PR makes significant improvements to the repository's security posture, particularly in addressing dangerous workflows and adding signed releases. However, critical gaps in signature verification and dependency integrity checks must be addressed before merging. Additionally, the potential breaking changes introduced by dependency pinning should be carefully documented to ensure backward compatibility.
Action Items
-
🔴 Address critical issues:
- Add Sigstore signature verification.
- Implement hash-based integrity checks for
pip install. - Isolate benchmark execution to prevent sandbox escapes.
-
🟡 Mitigate potential breaking changes:
- Document the rationale for dependency pinning.
- Consider using semantic versioning ranges where possible.
-
💡 Implement suggestions:
- Use stable version tags for GitHub Actions.
- Add tests for policy engine correctness.
- Provide context for MSRC case references.
🤖 AI Agent: security-scanner — Security Review of Pull RequestSecurity Review of Pull RequestThis pull request addresses several security-related improvements in the GitHub Actions workflows, including changes to address dangerous workflows, enforce signed releases, and pin dependencies. Below is a detailed security review of the changes. Findings1. Dangerous Workflow Mitigation
2. Signed Releases
3. Pinned Dependencies
4. Use of
|
| Finding | Risk | Recommendation |
|---|---|---|
| Dangerous Workflow Mitigation | 🔴 CRITICAL | Ensure no workflows use ref: head.sha in pull_request_target context. |
| Signed Releases | 🟠 HIGH | Add a verification step to confirm the validity of signatures after signing. |
| Pinned Dependencies | 🟡 MEDIUM | Use a dependency management tool to automate updates and audits of dependencies. |
Use of --no-cache-dir in pip install |
🔵 LOW | No further action required. |
| SPDX License Compliance | 🔵 LOW | Periodically audit dependencies for license compliance. |
Final Assessment
This pull request makes significant improvements to the security posture of the repository by addressing dangerous workflows, introducing signed releases, and pinning dependencies. However, there are still areas for improvement, particularly around verifying signed artifacts and automating dependency updates.
- Overall Risk Rating: 🟠 HIGH
- Approval Recommendation: Approve with the condition that a follow-up PR addresses the recommendations for signed release verification and dependency management.
…rosoft#356) * docs: add testing guide for external testers and customers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add regulatory alignment table and Purview positioning to README Add EU AI Act, Colorado AI Act, and GPAI obligations timeline with AGT coverage mapping. Reference Microsoft Purview DSPM for AI as complementary data governance layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): restore read-all at workflow level for Scorecard verification The Scorecard API rejects workflows with write permissions at the workflow level. id-token: write and security-events: write must be scoped to the job level only. Restores permissions: read-all at workflow level while keeping job-level write permissions intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add comprehensive docstrings to mcp_adapter.py classes (microsoft#324) Add Google-style docstrings with Args, Returns, Raises, Attributes, and Example sections to MCPMessageType, MCPAdapter, and MCPServer classes. Also enhances docstrings for key methods including handle_message, _handle_tools_call, _handle_resources_read, and _map_tool_to_action. Fixes microsoft#316 * ci: add markdown link checker workflow (microsoft#323) Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * feat: add policy evaluation heatmap to SRE dashboard (microsoft#309) (microsoft#326) * fix: remove unregistered PyPI packages from notebooks and requirements (dependency confusion) (microsoft#325) - Replace !pip install agent-os with !pip install -e ../.. in all 6 notebooks; agent-os is not on PyPI and installing it from PyPI is a dependency confusion vector - Replace zendesk-sdk/freshdesk-sdk with zenpy/freshdesk (the real published SDKs) in customer-service/requirements.txt - Remove hashlib-compat from healthcare-hipaa/requirements.txt; hashlib is stdlib and hashlib-compat is not a real PyPI package * fix(security): complete dependency confusion fix — replace all pip install agent-os with agent-os-kernel Replace all remaining instances of `pip install agent-os` (unregistered on PyPI) with `pip install agent-os-kernel` (the actual package) across docs, examples, TypeScript extensions, CLI source, tests, and SVG assets. Also fixes `pip install emk` references to point to `agent-os-kernel[full]` since emk is a submodule, not a standalone PyPI package. Completes the fix started in PR microsoft#325 which only covered notebooks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update framework star counts to current values Dify 65K→133K, AutoGen 42K→55K, CrewAI 28K→46K, Semantic Kernel 24K→27K, LangGraph 24K→27K, Haystack 22K→24K, Agent Framework 7.6K→8K. Added star counts for OpenAI Agents SDK (20K) and Google ADK (18K). Sorted by stars descending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(security): add dependency confusion pre-commit hook + weekly audit CI - scripts/check_dependency_confusion.py: Pre-commit hook that scans for pip install commands referencing unregistered PyPI packages. Maintains an allowlist of known registered packages. - .github/workflows/weekly-security-audit.yml: Weekly CI job running dependency confusion scan, security skills scan, and weak crypto check. Uploads reports as artifacts with 90-day retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(security): proactive audit — dependency confusion + MD5→SHA-256 (microsoft#349) * docs: add testing guide for external testers and customers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add regulatory alignment table and Purview positioning to README Add EU AI Act, Colorado AI Act, and GPAI obligations timeline with AGT coverage mapping. Reference Microsoft Purview DSPM for AI as complementary data governance layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): restore read-all at workflow level for Scorecard verification The Scorecard API rejects workflows with write permissions at the workflow level. id-token: write and security-events: write must be scoped to the job level only. Restores permissions: read-all at workflow level while keeping job-level write permissions intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add comprehensive docstrings to mcp_adapter.py classes (microsoft#324) Add Google-style docstrings with Args, Returns, Raises, Attributes, and Example sections to MCPMessageType, MCPAdapter, and MCPServer classes. Also enhances docstrings for key methods including handle_message, _handle_tools_call, _handle_resources_read, and _map_tool_to_action. Fixes microsoft#316 * ci: add markdown link checker workflow (microsoft#323) Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * feat: add policy evaluation heatmap to SRE dashboard (microsoft#309) (microsoft#326) * fix: remove unregistered PyPI packages from notebooks and requirements (dependency confusion) (microsoft#325) - Replace !pip install agent-os with !pip install -e ../.. in all 6 notebooks; agent-os is not on PyPI and installing it from PyPI is a dependency confusion vector - Replace zendesk-sdk/freshdesk-sdk with zenpy/freshdesk (the real published SDKs) in customer-service/requirements.txt - Remove hashlib-compat from healthcare-hipaa/requirements.txt; hashlib is stdlib and hashlib-compat is not a real PyPI package * fix(security): complete dependency confusion fix — replace all pip install agent-os with agent-os-kernel Replace all remaining instances of `pip install agent-os` (unregistered on PyPI) with `pip install agent-os-kernel` (the actual package) across docs, examples, TypeScript extensions, CLI source, tests, and SVG assets. Also fixes `pip install emk` references to point to `agent-os-kernel[full]` since emk is a submodule, not a standalone PyPI package. Completes the fix started in PR microsoft#325 which only covered notebooks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update framework star counts to current values Dify 65K→133K, AutoGen 42K→55K, CrewAI 28K→46K, Semantic Kernel 24K→27K, LangGraph 24K→27K, Haystack 22K→24K, Agent Framework 7.6K→8K. Added star counts for OpenAI Agents SDK (20K) and Google ADK (18K). Sorted by stars descending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Parsa Faraji Alamouti <165321600+parsa-faraji@users.noreply.github.com> Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com> Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Zeel Desai <72783325+zeel2104@users.noreply.github.com> Co-authored-by: Xavier Garceau-Aranda <xavier.garceau-aranda@posteo.net> * fix(security): replace unregistered package names + MD5→SHA-256 - agentmesh → agentmesh-platform (5 files) - agentmesh-governance → agent-governance-toolkit - agent-os-observability → agent-os-kernel[observability] - hashlib.md5 → hashlib.sha256 (3 files) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(scorecard): remove explicit ref in pull_request_target, add sigstore, pin CI deps Dangerous-Workflow: Remove ref: base.sha from pull_request_target workflows — default checkout IS the base branch, explicit ref triggers Scorecard false positive. Security preserved via persist-credentials: false + MSRC comments. Signed-Releases: Add sigstore/gh-action-sigstore-python to publish workflow for Python package signing alongside existing attest-build-provenance. Pinned-Dependencies: Pin pip install versions in CI workflows: - ci.yml: pytest==8.4.1, pytest-asyncio==0.26.0 - benchmarks.yml: add --no-cache-dir - security-scan.yml: pyyaml==6.0.2 - weekly-security-audit.yml: pyyaml==6.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Parsa Faraji Alamouti <165321600+parsa-faraji@users.noreply.github.com> Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com> Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Zeel Desai <72783325+zeel2104@users.noreply.github.com> Co-authored-by: Xavier Garceau-Aranda <xavier.garceau-aranda@posteo.net>
Full Scorecard sweep targeting 3 checks: Dangerous-Workflow 0->10 (removed explicit ref in pull_request_target), Signed-Releases 0->10 (added sigstore), Pinned-Dependencies 5->7+ (pinned CI pip versions). 11 files changed. Branch-Protection needs admin settings change separately.