Skip to content

feat: Screencap action for pipeline#1165

Merged
MistEO merged 4 commits intomainfrom
feat/screencap
Mar 2, 2026
Merged

feat: Screencap action for pipeline#1165
MistEO merged 4 commits intomainfrom
feat/screencap

Conversation

@MistEO
Copy link
Member

@MistEO MistEO commented Mar 1, 2026

Summary by Sourcery

在流水线系统中新增一种 Screencap 动作类型,用于将当前屏幕截取为图像文件,并在各框架和绑定中对其进行暴露。

新功能:

  • 在流水线动作定义中引入 Screencap 动作类型及其参数,支持 PNG 和 JPEG 格式,并可配置图像质量。
  • 在执行器中实现对控制器缓存图像的截取,将其保存到日志的 screencap 目录下,并返回包含文件路径在内的详细动作结果。
  • 在 Python 和 Node.js 绑定中暴露 Screencap 动作的参数和类型,使流水线可以以编程方式配置和使用该动作。

增强:

  • 扩展流水线解析、导出和默认参数管理,以支持 Screencap 动作及其配置字段。
  • 更新英文版流水线协议文档,描述 Screencap 动作及其选项,并将其添加到可用动作列表和版本历史中。
  • 在贡献者指南中记录 Markdown 换行约定,以确保多行列表项能够正确渲染。

测试:

  • 新增 Python 流水线测试,用于验证 Screencap 动作的解析、默认值和类型映射。
Original summary in English

Summary by Sourcery

Add a new Screencap action type to the pipeline system that captures the current screen to an image file and exposes it across frameworks and bindings.

New Features:

  • Introduce Screencap action type and parameters in pipeline action definitions, including support for PNG and JPEG formats with configurable quality.
  • Implement Actuator support to capture the controller’s cached image, save it under the log screencap directory, and return detailed action results including file path.
  • Expose Screencap action parameters and types in Python and Node.js bindings so pipelines can configure and use this action programmatically.

Enhancements:

  • Extend pipeline parsing, dumping, and default-parameter management to handle the Screencap action and its configuration fields.
  • Update English pipeline protocol documentation to describe the Screencap action, its options, and add it to the list of available actions and version history.
  • Document a Markdown line-break convention in contributor guidelines to ensure correct rendering of multi-line list items.

Tests:

  • Add Python pipeline tests to validate Screencap action parsing, default values, and type mapping.
Original summary in English

Summary by Sourcery

在流水线系统中新增一种 Screencap 动作类型,用于将当前屏幕截取为图像文件,并在各框架和绑定中对其进行暴露。

新功能:

  • 在流水线动作定义中引入 Screencap 动作类型及其参数,支持 PNG 和 JPEG 格式,并可配置图像质量。
  • 在执行器中实现对控制器缓存图像的截取,将其保存到日志的 screencap 目录下,并返回包含文件路径在内的详细动作结果。
  • 在 Python 和 Node.js 绑定中暴露 Screencap 动作的参数和类型,使流水线可以以编程方式配置和使用该动作。

增强:

  • 扩展流水线解析、导出和默认参数管理,以支持 Screencap 动作及其配置字段。
  • 更新英文版流水线协议文档,描述 Screencap 动作及其选项,并将其添加到可用动作列表和版本历史中。
  • 在贡献者指南中记录 Markdown 换行约定,以确保多行列表项能够正确渲染。

测试:

  • 新增 Python 流水线测试,用于验证 Screencap 动作的解析、默认值和类型映射。
Original summary in English

Summary by Sourcery

Add a new Screencap action type to the pipeline system that captures the current screen to an image file and exposes it across frameworks and bindings.

New Features:

  • Introduce Screencap action type and parameters in pipeline action definitions, including support for PNG and JPEG formats with configurable quality.
  • Implement Actuator support to capture the controller’s cached image, save it under the log screencap directory, and return detailed action results including file path.
  • Expose Screencap action parameters and types in Python and Node.js bindings so pipelines can configure and use this action programmatically.

Enhancements:

  • Extend pipeline parsing, dumping, and default-parameter management to handle the Screencap action and its configuration fields.
  • Update English pipeline protocol documentation to describe the Screencap action, its options, and add it to the list of available actions and version history.
  • Document a Markdown line-break convention in contributor guidelines to ensure correct rendering of multi-line list items.

Tests:

  • Add Python pipeline tests to validate Screencap action parsing, default values, and type mapping.

@MistEO MistEO marked this pull request as ready for review March 1, 2026 09:53
Copilot AI review requested due to automatic review settings March 1, 2026 09:53
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - 我发现了两个问题,并给出了一些整体层面的反馈:

  • Actuator::screencap 中,建议在调用 imwrite 之前,先确保目标目录 log_dir/screencap 已经存在(例如通过 create_directories),否则在目录缺失时该操作会静默失败。
  • 解析器和动作处理器目前接受任意的 format 字符串和任意的 quality 值;既然文档中已经约定为 png|jpg|jpeg0–100,更安全的做法是对这些字段做校验和/或钳制,并在提供无效值时进行日志记录。
  • Actuator::screencap 中的 imwrite 调用失败时,日志信息仍然是 screencap saved to;可以考虑记录更清晰的错误路径(如果有原因的话也一并记录),以便更容易定位截屏失败的原因。
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
-`Actuator::screencap` 中,建议在调用 `imwrite` 之前,先确保目标目录 `log_dir/screencap` 已经存在(例如通过 `create_directories`),否则在目录缺失时该操作会静默失败。
- 解析器和动作处理器目前接受任意的 `format` 字符串和任意的 `quality` 值;既然文档中已经约定为 `png|jpg|jpeg``0–100`,更安全的做法是对这些字段做校验和/或钳制,并在提供无效值时进行日志记录。
-`Actuator::screencap` 中的 `imwrite` 调用失败时,日志信息仍然是 `screencap saved to`;可以考虑记录更清晰的错误路径(如果有原因的话也一并记录),以便更容易定位截屏失败的原因。

## Individual Comments

### Comment 1
<location path="source/MaaFramework/Task/Component/Actuator.cpp" line_range="532-537" />
<code_context>
+    std::string filename = param.filename.empty() ? std::format("{}_{}{}", format_now_for_filename(), name, ext) : param.filename + ext;
+    auto filepath = option.log_dir() / "screencap" / path(filename);
+
+    std::vector<int> encode_params;
+    if (param.format == "jpg" || param.format == "jpeg") {
+        encode_params = { cv::IMWRITE_JPEG_QUALITY, param.quality };
+    }
+
+    bool ret = imwrite(filepath, image, encode_params);
+    LogInfo << "screencap saved to" << filepath << VAR(ret);
+
</code_context>
<issue_to_address>
**suggestion:**`format``quality` 增加校验/钳制逻辑,以避免未定义或依赖后端的行为。

`param.format` 只和小写的 "jpg"/"jpeg" 做匹配,其他任何值(或不同大小写的情况)都会退回到 OpenCV 的默认设置且不带质量参数。`param.quality` 也没有做任何检查,超出预期 `[0, 100]` 范围的数值可能会表现得不一致。

建议在构造 `encode_params` 之前,先将 `param.format` 进行标准化(例如转成小写)、只显式支持有限的几种格式,并将 `param.quality` 钳制在一个明确的范围内。对于不支持的格式,可以选择快速失败(`ret = false`)或记录一条清晰的警告日志,从而使行为更加可预期。

Suggested implementation:

```cpp
    const auto& option = MAA_GLOBAL_NS::OptionMgr::get_instance();

    // normalize format to lowercase to avoid casing issues
    std::string normalized_format = param.format;
    std::transform(normalized_format.begin(), normalized_format.end(), normalized_format.begin(),
                   [](unsigned char c) { return static_cast<char>(std::tolower(c)); });

    std::string ext = "." + normalized_format;
    std::string filename = param.filename.empty()
                               ? std::format("{}_{}{}", format_now_for_filename(), name, ext)
                               : param.filename + ext;
    auto filepath = option.log_dir() / "screencap" / path(filename);

    // clamp quality into [0, 100] to keep behavior well-defined across backends
    const int clamped_quality = std::clamp(param.quality, 0, 100);

    std::vector<int> encode_params;
    if (normalized_format == "jpg" || normalized_format == "jpeg") {
        encode_params = { cv::IMWRITE_JPEG_QUALITY, clamped_quality };
    } else if (normalized_format != "png" && normalized_format != "bmp") {
        // Log explicitly for unsupported formats so behavior is predictable
        LogWarn << "Unsupported screencap format: " << param.format
                << ", falling back to OpenCV defaults without explicit quality parameter";
    }

    bool ret = imwrite(filepath, image, encode_params);
    LogInfo << "screencap saved to" << filepath << VAR(ret);


MAA_TASK_NS_BEGIN

```

为了确保可以成功编译,请注意:

1. 在 `source/MaaFramework/Task/Component/Actuator.cpp` 文件顶部需要包含 `#include <algorithm>`(用于 `std::transform` 和 `std::clamp`):
   ```cpp
   #include <algorithm>
   ```
   如果当前还没有,请将其加入到其他标准库头文件附近。

2. 如果你的代码库对警告日志有自定义宏,请确认 `LogWarn` 是否为正确的宏;若不是,请替换为该文件中其他地方使用的警告级别日志宏。
</issue_to_address>

### Comment 2
<location path="source/MaaFramework/Resource/PipelineTypesV2.h" line_range="307" />
<code_context>
+{
+    std::string filename;
+    std::string format;
+    int quality = 0;
+
+    MEO_TOJSON(filename, format, quality);
</code_context>
<issue_to_address>
**issue (bug_risk):** 让 JScreencap 的 `quality` 默认值与 C++ 和 Python 中的默认值保持一致。

`Action::ScreencapParam` 和 Python 中的 `JScreencap` 数据类都将 `quality` 默认设为 `100`,但该结构体的默认值是 `0`。因此,由 dumper 生成的 V2 JSON 会包含 `quality = 100`,而在代码构造 V2 JSON 且不包含 `quality` 字段时,则会默认使用 `0`。为了保持一致性,请将这里的默认值改为 `100`。
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
帮我变得更有用!请在每条评论上点击 👍 或 👎,我会根据你的反馈改进之后的评审。
Original comment in English

Hey - I've found 2 issues, and left some high level feedback:

  • In Actuator::screencap, consider ensuring the target log_dir/screencap directory exists (e.g., via create_directories) before calling imwrite, otherwise the action can silently fail on a missing directory.
  • The parser and action handler accept any format string and any quality value; since the docs specify png|jpg|jpeg and 0–100, it would be safer to validate and/or clamp these fields and log when invalid values are provided.
  • When imwrite fails in Actuator::screencap, the log message still reads screencap saved to; consider logging a clearer error path (with reason if available) to make diagnosing screencap failures easier.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `Actuator::screencap`, consider ensuring the target `log_dir/screencap` directory exists (e.g., via `create_directories`) before calling `imwrite`, otherwise the action can silently fail on a missing directory.
- The parser and action handler accept any `format` string and any `quality` value; since the docs specify `png|jpg|jpeg` and `0–100`, it would be safer to validate and/or clamp these fields and log when invalid values are provided.
- When `imwrite` fails in `Actuator::screencap`, the log message still reads `screencap saved to`; consider logging a clearer error path (with reason if available) to make diagnosing screencap failures easier.

## Individual Comments

### Comment 1
<location path="source/MaaFramework/Task/Component/Actuator.cpp" line_range="532-537" />
<code_context>
+    std::string filename = param.filename.empty() ? std::format("{}_{}{}", format_now_for_filename(), name, ext) : param.filename + ext;
+    auto filepath = option.log_dir() / "screencap" / path(filename);
+
+    std::vector<int> encode_params;
+    if (param.format == "jpg" || param.format == "jpeg") {
+        encode_params = { cv::IMWRITE_JPEG_QUALITY, param.quality };
+    }
+
+    bool ret = imwrite(filepath, image, encode_params);
+    LogInfo << "screencap saved to" << filepath << VAR(ret);
+
</code_context>
<issue_to_address>
**suggestion:** Add validation/clamping for `format` and `quality` to avoid undefined or backend-dependent behavior.

`param.format` is only matched against lowercase "jpg"/"jpeg" and anything else (or different casing) falls back to OpenCV defaults with no quality parameter. `param.quality` is also passed through unchecked, so values outside the expected `[0, 100]` range may behave inconsistently.

Consider normalizing `param.format` (e.g., lowercase), explicitly handling only supported formats, and clamping `param.quality` to a defined range before constructing `encode_params`. For unsupported formats, either fail fast (`ret = false`) or log a clear warning so behavior is predictable.

Suggested implementation:

```cpp
    const auto& option = MAA_GLOBAL_NS::OptionMgr::get_instance();

    // normalize format to lowercase to avoid casing issues
    std::string normalized_format = param.format;
    std::transform(normalized_format.begin(), normalized_format.end(), normalized_format.begin(),
                   [](unsigned char c) { return static_cast<char>(std::tolower(c)); });

    std::string ext = "." + normalized_format;
    std::string filename = param.filename.empty()
                               ? std::format("{}_{}{}", format_now_for_filename(), name, ext)
                               : param.filename + ext;
    auto filepath = option.log_dir() / "screencap" / path(filename);

    // clamp quality into [0, 100] to keep behavior well-defined across backends
    const int clamped_quality = std::clamp(param.quality, 0, 100);

    std::vector<int> encode_params;
    if (normalized_format == "jpg" || normalized_format == "jpeg") {
        encode_params = { cv::IMWRITE_JPEG_QUALITY, clamped_quality };
    } else if (normalized_format != "png" && normalized_format != "bmp") {
        // Log explicitly for unsupported formats so behavior is predictable
        LogWarn << "Unsupported screencap format: " << param.format
                << ", falling back to OpenCV defaults without explicit quality parameter";
    }

    bool ret = imwrite(filepath, image, encode_params);
    LogInfo << "screencap saved to" << filepath << VAR(ret);


MAA_TASK_NS_BEGIN

```

To compile successfully, ensure:

1. `#include <algorithm>` is present at the top of `source/MaaFramework/Task/Component/Actuator.cpp` (for `std::transform` and `std::clamp`):
   ```cpp
   #include <algorithm>
   ```
   If it is not there yet, add it alongside the other standard library includes.

2. If your codebase uses a custom logging macro for warnings, confirm that `LogWarn` is the correct one; if not, replace `LogWarn` with the appropriate warning-level logging macro used elsewhere in this file.
</issue_to_address>

### Comment 2
<location path="source/MaaFramework/Resource/PipelineTypesV2.h" line_range="307" />
<code_context>
+{
+    std::string filename;
+    std::string format;
+    int quality = 0;
+
+    MEO_TOJSON(filename, format, quality);
</code_context>
<issue_to_address>
**issue (bug_risk):** Align JScreencap `quality` default with the C++ and Python defaults.

`Action::ScreencapParam` and the Python `JScreencap` dataclass both default `quality` to `100`, but this struct defaults it to `0`. As a result, V2 JSON produced by the dumper will include `quality = 100`, while code constructing V2 JSON without a `quality` field will assume `0`. Please change this default to `100` for consistency.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

{
std::string filename;
std::string format;
int quality = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): 让 JScreencap 的 quality 默认值与 C++ 和 Python 中的默认值保持一致。

Action::ScreencapParam 和 Python 中的 JScreencap 数据类都将 quality 默认设为 100,但该结构体的默认值是 0。因此,由 dumper 生成的 V2 JSON 会包含 quality = 100,而在代码构造 V2 JSON 且不包含 quality 字段时,则会默认使用 0。为了保持一致性,请将这里的默认值改为 100

Original comment in English

issue (bug_risk): Align JScreencap quality default with the C++ and Python defaults.

Action::ScreencapParam and the Python JScreencap dataclass both default quality to 100, but this struct defaults it to 0. As a result, V2 JSON produced by the dumper will include quality = 100, while code constructing V2 JSON without a quality field will assume 0. Please change this default to 100 for consistency.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new Screencap action to the pipeline system so tasks can save the current controller image to disk, and exposes the action across the JSON schema, C++ runtime, Python/NodeJS bindings, docs, and tests.

Changes:

  • Introduce Screencap action type + parameters (filename, format, quality) in pipeline types/schema, parser, dumper, and default-param lookup.
  • Implement Actuator::screencap() to write the cached controller image into the log directory and return action details (including filepath).
  • Expose the action in Python/NodeJS bindings and add Python tests + documentation updates.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/pipeline.schema.json Adds Screencap schema/defs and action enum entries for v1/v2 pipelines.
test/python/pipeline_test.py Adds parsing/type/defaults tests for the new Screencap action.
source/binding/Python/maa/pipeline.py Adds JActionType.Screencap and JScreencap dataclass + parsing support.
source/binding/NodeJS/src/apis/pipeline.d.ts Adds ActionScreencap and integrates it into TS action unions.
source/MaaFramework/Task/Component/Actuator.h Declares Actuator::screencap() action handler.
source/MaaFramework/Task/Component/Actuator.cpp Implements Screencap execution: build filename, encode params, write file, return detail.
source/MaaFramework/Resource/ResourceMgr.cpp Adds Screencap default action param retrieval.
source/MaaFramework/Resource/PipelineTypesV2.h Adds PipelineV2::JScreencap and includes it in the JActionParam variant.
source/MaaFramework/Resource/PipelineTypes.h Adds Action::Type::Screencap, ScreencapParam, and type maps.
source/MaaFramework/Resource/PipelineParser.h Declares parse_screencap().
source/MaaFramework/Resource/PipelineParser.cpp Parses Screencap params (filename/format/quality).
source/MaaFramework/Resource/PipelineDumper.cpp Dumps Screencap params into v2 JSON shape.
docs/en_us/3.1-PipelineProtocol.md Documents Screencap action and adds it to the v1 action list + version table.
AGENTS.md Tightens doc authoring guidance for Markdown hard line breaks.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Windsland52
Copy link
Member

不能自定义保存的路径吗

@Windsland52
Copy link
Member

怎么感觉之前没支持的自定义要一个一个加进来了(

@MistEO
Copy link
Member Author

MistEO commented Mar 1, 2026

不能自定义保存的路径吗

能啊,不是 filename 字段吗

@MistEO MistEO merged commit d733a43 into main Mar 2, 2026
19 checks passed
@MistEO MistEO deleted the feat/screencap branch March 2, 2026 06:45
@Windsland52
Copy link
Member

不能自定义保存的路径吗

能啊,不是 filename 字段吗

我看文档是这样写的:

保存当前截图到文件。截图保存在 `log_dir/screencap/` 目录下。

该动作属性需额外部分字段:

- `filename`: *string*  
    保存截图的文件名(不含扩展名)。可选,默认使用 `时间戳_节点名` 格式。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants