Skip to content

Fix ComfyUI prompt extraction for ImpactWildcardProcessor#933

Merged
zanllp merged 3 commits into
zanllp:mainfrom
osirigunso:fix/comfyui-populated-text
Mar 23, 2026
Merged

Fix ComfyUI prompt extraction for ImpactWildcardProcessor#933
zanllp merged 3 commits into
zanllp:mainfrom
osirigunso:fix/comfyui-populated-text

Conversation

@osirigunso

Copy link
Copy Markdown
Contributor

Summary

Fix ComfyUI prompt extraction for workflows using ImpactWildcardProcessor.

What changed

  • prefer populated_text over wildcard_text
  • avoid extracting unresolved wildcard template text
  • recursively resolve upstream prompt nodes
  • support intermediate conditioning nodes such as FluxGuidance

Why

wildcard_text contains the template before wildcard expansion, while populated_text contains the final prompt actually used for generation.

This change ensures the extracted prompt matches the real generated prompt.

@zanllp

zanllp commented Mar 14, 2026

Copy link
Copy Markdown
Owner

Please provide a few sample image files so I can test.

@osirigunso

Copy link
Copy Markdown
Contributor Author
103920_aiponyanime_v2 safetensors_1122176594524013_00001_ 120732_aiponyanime_v2 safetensors_903128333946712_00001_ I confirmed that these images can be properly acquired.

@zanllp

zanllp commented Mar 15, 2026

Copy link
Copy Markdown
Owner

Compared to master, the positive prompt is missing when viewing this image in your PR branch.

comfyui_2x_403061063-94099f3e-0084-4f68-a1a1-54a38a10c8e8

@osirigunso

Copy link
Copy Markdown
Contributor Author

This image is not one that was expected to be compatible because it contains an SD3 workflow. However, it appears that both the original tool.py and the modified tool.py are obtaining equivalent results in my environment.
original
https://gyazo.com/22a844815ec49e0b7511aac48c97ec3f
modified
https://gyazo.com/1d945db98162d86f3a8f884f6e336319

@zanllp

zanllp commented Mar 16, 2026

Copy link
Copy Markdown
Owner

My apologies - I was testing with inconsistent environment variables. However, I've identified another issue that needs to be addressed: there's a problem where the negative prompt field is being read as the model name. This shouldn't be happening since the node name clearly indicates it's a model node.

image image

Additionally, I believe all parser improvements should maintain backward compatibility with existing images.

@osirigunso

Copy link
Copy Markdown
Contributor Author

I had also noticed that negative prompt, which seems odd to me as well. However, since the main focus this time was on adapting to ImpactWildcardProcessor, and since the issue also occurs in the original tool.py, I believe this fix—which only affects cases where wildcard_text is present—is working as intended.
I believe that for systems like SD3, flux, and z-image—which encompass a wide variety of workflow variations—it may be acceptable for them to remain generally incompatible unless official workflows or nodes are standardized. This is because users can individually adapt by selecting the appropriate workflows and nodes to ensure compatibility with their own image management methods.
Furthermore, this issue is also influenced by differences in JSON structure—such as whether it represents the entire workflow or a simplified list of nodes—as well as its location like prompt, workflowapiJson, workflow, so on.
I feel we're still at a stage where we can only take ad hoc measures.

@osirigunso

Copy link
Copy Markdown
Contributor Author

Speaking of parsers, I wonder if the difference in whether prompts are highlighted with squares is determined by detecting and sorting the difference between tags and natural language? There are some images where spaces unintentionally became tab indents, and I feel like it breaks down in those cases.

@zanllp

zanllp commented Mar 17, 2026

Copy link
Copy Markdown
Owner

Speaking of parsers, I wonder if the difference in whether prompts are highlighted with squares is determined by detecting and sorting the difference between tags and natural language? There are some images where spaces unintentionally became tab indents, and I feel like it breaks down in those cases.

Yes, Tag-style prompts get highlighted with those boxes.

The logic checks the tag lengths - if tags are short and concise, it treats them as tag-style and highlights them. If there are long tags or the average length is high, it assumes natural language and doesn't highlight.

image

Check the code I pasted - that's the exact function doing this detection.

The spaces becoming tab indents are likely just a frontend display styling issue and won't affect the actual parsing logic.

@zanllp

zanllp commented Mar 17, 2026

Copy link
Copy Markdown
Owner

Thank you for the detailed explanation about your approach. However, I'd like to address some concerns and propose an alternative implementation strategy.

Issue with Current Approach

Looking at the screenshots I provided earlier, notice that the negative prompt issue you mentioned (where negative prompt contains the model name) does not occur in the master branch based on my testing. This appears to be a new issue introduced by the changes.

Concerns About the Implementation

  1. Overly Complex Changes: Your refactoring significantly changes the logic for prompt extraction, affecting all ComfyUI workflows, not just those using ImpactWildcardProcessor

  2. Breaking Existing Functionality: The current code is simple and works for a very large portion of ComfyUI prompt extraction. The changes risk breaking this compatibility

  3. Scope Creep: While the PR's stated goal is to fix ImpactWildcardProcessor extraction, it actually modifies the entire prompt extraction logic

Proposed Alternative Approach

I believe a more conservative approach would be better:

1. Keep the current implementation intact in get_comfyui_exif_data

2. Create a separate global function for wildcard handling

For example:

def extract_comfyui_prompt_with_wildcard_support(
    data: Dict,
    KSampler_entry: Dict
) -> Tuple[str, str]:
    """
    Enhanced prompt extraction for workflows using ImpactWildcardProcessor.

    This function handles:
    - populated_text over wildcard_text
    - Recursive resolution of upstream prompt nodes
    - Intermediate conditioning nodes such as FluxGuidance

    Returns:
        Tuple of (positive_prompt, negative_prompt)
    """
    # Your enhanced implementation here
    # Include the recursive resolution and populated_text logic
    pass

3. Add detection points in get_comfyui_exif_data to conditionally call the new function

Detection Point 1 - Check for ImpactWildcardProcessor nodes (around L520):
If detected, immediately return using the enhanced extraction function

def get_comfyui_exif_data(img: Image):
    # ... existing code up to line ~518 ...

    # Detection Point 1: Check if workflow contains ImpactWildcardProcessor
    # If yes, immediately use the enhanced extraction and return
    if any(
        node_data.get("class_type") == "ImpactWildcardProcessor"
        for node_data in data.values()
    ):
        pos_prompt, neg_prompt = extract_comfyui_prompt_with_wildcard_support(
            data, KSampler_entry
        )
        pos_prompt_arr = unique_by(parse_prompt(pos_prompt)["pos_prompt"])
        return {
            "meta": meta,
            "pos_prompt": pos_prompt_arr,
            "pos_prompt_raw": pos_prompt,
            "neg_prompt_raw": neg_prompt
        }

    # Continue with current extraction logic
    extract_all_prompts = os.getenv("IIB_COMFYUI_EXTRACT_ALL_PROMPTS", "false").lower() == "true"

    if extract_all_prompts:
        # ... existing extract_all_prompts logic ...
    else:
        # ... existing prompt extraction logic ...
        pos_prompt = get_text_from_clip(KSampler_entry["positive"][0])
        neg_prompt = get_text_from_clip(KSampler_entry["negative"][0])

    # Detection Point 2: Fallback if no prompts were extracted
    pos_prompt_arr = unique_by(parse_prompt(pos_prompt)["pos_prompt"])

    # If standard extraction failed, try the enhanced method
    if not pos_prompt_arr or not pos_prompt.strip():
        pos_prompt_fallback, neg_prompt_fallback = extract_comfyui_prompt_with_wildcard_support(
            data, KSampler_entry
        )

        if pos_prompt_fallback:
            pos_prompt = pos_prompt_fallback
            pos_prompt_arr = unique_by(parse_prompt(pos_prompt_fallback)["pos_prompt"])

        if neg_prompt_fallback:
            neg_prompt = neg_prompt_fallback

    return {
        "meta": meta,
        "pos_prompt": pos_prompt_arr,
        "pos_prompt_raw": pos_prompt,
        "neg_prompt_raw": neg_prompt
    }

Could you please revise the PR to implement this more targeted approach? This would ensure that we gain support for ImpactWildcardProcessor without sacrificing compatibility with the vast majority of existing ComfyUI workflows.

@osirigunso

Copy link
Copy Markdown
Contributor Author

First of all, I deeply appreciate your kind and thorough explanation and guidance. Thanks to that, I believe the corrections have been completed successfully without any issues.

To be honest and swallowing my embarrassment, since I'm only a hobby-level programmer, this might be beyond my skill level.

Anyway, it seems to be working properly now, so please check it out. :D

@zanllp zanllp merged commit 5694044 into zanllp:main Mar 23, 2026
@zanllp

zanllp commented Mar 23, 2026

Copy link
Copy Markdown
Owner

Thanks for your contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants