…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: arshidkv12 <arshidkv12@gmail.com>

Issue: PR valkey-io#3088 matched Redis #14243 perfectly on both files (patch-id),
but was being filtered out by line percentage check (7.7% < 50%).

Fixes:
1. Bypass line percentage filter for patch-id (exact) matches
2. Sort matches by same_files_count first, then similarity
   - PRs matching multiple files ranked higher
   - Ensures #14243 (2-file match) appears before single-file matches

Result: PR valkey-io#3088 now correctly shows #14243 as primary match
Signed-off-by: Ping Xie <pingxie@outlook.com>

Issue: PR valkey-io#1 falsely matched Redis #13157 due to patch-id collision.
- Valkey PR valkey-io#1: 2-file test PR touching dict.h and t_zset.c
- Redis PR #13157: 36-file license change PR
- Both happened to have same patch-id (efdd4d9b5640) for t_zset.c

Fix: Add file count ratio check for patch-id matches.
- If file counts differ by >3x, treat as collision (suppress match)
- Example: 2 files vs 36 files = 18x ratio -> filtered out
- Example: 2 files vs 2 files = 1x ratio -> kept

Result:
- PR valkey-io#1 now passes (no false positive from license change PR)
- PR valkey-io#3088 still correctly detects #14243 (2 files vs 2 files)

Signed-off-by: Ping Xie <pingxie@outlook.com>

Implements line-by-line comparison of actual diff content after initial
fingerprint matching to validate matches are truly similar.

Features:
- deep_compare_diffs(): Compares normalized diffs using Jaccard + sequence similarity
- Validates top candidates only (up to 2x report limit) to minimize API calls
- Adaptive thresholds: 50% for patch-id matches, 70% for simhash matches
- Graceful fallback: accepts fingerprint match if deep comparison fails

Benefits:
- Filters patch-id hash collisions (different changes, same hash)
- Validates simhash matches are semantically similar
- PR valkey-io#3088: correctly detects #14243, rejects 40+ collision false positives
- PR valkey-io#1: still passes (no false matches)

Performance:
- Before: checked every match (~60+ API calls for PR valkey-io#3088)
- After: checks top 10 candidates (~10 API calls)
- Speedup: 9s vs 60+s for PR valkey-io#3088

Signed-off-by: Ping Xie <pingxie@outlook.com>

Issue: PR valkey-io#3105 was passing even though it's a subset of Redis #14435.
- Valkey: Adds IFNE option (132 lines, 7 files)
- Redis: Adds IFEQ/IFNE/IFNEQ + xxhash + digest (9843 lines, 18 files)
- Jaccard similarity only 20% due to size difference
- But 90-100% of Valkey tokens exist in Redis (subset relationship)

Fix: Check subset ratio in addition to Jaccard similarity.
- Subset ratio = |Valkey ∩ Redis| / |Valkey|
- Detects when smaller PR cherry-picks from larger PR
- Final similarity = max(weighted_jaccard, subset_ratio)

Per-file analysis of PR valkey-io#3105 vs #14435:
- src/commands.def: 92.9% subset ratio
- src/commands/set.json: 93.8% subset ratio
- src/t_string.c: 95.5% subset ratio
- tests/unit/type/string.tcl: 100% subset ratio

Result:
- PR valkey-io#3105 now correctly detects Redis #14435 match
- PR valkey-io#3088 still works (detects #14243)
- PR valkey-io#1 still passes (no false positives)

Signed-off-by: Ping Xie <pingxie@outlook.com>

Remove patch-id completely - it was causing false positives from:
- Hash collisions on small changes
- Trivial whitespace/formatting fixes
- Unrelated changes to same code

New approach:
- Layer 1: Liberal candidate generation (SimHash > 0.70, local DB lookup)
- Layer 2: Deep content validation (Jaccard+subset > 0.75, requires API)

Benefits:
- Simpler logic: Two clear functions, no special cases
- Fewer false positives: No patch-id collisions
- Same API call count: ~6-11 calls per Valkey PR
- Better accuracy: True copies still detected, false positives filtered

Changes:
- common.py: Removed compute_patch_id(), compute_similarity()
- common.py: Added compute_simhash_similarity(), updated thresholds
- check_provenance.py: Replaced complex find_matches() with layer1_find_candidates() + layer2_validate_candidate()
- refresh_redis_prs.py: Removed patch-id computation
- bootstrap_redis_commits.py: Removed patch-id computation
- Rebuilt database without patch-id fields
- Updated all tests to remove patch-id references

Validation:
- All 30 tests pass
- PR valkey-io#3088 correctly detected (0.891 similarity with #14243)
- PR #14268 patch-id collision false positive eliminated

Signed-off-by: Ping Xie <pingxie@outlook.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

valkey-io#3088)"

This reverts commit 2e2eb41.

valkey-io#3088)"

This reverts commit 2e2eb41.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

Reverts "Fix the memory leak in the VM_GetCommandKeysWithFlags function
(#3088)"

This reverts commit 2e2eb41.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

valkey-io#3088)"

This reverts commit 7b88723.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

valkey-io#3088)"

This reverts commit 457d032.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

Reverts "Fix the memory leak in the VM_GetCommandKeysWithFlags function
(valkey-io#3088)"

This reverts commit 2e2eb41.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>

Reverts "Fix the memory leak in the VM_GetCommandKeysWithFlags function
(valkey-io#3088)"

This reverts commit 2e2eb41.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…y-io#3088)

Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes valkey-io#3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>

Reverts "Fix the memory leak in the VM_GetCommandKeysWithFlags function
(valkey-io#3088)"

This reverts commit 2e2eb41.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>