Summary
The blocking algorithm is too liberal in including all patients from overlapping person clusters. We should include those patients that don't match because of missing key values, but we should exclude patients with different values.
Acceptance Criteria
Details / Tasks
There are likely two ways to solve this, a) modify the query b) keep the query as is and filter the dataset in memory. It's not obvious which strategy is better from a performance standpoint, please consider both when implementing and include some documentation / comments on why that strategy was taken.
Background / Context
Take a simple blocking example of finding all patients with a matching BIRTHDATE of '1999-01-01'. Currently, we find all patient records in the database with that matching BIRTHDATE in the blocking values table, we then reduce that to a unique set of person clusters (based on what clusters those records belong to). We then return all patients in those matching person clusters, this creates 3 groups of data in the result set.
- Patients that exactly match on '1999-01-01'
- Patients that have no BIRTHDATE value
- Patients that have a BIRTHDATE value, but does not match '1999-01-01'
The last group in the result set is problematic, it clearly doesn't block, and therefore shouldn't be present in our evaluation. Removing this group from our result set, should provide more accurate calculations on cluster membership.
Related Issues/PRs
#230
Summary
The blocking algorithm is too liberal in including all patients from overlapping person clusters. We should include those patients that don't match because of missing key values, but we should exclude patients with different values.
Acceptance Criteria
get_block_datamethod to exclude patients that don't match the blocking criteriaDetails / Tasks
There are likely two ways to solve this, a) modify the query b) keep the query as is and filter the dataset in memory. It's not obvious which strategy is better from a performance standpoint, please consider both when implementing and include some documentation / comments on why that strategy was taken.
Background / Context
Take a simple blocking example of finding all patients with a matching BIRTHDATE of '1999-01-01'. Currently, we find all patient records in the database with that matching BIRTHDATE in the blocking values table, we then reduce that to a unique set of person clusters (based on what clusters those records belong to). We then return all patients in those matching person clusters, this creates 3 groups of data in the result set.
The last group in the result set is problematic, it clearly doesn't block, and therefore shouldn't be present in our evaluation. Removing this group from our result set, should provide more accurate calculations on cluster membership.
Related Issues/PRs
#230