mraba/reflection-optimisation: skip schema resolution for single table#656
Merged
Conversation
737c2aa to
ca3d80e
Compare
… optimise reflection - Add get_multi_columns, get_multi_pk_constraint, get_multi_unique_constraints, get_multi_foreign_keys for SA 2.x bulk reflection (one schema-wide query per reflection pass instead of one query per table). - SA 2.x get_columns uses DESC TABLE directly (IS_VERSION_20 guard); get_multi_columns handles all bulk reflection including temp/dynamic tables via DESC TABLE fallback. - Fix schema key mismatch in all get_multi_* methods: use effective_schema for SQL queries, preserve original schema (possibly None) as the return dict key so SA's _reflect_info lookup succeeds. - Remove _should_use_table_specific_query; get_columns uses _is_single_table_reflection. - Fix SHOW INDEXES IN TABLE replacing SHOW TABLES LIKE for single-table index reflection. - Add _always_quote_join helper for correctly-quoted per-table SQL identifiers. - Add shared row-parsing helpers (_parse_pk_rows, _parse_uk_rows, _parse_fk_rows). - Fix foreign key referred_schema resolution (same_schemas includes explicit schema). - Update README Cache Column Metadata section with accurate SA 2.x dispatch model. - Update DESCRIPTION.md unreleased notes.
7216ee1 to
b706b59
Compare
sfc-gh-jcieslak
approved these changes
Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please answer these questions before submitting your pull requests. Thanks!
What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes #SNOW-689531
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Optimise reflection performance (SNOW-689531)
Summary
Adds SQLAlchemy 2.x get_multi_* bulk reflection hooks and per-table SHOW … IN TABLE / DESC TABLE reflection paths. Schema-wide queries run once per MetaData.reflect() pass instead of once per table.
Single-table Inspector calls use fast per-table queries that work correctly for all table types including temporary and dynamic tables.
Changes
Performance
• Add get_multi_columns, get_multi_pk_constraint, get_multi_unique_constraints, get_multi_foreign_keys — SA 2.x bulk hooks that each issue one schema-wide query per reflection pass instead of one
query per table.
• cache_column_metadata=True opt-in enables per-table SHOW … IN TABLE queries for PK, UK, FK, and indexes on both SA versions.
Correctness
• Fix FK referred_schema resolution: both per-table and schema-wide paths now include the explicitly reflected schema in the same-schema set, preventing incorrect cross-schema references when
reflecting a non-default schema.
• Fix unique constraint keying: constraints are now keyed by (table_name, constraint_name) instead of constraint_name alone, preventing silent overwrites when two tables have identically-named
constraints.
• Add _always_quote_join helper that always quotes denormalised identifiers, ensuring correct SQL for case-sensitive table and schema names.
• Replace SHOW TABLES LIKE with SHOW INDEXES IN TABLE for single-table index reflection, eliminating SQL LIKE wildcard false-positives and case-sensitivity bugs.
Robustness
• Narrow exception handling in all per-table reflection methods (_get_table_primary_keys, _get_table_unique_constraints, _get_table_foreign_keys, _get_table_indexes) from sa_exc.DBAPIError to
sa_exc.ProgrammingError — connection and operational errors now propagate instead of being silently swallowed.
• Narrow _execute_desc from bare except Exception to sa_exc.ProgrammingError — only SQL-level errors (e.g. table dropped by another session) are swallowed; network/permission errors propagate with
actionable diagnostics.
• Add logger.debug to all swallowed exceptions for diagnosability.
Code quality
• Add shared row-parsing helpers (_parse_pk_rows, _parse_uk_rows, _parse_fk_rows, _parse_index_rows) so correctness fixes propagate to both per-table and schema-wide paths.
• Hoist _StructuredTypeInfoManager allocation outside the fallback loop in get_multi_columns.
• Fix misleading docstrings on get_table* methods that claimed "results are not cached" when the calling methods apply @reflection.cache.
Documentation
• README: document SA 2.x vs 1.4 reflection routing, cache_column_metadata opt-in, performance implications, and best practices for different schema sizes.
• README: note that Inspector.get_columns() on SA 2.x uses DESC TABLE, which includes Snowflake's resolved default sizes in type str() representations (e.g. BINARY(8388608)). Type objects are
functionally identical; use isinstance() for type checks.
Behavioural notes
• SA 2.x
Inspector.get_columns()now uses DESC TABLE instead of information_schema. Reflected type objects are functionally identical, but str() output may include explicit default sizes (e.g.BINARY(8388608) vs BINARY). MetaData.reflect() and Alembic are unaffected — they use get_multi_columns which still queries information_schema.
• SA 2.x
get_columns()for missing tables returns [] instead of raising NoSuchTableError, aligning with the SA 2.0 convention of returning falsy values. reflect_table handles the empty-columnscase separately.
• Per-table reflection methods now let non-ProgrammingError exceptions (network failures, permission errors) propagate instead of returning empty metadata silently.