Skip to content

mraba/reflection-optimisation: skip schema resolution for single table#656

Merged
sfc-gh-mraba merged 7 commits into
mainfrom
mraba/reflection-optimisation
Apr 15, 2026
Merged

mraba/reflection-optimisation: skip schema resolution for single table#656
sfc-gh-mraba merged 7 commits into
mainfrom
mraba/reflection-optimisation

Conversation

@sfc-gh-mraba

@sfc-gh-mraba sfc-gh-mraba commented Feb 25, 2026

Copy link
Copy Markdown
Collaborator

Please answer these questions before submitting your pull requests. Thanks!

  1. What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes #SNOW-689531

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
    • I am adding new logging messages
    • I am adding new credentials
    • I am adding a new dependency
  3. Please describe how your code solves the related issue.

Optimise reflection performance (SNOW-689531)

Summary

Adds SQLAlchemy 2.x get_multi_* bulk reflection hooks and per-table SHOW … IN TABLE / DESC TABLE reflection paths. Schema-wide queries run once per MetaData.reflect() pass instead of once per table.
Single-table Inspector calls use fast per-table queries that work correctly for all table types including temporary and dynamic tables.

Changes

Performance

• Add get_multi_columns, get_multi_pk_constraint, get_multi_unique_constraints, get_multi_foreign_keys — SA 2.x bulk hooks that each issue one schema-wide query per reflection pass instead of one
query per table.
• cache_column_metadata=True opt-in enables per-table SHOW … IN TABLE queries for PK, UK, FK, and indexes on both SA versions.

Correctness

• Fix FK referred_schema resolution: both per-table and schema-wide paths now include the explicitly reflected schema in the same-schema set, preventing incorrect cross-schema references when
reflecting a non-default schema.
• Fix unique constraint keying: constraints are now keyed by (table_name, constraint_name) instead of constraint_name alone, preventing silent overwrites when two tables have identically-named
constraints.
• Add _always_quote_join helper that always quotes denormalised identifiers, ensuring correct SQL for case-sensitive table and schema names.
• Replace SHOW TABLES LIKE with SHOW INDEXES IN TABLE for single-table index reflection, eliminating SQL LIKE wildcard false-positives and case-sensitivity bugs.

Robustness

• Narrow exception handling in all per-table reflection methods (_get_table_primary_keys, _get_table_unique_constraints, _get_table_foreign_keys, _get_table_indexes) from sa_exc.DBAPIError to
sa_exc.ProgrammingError — connection and operational errors now propagate instead of being silently swallowed.
• Narrow _execute_desc from bare except Exception to sa_exc.ProgrammingError — only SQL-level errors (e.g. table dropped by another session) are swallowed; network/permission errors propagate with
actionable diagnostics.
• Add logger.debug to all swallowed exceptions for diagnosability.

Code quality

• Add shared row-parsing helpers (_parse_pk_rows, _parse_uk_rows, _parse_fk_rows, _parse_index_rows) so correctness fixes propagate to both per-table and schema-wide paths.
• Hoist _StructuredTypeInfoManager allocation outside the fallback loop in get_multi_columns.
• Fix misleading docstrings on get_table* methods that claimed "results are not cached" when the calling methods apply @reflection.cache.

Documentation

• README: document SA 2.x vs 1.4 reflection routing, cache_column_metadata opt-in, performance implications, and best practices for different schema sizes.
• README: note that Inspector.get_columns() on SA 2.x uses DESC TABLE, which includes Snowflake's resolved default sizes in type str() representations (e.g. BINARY(8388608)). Type objects are
functionally identical; use isinstance() for type checks.

Behavioural notes

• SA 2.x Inspector.get_columns() now uses DESC TABLE instead of information_schema. Reflected type objects are functionally identical, but str() output may include explicit default sizes (e.g.
BINARY(8388608) vs BINARY). MetaData.reflect() and Alembic are unaffected — they use get_multi_columns which still queries information_schema.
• SA 2.x get_columns() for missing tables returns [] instead of raising NoSuchTableError, aligning with the SA 2.0 convention of returning falsy values. reflect_table handles the empty-columns
case separately.
• Per-table reflection methods now let non-ProgrammingError exceptions (network failures, permission errors) propagate instead of returning empty metadata silently.

@sfc-gh-mraba sfc-gh-mraba self-assigned this Feb 25, 2026
@sfc-gh-mraba sfc-gh-mraba force-pushed the mraba/reflection-optimisation branch from 737c2aa to ca3d80e Compare March 25, 2026 15:35
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
Comment thread src/snowflake/sqlalchemy/snowdialect.py Outdated
… optimise reflection

- Add get_multi_columns, get_multi_pk_constraint, get_multi_unique_constraints,
  get_multi_foreign_keys for SA 2.x bulk reflection (one schema-wide query per
  reflection pass instead of one query per table).
- SA 2.x get_columns uses DESC TABLE directly (IS_VERSION_20 guard); get_multi_columns
  handles all bulk reflection including temp/dynamic tables via DESC TABLE fallback.
- Fix schema key mismatch in all get_multi_* methods: use effective_schema for SQL
  queries, preserve original schema (possibly None) as the return dict key so SA's
  _reflect_info lookup succeeds.
- Remove _should_use_table_specific_query; get_columns uses _is_single_table_reflection.
- Fix SHOW INDEXES IN TABLE replacing SHOW TABLES LIKE for single-table index reflection.
- Add _always_quote_join helper for correctly-quoted per-table SQL identifiers.
- Add shared row-parsing helpers (_parse_pk_rows, _parse_uk_rows, _parse_fk_rows).
- Fix foreign key referred_schema resolution (same_schemas includes explicit schema).
- Update README Cache Column Metadata section with accurate SA 2.x dispatch model.
- Update DESCRIPTION.md unreleased notes.
@sfc-gh-mraba sfc-gh-mraba force-pushed the mraba/reflection-optimisation branch from 7216ee1 to b706b59 Compare March 26, 2026 15:22
@sfc-gh-mraba sfc-gh-mraba marked this pull request as ready for review April 13, 2026 09:41
@sfc-gh-mraba sfc-gh-mraba requested a review from a team as a code owner April 13, 2026 09:41
@sfc-gh-mraba sfc-gh-mraba merged commit 4407b9b into main Apr 15, 2026
62 checks passed
@sfc-gh-mraba sfc-gh-mraba deleted the mraba/reflection-optimisation branch April 15, 2026 12:20
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 15, 2026
@sfc-gh-mraba sfc-gh-mraba restored the mraba/reflection-optimisation branch April 15, 2026 14:39
@sfc-gh-mraba sfc-gh-mraba deleted the mraba/reflection-optimisation branch April 15, 2026 14:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants