Skip to content

fix: apply changelogs 1.76-1.78, fix 32 views with missing data, regenerate schema#8479

Merged
pethers merged 6 commits intomasterfrom
copilot/update-sample-data-analysis
Mar 20, 2026
Merged

fix: apply changelogs 1.76-1.78, fix 32 views with missing data, regenerate schema#8479
pethers merged 6 commits intomasterfrom
copilot/update-sample-data-analysis

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 20, 2026

Description

Fixes 4 systemic root causes that left 32 advanced analytical views returning zero/NULL data despite underlying tables having millions of rows. Applied 22 new Liquibase changesets (1.76–1.78), verified application startup, and regenerated full_schema.sql via pg_dump.

Root causes:

# Bug Effect Blast radius
1 Vote comparisons use title-case ('Ja') but data is uppercase ('JA') absence_rate=0, rebel_rate=NULL everywhere view_risk_score_evolution → 6 downstream views
2 full_schema.sql stale: 1.76-011 fix marked EXECUTED but view definition reverted Wrong rebel calculation (vote vs party code) view_politician_risk_summary → 3 downstream views
3 rule_group filters use non-existent enums ('ABSENTEEISM') instead of actual DRL values ('Behavior', 'Assignments', 'Experience', 'Party', 'Role') All violation subcategories = 0 view_politician_risk_summary
4 Wrong status/type filters: status='active' (actual: Swedish strings), rv.status='ACTIVE' (actual: MINOR/MAJOR/CRITICAL), document_type='Motion' (actual: 'mot') active_members=0, motions_count=0 for all parties view_party_performance_metrics → 3 downstream views

Example fix (vote case sensitivity):

-- Before (matches 0 of 3.7M rows):
WHEN vote_data.vote::text = 'Ja' THEN 1
-- After:
WHEN UPPER(vote_data.vote::text) = 'JA' THEN 1

Changesets applied:

  • 1.76 (11): Materialized sub-views for decision intelligence timeout, coalition evolution fix, voting anomaly detection, party transition history rebuild, risk summary rebel calculation
  • 1.77 (4): Party transition history using assignment_data org_codes, defector/switcher analysis, anomaly pattern Cartesian join elimination
  • 1.78 (7): UPPER() vote normalization, risk summary re-apply + rule_group mapping, party performance metrics (status/type/violation filters), effectiveness trends, 3 CASCADE-dropped view recreations

Verified: Application starts (Jetty port 28443, healthcheck OK). Schema regenerated with pg_dump per maintenance guide.

Type of Change

Primary Changes

  • 🐛 Bug Fix

Political Analysis

  • 📊 Political Data Analysis
    • Party Analysis
    • Riksdagen Integration
  • 📈 Analytics & Metrics
    • Performance Metrics
    • Risk Assessment
    • Decision Analysis
    • Transparency Metrics

Technical Changes

  • 🏗️ Infrastructure
    • Database Changes
    • Performance Optimization

Impact Analysis

Political Analysis Impact

  • Impact on data quality: 32 views with previously-zero columns now populate from actual data (3.7M vote rows, 2.3K rule violations, 94K motions)
  • Impact on analysis accuracy: Risk scores, rebel rates, absence rates, violation breakdowns, party member counts, document metrics all corrected
  • Impact on transparency features: Party performance, electoral trends, and longitudinal analysis reflect real parliamentary activity

Technical Impact

  • Performance impact: Anomaly pattern view reduced from timeout (>5min) to <5s via Cartesian join elimination and materialized sub-views
  • Security implications: None — SQL view definition fixes only
  • Dependency changes: None

Testing

  • Political data validation completed
  • Security compliance verified
  • Performance testing completed

All 22 changesets validated via Liquibase update, application startup verified (healthcheck OK), schema regenerated and verified.

Documentation

  • Package/module documentation updated

Rule group mapping documented inline in SQL comments for backward-compatible column aliasing.

Screenshots

N/A — database view fixes, no UI changes.

Related Issues

Checklist

  • Code follows project coding standards
  • Comments are clear and helpful
  • Documentation is updated
  • Tests are passing
  • Security compliance is maintained
  • Performance impact is acceptable
  • Breaking changes are documented
  • Changes are backward compatible

Additional Notes

Data sources used for validation:

  • vote_data.vote: JA(2.08M), FRÅNVARANDE(742K), NEJ(677K), AVSTÅR(203K)
  • rule_violation: 2,301 rows, rule_group ∈ {Behavior, Assignments, Experience, Party, Role, Attribute, Structure}
  • rule_violation.status ∈ {MINOR, MAJOR, CRITICAL} — not ACTIVE
  • document_data.document_type ∈ {mot, bet, prop, kammakt} — not Motion/Proposition
  • person_data.status: Swedish strings — not active/inactive

Security Considerations

  • No sensitive data exposed
  • Security best practices followed
  • Compliance requirements met

Release Notes

Fixed 32 advanced analytical views returning empty/zero data due to case-sensitivity bugs in vote comparisons, incorrect enum values for rule violations and document types, wrong person status filters, and stale schema definitions. Eliminated timeout in election cycle anomaly pattern view. Applied 22 Liquibase changesets (1.76–1.78) and regenerated full_schema.sql.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 20, 2026

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 3b6e6a1.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

…out anomaly view

Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Copilot AI changed the title [WIP] Update analysis of advanced views in extraction summary report fix: resolve 3 empty party views and anomaly pattern timeout in advanced analytics Mar 20, 2026
Copilot AI requested a review from pethers March 20, 2026 18:00
Copilot AI changed the title fix: resolve 3 empty party views and anomaly pattern timeout in advanced analytics fix: resolve missing data in 32 advanced views from 4 systemic root causes Mar 20, 2026
@sonarqubecloud
Copy link
Copy Markdown

Copilot AI changed the title fix: resolve missing data in 32 advanced views from 4 systemic root causes fix: apply changelogs 1.76-1.78, fix 32 views with missing data, regenerate schema Mar 20, 2026
@pethers pethers marked this pull request as ready for review March 20, 2026 19:23
@pethers pethers requested a review from Copilot March 20, 2026 19:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the database layer (Liquibase + regenerated full_schema.sql) to fix systemic filter/enum/case-sensitivity issues that caused many analytics views to return empty/NULL/zeroed results, and introduces performance improvements via new materialized sub-views.

Changes:

  • Added Liquibase changelogs 1.77 and 1.78 and wired them into the root db-changelog.xml.
  • Regenerated full_schema.sql to reflect applied changesets, including corrected view definitions and new materialized views/indexes.
  • Updated multiple view definitions to use correct data conventions (e.g., vote casing, rule_group/status/document_type filters) and removed a Cartesian-join timeout source.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
service.data.impl/src/main/resources/full_schema.sql Regenerated schema dump reflecting updated/added analytical views, matviews, and databasechangelog entries through 1.78.
service.data.impl/src/main/resources/db-changelog.xml Includes the new 1.77 and 1.78 Liquibase changelog files in sequence.
service.data.impl/src/main/resources/db-changelog-1.77.xml Adds view rebuilds/fixes for party transition/defection/switcher views and eliminates anomaly-pattern Cartesian join timeout.
service.data.impl/src/main/resources/db-changelog-1.78.xml Fixes vote casing, risk summary rule_group mapping, party performance metrics filters, and recreates dependent views after CASCADE drops.

ELSE false
END AS is_post_election_period
FROM windowed_statistics ws
ORDER BY party, election_cycle_id, cycle_year, semester;;
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CREATE VIEW statement ends with ;;. To avoid Liquibase executing an empty SQL statement after splitting, keep only a single trailing semicolon.

Suggested change
ORDER BY party, election_cycle_id, cycle_year, semester;;
ORDER BY party, election_cycle_id, cycle_year, semester;

Copilot uses AI. Check for mistakes.
ELSE false
END AS is_election_cycle_end
FROM windowed_statistics ws
ORDER BY party, election_cycle_id, cycle_year, semester;;
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CREATE VIEW statement ends with ;;. Liquibase can treat this as an extra empty statement; please remove the extra semicolon and keep just one terminator.

Suggested change
ORDER BY party, election_cycle_id, cycle_year, semester;;
ORDER BY party, election_cycle_id, cycle_year, semester;

Copilot uses AI. Check for mistakes.
END AS performance_trend,
round((((performance_score * 0.4) + (party_win_rate * 0.3)) + (party_participation_rate * 0.3)), 2) AS discipline_score,
round((((rank_by_performance)::numeric / (NULLIF(cycle_party_count, 0))::numeric) * (100)::numeric), 2) AS competitiveness_index
FROM windowed w;;
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This SQL ends with a double semicolon (;;). Liquibase may split statements on semicolons and attempt to execute an empty statement, which can break migrations depending on configuration/driver. Remove the extra semicolon so the CREATE VIEW ends with a single ;.

Suggested change
FROM windowed w;;
FROM windowed w;

Copilot uses AI. Check for mistakes.
@pethers pethers merged commit bddbd68 into master Mar 20, 2026
16 checks passed
@pethers pethers deleted the copilot/update-sample-data-analysis branch March 20, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants