Skip to content

fix: show General forum topic messages (coalesce NULL reply_to_top_id)#122

Merged
GeiserX merged 2 commits intoGeiserX:mainfrom
tondeaf:fix/general-topic-null-messages
Apr 25, 2026
Merged

fix: show General forum topic messages (coalesce NULL reply_to_top_id)#122
GeiserX merged 2 commits intoGeiserX:mainfrom
tondeaf:fix/general-topic-null-messages

Conversation

@tondeaf
Copy link
Copy Markdown
Contributor

@tondeaf tondeaf commented Apr 18, 2026

Summary

  • Apply the same coalesce(reply_to_top_id, 1) == topic_id predicate in get_messages_paginated() that get_forum_topics() already uses for counting, so the General topic pane stops returning empty when its messages have NULL reply_to_top_id (pre-v6.2.0 / pre-forum) or explicit reply_to_top_id=1 (post-forum-enable).
  • Three regression tests covering NULL+explicit-1 General messages, strict filtering for non-General topics, and unfiltered no-topic queries.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that causes existing functionality to change)
  • Documentation update
  • Infrastructure/CI change

Database Changes

  • Schema changes (Alembic migration required)
  • Data migration script added in scripts/
  • No database changes

Data Consistency Checklist

  • All chat_id values use marked format (via _get_marked_id()) — N/A, no chat_id construction
  • All datetime values pass through _strip_tz() before DB operations — N/A, no datetime handling
  • INSERT and UPDATE operations handle the same fields identically — N/A, change is read-side only

Testing

  • Tests pass locally (python -m pytest tests/ -v)
  • Linting passes (ruff check .)
  • Formatting passes (ruff format --check .)
  • Manually tested in development environment

Security Checklist

  • No secrets or credentials committed
  • User input properly validated/sanitized — topic_id flows through SQLAlchemy bound parameters
  • Authentication/authorization properly checked — same auth path as other paginated reads

Deployment Notes

  • No image rebuild needed beyond a normal release; the change is a query predicate and takes effect immediately on rolled code.

The bug

In the web viewer, a forum chat's topic list correctly shows the General topic with its true message count (e.g. "67 messages"), but clicking the General tab renders an empty message pane.

Reproduce:

  1. Enable a forum on a group chat that has pre-existing (pre-forum) messages, or keep using a forum chat whose early messages predate v6.2.0.
  2. Run the archiver + viewer.
  3. Open the chat → General topic.

Sidebar shows the correct count; main pane is empty.

Root cause

get_forum_topics() in src/db/adapter.py already handles this correctly when counting General messages:

effective_topic_id = func.coalesce(Message.reply_to_top_id, 1).label("effective_topic_id")

…with a comment that explains: "Messages with reply_to_top_id=NULL are treated as General topic (id=1), since pre-v6.2.0 messages and pre-forum messages lack topic assignment and Telegram's client displays them under General."

But get_messages_paginated() uses a strict equality filter:

if topic_id is not None:
    stmt = stmt.where(Message.reply_to_top_id == topic_id)

So ?topic_id=1 never matches the NULL-top rows, and the General pane is empty despite the sidebar claim.

The fix

Apply the same coalesce(reply_to_top_id, 1) == topic_id filter in get_messages_paginated(). General sees its NULL-top messages; non-General topics still filter strictly.

One-line change + three regression tests (tests/test_general_topic_messages.py) covering:

  • topic_id=1 returns NULL-top and explicit reply_to_top_id=1 messages (post-review fix-up)
  • non-General topic IDs still filter strictly
  • no topic_id returns everything regardless of reply_to_top_id

Tests use Base.metadata.create_all on an in-memory SQLite engine so column drift can't silently break them.

Verification against a real DB

On the affected forum chat (pre-forum + forum messages mixed):

  • GET /api/chats/<id>/messages?topic_id=1 — before: []; after: returns the General messages.
  • GET /api/chats/<id>/messages?topic_id=<other> — still returns only rows matching that exact topic.

Test plan

  • pytest tests/test_general_topic_messages.py — 3/3 pass
  • Manually verified General tab now renders messages in the viewer
  • Manually verified non-General topics still filter strictly

The viewer's topic list already displays General with its true message
count via ``coalesce(reply_to_top_id, 1)`` in ``get_forum_topics()``, but
clicking the General tab rendered an empty pane even when the sidebar
said "N messages".

Cause: ``get_messages_paginated()`` filters with strict
``Message.reply_to_top_id == topic_id``, so ``?topic_id=1`` never matches
messages whose ``reply_to_top_id`` is NULL. Pre-v6.2.0 messages and
pre-forum messages all have NULL — Telegram's own client groups them
under General.

Fix: apply the same ``coalesce(reply_to_top_id, 1) == topic_id`` filter
in the messages query so General sees its NULL-top rows while
non-General topics still filter strictly.

Regression tests in ``tests/test_general_topic_messages.py`` spin up an
in-memory SQLite DB via the ORM metadata and cover:
- ``topic_id=1`` returns NULL-top messages
- non-General topic IDs still filter strictly
- no ``topic_id`` returns all messages regardless of ``reply_to_top_id``
@tondeaf tondeaf requested a review from GeiserX as a code owner April 18, 2026 21:50
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 18, 2026

📝 Walkthrough

Walkthrough

Updated get_messages_paginated() to treat NULL Message.reply_to_top_id as General topic (id 1) via coalesce. Added async tests that verify General-topic inclusion of NULL and explicit 1, strict equality for other topics, and unfiltered retrieval when no topic_id is provided.

Changes

Cohort / File(s) Summary
Database Query Logic
src/db/adapter.py
Replaced strict equality predicate (Message.reply_to_top_id == topic_id) with func.coalesce(Message.reply_to_top_id, 1) == topic_id in forum message pagination to treat NULL as topic id 1.
Test Coverage
tests/test_general_topic_messages.py
Added async test module with helper to seed an in-memory SQLite DB and three tests verifying: General topic includes NULL and explicit 1, non-General topics filter by strict equality, and no-topic returns all messages.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and concisely summarizes the main fix: using coalesce to treat NULL reply_to_top_id as topic 1 (General) in message pagination.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed PR description covers all required template sections with clear bug explanation, fix details, and comprehensive testing verification.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
tests/test_general_topic_messages.py (2)

29-61: Shared fixture + type hint would trim duplication.

All three tests repeat the adapter, engine, chat_id = await _make_adapter_with_messages() / try/finally engine.dispose() dance. A pytest_asyncio.fixture yielding the tuple and disposing on teardown would drop ~15 lines and remove the chance of forgetting dispose() in a future test. Also, the helper is missing a return type annotation (per repo guideline: type hints for function signatures).

As per coding guidelines: "Use type hints for function signatures".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/test_general_topic_messages.py` around lines 29 - 61, Add a
pytest_asyncio.fixture that yields the (adapter, engine, chat_id) tuple created
by the existing helper and disposes the engine in the fixture teardown; update
tests to use that fixture instead of calling _make_adapter_with_messages() and
manual try/finally engine.dispose(). Also add a return type annotation to the
helper _make_adapter_with_messages (e.g., -> Tuple[DatabaseAdapter, AsyncEngine,
int]) to satisfy the repo typing guideline and keep the helper for reuse; ensure
the fixture imports pytest_asyncio and references DatabaseAdapter and
engine.dispose() for cleanup.

53-54: Nit: rename the placeholder text.

"sexy topic msg" / "another sexy msg" reads oddly in a regression test. Something neutral like "topic-47 msg #1" keeps future readers focused on the assertion, not the wording.

Proposed tweak
-            (3, datetime(2024, 3, 1, 10), "sexy topic msg", 47),
-            (4, datetime(2024, 3, 2, 10), "another sexy msg", 47),
+            (3, datetime(2024, 3, 1, 10), "topic-47 msg `#1`", 47),
+            (4, datetime(2024, 3, 2, 10), "topic-47 msg `#2`", 47),
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/test_general_topic_messages.py` around lines 53 - 54, Replace the odd
placeholder strings in the test data tuples (currently "sexy topic msg" and
"another sexy msg") with neutral, descriptive placeholders so the test reads
clearly; update the two tuples (the ones with message text and topic id 47) to
use something like "topic-47 msg `#1`" and "topic-47 msg `#2`" to keep focus on the
assertion rather than the wording.
src/db/adapter.py (1)

1101-1104: LGTM — coalesce matches get_forum_topics counting logic.

Filter now treats NULL reply_to_top_id as General (id=1), so sidebar counts and message pane agree. Non-General topics remain strict since coalesce(NULL, 1) == 47 is false.

One heads-up: if an index exists on reply_to_top_id, wrapping it in coalesce(...) will prevent its use on most engines. Likely negligible given the chat_id filter already narrows the scan, but worth keeping in mind if message volumes grow.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/db/adapter.py` around lines 1101 - 1104, The current use of
func.coalesce(Message.reply_to_top_id, 1) in stmt.where disables index use on
reply_to_top_id; replace the coalesce predicate with an explicit conditional
that preserves indexability: filter by Message.reply_to_top_id == topic_id for
non-General topics, and for topic_id == 1 add an additional OR branch that
matches Message.reply_to_top_id IS NULL (i.e., treat NULL as General) so the
query uses a direct column comparison on Message.reply_to_top_id instead of
wrapping it in coalesce; update the stmt.where call accordingly (referencing
Message.reply_to_top_id, stmt.where, and the current topic_id handling).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/db/adapter.py`:
- Around line 1101-1104: The current use of
func.coalesce(Message.reply_to_top_id, 1) in stmt.where disables index use on
reply_to_top_id; replace the coalesce predicate with an explicit conditional
that preserves indexability: filter by Message.reply_to_top_id == topic_id for
non-General topics, and for topic_id == 1 add an additional OR branch that
matches Message.reply_to_top_id IS NULL (i.e., treat NULL as General) so the
query uses a direct column comparison on Message.reply_to_top_id instead of
wrapping it in coalesce; update the stmt.where call accordingly (referencing
Message.reply_to_top_id, stmt.where, and the current topic_id handling).

In `@tests/test_general_topic_messages.py`:
- Around line 29-61: Add a pytest_asyncio.fixture that yields the (adapter,
engine, chat_id) tuple created by the existing helper and disposes the engine in
the fixture teardown; update tests to use that fixture instead of calling
_make_adapter_with_messages() and manual try/finally engine.dispose(). Also add
a return type annotation to the helper _make_adapter_with_messages (e.g., ->
Tuple[DatabaseAdapter, AsyncEngine, int]) to satisfy the repo typing guideline
and keep the helper for reuse; ensure the fixture imports pytest_asyncio and
references DatabaseAdapter and engine.dispose() for cleanup.
- Around line 53-54: Replace the odd placeholder strings in the test data tuples
(currently "sexy topic msg" and "another sexy msg") with neutral, descriptive
placeholders so the test reads clearly; update the two tuples (the ones with
message text and topic id 47) to use something like "topic-47 msg `#1`" and
"topic-47 msg `#2`" to keep focus on the assertion rather than the wording.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ffdd2f96-3f14-492c-b1a7-22625e4b735d

📥 Commits

Reviewing files that changed from the base of the PR and between d76873a and e7d85bd.

📒 Files selected for processing (2)
  • src/db/adapter.py
  • tests/test_general_topic_messages.py

@GeiserX
Copy link
Copy Markdown
Owner

GeiserX commented Apr 18, 2026

Hey @tondeaf, thanks for this excellent bug fix! The root cause analysis in the PR description is really well done — identifying the mismatch between get_forum_topics() (coalesce) and get_messages_paginated() (strict equality) is spot on.

I ran a thorough review (code quality, security, test adequacy) and the fix looks great overall. A few items to address before merge:

Code Quality (1 low-severity note)

LOW — coalesce() may prevent index usage on idx_messages_topic
COALESCE(reply_to_top_id, 1) = ? wraps the column in a function, which prevents the B-tree index from being used directly. Pragmatically this is unlikely to matter (the chat_id predicate narrows first, and LIMIT 50 ORDER BY date DESC stops early), but worth noting. An alternative that preserves index usage for non-General topics:

if topic_id == 1:
    stmt = stmt.where(or_(Message.reply_to_top_id == 1, Message.reply_to_top_id.is_(None)))
else:
    stmt = stmt.where(Message.reply_to_top_id == topic_id)

That said, the current approach is consistent with get_forum_topics() which already uses the same coalesce pattern, so I'd understand keeping it as-is for consistency. Your call.

Test Adequacy (1 missing edge case)

Please add a test for explicit reply_to_top_id=1: The current seed data only has NULL for General messages, but in production, after forums are enabled, Telegram sets reply_to_top_id=1 on new General messages. Add a seed row like:

(6, datetime(2024, 5, 1, 10), "explicit general msg", 1),

Then update test_general_topic_includes_null_reply_to_top_id to assert ids == [1, 2, 6] — proving both NULL and explicit-1 messages appear together under topic_id=1.

CodeRabbit

CodeRabbit flagged that the PR description doesn't follow the repo's PR template (Type of Change checkboxes, Database Changes section, etc.). Would be nice to restructure the description to match the template format.

Security

No concerns — the change is a parameterized SQLAlchemy expression, chat_id filter is unconditional, and auth is enforced before the query runs. Clean.

Thanks again for the contribution! 🙏

Address PR GeiserX#122 review: the existing seed data only had NULL
reply_to_top_id rows for the General topic, but post-forum-enable
Telegram sets reply_to_top_id=1 explicitly on new General messages.
Add a row with reply_to_top_id=1 and assert it appears alongside
the NULL rows under topic_id=1, proving the coalesce predicate
matches both representations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/test_general_topic_messages.py`:
- Around line 29-64: Replace the fragile DatabaseManager.__new__ pattern in
_make_adapter_with_messages with a normal constructor call using
DatabaseManager(database_url="sqlite+aiosqlite://") so __init__ runs and
attributes are set correctly, and add an explicit return type hint on
_make_adapter_with_messages: async def _make_adapter_with_messages() ->
tuple[DatabaseAdapter, AsyncEngine, int]: to reflect the returned
(DatabaseAdapter, engine, chat_id) tuple; keep the rest of the seeding logic
unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 93c58d48-8952-40db-bfc3-4ea4c2fd0bda

📥 Commits

Reviewing files that changed from the base of the PR and between e7d85bd and 8554478.

📒 Files selected for processing (1)
  • tests/test_general_topic_messages.py

Comment on lines +29 to +64
async def _make_adapter_with_messages():
"""Spin up an in-memory SQLite DB seeded with a forum chat + mixed-topic messages."""
engine = create_async_engine(
"sqlite+aiosqlite://",
poolclass=StaticPool,
connect_args={"check_same_thread": False},
)

async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)

db_manager = DatabaseManager.__new__(DatabaseManager)
db_manager.engine = engine
db_manager.database_url = "sqlite+aiosqlite://"
db_manager._is_sqlite = True
db_manager.async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)

chat_id = -1002010842190
async with db_manager.async_session_factory() as session:
session.add(Chat(id=chat_id, type="group", title="Forum Chat", is_forum=1))
rows = [
# (id, date, text, reply_to_top_id)
(1, datetime(2024, 1, 1, 10), "pre-forum general #1", None),
(2, datetime(2024, 1, 2, 10), "pre-forum general #2", None),
(3, datetime(2024, 3, 1, 10), "sexy topic msg", 47),
(4, datetime(2024, 3, 2, 10), "another sexy msg", 47),
(5, datetime(2024, 4, 1, 10), "photos topic", 144),
# Post-forum-enable: Telegram sets reply_to_top_id=1 explicitly on
# new General messages. Must appear alongside NULL rows under topic_id=1.
(6, datetime(2024, 5, 1, 10), "post-forum general (explicit 1)", 1),
]
for mid, dt, body, top in rows:
session.add(Message(id=mid, chat_id=chat_id, date=dt, text=body, reply_to_top_id=top))
await session.commit()

return DatabaseAdapter(db_manager), engine, chat_id
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
ast-grep --pattern $'class DatabaseManager:
  $$$
'
fd -e py . tests | xargs rg -nP -C2 'DatabaseManager\.__new__|DatabaseManager\('

Repository: GeiserX/Telegram-Archive

Length of output: 27480


Replace DatabaseManager.__new__ with normal instantiation and add return type hint.

The DatabaseManager.__new__(DatabaseManager) plus manual attribute injection is unnecessary and fragile. DatabaseManager.__init__ accepts a database_url parameter, so just use:

db_manager = DatabaseManager(database_url="sqlite+aiosqlite://")

This avoids circumventing initialization and is clearer.

Also add the return type hint to the helper:

async def _make_adapter_with_messages() -> tuple[DatabaseAdapter, AsyncEngine, int]:

(Optional: The three tests rebuild engine + schema + seed each time. A pytest_asyncio fixture would reduce boilerplate and consolidate cleanup, but the current approach works.)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/test_general_topic_messages.py` around lines 29 - 64, Replace the
fragile DatabaseManager.__new__ pattern in _make_adapter_with_messages with a
normal constructor call using
DatabaseManager(database_url="sqlite+aiosqlite://") so __init__ runs and
attributes are set correctly, and add an explicit return type hint on
_make_adapter_with_messages: async def _make_adapter_with_messages() ->
tuple[DatabaseAdapter, AsyncEngine, int]: to reflect the returned
(DatabaseAdapter, engine, chat_id) tuple; keep the rest of the seeding logic
unchanged.

@tondeaf
Copy link
Copy Markdown
Contributor Author

tondeaf commented Apr 25, 2026

Addressed your feedback in the latest commit. Summary of what changed and why:

Test row for explicit reply_to_top_id=1

  • Added a row with reply_to_top_id=1 (post-forum-enable General) and updated test_general_topic_includes_null_and_explicit_one (renamed) to expect [1, 2, 6]. test_no_topic_filter_returns_all updated to [1, 2, 3, 4, 5, 6].

Index-usage alternative (LOW)

  • Left as-is. As you noted, the current approach matches get_forum_topics() and the chat_id predicate + LIMIT make the practical impact negligible. Happy to switch to the or_(eq, is_null) form if you'd rather have it.

PR description template

  • Not yet — wanted to keep this commit narrowly scoped. I'll restructure the body if you'd like before merge.

Ready for another look.

@GeiserX GeiserX merged commit e0a7df2 into GeiserX:main Apr 25, 2026
2 checks passed
@GeiserX
Copy link
Copy Markdown
Owner

GeiserX commented Apr 25, 2026

Thanks @tondeaf! This fix is now live in v7.6.4. 🎉

The coalesce alignment between get_forum_topics() and get_messages_paginated() was a great catch — General topic users were seeing a message count in the sidebar but getting zero results when clicking in.

I also shipped a follow-up in #132 that built on your test suite: renamed the test data to neutral names, converted to a pytest fixture, removed redundant decorators, and added edge case tests for nonexistent topics, topic_id=0, and topic+search interaction. Your original 3 tests were solid foundations for all of that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants