Skip to content

[RFC] Fix Type Conflict Schema Merging for Cross-Index Fields in Append and Multisearch Commands #4383

@ahkcs

Description

@ahkcs

Problem Statement

Both append and multisearch commands currently handle schema merging with type conflicts using a field renaming workaround instead of implementing intelligent type resolution. While both commands now work functionally, they create confusing user experiences with duplicated semantic fields.

Current Behavior Analysis

Both append and multisearch commands now work around type conflicts by renaming conflicting fields:

Example with Type Conflict:

// Append syntax:
source=account | fields firstname, age, balance | head 2 |
append [ source=locations | fields description, age, place_id | head 2 ]

// Multisearch syntax:
| multisearch
  [search source=account | fields firstname, age, balance | head 2]
  [search source=locations | fields description, age, place_id | head 2]

Current Result Schema (Both Commands):

  • firstname (string) - from account
  • age (bigint) - from account (original)
  • balance (bigint) - from account
  • description (string) - from locations
  • age0 (string) - from locations (renamed to avoid conflict)
  • place_id (int) - from locations

Problem: This creates confusing column names and duplicated semantic fields (age and age0 both represent the same logical concept).

Test Index Schemas Used

ACCOUNT Index:

  • firstname (string), age (bigint), balance (bigint), account_number (bigint)

LOCATIONS_TYPE_CONFLICT Index:

  • description (string), age (string), place_id (integer)

Long-Term Goals

Expected Behavior: Both commands should intelligently merge schemas with type conflicts by:

  1. No Type Conflict: Automatically resolve type conflicts

    • age (bigint) + age (string) → age
    • balance (int) + balance (double) → balance
  2. Consistent Behavior: Both append and multisearch should use identical schema merging logic

Potential solution

#4349

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languageRFCRequest For Commentscalcitecalcite migration releatedenhancementNew feature or request

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions