-
-
Notifications
You must be signed in to change notification settings - Fork 48
Closed
Milestone
Description
Sometimes when converting from one data format to another some metadata is missing, for example, when converting from PHP int into Parquet int we can't say for sure if it's INT32 or INT96.
The same applies for the databases, we never know what's the maximum length of a String column.
The solution for that problem would be to let each Adapter to provide and handle custom schema metadata that can be used to improve transformation precision.
For example Doctrine Adapter could expose metadata like:
class DoctrineMatadata
{
public const STRING_LENGTH = 'string_length';
}
$schema = schema(
str_schema('name', nullable: true, metadata: Metadata::fromArray([DoctrineMetadata::STRING_LENGTH => 255])),
);Of course, we would need to build a mechanism that would covert schemas from Flow into Doctrine DBAL first to support that metadata fully.
But it's a generic concept that could be applied for example in Parquet or in the #1353