Skip to content

Record types#6679

Merged
bentsherman merged 1 commit intomasterfrom
record-types
Mar 9, 2026
Merged

Record types#6679
bentsherman merged 1 commit intomasterfrom
record-types

Conversation

@bentsherman
Copy link
Member

@bentsherman bentsherman commented Dec 19, 2025

This PR implements the first preview of record types:

  • Add Record type to standard types
  • Add record() function to create a record from named arguments
  • Add record types via record keyword for validating records
  • Add ability to destructure a record in a process input
  • Fix hashing of records
  • Add ability to include records across modules
  • Add ability to stage record type

Cache breaking: This PR changes the way that map values are hashed. It now includes both the keys and values instead of only the values. It also prioritizes CacheFunnel over Map, so that custom classes can implement Map while also implementing CacheFunnal in order to define custom hashing behavior.

@netlify
Copy link

netlify bot commented Dec 19, 2025

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit e28ef7f
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69af2e622c5a1e00087c29ea
😎 Deploy Preview https://deploy-preview-6679--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@bentsherman
Copy link
Member Author

This PR does not make any changes to operators, so you need some extra logic in some cases to pass records through an operator.

For example, given two record channels that you want to join:

samples_ch = fastqc_ch.join(quant_ch, by: 'id')

The above syntax is what we'd like to do, but join doesn't work with records yet, so you have to wrap the records as tuples before the join and unwrap after the join:

samples_ch = fastqc_ch.map { r -> tuple(r.id, r) }
    .join( quant_ch.map { r -> tuple(r.id, r) } )
    .map { id, r1, r2 -> r1 + r2 }

@bentsherman bentsherman added the cache-breaking Changes that will break everyone's task cache label Dec 19, 2025
@bentsherman bentsherman force-pushed the record-types branch 2 times, most recently from 4f335a9 to 01e71db Compare January 16, 2026 19:42
@bentsherman bentsherman marked this pull request as ready for review January 21, 2026 21:39
@bentsherman bentsherman requested review from a team as code owners January 21, 2026 21:39
Copy link
Collaborator

@christopher-hakkaart christopher-hakkaart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs are excellent. No comments.

@bentsherman bentsherman force-pushed the record-types branch 2 times, most recently from 0cbf80f to 18ae371 Compare January 23, 2026 19:09
@bentsherman bentsherman added this to the 26.04 milestone Feb 6, 2026
@bentsherman bentsherman force-pushed the record-types branch 2 times, most recently from d8d92b3 to 5880e05 Compare February 11, 2026 19:02
pinin4fjords added a commit to pinin4fjords/rnaseq that referenced this pull request Feb 13, 2026
Demonstrates how record types (nextflow-io/nextflow#6679) would
simplify the workflow outputs pattern by replacing groups of related
tuple-based channels with single record-typed channels.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@pditommaso pditommaso force-pushed the master branch 2 times, most recently from d9fa5cd to d752bc2 Compare February 28, 2026 13:10
@bentsherman bentsherman requested a review from pditommaso March 2, 2026 19:21
@christopher-hakkaart christopher-hakkaart self-requested a review March 3, 2026 03:41
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review of the record types feature. Overall well-structured implementation following existing codebase patterns. Flagging 5 issues ranging from a mutability/data-race risk to minor UX improvements.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman merged commit d54ff29 into master Mar 9, 2026
24 checks passed
@bentsherman bentsherman deleted the record-types branch March 9, 2026 21:04
@pditommaso
Copy link
Member

Does this change the hashing irrespective the use of record types? if so could be confined only when the feature is enabled?

@bentsherman
Copy link
Member Author

The hashing change was to hash maps by their keys and values instead of only their values

It made more sense to apply this change for all records and maps because it fixes a bug (#4916) where e.g. a meta-map key would be renamed but the task wouldn't re-execute

@pditommaso
Copy link
Member

pditommaso commented Mar 11, 2026

Considering the impact breaking the cache in production deployment it may be worth considering making this configurable. Default should use the new version, however it could be possible to back to the previous caching. it should be fairly simple to have a caching strategy as a singleton in the session or process level.

@bentsherman
Copy link
Member Author

bentsherman commented Mar 11, 2026

I'm not sure that anyone is asking for that, but if we were to do anything about it, I would do something like #6353 where we add an environment variable to the STABLE/26.04.x branch after it is released, so that it doesn't clutter the codebase going forward

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cache-breaking Changes that will break everyone's task cache

Projects

None yet

4 participants