Skip to content

Conversation

@NathanBaulch
Copy link
Contributor

@NathanBaulch NathanBaulch commented Sep 24, 2025

I finally got around to implementing iter support, as discussed last year in #439. This turned into a pretty big task, with over 160 new helper functions based on existing slice oriented helpers in the iter sub-package! ✨

Implementation notes:

  1. Requires Go 1.23 - might need to be part of a v2 release. Alternatively it could be a sub-module.
  2. I've followed the I suffix convention used in the mutable package for helper callbacks that include an index parameter.
  3. Variadic parameters have been left as is - converting to sequences made helpers awkward to use.
  4. Difference, FilterReject, Unzip* not supported - no reasonable way to fan-out to multiple sequences.
  5. Reverse, ReduceRight not supported - would need to buffer entire sequence.
  6. Sample*, Shuffle not supported - sequence length is unknown and no way to seek randomly forwards/backwards.
  7. OmitBy*, PickBy* not supported - picking/omitting from a map based on a sequence of keys/values doesn't seem useful.
  8. Words not supported - not worthwhile with the current regex approach which appears to be intended for splitting relatively small identifiers.
  9. DropByIndex, Subset, *Nth*, Splice don't support negative indexes.
  10. GroupBy, PartitionBy, Mode return slices - they already have to scan the full input sequence.
  11. Chunk returns a sequence of slices - feels awkward returning a sequence of sequences. Can we reuse the allocated slice?
  12. FindUniques* is implemented but needs to fully iterate the input sequence before yielding anything - not sure if reasonable.
  13. FindDuplicates* yields items in order of the first duplicate - this order differs slightly from the slice version.
  14. Fill is implemented but the name feels misleading - there's no pre-allocated memory being "filled".
  15. DropByIndex, WithoutNth are identical - the only difference to the slice version is DropByIndex supports negative indexes.
  16. DropRight allocates an internal buffer slice of size n.
  17. DropRightWhile allocates an internal buffer slice that grows to the longest sub-sequence of items for which the predicate returns true.
  18. Map helpers use iter.Seq2 and do not use lo.Entry - more idiomatic.
  19. Entries, ToPairs, FromEntries, FromPairs accept multiple variadic inputs similar to Keys and Values - otherwise these helpers wouldn't offer any benefit over maps.All and maps.Collect.
  20. Users will need to be careful about passing infinite sequences to helpers like lo.Sum - will block forever.
  21. Passing a nil sequence panics - this is consistent with slices.Collect.
  22. I've used signed int or constraints.Integer everywhere even if negative arguments are not supported.

@codecov
Copy link

codecov bot commented Sep 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.51%. Comparing base (64bb58c) to head (d2aa9b3).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #672   +/-   ##
=======================================
  Coverage   94.51%   94.51%           
=======================================
  Files          18       18           
  Lines        3429     3429           
=======================================
  Hits         3241     3241           
  Misses        174      174           
  Partials       14       14           
Flag Coverage Δ
unittests 94.51% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@NathanBaulch
Copy link
Contributor Author

I've had a lot of issues getting unit tests passing reliably due to timing dependencies in the channel/concurrency/retry tests - nothing to do with the specific changes introduced in this PR.
FWIW, I managed to get them passing in my fork after a few manual retries: https://github.com/NathanBaulch/lo/actions/runs/17966454606

@samber
Copy link
Owner

samber commented Sep 24, 2025

Hey @NathanBaulch

A big thank you for this work. 👏


  • 1. Instead of supporting only go >= 1.23, what about a directive similar to the internal/rand/ package ? I maintain this library as backward compatible as possible, for broad usage and LTS.
  • 3. I agree.
  • 5. + 6. Since you added GroupBy+PartitionBy+Mode, i don't feel uncomfortable adding such helpers, especially if this is properly documented.
  • 11. It does not feel safe to me to reuse the slice. The spirit of this library is to offer easy-to-use helpers. If you need to optimize something: build it yourself. This is the Pareto Principle.
  • 18. Good. I have never been satisfied with the lo.Entry type. I should have used a Tuple2.
  • 22. Good. This is consistent with the rest of the library.

I wonder if it could be useful to duplicate some helpers to receive/return iter.Seq2 iterators. Chaining multiple lo helpers without having the iter.Seq2 variant seems complicated. It can be done in a different PR, but i'm asking this now to prevent any breaking change in method naming.

If we do such a variant, we have to think about the meaning of the

Anyway, 2 more helpers would be nice: Seq1ToSeq2 and Seq2ToSeq1, with K being a forever-increasing int in Seq1ToSeq2. WDYT?

Other proposal: add a Void or Flush method for consuming the iterator.


I just figured out that iter might not be the right name for this package, as suggested in #439. In fact, you cannot import twice a package called "iter" in a go file (iter std package + github.com/samber/lo/iter).

May I suggest "it"? Any idea?


No need to create examples on https://play.golang.org/. First, because we need to merge+release this contribution before, but also because I made an MCP server for creating all of them in 1 prompt.

I think it's time for me to create a dedicated website for documentation. Mixing basic helpers with mutable, parallel, and iterable helpers will make this README.md unreadable.


FYI, I will be releasing a reactive programming lib very soon (next week?), with an API similar to rxjs.

The API will be very different and stream-oriented (eg: Sum won't return a single value at the end, but as many values sent to it).

@NathanBaulch NathanBaulch force-pushed the iter branch 2 times, most recently from 638f431 to 1e67376 Compare September 25, 2025 10:53
@NathanBaulch
Copy link
Contributor Author

Since you added GroupBy+PartitionBy+Mode, i don't feel uncomfortable adding such helpers, especially if this is properly documented.

Just to clarify, you're saying I should implement Reverse, ReduceRight, Sample*, Shuffle (with a disclaimers in the docs) even though they need to scan the full input sequence?

If we do such a variant, we have to think about the meaning of the

Heh, where was this sentence going?

2 more helpers would be nice: Seq1ToSeq2 and Seq2ToSeq1, with K being a forever-increasing int in Seq1ToSeq2.

Would we need two versions of Seq2ToSeq1, one for keys and one for values?

Other proposal: add a Void or Flush method for consuming the iterator.

I would propose Drain.

@samber
Copy link
Owner

samber commented Sep 25, 2025

Just to clarify, you're saying I should implement Reverse, ReduceRight, Sample*, Shuffle (with a disclaimers in the docs) even though they need to scan the full input sequence?

Yes. Some helpers you built scan the full sequence as well. I would consider that developers understand that reversing a lazily produced slice is not possible without reading it 100%.

Heh, where was this sentence going?

I don't remember 😅

Would we need two versions of Seq2ToSeq1, one for keys and one for values?

Oh yes, definitely.

I would propose Drain.

Perfect!

@NathanBaulch
Copy link
Contributor Author

NathanBaulch commented Sep 25, 2025

Some helpers you built scan the full sequence as well.

I just realized that I haven't explained how I decided which full-scan helpers to exclude. It's basically about memory - I skipped the ones that need to scan the full sequence into a slice internally. So Sum, Reduce, IsSorted are fine, but something like Reverse is just going to be a thin wrapper around the slice based Reverse helper.

func Reverse[T any, I ~func(func(T) bool)](collection I) I {
	s := slices.Collect(iter.Seq[T](collection))
	mutable.Reverse(s)
	return I(slices.Values(s))
}

Anyway, happy to add them with a suitable disclaimer in the docs.

@samber
Copy link
Owner

samber commented Sep 26, 2025

We have too many flaky tests in this project... 🤬

image

@NathanBaulch
Copy link
Contributor Author

Pretty sure I've addressed all outstanding feedback - time for an update!

  1. Added Seq to the names of helpers corresponding to existing helpers with Slice in their name.
  2. Replaced Right with Last and Left with First.
  3. Implemented new helpers Drain, Seq2Channel, SeqToSeq2, Seq2KeyToSeq, Seq2ValueToSeq.
  4. Ported helpers that iterate the full sequence Reverse, ReduceRight, Sample, Shuffle.
  5. Reviewed all helpers and added documentation if they fully iterate before returning or they allocate internal slices/maps.
  6. Ported recent additions HasPrefix, HasSuffix, Cut*, Trim*.
    Cut isn't supported since it would need to return multiple sequences, similar to Difference.
  7. Implemented new helper Length which corresponds to len(myslice). Can rename to Len if preferred.
  8. Intersect is now variadic, taking any number of input sequences. Perhaps the slice version can also in v2.
  9. Renamed IsSortedByKey to IsSortedBy to be more consistent with other *By helpers. Another idea for v2.
  10. Renamed all map/filter callback parameters to transform/predicate respectively. "iteratee" is a pretty obscure word and hasn't been used consistently. For example, lo.ForEachWhile and lo.RepeatBy callbacks are named incorrectly.
  11. Significantly reduced duplicate code by using helpers inside helpers. Iterators are well suited to this since they don't allocate return slices.
    For example, Subset calls Slice, Replace, ToAnySeq call Map, Compact calls Filter, etc.
  12. Lots of performance tweaks, especially around reducing allocations and multiple iteration.
  13. DropByIndex and WithoutNth are identical, Contains is the same as Some if only passed a single item.

@samber
Copy link
Owner

samber commented Oct 2, 2025

Fantastic!

It's time to merge this PR. Thanks for your work on this long-awaited feature!

Next steps for me:

  • release of v1.52.0
  • publish lo.samber.dev
  • fix these flaky tests

@samber samber merged commit 43cef1f into samber:master Oct 2, 2025
66 of 107 checks passed
@NathanBaulch NathanBaulch deleted the iter branch October 2, 2025 21:03
@samber
Copy link
Owner

samber commented Oct 6, 2025

Documentation is available here: https://lo.samber.dev/docs/iter/map

The runnable examples will be published on Go playground tomorrow or on Wednesday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants