Does this broadcast arrays/sclars?

@charris: No, this does not broadcast at all, as concatenate does not. All it does is promote dimensionality, so that (1, 2) is the same as (1, 1, 1, 2).

As I mention in the front-matter though, broadcasting would be desirable in future, but I think most of the utility is already there without it.

Can I have some feedback about the restrictions I proposed adding regarding list nesting and forbidding tuples?

@eric-wieser The list nesting restriction seems reasonable. It never hurts to make things explicit, especially if you plan on adding possibilities.

Why do you want to restrict tuples? Personally, that's how I prefer to pass in a sequence of arrays, so I would be against that particular restriction.

Also, as far as your latest proposed enhancement goes (an array object containing other arrays), perhaps you could just treat an ndarray input with dtype=object the same as list and tuple (e.g. by expanding is_element)?

Why do you want to restrict tuples?

• Deliberately make np.block look similar to the usual invocation of np.array

• Leave room for future expansion. For instance, structured arrays:

  dt = np.dtype([('x', int), ('y', int)])
  A = np.array([(1, 2), (3, 4)], dtype=dt)
  
  # now
  B = np.block(A, np.array((5, 6), dtype=dt))
  
  # possibly in future
  B = np.block([A, (5, 6)])  # requires us to not "use up" the `()` syntax
  B = np.block([A, (5, 6)], dtype=dt)  # alternative option with a flag to disable that syntax
                                       # may still surprise the user, but not incompatible

perhaps you could just treat an ndarray input with dtype=object the same as list and tuple

This would have bad effects on actually trying to concatenate object arrays. Also, I'd like this code path to forbid the kind of stacking that produces:

AABB
CCCC

I don't think item 1 is a particularly good reason, but item 2 is pretty convincing.

Another item you may want to add to the expansion list is adding an explicit dtype parameter as your example suggests. That would probably go hand-in-hand with implementing preallocation and moving away from concatenate.

That would probably go hand-in-hand with implementing preallocation and moving away from concatenate.

I think the correct path here is to add a dtype argument to concatenate, but that's for another time.

Regarding preallocation, I think that would be best solved with an out argument in concatenate

@eric-wieser - this is a very nice improvement on what was already a nice idea. I have only skimmed the code, but I find the notation without matching levels of [] very unclear, so would agree with your proposed restriction that levels of [ should match (this is also consistent with np.array -- e.g., np.array([[1,2],3]) returns a 2-item object array). To me, it also makes sense to forbid tuples for now, and I think your analogy of np.array is a good reason in itself (and anyway, it can be enabled later if needed, but would not be possible to revert).

Finally, I find your example in the release notes clearer than some of the examples in the docstring, so would suggest repeating it there.

@mhvk I've committed a version that allows toggling the bracket-matching with a flag. I could make some cleanup by removing that flag.

Do you want me to go ahead and enforce that always?

Thinking about it more, I can see how [[a,b], c] is still reasonably obvious, as it reads "stacked one set", now stack next axis with another", but oddly I find [c, [a,b]] more mind-jarring. In contrast, just reading every instance of a brace pair as "now go one further dimension out" seems much simpler.
E.g., [[[a1]], [[a2]]] seems quite a bit better than [a1, [[a2]]].

Overall, I'd just go for simplicity and remove the toggle: one obvious way to do it and all that.

oddly I find [c, [a,b]] more mind-jarring

Try np.block([a, b, c, [[[ [[[ ]]] ]]] ]) out for size then, which means concatenate(map(atleast7d, [a, b, c]), axis=-7). I think another argument against this input being allowed is that you have to read right to the end to understand what's going on - rather than just to the first element

Overall, I'd just go for simplicity and remove the toggle: one obvious way to do it and all that

I agree with this - I only kept it because it was there in the original PR.

Huge implementation overhaul here. Gotten rather functional in order to avoid writing the same recursion over and over again.

A couple high level comments:

• This looks very nice now -- like something I might actually use if I can wrap my head around it!
• Documentation should mention how `block uses broadcasting to expand array dimensions.
• +1 for only treating list in a special way. NumPy could use less type overloading.

Something like this then? ;)

how block uses broadcasting to expand array dimensions.

It does not.

I did think I covered this with "Elements shapes must match along the appropiate axes (without
broadcasting)". but perhaps it needs to be more obvious

+1 for only treating list in a special way

Things get pretty weird if I only treat list in this way - namely, suddenly tuple gets treated like ndarray! (because we just throw asanyarray at everything else)

This has some somewhat useful consequences:

>>> np.block([
...    ([1],
...     [1]), ([1,2,3],
...            [4,5,6])      # () guards these from np.block, and passes them
                             # to the array constructor instead
... ])
array([[1, 1, 2, 3],
       [1, 4, 5, 6]])

how block uses broadcasting to expand array dimensions.

It's not broadcasting per-se, but it sure looks a lot like broadcasting in the way it supports promoting scalars to higher dimensions.

Things get pretty weird if I only treat list in this way - namely, suddenly tuple gets treated like ndarray! (because we just throw asanyarray at everything else)

OK, this is pretty confusing...

Questions regarding that confusing thing:

1. Is this useful
2. Is this an argument for using tuple instead of list, to make the ndarray ctors more obvious

   >>> np.block((  # was previously ([
   ...    [[1],
   ...     [1]], [[1,2,3],
   ...            [4,5,6]]      # now [] guards instead of ()
   ... ))  # was previously ])

3. Should I just throw TypeError: implicit conversion from tuple to ndarray not allowed within block if anyone tries to pull this on us, because it's super-confusing?

@charris: I was hoping to add a default grid=True parameter first - with the reasoning that adding it as an afterthought would be a breaking change. But maybe that shouldn't be a concern, as if it did turn out to be worthwhile, it wouldn't be a difficult deprecation cycle

But yes, I'll squash the history down in some form, and I definitely intend to preserve @sotte's commit.

I don't know how entangled the various commits are, but with so many it might be easiest to soft reset to after @sotte's commit and then commit the five files in some reasonable order.

@charris: I'll work something out - my main issue with rebases is that something (typically Sublime Text) ends up sitting on a file handle, which messes up a git rebase -i, requiring me to start over.

@eric-wieser You work on windows, yes? The virtue of a soft reset is that all the files are there in their current state and you just need to commit them (again).

Indeed, soft resetting has some advantages, although I'd then have to rewrite my commit messages rather than just combine/edit them. Don't worry about it, I'll work something out

Anyway, are you -1 on putting grid=True in this release (eric-wieser#2)?

I think the grid keyword can be added later if the default is False. That default was also @madphysicist's suggestion, but I don't really feel strongly either way. Might be good to hear from @mhvk and @shoyer on that.

Relatedly, it might be handy to add np.b_[...] == np.block([...]), as suggested in #8899

I just want to say that I'm very happy that someone picked this up! Thanks guys!

@charris: Trimmed off 7 commits, and reordered them to put fixups to @sotte's code nearer the beginning (where possible) - trying to keep those separate from my changes, without creating too much work for myself. I've verified the final commit matches the previous one exactly, but I haven't done too much to check that the intermediate commits are still somewhat valid

Short enough now?

My sense would be to keep the PR as is (which is very nice), and leave improvements (#8899) to 1.14, hopefully informed by feedback.

@mhvk: Do we want to slip a deprecation of np.bmat into 1.13 to direct people to np.block, to try and increase awareness and therefore feedback?

Seems a bit much to deprecate bmat -- unless we're also deprecating matrix more generally...

If there is no more feedback, I'll put this in tomorrow.

Thanks Eric.