As per your comment below, map_scalar.is_valid needs to be checked before casting, but in that case, how could a NullScalar of an appropriate type be made (which requires item_type)?

Ah sorry, the cast from Scalar to MapScalar is fine - it's the cast from MapScalar.value to StructArray that is failing, because value is nullptr when is_valid == false.

I would suggest looking at this in a debugger if you haven't already, just to see what this case looks like.

Doing something like:

auto descrs = batch.GetDescriptors();
std::shared_ptr<DataType> type = descrs.front().type;
std::shared_ptr<DataType> item_type = checked_cast<const MapType&>(*type).item_type();

Like in ResolveMapArrayLookupType okay?

checked_cast<const MapType&>(*batch[0].type()) is enough, a Datum has its own type.

This does the query_key null checking part, right? @lidavidm
The Errors test also tests for this.

No, this tests if the shared_ptr is null, but a valid Scalar may contain a null value (Scalar.is_valid) - that needs to be checked as well

(Also, we should break out the error separately for !query_key || !query_key->is_valid so that we give a relevant error message. This will crash if we give a nullptr query_key since the error message tries to read query_key->type.)

Just a reminder to revisit and implement this.

To do so, you'll need to:

1. Add the options class definition to the Cython pxd:

   arrow/python/pyarrow/includes/libarrow.pxd

   Lines 2147 to 2150 in 39367db

   	cdef cppclass CIndexOptions \
   	"arrow::compute::IndexOptions"(CFunctionOptions):
   	CIndexOptions(shared_ptr[CScalar] value)
   	shared_ptr[CScalar] value

2. Add a Python options class wrapper to the Cython code:

   arrow/python/pyarrow/_compute.pyx

   Lines 1317 to 1333 in 39367db

   	cdef class _IndexOptions(FunctionOptions):
   	def _set_options(self, scalar):
   	self.wrapped.reset(new CIndexOptions(pyarrow_unwrap_scalar(scalar)))
   	class IndexOptions(_IndexOptions):
   	"""
   	Options for the `index` function.
   	Parameters
   	----------
   	value : Scalar
   	The value to search for.
   	"""
   	def __init__(self, value):
   	self._set_options(value)

3. Add an import so it's in the public API:

   arrow/python/pyarrow/compute.py

   Line 18 in 39367db

   from pyarrow._compute import ( # noqa

4. Add the options class to the serialization test:

   arrow/python/pyarrow/tests/test_compute.py

   Line 138 in 39367db

   pc.IndexOptions(pa.scalar(1)),

   (just add it to the list there, no need to add additional assertions to the test)
5. Add a brief test to make sure everything works (no need to add detailed tests, we just want to make sure the bindings were set up):

   arrow/python/pyarrow/tests/test_compute.py

   Lines 2081 to 2092 in 39367db

   	def test_index():
   	arr = pa.array([0, 1, None, 3, 4], type=pa.int64())
   	assert pc.index(arr, pa.scalar(0)).as_py() == 0
   	assert pc.index(arr, pa.scalar(2, type=pa.int8())).as_py() == -1
   	assert pc.index(arr, 4).as_py() == 4
   	assert arr.index(3, start=2).as_py() == 3
   	assert arr.index(None).as_py() == -1
   	arr = pa.chunked_array([[1, 2], [1, 3]], type=pa.int64())
   	assert arr.index(1).as_py() == 0
   	assert arr.index(1, start=2).as_py() == 2
   	assert arr.index(1, start=1, end=2).as_py() == -1

I'm really sorry to bikeshed here, but now that I look at it…would "map_lookup" be a better kernel name? I think we only use "array" in a kernel name when the kernel only works on arrays, but this works on arrays and scalars.

Not a problem, better to be rigorous here than have an inconsistent codebase. We just will have to be on the lookout for anything that I've missed renaming.