Take initial parameters from parameters yaml files and constructor arguments.#225
Take initial parameters from parameters yaml files and constructor arguments.#225nuclearsandwich merged 35 commits intomasterfrom
Conversation
sloretz
left a comment
There was a problem hiding this comment.
There are a few places where error checking is needed. Most cpython APIs can set an exception and return NULL.
rclpy/src/rclpy/_rclpy.c
Outdated
| PyErr_Format(PyExc_RuntimeError, "Failed to parse yaml params file: %s", param_files[i]); | ||
| return false; | ||
| } | ||
| allocator.deallocate(param_files[i], allocator.state); |
There was a problem hiding this comment.
I think this needs a call to allocator.deallocate(param_files, allocator.state) after the array members have been deallocated.
rclpy/src/rclpy/_rclpy.c
Outdated
| PyObject * value; | ||
| if (variant->bool_value) { | ||
| type_enum_value = 1; | ||
| value = variant->bool_value ? Py_True : Py_False; |
There was a problem hiding this comment.
Need to Py_INCREF(Py_True) or Py_INCREF(Py_False).
| value = PyUnicode_FromString(variant->string_value); | ||
| } else if (variant->byte_array_value) { | ||
| type_enum_value = 5; | ||
| value = PyList_New(variant->byte_array_value->size); |
There was a problem hiding this comment.
Need to check for NULL return here
There was a problem hiding this comment.
Same comment in a couple places below
There was a problem hiding this comment.
NULL check is now just below.
There was a problem hiding this comment.
Looks like this returns a list of bytes() instances. It might be more convenient for users if this was one bytes() object with all the bytes in it. If that sounds reasonable I think PyBytes_FromStringAndSize could convert byte_array_value in one go.
There was a problem hiding this comment.
It might be more convenient for users if this was one bytes() object with all the bytes in it.
It would be more convenient but it would violate the interface specification which states that a byte should be represented as a bytes of length one and that any of the array types should be represented by a Python list of the scalar type. http://design.ros2.org/articles/generated_interfaces_python.html
@mikaelarguedas and I talked about this and they may have a link to further discussion material.
rclpy/src/rclpy/_rclpy.c
Outdated
| type_enum_value = 5; | ||
| value = PyList_New(variant->byte_array_value->size); | ||
| for (size_t i = 0; i < variant->byte_array_value->size; i++) { | ||
| PyList_SetItem(value, i, PyBytes_FromFormat("%u", variant->byte_array_value->values[i])); |
There was a problem hiding this comment.
Since list is new you can use PyList_SET_ITEM here
There was a problem hiding this comment.
The docs don't mention it but it looks like PyBytes_FromFormat can error and return NULL.
There was a problem hiding this comment.
Same comments about PyList_SET_ITEM below
rclpy/src/rclpy/_rclpy.c
Outdated
| type_enum_value = 6; | ||
| value = PyList_New(variant->bool_array_value->size); | ||
| for (size_t i = 0; i < variant->bool_array_value->size; i++) { | ||
| PyList_SetItem(value, i, variant->bool_array_value->values[i] ? Py_True : Py_False); |
There was a problem hiding this comment.
Same comment about needing to incref True and false. PyList_SetItem and PyList_SET_ITEM steal a reference
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
| } | ||
|
|
||
| PyObject * type = PyObject_CallObject(parameter_type_cls, Py_BuildValue("(i)", type_enum_value)); |
There was a problem hiding this comment.
Need to check Py_BuildValue didn't return NULL, as well as PyObject_CallObject.
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
|
|
||
| rcl_node_t * node = (rcl_node_t *)PyCapsule_GetPointer(node_capsule, "rcl_node_t"); | ||
| const rcl_node_options_t * node_options = rcl_node_get_options(node); |
There was a problem hiding this comment.
This should be moved below the if (!node) check
rclpy/src/rclpy/_rclpy.c
Outdated
| const rcl_node_options_t * node_options = rcl_node_get_options(node); | ||
| const rcl_allocator_t allocator = node_options->allocator; | ||
| if (!node) { | ||
| PyErr_Format(PyExc_RuntimeError, "Failed to retrieve rcl node from capsule"); |
There was a problem hiding this comment.
PyCapsule_GetPointer would have already set an exception here. Returning NULL is enough.
rclpy/src/rclpy/_rclpy.c
Outdated
| return PyList_New(0); | ||
| } | ||
|
|
||
| PyObject * parameter_type_cls = PyObject_GetAttrString(parameter_cls, "Type"); |
rclpy/rclpy/node.py
Outdated
| def __init__( | ||
| self, node_name, *, cli_args=None, namespace=None, use_global_arguments=True, | ||
| start_parameter_services=True | ||
| start_parameter_services=True, initial_parameters=[] |
There was a problem hiding this comment.
The same list instance is used for every Node object initialized with the default initial_parameters. In this case it should be fine since no code modifies the list. I recommend using either None or tuple() as the default to be defensive against that though.
There was a problem hiding this comment.
Of course. 😁 I think we were even talking about this together last week.
rclpy/src/rclpy/_rclpy.c
Outdated
| * false when there was an error during parsing and a Python exception was raised. | ||
| * | ||
| */ | ||
| static bool _parse_param_files( |
There was a problem hiding this comment.
nit, the rest of the file has return type on its own line.
| for (size_t i = 0; i < variant->byte_array_value->size; i++) { | ||
|
|
||
| member_value = PyBytes_FromFormat("%u", variant->byte_array_value->values[i]); | ||
| if (NULL == member_value) { |
There was a problem hiding this comment.
The list still has its reference count incremented. Need to Py_DECREF(value) before returning.
| value = PyUnicode_FromString(variant->string_value); | ||
| } else if (variant->byte_array_value) { | ||
| type_enum_value = 5; | ||
| value = PyList_New(variant->byte_array_value->size); |
There was a problem hiding this comment.
Looks like this returns a list of bytes() instances. It might be more convenient for users if this was one bytes() object with all the bytes in it. If that sounds reasonable I think PyBytes_FromStringAndSize could convert byte_array_value in one go.
rclpy/src/rclpy/_rclpy.c
Outdated
| return NULL; | ||
| } | ||
| for (size_t i = 0; i < variant->integer_array_value->size; i++) { | ||
| PyList_SET_ITEM(value, i, PyLong_FromLong(variant->integer_array_value->values[i])); |
There was a problem hiding this comment.
PyLong_FromLong can return NULL if it fails to create a new python object.
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
| PyObject * parameter_type_cls = PyObject_GetAttrString(parameter_cls, "Type"); | ||
| if (NULL == parameter_type_cls) { | ||
| PyErr_Format(PyExc_RuntimeError, "Error getting 'Type' attribute from Parameter class"); |
There was a problem hiding this comment.
I expect PyObject_GetAttrString would have already raised AttributeError
There was a problem hiding this comment.
You would appear to be correct. https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Objects/object.c#L880
|
|
||
| if (node_options->use_global_arguments) { | ||
| if (!_parse_param_files(rcl_get_global_arguments(), allocator, params)) { | ||
| return NULL; |
There was a problem hiding this comment.
I thought so too but rcl_parse_yaml_file will actually fini the passed in struct on error https://github.com/ros2/rcl/blob/adc0190259a4eb725480e7b32a90b902bc5279b0/rcl_yaml_param_parser/src/parser.c#L1382.
There was a problem hiding this comment.
c08b236 makes _parse_param_files fini the params struct in both error situations so that the caller does not need to.
| } | ||
|
|
||
| if (!_parse_param_files(&(node_options->arguments), allocator, params)) { | ||
| return NULL; |
There was a problem hiding this comment.
same about freeing params here
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
|
|
||
| if (!PyObject_HasAttrString(parameter_cls, "Type")) { | ||
| PyErr_Format(PyExc_RuntimeError, "Parameter class is missing 'Type' attribute"); |
There was a problem hiding this comment.
need to fini params here and a few places below
| rcl_node_params_t node_params = params->params[node_index]; | ||
| PyObject * parameter_list = PyList_New(node_params.num_params); | ||
| if (NULL == parameter_list) { | ||
| return NULL; |
There was a problem hiding this comment.
Py_DECREF(parameter_type_cls) and in another place below
rclpy/src/rclpy/_rclpy.c
Outdated
| if (RCL_RET_OK != ret) { | ||
| PyErr_Format(PyExc_RuntimeError, "Failed to get initial parameters: %s", | ||
| rcl_get_error_string_safe()); | ||
| return false; |
There was a problem hiding this comment.
Replying to https://github.com/ros2/rclpy/pull/225/files#r211049548 , since the calling code doesn't fini params if this errors then I think params needs to be fini'd here.
There was a problem hiding this comment.
With da7da1a the implementation no longer takes an rcl_params_t as input and instead creates and uses the data structure internally only if parameter arguments are found without error so the params object no longer exists at this point.
rclpy/src/rclpy/_rclpy.c
Outdated
| return NULL; | ||
| } | ||
|
|
||
| int node_index = -1; |
There was a problem hiding this comment.
MSBuild (rightly) complains that intmight lose information when assigned from a size_t below. But changing to a size_t makes the sigil value -1 invalid. Do we have a pattern we use for this sort of situation like storing a second node_index_found value and setting and checking it along with the index?
|
While setting up system tests I discovered that the yaml parameter parser doesn't preserve any parameters currently in the struct as I though. So I'll need to refactor this to extract parameters one file as a time and store them in a Python dict rather than a list. |
This refactor is complete and the implementation is now working against the existing tests in system_tests/test_cli. |
rclpy/rclpy/node.py
Outdated
| self.__executor_weakref = None | ||
|
|
||
| node_parameters = _rclpy.rclpy_get_node_parameters(Parameter, self.handle) | ||
| # use the set_parameters API so parameter events are publish. |
There was a problem hiding this comment.
I tried to read the code to understand this but I'm a bit lost I think. In this case aren't the parameters potentially set twice, once as the "node parameters" and then again using the initial parameters?
And wouldn't that generate two events? Is that the intended behavior? I could see it either way: "I'd like to see the evolution of the values on startup" or "I just care about what it was initialized with not what went into determining the initial value".
There was a problem hiding this comment.
I thought I was matching the rclcpp behavior in sending events for parameters changed by subsequent parameter files but I must've been seeing double as ros2 topic echo /parameter_events for a cpp node only sends events for the end state of initial parameters.
|
Looks like this comment is still pending right? |
It was addressed indirectly. Added an explanation. |
rclpy/rclpy/node.py
Outdated
| elif node_parameters: | ||
| self.set_parameters(node_parameters) | ||
| elif initial_parameters: | ||
| self.set_parameters(initial_parameters) |
There was a problem hiding this comment.
At first glance, it looks like this could be simplified quite a bit by setting default values for node_parameters and initial_parameters.
(I didnt check the implementation so this is hypothetical)
But if:
initial_parameters defaulted to an empty list and node_parameters to an empty dict.
Could we get rid of the conditional logic and just have the following?
params_by_name = {p.name: p for p in node_parameters}
params_by_name.update({p.name: p for p in initial_parameters})
self.set_parameters(params_by_name.values())
I wouldnt expect a big performance hit if these are empty.
mikaelarguedas
left a comment
There was a problem hiding this comment.
looks like a few comments were not addressed, added a few comments inline as well
rclpy/src/rclpy/_rclpy.c
Outdated
|
|
||
| /// Create an rclpy.parameter.Parameter from an rcl_variant_t | ||
| /** | ||
| * On failure a Python exception is raised and false is returned if: |
There was a problem hiding this comment.
NULL is returned on failure (which is the right thing to return 👍 we should just update the docblock).
not sure what the Matches the rest of the file, pretty surprising sentence though but out of scope of this PRif: is referring to ? are we missing a sentence here?
rclpy/src/rclpy/_rclpy.c
Outdated
| * \param[in] parameter_type_cls The PythonObject for the Parameter.Type class. | ||
| * | ||
| * Returns a pointer to an rclpy.parameter.Parameter with the name, type, and value from | ||
| * the variant or NULL when raising a python exception. |
There was a problem hiding this comment.
Nit: python -> Python same multiple times below
rclpy/src/rclpy/_rclpy.c
Outdated
| PyObject * value; | ||
| PyObject * member_value; | ||
| if (variant->bool_value) { | ||
| type_enum_value = 1; |
There was a problem hiding this comment.
are these type_enum_values matching the types defined in https://github.com/ros2/rcl_interfaces/blob/db27f0e8619460848d80c1442f7fec0c56ee63e5/rcl_interfaces/msg/ParameterType.msg ?
If yes, any reason for not reusing these enums?
rclpy/src/rclpy/_rclpy.c
Outdated
| int param_files_count = rcl_arguments_get_param_files_count(args); | ||
| rcl_ret_t ret; | ||
| bool successful = true; | ||
| if (param_files_count > 0) { |
There was a problem hiding this comment.
we could reduce nesting by just returning false if param_files_count <= 0
rclpy/src/rclpy/_rclpy.c
Outdated
| rcl_ret_t ret; | ||
| bool successful = true; | ||
| if (param_files_count > 0) { | ||
| ret = rcl_arguments_get_param_files(args, allocator, ¶m_files); |
There was a problem hiding this comment.
as ret is not used anywhere else we could just check the return value of rcl_arguments_get_param_files and return on failure
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
| PyObject * param = PyObject_CallObject(parameter_cls, args); | ||
| Py_DECREF(args); | ||
| Py_DECREF(type); |
rclpy/src/rclpy/_rclpy.c
Outdated
| * Raises RuntimeError if the parameters file fails to parse | ||
| * | ||
| * \param[in] parameter_cls The rclpy.parameter.Parameter class object. | ||
| * \param[in] node_handle Capsule pointing to the node handle |
There was a problem hiding this comment.
Nit: node_handle -> node_capsule
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
|
|
||
| rcl_node_t * node = (rcl_node_t *)PyCapsule_GetPointer(node_capsule, "rcl_node_t"); | ||
| if (!node) { |
rclpy/src/rclpy/_rclpy.c
Outdated
| } | ||
| PyObject * parameter_type_cls = PyObject_GetAttrString(parameter_cls, "Type"); | ||
| if (NULL == parameter_type_cls) { | ||
| /* PyObject_GetAttrString raises AttributeError on failure. */ |
There was a problem hiding this comment.
Nit: please use // for single line comments
| } | ||
| Py_DECREF(parameter_type_cls); | ||
|
|
||
| const char * node_namespace = rcl_node_get_namespace(node); |
There was a problem hiding this comment.
is this deallocated anywhere?
There was a problem hiding this comment.
No. Since this is a const "borrowed" reference to the internal node string. It's lifetime is bound to the node it is from.
| { | ||
| "rclpy_convert_from_py_qos_policy", rclpy_convert_from_py_qos_policy, METH_VARARGS, | ||
| "Convert a QoSPolicy python object into a rmw_qos_profile_t." | ||
| "Convert a QoSPolicy Python object into a rmw_qos_profile_t." |
There was a problem hiding this comment.
@mikaelarguedas had me turn all my python references to Python so I brought this one along for the ride. Has nothing else to do with this PR.
Uses rcl_yaml_param_parser to find and parse parameter yaml files and supply initial parameters to rclpy nodes.
rclpy nodes can now take a list of initial parameters via their constructor and will check rcl arguments for parameter files returning any parameters specific to that node.
This also fixes an issue where early return would prevent deallocation of later array members.
The previous implementation relied on the (later debunked) assumption that calling rcl_parse_yaml_file repeatedly with the same rcl_params_t would yield a struct that contains the cumulative parameters from those files. That isn't actually the case so we need to create an intermediate storage which can be used instead. In rclcpp a std::map is used but we have no such datatype here. Instead we'll use a PyDict which maps node names to a nested dict of rclpy.parameter.Parameters by parameter name and then query that struct for parameters to return to the node.
This was reincorporated unintentionally during merge conflict resolution.
b5fd549 to
2a60867
Compare
|
FYI for any reviewers who have pulled locally I've just rebased and pushed in order to resolve merge conflicts. |
| return false; | ||
| } | ||
| PyObject * parameter_dict; | ||
| if (!PyDict_Contains(node_params_dict, py_node_name)) { |
There was a problem hiding this comment.
the docs say this can set an exception and return -1 , though I'm not sure under what circumstances
There was a problem hiding this comment.
I did a bit of empirical testing and it can return -1 when the item you pass in as a dict isn't. Since we create the dict that doesn't seem like a case worth checking at runtime.
| } | ||
| if (-1 == PyDict_SetItem(node_params_dict, py_node_name, parameter_dict)) { | ||
| Py_DECREF(py_node_name); | ||
| return false; |
There was a problem hiding this comment.
Py_DECREF(parameter_dict) too
| parameter_dict = PyDict_GetItem(node_params_dict, py_node_name); | ||
| if (NULL == parameter_dict) { | ||
| Py_DECREF(py_node_name); | ||
| return false; |
There was a problem hiding this comment.
Confusingly, PyDict_GetItem does not set an exception. In the rest of the places this function returns false a python exception is already set before returning.
rclpy/src/rclpy/_rclpy.c
Outdated
| Py_INCREF(parameter_dict); | ||
| } | ||
| rcl_node_params_t node_params = params->params[i]; | ||
| for (size_t ii = 0; ii < node_params.num_params; ii++) { |
There was a problem hiding this comment.
Nit, use prefix increment ++ii, though I'm sure the compiler is smart enough to not make a copy here anyways.
| * \param[in] args The arguments to parse for parameter files | ||
| * \param[in] allocator Allocator to use for allocating and deallocating within the function. | ||
| * \param[out] params_by_node_name A Python dict object to place parsed parameters into. | ||
| * |
There was a problem hiding this comment.
nit, missing documentation of parameter_cls, and parameter_type_cls
rclpy/src/rclpy/_rclpy.c
Outdated
| rcl_get_error_string_safe()); | ||
| return false; | ||
| } | ||
| for (int i = 0; i < param_files_count; i++) { |
rclpy/src/rclpy/_rclpy.c
Outdated
| params, allocator, parameter_cls, parameter_type_cls, params_by_node_name)) | ||
| { | ||
| PyErr_Format(PyExc_RuntimeError, | ||
| "Failed to fill params dict from file: %s", param_files[i]); |
There was a problem hiding this comment.
Except as noted about PyDict_GetItem, an exception is already set here.
rclpy/src/rclpy/_rclpy.c
Outdated
| PyErr_Format(PyExc_RuntimeError, | ||
| "Failed to fill params dict from file: %s", param_files[i]); | ||
| rcl_yaml_node_struct_fini(params); | ||
| return false; |
There was a problem hiding this comment.
I think param_files needs to be deallocated before returning.
rclpy/src/rclpy/_rclpy.c
Outdated
| Py_DECREF(params_by_node_name); | ||
| return NULL; | ||
| } | ||
| allocator.deallocate(node_name_with_namespace, allocator.state); |
There was a problem hiding this comment.
Nit, could deduplicate this line by putting it before the NULL check above.
rclpy/src/rclpy/_rclpy.c
Outdated
| // PyDict_GetItem is a borrowed reference. INCREF so we can return a new one. | ||
| Py_INCREF(node_params); | ||
|
|
||
| Py_DECREF(parameter_type_cls); |
There was a problem hiding this comment.
I think this was already decref'd
There was a problem hiding this comment.
Need to Py_DECREF(py_node_name_with_namespace);
|
Thank you @sloretz and @mikaelarguedas. This PR wouldn't have been realized without the time you spent providing clear feedback and meticulous review. |
With this pull request initial parameters can be passed via yaml parameters file with the
__params:=and as a list of parameters in theinitial_parameterskwarg of the Node constructor.Parameters in the constructor will override parameters from a parameters file.
If at least one reviewer could keep a close eye on my use of Py_DECREF and make sure that I have employed everywhere I should and nowhere I shouldn't have I'd be grateful.. I'm fairly confident in my usage but it's also my first time in Python's C API.
To test this PR I've been using the params.yaml in this gist https://gist.github.com/nuclearsandwich/8753121711671bfa8d9cb5f718ce09bc and the talker node from demo_nodes_py as well as the parameter services behavior just merged from #214
Connects to #202