BOSH errand rotate_cc_database_key fails (log output too long) #114
Description
Issue
When running the rotate_cc_database_key errand on a somewhat busier CF landscape, we get the following error:
$ bosh -d cf run-errand rotate_cc_database_key
Using environment 'https://192.168.1.11:25555' as client 'admin'
Using deployment 'cf'
Task 26401
Task 26401 | 11:33:44 | Preparing deployment: Preparing deployment
Task 26401 | 11:34:06 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 26401 | 11:34:06 | Preparing deployment: Preparing deployment (00:00:22)
Task 26401 | 11:34:06 | Creating missing vms: rotate-cc-database-key/121b24bb-8928-40aa-9ed7-1d1b89fb28cc (0) (00:01:17)
Task 26401 | 11:35:23 | Updating instance rotate-cc-database-key: rotate-cc-database-key/121b24bb-8928-40aa-9ed7-1d1b89fb28cc (0) (canary) (00:00:51)
Task 26401 | 11:36:14 | Running errand: rotate-cc-database-key/121b24bb-8928-40aa-9ed7-1d1b89fb28cc (0) (00:00:10)
L Error: Response exceeded maximum allowed length
Task 26401 | 11:36:33 | Error: Response exceeded maximum allowed length
Task 26401 Started Thu Dec 6 11:33:44 UTC 2018
Task 26401 Finished Thu Dec 6 11:36:33 UTC 2018
Task 26401 Duration 00:02:49
Task 26401 error
Running errand 'rotate_cc_database_key':
Expected task '26401' to succeed but state is 'error'
Exit code 1
Context
We tried to rotate an encrypted CC database following the instructions on https://docs.cloudfoundry.org/adminguide/encrypting-cc-db.html. On a smaller development landscape, everything works fine. When we move to larger productive landscapes, we get the error above. Note that on our landscapes, the cc.log_db_queries parameter is true. When setting the parameter to false, the logs are shorter and the errand succeeds.
Steps to Reproduce
Configure a CF landscape with cc.log_db_queries to true and run bosh -d cf run-errand rotate_cc_database_key. The CC database must have a sufficiently large number of tables and rows (high number of apps / blobs / service bindings).
Expected result
The errand should always succeed. Database operations should only be logged if an error occurs. Successful updates don't need to be logged.
Current result
Errand fails because output is too large.
Possible Fix
Make the DB log level configurable with a separate parameter (e.g. cc.key_rotate_logging_level):
Thanks for your support!