Server returned (400): IO error: lock /opt/influxdb/shared/data/db/shard_db_v2/03773/LOCK: No locks available #1013
Description
yet another issue appears while using https://github.com/vimeo/whisper-to-influxdb to populate influxdb...
this time the clientside reports:
committed InfluxDB series 'stats.dfvimeopweb6.memcached_delete_fail' (196967 points)
committed InfluxDB series 'stats.dfvimeopweb6.memcached_set.17' (1 points)
Failed to write InfluxDB series 'stats.dfvimeopweb6.memcached_objects2_get' (218369 points): Server returned (400): IO error: lock /opt/influxdb/shared/data/db/shard_db_v2/03773/LOCK: No locks available (operation took 4.035181891s)
snippet from server log:
[dieter.plaetinck@dfvimeographite1 shared]$ sudo grep -C 5 LOCK influxdb.log.007
[2014/10/08 18:28:31 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).DeleteShard:254) DATASTORE: dropping shard /opt/influxdb/shared/data/db/shard_db_v2/03771
[2014/10/08 18:28:31 EDT] [DEBG] (github.com/influxdb/influxdb/api/graphite.func·002:188) GraphiteServer committing 3 series
[2014/10/08 18:28:31 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db_v2/03769
[2014/10/08 18:28:31 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).DeleteShard:254) DATASTORE: dropping shard /opt/influxdb/shared/data/db/shard_db_v2/03772
[2014/10/08 18:28:31 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db_v2/03773
[2014/10/08 18:28:31 EDT] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).CommitSeriesData:696) COORD error writing: %!(EXTRA rocksdb.DatabaseError=IO error: lock /opt/influxdb/shared/data/db/shard_db_v2/03773/LOCK: No locks available)
[2014/10/08 18:28:31 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db_v2/03771
[2014/10/08 18:28:32 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db_v2/03770
[2014/10/08 18:28:32 EDT] [DEBG] (github.com/influxdb/influxdb/api/graphite.func·002:188) GraphiteServer committing 135 series
[2014/10/08 18:28:32 EDT] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).DeleteShard:254) DATASTORE: dropping shard /opt/influxdb/shared/data/db/shard_db_v2/03773
[2014/10/08 18:28:32 EDT] [DEBG] (github.com/influxdb/influxdb/api/graphite.func·002:188) GraphiteServer committing 9 series
full log available on request
[root@dfvimeographite1 ~]# stat /opt/influxdb/shared/data/db/shard_db_v2/03773/LOCK:
stat: cannot stat `/opt/influxdb/shared/data/db/shard_db_v2/03773/LOCK:': No such file or directory
[root@dfvimeographite1 ~]# ls -alh /opt/influxdb/shared/data/db/shard_db_v2/03773/
total 8.6M
drwxr-xr-x 2 influxdb influxdb 4.0K Oct 8 18:28 .
drwxr--r-- 712 influxdb influxdb 20K Oct 8 20:00 ..
-rw-r--r-- 1 influxdb influxdb 45K Oct 8 18:28 000003.log
-rw-r--r-- 1 influxdb influxdb 16 Oct 8 18:28 CURRENT
-rw-r--r-- 1 influxdb influxdb 37 Oct 8 18:28 IDENTITY
-rw-r--r-- 1 influxdb influxdb 0 Oct 8 18:28 LOCK
-rw-r--r-- 1 influxdb influxdb 10K Oct 8 18:28 LOG
-rw-r--r-- 1 influxdb influxdb 13 Oct 8 18:28 MANIFEST-000001
-rw-r--r-- 1 influxdb influxdb 7 Oct 8 18:28 type