Skip to content

row.exists doesn't limit cell versions, can cause out of memory exceptions #608

@jschmidt-metacx

Description

@jschmidt-metacx

Please run down the following list and make sure you've tried the usual "quick fixes":

If you are still having issues, please be sure to include as much information as possible:

Environment details

  • OS: macos 10.14.6, linux (on GKE)
  • Node.js version: all (discovered while on v8.16.2)
  • npm version: all (discovered while on 6.4.1)
  • @google-cloud/bigtable version: all (discovered via 1.0.1)

Steps to reproduce

  1. write a very large number of increments to a single row (a simple loop that sends increment mutations should work)
  2. run row.exists() against the row, monitor memory usage

Expected result

row.exists() should query enough information to accurately determine if the row exists, but not pull all versions of all cells within the row.

Actual result

With very active rows which are creating many versions due to increment operations, row.exists() fetches every version of the cell, consuming a massive amount of memory (potentially crashing node) and taking a long time to execute (while waiting for the versions) before finally resolving with the boolean result. This can cause server failures and API call timeouts.

Analysis

row.exists() fetches the entire row, including all versions of all cells and doesn’t accept an options object, so it cannot be made to limit the amount of data fetched.

Specifically, row.exists:

  • accepts no parameters/options
  • forces gaxOptions to an empty object
  • calls row.getMetadata
  • which calls table.getRows
  • which calls createReadStream, then pipes all results in-memory to be concatenated in a single array

Metadata

Metadata

Assignees

Labels

api: bigtableIssues related to the googleapis/nodejs-bigtable API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions