Skip to content

Buffering logs into memory can make debugging difficult #564

@crwood

Description

@crwood

Currently, Gridsync collects multiple log streams from several sources (i.e., Tahoe-LAFS, Magic-Folder, itself) and buffers them into an in-memory data-structure that can later be inspected or exported to disk (via the "Help" -> "Export Debug Information..." menu action). The original intent behind this implementation was to avoid storing additional metadata and/or secrets directly to disk, however, there are several shortcomings with this approach:

  • In the event of an application crash or exit, whatever logs would be buffered into memory would naturally become lost. This can make it more difficult to gain insights about the behaviors where such insights are typically needed most.
  • (Most of?) the metadata/secrets contained in such logs can be deduced or inferred from other information that is already existing elsewhere on disk. The magic-folder sqlite database and other files in the Tahoe-LAFS nodedir's "private" directory, for example, contain timestamps for upload/download operations, cryptographic capabilities, and so on. Not writing such secrets to disk a second time (in the form of logs) does nothing to conceal or redact the original copies of those things already found in the Tahoe-LAFS node directory.
  • Memory (in the form of RAM) is typically more scarce/expensive than storage. Reducing the runtime memory footprint of Gridsync by writing whatever logs would normally be buffered into memory to disk instead would be welcome as it allows users to run more applications simultaneously, reducing the likelihood that they would need to exit a memory-hungry Gridsync in order to accomplish some other task. Most logging handlers also provide mechanisms for compressing logs, rotating logs, and so on, allowing for finer-grained control over the amount of storage consumed -- dials which can later be exposed to users.

Accordingly, it seems advantageous -- both for developers and for users -- to stop buffering logs into memory and, instead, to store whatever logs Gridsync already collects to disk. Currently, Gridsync attempts to capture the following streams:

  • The Gridsync twisted/stdlib debug logs
  • Tahoe-LAFS' eliot logs
  • Messages written by Tahoe-LAFS to stdout/stderr
  • The Magic-Folder eliot logs
  • Messages written by Magic-Folder to stdout/stderr

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions