We discussed how to make shadow's logging and data output more consistent and useful. Many of these items will require a major version bump.

FAQ:

• what format do we want to log the data in?
  • jsonl, compressed
  • also in log file to help debugging
• where do we want to log the data?
  • shadow.data/
  • keep each type of data in separate filenames as much as possible
• should it be logged in a way that external tools can monitor it while the simulation is running?
  • ideally yes; we may need to check the compression algorithm and/or library, and arrange for periodic flushes to disk
• how should we support multiple versions in tornettools(?)
  • will become easier when we have json files
  • don't support old versions eg. that require log message parsing

user-facing statistics

• guiding rule: stable, supported for user parsing, don't change often
• jsonl format, compressed since we'll be repeating field names
• store in shadow.data/
  • keep each type of data in separate filenames as much as possible
• data types:
  • network statistics
    • shadow-global: nothing for now, could be added later
    • host-specific:
      • bytes sent, bytes received
      • count of total packets sent and received
      • count of retransmitted packets sent and received
      • total since last write, write period configurable
    • plugin-specific or socket-specific could be added later
  • memory usage statistics
    • shadow-global usage, data like what free provides, currently DNE in shadow
    • host-specific and plugin-specific could be added later
  • pcap files (exception; not in json format)
  • processed config yaml file
  • strace-style logging
  • syscall counts
    • only simulation-wide syscall counts
  • heartbeat info:
    • real timestamp and simulated timestamp every second

dev-facing info (useful for debugging)

• guiding rule: users SHOULD NOT PARSE log file, we will change randomly and often
• standard log messages
• can repeat some jsonl user-facing stats in log file for debugging
• data types:
  • log statements (warn trace debug etc)
    • log warnings if resource usage is high (ex: open file count)
  • packet states
    • written to shadow log file only, at trace level
    • log as a json string, include timestamp, prefix is "PACKET_JSON: {}"
    • packet id should be grepable, include hostname
    • removing printing full list of states, just log current state on state-transition
    • payloads enabled through an experimental option
  • object allocation counts
    • print at the end of the log
  • build/version information

other (leave as-is)

• simulation success status (num failed plugins, etc)
• progress bar

items to remove

• don't keep current thing, can add for devs later in ad-hoc manner if needed
• data types:
  • network stats:
    • control vs data packet counts
  • socket stats:
    • remove shadow's current (possibly broken) version
    • if we want this in the future, we can support enough of netlink that we can support some of the ss tool in shadow (see sock_diag(7))