Skip to content

Latest commit

 

History

History
232 lines (197 loc) · 7.3 KB

File metadata and controls

232 lines (197 loc) · 7.3 KB

Roadmap

Scanning

Targets

  • Scan Github Gists

  • Scan Gitlab.com Group Repos (needs a group ID)

  • Scan Gitlab.com User Repos (needs a user name)

  • Scan Gitlab Snippets

  • Scan Gitlab On-Prem Org Repos

  • Scan Gitlab On-Prem User Repos

  • Scan Bitbucket.com Project Repos

  • Scan Bitbucket.com User Repos

  • Scan Bitbucket On-Prem Org Repos

  • Scan Bitbucket On-Prem User Repos

  • Scan AWS Code Commit

  • Scan Azure DevOps

  • Scan Wiki's

  • Scan Pastebin

  • Scan Confluence

  • Scan OneDrive

  • Scan Dropbox

  • Scan GoogleDrive

  • Scan iCloud

  • Scan S3 Buckets

  • Scan MS Office Docs

  • Scan OpenOffice/LibreOffice Docs

  • Scan Evernote

  • Scan GSuite

  • Scan Quip

Milestone 1

Bugs

  • Still have a lot of missing files when scanning (turn on debug)
  • In memory clone returns no findings (I think this has something to do with the path not being found)
  • Fix how all repos are gathered (org repos is threaded and general, user repos is not threaded and github specific)
  • Gitlab client does not follow redirects
  • Github does not follow redirects
  • In gitlab you can have a project w/ no repo, this will error out
  • Expanding orgs is not working
  • Number of dirty commits is wrong, it should be more than is showing

Documentation

  • Document all flags
  • Document all code completely
  • Document how to add a new command or source
  • Document all stats
  • Create a developer doc with the design and code execution flow
  • Contributing.md
    • wraith
    • wraith-tests
    • wraith-signatures
  • README.md
    • wraith
    • wraith-tests
    • wraith-signatures
  • Security.txt
    • wraith
    • wraith-tests
    • wraith-signatures
  • Initial blog post
  • Detailed documentation published on the net and with source control
  • Changelog.md
    • [X] wraith
    • wraith-tests
    • [X] wraith-signatures
  • Go doc strings
    • core
    • version
  • [X] Issue template
  • [X] PR template
  • Submit story to hackernews
  • Submit story to changelog.com
  • Add a built w/ section
  • [X] Call out individual contributers after N merges

Testing

  • Code review and remove debug statements
  • Ensure that an error status will exit the program, if not swap to a warning status so it does
  • Sanity check testing plan
  • Copy existing tests to the new codebase
  • Make sure we use https so keys are not necessary
  • Update CodeCov for Wraith
  • [X] Golint needs to pass
  • Convert tests to testify
  • Review all flags to ensure they are needed

Features

  • -1 Confidence level loads all signatures
  • Tor capability
  • IP switching to hide itself
  • Ability to use multiple tokens/keys for a single service (several GH keys)
  • Need to drop in the org in the realtime output
  • Add the status to all functions for use in the web interface*__*
  • Need to find gitlab api endpoint
  • Port gitlab to match G
  • Set the debug like G
  • Remove all dead code
  • Change id -> ID
  • Refactor how we do stats on commits
  • Need to be able to scan all the orgs a user is a member of
  • Need to be able to scan all the forks of a given repo that we can reach
  • Set the csv and json like G
  • Change name from threads to go routines or make that clear
  • Refactor how threads are calculated
  • Created a dedicated GPG key
  • Enforce https for all connections
  • Enforce https for the site
  • Fully Instrumented with Performance Stats
  • Entrophy Checks
  • If we find a .git directory in a localPath scan just ignore it and process the dir as localPath
  • Cleanup issues in summary output
  • Add more debuging info
  • [X] Need to list the flag defaults on the help screen
  • [X] If no arg's are given for a command, then list the help screen
  • Only export the functions and variables necessary
  • Capture the error if no sig file is given and the default does not exist
  • Break out checking if a file is to be scanned into a single function
  • Add a flag to de-dupe findings

Milestone 2

Bugs

  • web interface cannot handle local files (requires mucking with bindata.go)
  • web interface is gitlab specific by default (requires mucking with bindata.go)
  • why is the web interface using to old index.html (requires mucking with bindata.go)
  • web interface progress bar not working
  • web interface links to the file should be more detailed and point to the commit/line in the code
  • web interface is not dynamic, I need to refresh it manually
  • validate all user input

Documentation

Features

  • Scan specific branches
  • Scan since a given commit
  • Update Signatures command specific features
  • Implement threading for local path scans
  • Ability to use the .gitignore when scanning for ignoring paths and files

Testing

  • Make tech debt fail build process
  • Add config details to debug statement at the start of a run
  • Create stats for signatures
  • Structured Logging
  • Create a standard set of error codes
  • Error Handling
    • config
    • core
    • version
  • Security Scans
  • Sanitize user inputs
  • Code should be optimized into multiple packages

Milestone 3

Bugs

Features

  • working with local repos is not threaded
  • Exclude or include files based on mime type
  • Exclude a default path/extension default exclusion
  • Exclude specific branches or tags
  • Only scan selected branches or tags
  • Exclude public or private repos
  • Exclude Users or Repos in an org scan
  • Database Backend
  • Web Frontend For Configuration
  • Consistent search on all platforms
  • need to update the go git library used
  • Regex's are not performant
  • Move the repo count per target during a run to a debug statement
  • Create stats for signatures
  • add additional stats to web interface
  • Create new ascii art
  • Rebuild the web interface
  • Thread the scanning of commits
  • Check all urls point to the right repos (requires mucking with bindata.go)
  • Break out global vs command specific variables
  • Combine all shell scripts into Makefile
  • Need to make a flag in the Makefile to update the dependencies
  • Split out the web go code into a specific package
  • Swap to libgit2 where it makes sense for scaling
  • [X] 3PP Scans

Testing

  • Test all regexes
  • Unit tests for all code
    • common
    • config
    • core
    • github
    • gitlab
    • matching
    • version
    • rules
  • Code Test Coverage
  • Table driven tests

Documentation

Research

  • Do we want to add files,dirs,repos,etc to an ignore list when they are not found or they error out
  • Look at using the gitignore when scanning repos
  • what errors should stop the run
  • Mascot

Notes

  • Can we Go for the web front-end
  • Language Parsers
  • Convert Repository type, etc into github ones if needed
  • How do we deal with abuse msgs, and rate limiting