Documentation
Start typing to search…

API Discovery Tutorial

How to discover and document APIs - for security testing or general development.

API Discovery

Let's illuminate a REST API. We'll download OWASP Juice Shop, create the relevant resources in NightVision, and view the API in the Swagger editor.

Prerequisites

Install the NightVision CLI. Follow the guide here to install the NightVision CLI on your system: Installing the CLI

First, clone the repository:

git clone https://github.com/juice-shop/juice-shop
cd juice-shop

(1) Run API Discovery

Now run this command:

nightvision swagger extract . --lang js --no-upload --output nv-swagger.yml

Note: Please use the following string literal to specify the programming language:

  • Java: java
  • JavaScript: js
  • Python: python
  • C#: dotnet
  • Go: go
  • Ruby: ruby

The output will look like the following:

[2024-08-24 18:58:13] INFO Launching NightVision CLI
[2024-08-24 18:58:14] INFO Initializing language provider
[2024-08-24 18:58:15] INFO Finished initializing language provider
[2024-08-24 18:58:15] INFO Starting language provider execution
[2024-08-24 18:58:15] INFO Finished language provider execution
[2024-08-24 18:58:15] INFO Starting generating OpenAPI document
[2024-08-24 18:58:15] INFO OpenAPI document generated in 806.137541ms
[2024-08-24 18:58:15] Number of discovered paths: 87
[2024-08-24 18:58:15] Number of discovered classes: 0
[2024-08-24 18:58:15] INFO Generated the OpenAPI document.
[2024-08-24 18:58:15] INFO Successfully validated the output.

(2) Measuring API Coverage

Juice Shop comes with a Swagger doc here, but it only contains one path - /orders. NightVision can provide much higher coverage for scanning APIs via API discovery. Let's measure the improvement:

nightvision swagger diff swagger.yml nv-swagger.yml

As you can see, NightVision found 108 unique API endpoints (combinations of URL paths and HTTP methods) compared to just one original API endpoint.

Paths:
  Added: 108
  Modified: 0
  Unchanged: 0
  Undiscovered: 1
Schemas:
  Added: 0
  Modified: 0
  Unchanged: 0
  Undiscovered: 5

To view more details about the newly discovered endpoints, append --paths to the swagger diff command.

nightvision swagger diff swagger.yml nv-swagger.yml --paths

The output is below:

Added (108):
+ GET: /api/Addresss
+ POST: /api/Addresss
+ DELETE: /api/Addresss/{id}
+ GET: /api/Addresss/{id}
+ PUT: /api/Addresss/{id}
+ POST: /api/BasketItems
+ PUT: /api/BasketItems/{id}
+ GET: /api/Cards
+ POST: /api/Cards
+ DELETE: /api/Cards/{id}
+ GET: /api/Cards/{id}
+ PUT: /api/Cards/{id}
+ POST: /api/Challenges
+ GET: /api/Complaints
+ POST: /api/Complaints
+ GET: /api/Deliverys
+ GET: /api/Deliverys/{id}
+ POST: /api/Feedbacks
+ PUT: /api/Feedbacks/{id}
+ GET: /api/PrivacyRequests
+ POST: /api/PrivacyRequests
+ POST: /api/Products
+ DELETE: /api/Products/{id}
+ POST: /api/Quantitys
+ DELETE: /api/Quantitys/{id}
+ GET: /api/Recycles
+ POST: /api/Recycles
+ DELETE: /api/Recycles/{id}
+ GET: /api/Recycles/{id}
+ PUT: /api/Recycles/{id}
+ GET: /api/SecurityAnswers
+ POST: /api/SecurityQuestions
+ GET: /api/Users
+ POST: /api/Users
+ DELETE: /api/Users/{id}
+ GET: /api/Users/{id}
+ PUT: /api/Users/{id}
+ GET: /array_1065
+ POST: /b2b/v2/orders
+ GET: /dataerasure/
+ POST: /dataerasure/
+ POST: /file-upload
+ GET: /metrics
+ GET: /profile
+ POST: /profile
+ POST: /profile/image/file
+ POST: /profile/image/url
+ GET: /promotion
+ GET: /redirect
+ POST: /rest/2fa/disable
+ POST: /rest/2fa/setup
+ GET: /rest/2fa/status
+ POST: /rest/2fa/verify
+ GET: /rest/admin/application-configuration
+ GET: /rest/admin/application-version
+ GET: /rest/basket/{id}
+ POST: /rest/basket/{id}/checkout
+ PUT: /rest/basket/{id}/coupon/{coupon}
+ GET: /rest/captcha
+ POST: /rest/chatbot/respond
+ GET: /rest/chatbot/status
+ GET: /rest/continue-code
+ GET: /rest/continue-code-findIt
+ PUT: /rest/continue-code-findIt/apply/{continueCode}
+ GET: /rest/continue-code-fixIt
+ PUT: /rest/continue-code-fixIt/apply/{continueCode}
+ PUT: /rest/continue-code/apply/{continueCode}
+ GET: /rest/country-mapping
+ GET: /rest/deluxe-membership
+ POST: /rest/deluxe-membership
+ GET: /rest/image-captcha
+ GET: /rest/languages
+ GET: /rest/memories
+ POST: /rest/memories
+ GET: /rest/order-history
+ GET: /rest/order-history/orders
+ PUT: /rest/order-history/{id}/delivery-status
+ PATCH: /rest/products/reviews
+ POST: /rest/products/reviews
+ GET: /rest/products/search
+ GET: /rest/products/{id}/reviews
+ PUT: /rest/products/{id}/reviews
+ GET: /rest/repeat-notification
+ GET: /rest/saveLoginIp
+ GET: /rest/track-order/{id}
+ GET: /rest/user/authentication-details
+ GET: /rest/user/change-password
+ POST: /rest/user/data-export
+ POST: /rest/user/login
+ POST: /rest/user/reset-password
+ GET: /rest/user/security-question
+ GET: /rest/user/whoami
+ GET: /rest/wallet/balance
+ PUT: /rest/wallet/balance
+ GET: /rest/web3/nftMintListen
+ GET: /rest/web3/nftUnlocked
+ POST: /rest/web3/submitKey
+ POST: /rest/web3/walletExploitAddress
+ POST: /rest/web3/walletNFTVerify
+ GET: /snippets
+ POST: /snippets/fixes
+ GET: /snippets/fixes/{key}
+ POST: /snippets/verdict
+ GET: /snippets/{challenge}
+ GET: /the/devs/are/so/funny/they/hid/an/easter/egg/within/the/easter/egg
+ GET: /this/page/is/hidden/behind/an/incredibly/high/paywall/that/could/only/be/unlocked/by/sending/1btc/to/us
+ GET: /video
+ GET: /we/may/also/instruct/you/to/refuse/all/reasonably/necessary/responsibility

Undiscovered (1):
- POST: /orders
👍

API Discovery via Source Code

Source-code based API Discovery is the fastest way to discover and document your REST APIs at scale.

Other tools will take a traffic-based approach. However, source-code based API discovery has these advantages:

  1. It's faster: Source code analysis takes under a minute instead of days to create representative traffic.
  2. It won't miss shadow APIs: Traffic-based approaches will miss shadow APIs if there is no live traffic against the endpoints. Source code analysis will uncover shadow APIs.
  3. It's cheaper: Traffic-based approaches need to ingest and process a large amount of data, which can result in thousands of dollars in cloud provider egress fees. Scanning source code costs nothing.
  4. It doesn't raise security concerns: Traffic based approaches have full access to all data in your HTTP traffic, such as bearer tokens and customer data (like PII/PCI/HIPPA data).

More details are available in the API Discovery FAQ.

Onwards! 🚀

More Details

NightVision's API Discovery (patent pending) supports dozens of API frameworks and hundreds of framework components. For more details, see the following:

Updated 4 months ago