Sets bCapture* true only while showing subwindow #4147

nznaza · 2021-11-06T13:24:11Z

Sets bAlwaysPersistRenderingState to true. This fixes bugs related to capture while on NoDisplay.
Sets bCaptureOnMovement and bCaptureEveryFrame to true only when they are going to be viewed on a subwindow

Fixes: #1724
Fixes: #2967
Fixes: #3504
Fixes: #3070

Partially fixes #4112

Should fix most of the FPS while capturing problems.

About

This PR fixes the NoDisplay/Headless bug by setting bAlwaysPersistRenderingState to true.

Setting bCaptureOnMovement and bCaptureEveryFrame to true only while it's on an active subwindow camera improves the performance of image capture/recording and python simGetImages

How Has This Been Tested?

A python script was used to display the captured images to corroborate the accurate capture of the desired captures; the same script was used for all captures using ComputerVision.

import airsim
from cv2 import data
import numpy as np
from airsim import *
import cv2 
import datetime

def ImageResponsetoImage(selected):
    isFloat = selected.pixels_as_float
    if isFloat:
        img1d = np.array(selected.image_data_float)
    else:
        img1d = np.frombuffer(selected.image_data_uint8, dtype=np.uint8)

    width = selected.width
    height = selected.height
    channels = int(img1d.size / (height * width))
    img = img1d.reshape(height, width, channels)
    if (isFloat):
        depthinmm = img*1000
        depth_16bit =  np.clip(depthinmm, 0,65535) 
        return depth_16bit.astype(np.uint16)

    else:
        return img.astype(np.uint8)

if __name__ == "__main__":
    client = MultirotorClient()
    client.confirmConnection()

    i = 0
    inc = True
    while True:
        responses = client.simGetImages(
            [
                ImageRequest("front_left", ImageType.Scene, False, False),
                ImageRequest("front_left", ImageType.DepthPerspective, True, False),
                ImageRequest("front_left", ImageType.Segmentation, False, False),
                
                ImageRequest("front_center", ImageType.Scene, False, False),
                ImageRequest("front_center", ImageType.DepthPerspective, True, False),
                ImageRequest("front_center", ImageType.Segmentation, False, False),
                
                ImageRequest("front_right", ImageType.Scene, False, False),
                ImageRequest("front_right", ImageType.DepthPerspective, True, False),
                ImageRequest("front_right", ImageType.Segmentation, False, False),
                
                ImageRequest("bottom_center", ImageType.Scene, False, False),
                ImageRequest("bottom_center", ImageType.DepthPerspective, True, False),
                ImageRequest("bottom_center", ImageType.Segmentation, False, False),

                ImageRequest("back_center", ImageType.Scene, False, False),
                ImageRequest("back_center", ImageType.DepthPerspective, True, False),
                ImageRequest("back_center", ImageType.Segmentation, False, False),
                
            ]
        )
        tm = datetime.datetime.now().second % 10
        
        if tm % 2 == 0 :
            if inc:
                i += 1
            inc = False
        else:
            inc = True   
        if i >= len(responses):
            i = 0
        selected = responses[i]

        cv2.imshow("", ImageResponsetoImage(selected))
        cv2.waitKey(1)

Performance goes back up after stopping capture while on NoDisplay, but stays the same after recording or using the python simGetImages, but this can be fixed by displaying and hiding a subwindow.

settings.json

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "ViewMode": "NoDisplay",
  "SimMode": "ComputerVision",
  "Recording": {
    "RecordOnMove": true,
    "RecordInterval": 0.05,
    "Folder": "",
    "Enabled": false,
    "Cameras": [
        { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false,  "VehicleName": "ComputerVision", "Compress": true },
        { "CameraName": "0", "ImageType": 1, "PixelsAsFloat": false,  "VehicleName": "ComputerVision", "Compress": true },
        { "CameraName": "0", "ImageType": 2, "PixelsAsFloat": false,  "VehicleName": "ComputerVision", "Compress": true },
        { "CameraName": "0", "ImageType": 7, "PixelsAsFloat": false,  "VehicleName": "ComputerVision", "Compress": true }
    ]
  },
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 640,
        "Height": 480
      },
      {
        "ImageType": 2,
        "Width": 640,
        "Height": 480
      },
      {
        "ImageType": 7,
        "Width": 640,
        "Height": 480
      },
      {
        "ImageType": 8,
        "Width": 640,
        "Height": 480
      },
      {
        "ImageType": 9,
        "Width": 640,
        "Height": 480
      },
      {
        "ImageType": 5,
        "Width": 640,
        "Height": 480
      }
    ],
    "NoiseSettings": [
      {
        "Enabled": false,
        "ImageType": 7,

        "RandContrib": 0.2,
        "RandSpeed": 100000.0,
        "RandSize": 500.0,
        "RandDensity": 2,

        "HorzWaveContrib":0.03,
        "HorzWaveStrength": 0.08,
        "HorzWaveVertSize": 1.0,
        "HorzWaveScreenSize": 1.0,

        "HorzNoiseLinesContrib": 1.0,
        "HorzNoiseLinesDensityY": 0.01,
        "HorzNoiseLinesDensityXY": 0.5,

        "HorzDistortionContrib": 1.0,
        "HorzDistortionStrength": 0.002
      }
    ]
  },
  "SegmentationSettings": {
    "InitMethod": "",
    "MeshNamingMethod": "StaticMeshName",
    "OverrideExisting": true
  }
}

Screenshots (if appropriate)

NoDisplay

RGB
Previous:

Depth:
Previous:

After:

Segmentation
Previous:

After:

Performance captures

Recording previous to fix

After recording and showing and hiding subwindow (small gain)

Recording after fix (Lower timings, )

After recording and showing and hiding subwindow (Back to normal)

Python capture before fix

Python capture post fix

Python capture before fix (Only front_center)

Python capture after fix (Only front_center)

Python capture before fix (Only front_center Scene)

Python capture after fix (Only front_center Scene)

While capturing with python's simGetImages the performance after the fix are lower, but measurable: 205 ms vs 185ms, 70ms vs 54, single images are within margin error, but fix is still lower

For comparison here are the timings of all cameras and NoDIsplay

While capturing

After stopping capturing, the levels go back to normal without needing to enable a subwindow.

Sets bAlwaysPersistRenderingState to true. This fixes bugs of capture Sets bCaptureOnMovement and bCaptureEveryFrame to true only when they are going to be viewed on as subwindow

ghost · 2021-11-06T13:24:23Z

All CLA requirements met.

rajat2004

Still need to test, but any idea why the performance for simGetImages dropped?

rajat2004 · 2021-11-27T12:25:38Z

Unreal/Plugins/AirSim/Source/CameraDirector.cpp

-    else {
-        gameViewport->bDisableWorldRendering = 0;
-    }
+    gameViewport->bDisableWorldRendering = (uint32)mode_;


You can just use the nodisplay variable instead of casting mode_

Correct, I was removing the branch assignation and used the wrong variable, will issue a commit to address this, thanks.

rajat2004 · 2021-11-27T12:29:17Z

Unreal/Plugins/AirSim/Source/PIPCamera.cpp

    }

-    onViewModeChanged(false);
+    onViewModeChanged(true);


So in the beginning, we set all cameras to NoDisplay mode to improve performance?
Would be great if you could add some comments in the code also as to why this helps

We initialize all cameras with bCaptureOnMovement and bCaptureEveryFrame to false.

This improves performance because the capture components are no longer updating every frame and only update while requesting an image (and are set to true while we are showing them on a sub-window because sub-window captures don’t count as a request)

Unreal/Plugins/AirSim/Source/CameraDirector.cpp

nznaza · 2021-11-30T18:34:05Z

Still need to test, but any idea why the performance for simGetImages dropped?

Sorry I don’t quite get the question, if you mean why is performance still drops while simGetImages is called: it’s because we still need to acquire the images (this is what causes the performance drop) this will commit will only drop while there is a request for the camera instead of having all of them updating every frame

Fixed gameViewport->bDisableWorldRendering assignation Added comments explaining the reasons behind setting cameras to nodisplay

rajat2004 · 2021-12-01T02:19:35Z

Ah, I think I misunderstood the issue, I thought the FPS of simGetImages itself dropped, rather than the viewport. If possible, could you try the image_benchmarker.py script with and without your fix, just to get a sense of performance improvement?

zimmy87 · 2021-12-01T08:32:09Z

Thanks for submitting this fix @nznaza! Tested this locally using the repro steps for #1724 and #2967. Confirmed #1724 and #3504 (which is a duplicate of #1724) are fixed now. Unfortunately, I was still able to repro #2967 and I didn't see any significant performance change using the image_benchmarker.py script (I was getting around 40 fps for the Blocks map both with and without this fix), so I suspect #3070 still repros. Considering this does fix some of the issues, I'm going to go ahead and merge this, we can address the other issues in a separate PR.

Sets bCapture* true only while showing subwindow

fa0c554

Sets bAlwaysPersistRenderingState to true. This fixes bugs of capture Sets bCaptureOnMovement and bCaptureEveryFrame to true only when they are going to be viewed on as subwindow

rajat2004 reviewed Nov 27, 2021

View reviewed changes

nznaza added 2 commits November 30, 2021 13:24

Viewport branchless assignation

f588aae

Fixed gameViewport->bDisableWorldRendering assignation Added comments explaining the reasons behind setting cameras to nodisplay

Fix clang-format violation

8524e35

zimmy87 merged commit 51e632a into microsoft:master Dec 1, 2021

This was referenced Dec 1, 2021

Cannot obtain Depth Images under NoDisplay mode #2967

Open

low image fps while using simGetImage python API #3070

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Sets bCapture* true only while showing subwindow #4147

Sets bCapture* true only while showing subwindow #4147

Uh oh!

nznaza commented Nov 6, 2021 •

edited

Loading

Uh oh!

ghost commented Nov 6, 2021 •

edited by ghost

Loading

Uh oh!

rajat2004 left a comment

Uh oh!

rajat2004 Nov 27, 2021

Uh oh!

nznaza Nov 30, 2021

Uh oh!

rajat2004 Nov 27, 2021

Uh oh!

nznaza Nov 30, 2021

Uh oh!

Uh oh!

nznaza commented Nov 30, 2021

Uh oh!

rajat2004 commented Dec 1, 2021

Uh oh!

zimmy87 commented Dec 1, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sets bCapture* true only while showing subwindow #4147

Sets bCapture* true only while showing subwindow #4147

Uh oh!

Conversation

nznaza commented Nov 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

How Has This Been Tested?

Screenshots (if appropriate)

Uh oh!

ghost commented Nov 6, 2021 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rajat2004 left a comment

Choose a reason for hiding this comment

Uh oh!

rajat2004 Nov 27, 2021

Choose a reason for hiding this comment

Uh oh!

nznaza Nov 30, 2021

Choose a reason for hiding this comment

Uh oh!

rajat2004 Nov 27, 2021

Choose a reason for hiding this comment

Uh oh!

nznaza Nov 30, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nznaza commented Nov 30, 2021

Uh oh!

rajat2004 commented Dec 1, 2021

Uh oh!

zimmy87 commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nznaza commented Nov 6, 2021 •

edited

Loading

ghost commented Nov 6, 2021 •

edited by ghost

Loading

zimmy87 commented Dec 1, 2021 •

edited

Loading