Unable to use vision capabilities and prompt in custom extension #239976
Description
Type: Bug
I am trying to include functionality in my custom extension to leverage the new ability to use vision capabilities of an AI model. The testing I have done manually in the Chat UI show great results but I am trying to get it to work in a custom command in my extension. I am able to add an image to the chat context successfully but when I programmatically sent a prompt to the AI model it doesn't seem to be able to answer questions about the image. I may be missing something though. Is there something specific you need to do to get the prompt to be able to reference the image? I can see it is part of the request but not sure if I need to reference it somehow specifically in the code or if the prompt should automatically leverage it as part of the chat context.
VS Code version: Code - Insiders 1.97.0-insider (e72728b, 2025-02-03T22:42:32.123Z)
OS version: Windows_NT x64 10.0.26100
Modes:
System Info
| Item | Value |
|---|---|
| CPUs | 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995) |
| GPU Status | 2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off |
| Load (avg) | undefined |
| Memory (System) | 31.71GB (6.22GB free) |
| Process Argv | |
| Screen Reader | no |
| VM | 0% |
Extensions (3)
| Extension | Author (truncated) | Version |
|---|---|---|
| copilot | Git | 1.266.1363 |
| copilot-chat | Git | 0.24.2025020602 |
| ms-learn | ms- | 0.3.0 |
A/B Experiments
vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vsc_aacf:30263846
vscod805:30301674
vsaa593cf:30376535
py29gd2263:31024238
13748-chatv2:31230757
c4g48928:30535728
962ge761:30841072
pythonnoceb:30776497
dsvsc014:30777825
dsvsc015:30821418
pythonmypyd1:30859725
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:30980852
pythonait:30973460
dwnewjupytercf:31046870
nativerepl1:31134653
pythonrstrctxt:31093868
nativeloc1:31118317
e80f6927:31120813
iacca1:31150324
notype1:31143044
dwcopilot:31158714
h409b430:31177054
5b1c1929:31184661
6074i472:31201624
dwoutputs:31217127
9064b325:31222308
copilot_t_ci:31222730
9hfj7396:31230372
g012b348:31231168