Conversation
…OCM" This reverts commit e6ff323.
…ytorch#156903)" This reverts commit 823e223.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/161754
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 008f831 with merge base 403a3a3 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| if(${__AOTRITON_RUNTIME_INDEX} LESS 0) | ||
| aotriton_build_from_source(ON aotriton_runtime) | ||
| message(STATUS "Cannot find AOTriton runtime for ROCM ${__AOTRITON_SYSTEM_ROCM}. \ | ||
| Build runtime from source") |
There was a problem hiding this comment.
@xinyazhang Should this message appear above the call to aotriton_build_from_source so that the build flow will give the user an indication of what it is doing (I'm assuming aotriton_build_from_source is a significant time-taking step)?
There was a problem hiding this comment.
Yes it should be put one line above
|
rocm-mi300 workflow passed (except for one unrelated timeout): https://hud.pytorch.org/pytorch/pytorch/pull/161754?sha=0add8c2ad827f5562b94b22a1e17aa3d5092951d#rocm-mi300 rocm workflow passed: https://hud.pytorch.org/pytorch/pytorch/pull/161754?sha=0add8c2ad827f5562b94b22a1e17aa3d5092951d#rocm trunk passed: https://hud.pytorch.org/pytorch/pytorch/pull/161754?sha=0add8c2ad827f5562b94b22a1e17aa3d5092951d#trunk Latest commit 008f831 merely moves the message, so test results from previous commit should be good enough. Merging to allow sufficient time for internal builds to adjust to aotriton 0.11b changes. |
|
@jeffdaily please approve and merge |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
| CMAKE_CACHE_ARGS | ||
| -DAOTRITON_TARGET_ARCH:STRING=${PYTORCH_ROCM_ARCH} | ||
| -DCMAKE_INSTALL_PREFIX:FILEPATH=${__AOTRITON_INSTALL_DIR} | ||
| CMAKE_ARGS |
There was a problem hiding this comment.
This aotriton_build_from_source function is at least going to need -DHIP_PLATFORM=amd set as well, to avoid errors on Linux and Windows like https://github.com/ROCm/TheRock/actions/runs/17467234227/job/49606011053#step:11:66811
[1902/7918] Performing configure step for 'aotriton_runtime'
CMake Error at /opt/python/cp313-cp313/lib/python3.13/site-packages/_rocm_sdk_devel/lib/cmake/hip/hip-config.cmake:144 (message):
Unexpected HIP_PLATFORM:
Call Stack (most recent call first):
CMakeLists.txt:64 (find_package)
The source for that hip-config.cmake is https://github.com/ROCm/rocm-systems/blob/2202dcfe806766804648a9f38de35f555351e7fa/projects/clr/hipamd/hip-config.cmake.in#L111-L121
if(HIP_PLATFORM STREQUAL "amd")
set(HIP_RUNTIME "rocclr")
set(HIP_COMPILER "clang")
include( "${hip_LIB_INSTALL_DIR}/cmake/hip/hip-config-amd.cmake" )
elseif(HIP_PLATFORM STREQUAL "nvidia")
set(HIP_RUNTIME "cuda")
set(HIP_COMPILER "nvcc")
include( "${hip_LIB_INSTALL_DIR}/cmake/hip/hip-config-nvidia.cmake" )
else()
message(FATAL_ERROR "Unexpected HIP_PLATFORM: " ${HIP_PLATFORM})
endif()I'm working on rebasing TheRock's downstream patch https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/patches/pytorch/main/pytorch/hipified/0002-Support-FLASH_ATTENTION-MEM_EFF_ATTENTION-via.-aotri.patch . I'll try to split it into baseline Linux fixes like that one and the deeper changes needed for Windows support.
There was a problem hiding this comment.
The real problem is
execute_process(COMMAND ${hip_HIPCONFIG_EXECUTABLE} --platform
OUTPUT_VARIABLE HIP_PLATFORM
OUTPUT_STRIP_TRAILING_WHITESPACE)
does not work on Windows as expected. Normally setting HIP_PLATFORM is not necessary.
There was a problem hiding this comment.
Normally setting
HIP_PLATFORMis not necessary
In addition, it's not due to missing GPU installation.
I checked with 7.0RC2 docker image. CPU only instances can correctly report the platform as amd.
There was a problem hiding this comment.
Huh... we're seeing issues on both Linux and Windows there. I'll debug a bit. I see what you're saying - the logic in there should infer the HIP_PLATFORM CMake variable via hipconfig --platform.
There was a problem hiding this comment.
The interesting part is HIP_PLATFORM becomes mandatory even on Linux. From my experience this variable is always auto-configured.
There was a problem hiding this comment.
🤦
if("@HIP_INSTALLS_HIPCC@")is getting templated as this in TheRock's builds:
if("OFF")
if (WIN32)
set_and_check(hip_HIPCC_EXECUTABLE "${hip_BIN_INSTALL_DIR}/hipcc.exe")
set_and_check(hip_HIPCONFIG_EXECUTABLE "${hip_BIN_INSTALL_DIR}/hipconfig.exe")
else()
set_and_check(hip_HIPCC_EXECUTABLE "${hip_BIN_INSTALL_DIR}/hipcc")
set_and_check(hip_HIPCONFIG_EXECUTABLE "${hip_BIN_INSTALL_DIR}/hipconfig")
endif()
endif()There was a problem hiding this comment.
Filed ROCm/TheRock#1402 to solve that. What you have here (omitting -DHIP_PLATFORM=amd) should be fine then.
We have other issues downstream to triage though: ROCm/TheRock#1401 (comment)
There was a problem hiding this comment.
We're investigating that and other failures on Linux when building using TheRock at ROCm/TheRock#1408
Notable new features/optimizations for SDPA operators on AMD systems from AOTriton 0.11b:
AOTriton 0.11b Release Page for more
details.
logsumexptensor, matching CUDA's behaviorCausalVariant.LOWER_RIGHTThe build system changes drastically along with new packaging scheme of
AOTriton 0.11
aotriton.cmakenow selectively downloads image packs according toPYTORCH_ROCM_ARCHaotriton.cmakenow only use pre-compiled runtime library that exactlymatches the ROCM in the build environment. For PyTorch builds with ROCm
versions not listed in the file, the build process will build AOTriton
runtime without GPU images from source
require submodules.
Bug fixes:
Known Problems:
due to accuracy issues. Triton compiler fixes are needed to restore the
support status.
This issue is under investigation.
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben