Flash Attention 3 (FA3) Support

[mgoin]

As of #12093 Flash Attention 3 is now supported in vLLM for Hopper GPUs (SM 9.0).

It can also be enabled for SM 8.0 and 8.7 using VLLM_FLASH_ATTN_VERSION=3.

For 8.6 and 8.9 its fully disabled since they don't have enough shared memory for the current implementation, some work needs to be done here.

This issue tracks the remaining features that have yet to be implemented

Hardware Support

• SM 8.9 Ada Lovelace (L4, L40s) Support
• SM 8.6 Ampere (A6000) Support

Optimizations

• FP8 Attention

[zhangjiekui]

As of #12093 Flash Attention 3 is now supported in vLLM for Hopper GPUs (SM 9.0).

It can also be enabled for SM 8.0 and 8.7 using VLLM_FLASH_ATTN_VERSION=3.

For 8.6 and 8.9 its fully disabled since they don't have enough shared memory for the current implementation, some work needs to be done here.

This issue tracks the remaining features that have yet to be implemented

Hardware Support

• SM 8.9 Ada Lovelace (L4, L40s) Support[ ] SM 8.6 Ampere (A6000) Support

Optimizations

• FP8 Attention

Watch

[aabbccddwasd]

emm， Now we have 4090 48GB so is there enough VRAM for FA3, isn't it?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[voycey]

@mgoin - Any news on 8.6 / 8.9 support for this? Looking at deploying on L40S and require Flash Attention 3, not sure why this is excluded specifically considering the available VRAM?