Add support for Python 3.11

[bdice]

RAPIDS currently supports Python 3.9 and 3.10. We would like to add support for Python 3.11. This issue documents the steps needed.

Each section should be fully completed before moving to the next section.

CI images

• Add Python 3.11 builds of miniforge-cuda. Drop CUDA 11.2 and 11.5, add Python 3.11 and 3.12. miniforge-cuda#55
• Add Python 3.11 CI images (ci-conda, ci-wheel, citestwheel) Drop CUDA 11.2 and 11.5, add Python 3.11 ci-imgs#96

CI workflows

Branch Strategy:

• Create a branch on shared-workflows called python-3.11
• Add Python 3.11 to the build matrix on the python-3.11 branch: https://github.com/rapidsai/shared-workflows/blob/e7ebbae5854727b897b65213cf51ff8b965f53c1/.github/workflows/conda-python-build.yaml#L56-L63
• Add Python 3.11 to the test matrix on the python-3.11 branch: https://github.com/rapidsai/shared-workflows/blob/e7ebbae5854727b897b65213cf51ff8b965f53c1/.github/workflows/conda-python-tests.yaml#L68-L82
  • When adjusting the test matrix, be aware of total GPU resource consumption. Build jobs are CPU only but test jobs require GPUs. We want to keep our GPU consumption roughly the same (don't double the test matrix size), even if it gets a bit sparser in its coverage. I would recommend changing a portion of the Python 3.10 jobs to use Python 3.11. Try to mix coverage across architectures, OS versions, Python versions, and CUDA versions, and be sure we cover configurations like "latest of everything (OS/Python/CUDA)" and "oldest of everything."

Experimental Strategy:

• @ajschmidt8 proposed that we could use GitHub Actions' continue-on-error to add Python 3.11 jobs that are allowed to fail, for all repos. Once all repos are passing, we could require the job to pass. This could be cleaner and less total work than our normal approach of the branch strategy above.

RAPIDS repositories

I copied a list of repositories from https://github.com/rapidsai/workflows/blob/dfd73ad47d977c57ee27b2349216f85946be757b/.github/workflows/nightly-pipeline.yaml#L27-L44 and sorted it roughly in dependency-tree order.

For each repo,

1. Update .github/workflows/*.yaml to point to the python-3.11 branch of shared-workflows
2. Update dependencies.yaml to add support for Python 3.11.
3. Review any pyproject.toml files for necessary changes (classifiers, etc.)
4. Update docs (README, etc) that reference a single Python version to point to the latest (3.11).
5. Once CI passes, merge the PR.

Most of this is easy to automate with rapids-reviser, and I've made an attempt at it here: https://github.com/rapidsai/rapids-reviser/pull/11. We still need to manually review the PRs for missing pieces.

Repos:

(checklist moved to #3 (comment))

Once all repos are migrated to the python-3.11 branch, the migration is complete. We merge python-3.11 into branch-24.02 on shared-workflows and then open follow-up PRs to each repo to reset the branches to branch-24.02. This "reset" is simple and should be automated with rapids-reviser.

Post-migration

• Update the ci-imgs repo's latest configuration to use Python 3.11. https://github.com/rapidsai/ci-imgs/blob/main/latest.yaml
  • The latest image is frequently used by CI jobs for building docs and testing notebooks. Be aware that issues may arise in those jobs.
  • PR: Use Python 3.11 in latest tag. ci-imgs#125
• Update the docker repo's matrix.yaml and matrix-test.yaml
  • PR: Add Python 3.11. docker#635
• (added by @jameslamb 2/27) update the build matrix in pypi-wheel-scripts so that Python 3.11 wheels are checked: https://github.com/rapidsai/pypi-wheel-scripts/blob/fa1e8744c8ec961a5b5e38ae172ae9c8c51b4280/release/check-wheels.sh#L41-L45
  • PR: https://github.com/rapidsai/pypi-wheel-scripts/pull/8

[jameslamb]

Next, let's merge the shared-workflow changes which introduce Python 3.11 jobs immediately, across all of RAPIDS, but using GitHub Actions' continue-on-error mechanism so they don't block merging PRs.

• merge add support for python 3.11 shared-workflows#176

I've proposed some changes to https://github.com/rapidsai/rapids-reviser/pull/11, would be nice to get that merged so we're working off one branch with the rapids-reviser changes:

• https://github.com/bdice/rapids-reviser/pull/1

Then let's get these jobs working, in RAPIDS dependency order. (will update this comment as more PRs are created).

• rapids-cmake
  • Add support for Python 3.11 rapids-cmake#547
• rmm
  • Add support for Python 3.11 rmm#1469
  • Requre NumPy 1.23+ rmm#1488
• pynvjitlink
  • Support Python 3.11 pynvjitlink#30
• kvikio
  • Add support for Python 3.11, require NumPy 1.23+ kvikio#350
• cudf
  • Add support for Python 3.11, require NumPy 1.23+ cudf#15111
• dask-cuda
  • Add support for Python 3.11 dask-cuda#1315
  • Requre NumPy 1.23+ dask-cuda#1316
• ucx-py
  • Add support for Python 3.11 ucx-py#1027
  • Requre NumPy 1.23+ ucx-py#1028
• ucxx
  • Add support for Python 3.11, require NumPy 1.23+ ucxx#199
• raft
  • Add support for Python 3.11, require NumPy 1.23+ raft#2200
• wholegraph
  • Add support for Python 3.11, require NumPy 1.23+ wholegraph#139
• cugraph
  • Add support for Python 3.11, require NumPy 1.23+ cugraph#4195
• cuml
  • Add support for Python 3.11, require NumPy 1.23+ cuml#5786
• cuspatial
  • Add support for Python 3.11, require NumPy 1.23+ cuspatial#1348
  • Fix Python 3.11 test failures cuspatial#1351
• cuxfilter
  • Add support for Python 3.11 cuxfilter#572
  • Requre NumPy 1.23+ cuxfilter#574
• cucim
  • Add support for Python 3.11, require NumPy 1.23+ cucim#704
• private repos
• integration
  • once everything's building run a job in integration repo confirming all the packages build together (example: [NO MRG] Test CI integration#700)
  • [NO MRG] Test Python 3.11 on CI integration#702