Skip to content

Commit 174c32a

Browse files
carmoccadmitsfBordaawaelchlidavidgilbertson
authored
Sync with master changes (#16230)
* [App] Introduce basic auth to Lightning CLI (#16105) * Introduce basic auth to Lightning CLI for app creation * Parsing creds added * Adding auth field to app instance body * Adding tests * Adding changelog entry * Adding more tests * Update runtime.py * Setting auth on update * Fix test * Update lightning-cloud dep * Apply suggestions from code review Co-authored-by: Adrian Wälchli <[email protected]> * Update runtime.py * Fix for release * Update base.txt Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> * docs: fix order of on_fit_start() hook (#16180) Fixes #16170 * docs: updated broken links (#16191) Co-authored-by: Shashwat <shashwat> Fixes #16186 * Always use the local rank zero imports (#16178) * update BYOC documentation with AWS details (#16044) Co-authored-by: Adrian Wälchli <[email protected]> * Using internal ip + port in a load balancer instead of URL exposed (#16119) Co-authored-by: thomas chaton <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove CUDA_LAUNCH_BLOCKING from Lite tests (#16177) * simplify torch.Tensor (#16190) * Fix inclusion of `model_parallel` document (#16197) fix link to gpu/advanced section * Update numpy requirement from <1.23.1,>=1.17.2 to >=1.17.2,<1.24.1 in /requirements (#16199) Update numpy requirement in /requirements Updates the requirements on [numpy](https://github.com/numpy/numpy) to permit the latest version. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v1.17.2...v1.24.0) --- updated-dependencies: - dependency-name: numpy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update s3fs requirement from <2022.8.3,>=2022.5.0 to >=2022.5.0,<2022.11.1 in /requirements (#16198) Update s3fs requirement in /requirements Updates the requirements on [s3fs](https://github.com/fsspec/s3fs) to permit the latest version. - [Release notes](https://github.com/fsspec/s3fs/releases) - [Changelog](https://github.com/fsspec/s3fs/blob/main/release-procedure.md) - [Commits](fsspec/s3fs@2022.5.0...2022.11.0) --- updated-dependencies: - dependency-name: s3fs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: upload only with release (#16194) * Update endpoint access examples: added info about accessing auth-protected apps (#16145) * Skip a failing Bagua test for manual optimization (#16225) * Fix type error when dividing chunk size in colossalai strategy (#16212) Co-authored-by: awaelchli <[email protected]> * Fix DDP on XLA (#16020) Co-authored-by: awaelchli <[email protected]> * Added support and test for custom artifact names in WandbLogger (#16173) * Update ipython[all] requirement from <8.6.1 to <8.7.1 in /requirements (#16220) Updates the requirements on [ipython[all]](https://github.com/ipython/ipython) to permit the latest version. - [Release notes](https://github.com/ipython/ipython/releases) - [Commits](ipython/ipython@rel-0.8.4...8.7.0) --- updated-dependencies: - dependency-name: ipython[all] dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improvements to checkpoint migration (#16233) Co-authored-by: Justus Schock <[email protected]> * Support arbitrary `Optimizable`s as optimizers (#16189) * Differences with master Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Dmitry Frolov <[email protected]> Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> Co-authored-by: David Gilbertson <[email protected]> Co-authored-by: Shashwat Agrawal <[email protected]> Co-authored-by: Raphael Randschau <[email protected]> Co-authored-by: thomas chaton <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: HELSON <[email protected]> Co-authored-by: Liyang90 <[email protected]> Co-authored-by: Manan Goel <[email protected]> Co-authored-by: Justus Schock <[email protected]>
1 parent a01cb22 commit 174c32a

File tree

86 files changed

+1013
-345
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

86 files changed

+1013
-345
lines changed

.github/CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ We recommend you follow this convention `<type>/<issue-id>_<short-name>` where t
199199

200200
We recommend creating a PR in a separate branch other than `master`, especially if you plan to submit several changes and do not want to wait until the first one is resolved (we can work on them in parallel).
201201

202-
First, make sure you have set [upstream](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork) by running:
202+
First, make sure you have set [upstream](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/configuring-a-remote-repository-for-a-fork) by running:
203203

204204
```bash
205205
git remote add upstream https://github.com/Lightning-AI/lightning.git

.github/workflows/release-pypi.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ defaults:
1313

1414
jobs:
1515

16-
1716
build-packages:
1817
uses: ./.github/workflows/_build-packages.yml
1918
with:
@@ -23,7 +22,7 @@ jobs:
2322
upload-packages:
2423
runs-on: ubuntu-20.04
2524
needs: build-packages
26-
if: startsWith(github.event.ref, 'refs/tags') || github.event_name == 'release'
25+
if: github.event_name == 'release'
2726
steps:
2827
- uses: actions/checkout@v3
2928
- uses: actions/download-artifact@v3

README.md

Lines changed: 15 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -367,36 +367,32 @@ ______________________________________________________________________
367367

368368
## Examples
369369

370-
###### Hello world
370+
###### Self-supervised Learning
371371

372-
- [MNIST hello world](https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/mnist-hello-world.html)
372+
- [CPC transforms](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#cpc-transforms)
373+
- [Moco v2 tranforms](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#moco-v2-transforms)
374+
- [SimCLR transforms](https://lightning-bolts.readthedocs.io/en/stable/transforms/self_supervised.html#simclr-transforms)
373375

374-
###### Contrastive Learning
376+
###### Convolutional Architectures
375377

376-
- [BYOL](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/self_supervised.html#byol)
377-
- [CPC v2](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/self_supervised.html#cpc-v2)
378-
- [Moco v2](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/self_supervised.html#moco-v2-api)
379-
- [SIMCLR](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/self_supervised.html#simclr)
380-
381-
###### NLP
382-
383-
- [GPT-2](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/convolutional.html#gpt-2)
384-
- [BERT](https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/text-transformers.html)
378+
- [GPT-2](https://lightning-bolts.readthedocs.io/en/stable/models/convolutional.html#gpt-2)
379+
- [UNet](https://lightning-bolts.readthedocs.io/en/stable/models/convolutional.html#unet)
385380

386381
###### Reinforcement Learning
387382

388-
- [DQN](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/reinforce_learn.html#dqn-models)
389-
- [Dueling-DQN](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/reinforce_learn.html#dueling-dqn)
390-
- [Reinforce](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/reinforce_learn.html#reinforce)
383+
- [DQN Loss](https://lightning-bolts.readthedocs.io/en/stable/losses.html#dqn-loss)
384+
- [Double DQN Loss](https://lightning-bolts.readthedocs.io/en/stable/losses.html#double-dqn-loss)
385+
- [Per DQN Loss](https://lightning-bolts.readthedocs.io/en/stable/losses.html#per-dqn-loss)
391386

392-
###### Vision
387+
###### GANs
393388

394-
- [GAN](https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/basic-gan.html)
389+
- [Basic GAN](https://lightning-bolts.readthedocs.io/en/stable/models/gans.html#basic-gan)
390+
- [DCGAN](https://lightning-bolts.readthedocs.io/en/stable/models/gans.html#dcgan)
395391

396392
###### Classic ML
397393

398-
- [Logistic Regression](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/classic_ml.html#logistic-regression)
399-
- [Linear Regression](https://lightning-bolts.readthedocs.io/en/stable/deprecated/models/classic_ml.html#linear-regression)
394+
- [Logistic Regression](https://lightning-bolts.readthedocs.io/en/stable/models/classic_ml.html#logistic-regression)
395+
- [Linear Regression](https://lightning-bolts.readthedocs.io/en/stable/models/classic_ml.html#linear-regression)
400396

401397
______________________________________________________________________
402398

dockers/README.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,6 @@ docker run --rm -it --gpus all pytorchlightning/pytorch_lightning:base-cuda-py3.
5050

5151
## Run Jupyter server
5252

53-
Inspiration comes from https://u.group/thinking/how-to-put-jupyter-notebooks-in-a-dockerfile
54-
5553
1. Build the docker image:
5654
```bash
5755
docker image build -t pytorch-lightning:v1.6.5 -f dockers/nvidia/Dockerfile --build-arg LIGHTNING_VERSION=1.6.5 .
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
:orphan:
2+
3+
.. _aws_cli:
4+
5+
6+
############################
7+
Create AWS role with AWS CLI
8+
############################
9+
10+
1. Install AWS CLI (see instructions `here <https://aws.amazon.com/cli/>`_).
11+
12+
2. Protect your role by creating a hard to guess password that will be used to authenticate Lightning (You will need to pass it to Lightning for authentication). In our example we will use `dummy`.
13+
14+
3. Create a role called `lightning-cloud` using the following command (replace <YOUR-HARD-TO-GUESS-PASSWORD> with your own):
15+
16+
.. code:: bash
17+
18+
aws iam create-role \
19+
--role-name lightning-cloud \
20+
--assume-role-policy-document '{"Statement":[{"Action":"sts:AssumeRole","Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::748115360335:root"}, "Condition": {"StringEquals": {"sts:ExternalId": "<YOUR-HARD-TO-GUESS-PASSWORD>"}}}]}' \
21+
--description " " \
22+
--max-session-duration 43200
23+
24+
4. Create a file `iam-policy.json` with the following permissions required for Lightning to manage cloud infrastructure for you:
25+
26+
.. code:: json
27+
28+
{
29+
"Version": "2012-10-17",
30+
"Statement": [
31+
{
32+
"Action": [
33+
"autoscaling:*",
34+
"cloudwatch:*",
35+
"ec2:*",
36+
"ecr:*",
37+
"eks:*",
38+
"elasticloadbalancing:*",
39+
"events:*",
40+
"guardduty:*",
41+
"iam:*",
42+
"logs:*",
43+
"route53resolver:*",
44+
"s3:*",
45+
"sns:*",
46+
"sqs:*",
47+
"tag:GetResources",
48+
"resource-groups:SearchResources"
49+
],
50+
"Effect": "Allow",
51+
"Resource": "*"
52+
},
53+
{
54+
"Effect": "Allow",
55+
"Action": "iam:CreateServiceLinkedRole",
56+
"Resource": "*",
57+
"Condition": {
58+
"StringLike": {
59+
"iam:AWSServiceName": [
60+
"guardduty.amazonaws.com",
61+
"malware-protection.guardduty.amazonaws.com"
62+
]
63+
}
64+
}
65+
},
66+
{
67+
"Effect": "Allow",
68+
"Action": "iam:CreateServiceLinkedRole",
69+
"Resource": "*",
70+
"Condition": {
71+
"StringEquals": {
72+
"iam:AWSServiceName": [
73+
"autoscaling.amazonaws.com",
74+
"ec2scheduled.amazonaws.com",
75+
"elasticloadbalancing.amazonaws.com",
76+
"spot.amazonaws.com",
77+
"spotfleet.amazonaws.com",
78+
"transitgateway.amazonaws.com"
79+
]
80+
}
81+
}
82+
}
83+
]
84+
}
85+
86+
5. Create a IAM policy and associate it with the role we just created, and pass in the path to your new file:
87+
88+
.. code:: bash
89+
90+
aws iam create-policy \
91+
--policy-name lightning-cloud \
92+
--description "policy granting lightning controlplane permissions" \
93+
--policy-document file:///my_dir/iam-policy.json
94+
95+
6. Fetch the role ARN so you can attach the policy:
96+
97+
.. code:: bash
98+
99+
aws iam get-role --role-name lightning-cloud --output json --query Role.Arn
100+
101+
7. Attach the policy to the IAM role you just created:
102+
103+
.. code:: bash
104+
105+
aws iam attach-role-policy \
106+
--role-name lightning-cloud \
107+
--policy-arn arn:aws:iam::1234567890:policy/lightning-cloud
108+
109+
------
110+
111+
**********************
112+
Next: Create a cluster
113+
**********************
114+
115+
You are now ready to create a Lightning cluster!
116+
117+
.. raw:: html
118+
119+
<div class="display-card-container">
120+
<div class="row">
121+
122+
.. Add callout items below this line
123+
124+
.. displayitem::
125+
:header: Create cluster
126+
:description: Create an AWS cluster for running ligthning apps, skip to step 2
127+
:button_link: create_cluster.html
128+
:col_css: col-md-12
129+
:height: 170
130+
131+
.. raw:: html
132+
133+
</div>
134+
</div>
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
:orphan:
2+
3+
.. _aws_console:
4+
5+
6+
###########################################
7+
Create AWS role with AWS management console
8+
###########################################
9+
10+
See the following video for instructions:
11+
12+
.. raw:: html
13+
14+
<video width="100%" max-width="800px" controls autoplay muted playsinline
15+
src="https://lightningaidev.wpengine.com/wp-content/uploads/2022/12/byoc.mp4"></video>
16+
17+
|
18+
19+
20+
1.1- Create a role
21+
------------------
22+
23+
The created role will give Lightning AI access (by specifying Ligthning account id), and will
24+
25+
1. Sign in to your AWS account. (If you don’t have an AWS account, create one).
26+
27+
2. On AWS console search for IAM > Roles > Create role.
28+
29+
In your new role set the following:
30+
31+
* Trusted entity type -> AWS account.
32+
33+
* An AWS account -> Another AWS account -> Account id: 748115360335. (This is Lightning's account id).
34+
35+
3. Protect your role by creating a hard to guess password that will be used to authenticate Lightning (You will need to pass it to Lightning for authnetication).
36+
37+
* Set Require external ID: <put-your-hard-to-guess-token>
38+
39+
4. Click "Next"
40+
41+
42+
1.2- Create a policy
43+
--------------------
44+
45+
1. Click "Create policy" -> A new tab opens to the Create policy page.
46+
47+
2. Click the JSON tab and copy-paste the `contents of this JSON file <https://gist.github.com/nicolai86/79b9caa5a3557d9d4f8a6e2227e34f2d>` into the JSON workspace.
48+
49+
.. note: Ignore the warnings generated by AWS.
50+
51+
3. Click "Next:Tags" (you can skip this step)
52+
53+
4. Complete policy creation by adding a meaningful name and description to the policy.
54+
55+
For example:
56+
57+
Name -> "lightning-cloud"
58+
59+
Description -> "permission to manage EC2 instances"
60+
61+
5. Add policy to the role: Return to the browser tab that you were using to create a role.
62+
Refresh the policy list and select the custom policy you just created (in this case “lai-byoc-policy”). The policy should appear at the top of the Policy List.
63+
64+
6. Click "Next".
65+
66+
7. Provide a meaningful name and description for the role, and click “Create Role”.
67+
68+
For example:
69+
70+
Name -> "lightning-cloud"
71+
72+
Description -> "Role for Lightning cloud permissions"
73+
74+
Congrats! You now successfully added the set up permissions needed for lightning to create a cluster.
75+
76+
77+
------
78+
79+
**********************
80+
Next: Create a cluster
81+
**********************
82+
83+
You are now ready to create a Lightning cluster!
84+
85+
.. raw:: html
86+
87+
<div class="display-card-container">
88+
<div class="row">
89+
90+
.. Add callout items below this line
91+
92+
.. displayitem::
93+
:header: Create cluster
94+
:description: Create an AWS cluster for running ligthning apps, skip to step 2
95+
:button_link: create_cluster.html
96+
:col_css: col-md-12
97+
:height: 170
98+
99+
.. raw:: html
100+
101+
</div>
102+
</div>

0 commit comments

Comments
 (0)