onnx · jcwchen · Jun 8, 2023 · Jun 8, 2023 · Jun 8, 2023 · Jun 8, 2023
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -3,140 +3,31 @@
 ## Description
 Description of model - What task does it address (i.e. object detection, image classification)? What is the main advantage or feature of this model's architecture?
 
-All ONNX models must pass the ONNX model checker before contribution. The snippet of code below can be used to perform the check. If any errors are encountered, it implies the check has failed.
+<!-- All ONNX models must pass the ONNX model checker before contribution. The snippet of code below can be used to perform the check. If any errors are encountered, it implies the check has failed.
 
 ```
 import onnx
 from onnx import checker
 model_proto = onnx.load("path to .onnx file")
-checker.check_model(model_proto)
+checker.check_model(model_proto, full_check=True)
 ```
-
-### Contribute a Gradio Demo to ONNX Organization on Hugging Face
-
-* Create an account on Hugging Face: https://huggingface.co/join
-* See list of models left to add to ONNX organization, please refer to the table with the [Models list](https://github.com/onnx/models#models)
-* Add Gradio Demo under your username, see blog for setting up Gradio Demo on Hugging Face: https://huggingface.co/blog/gradio-spaces
-* Request to join ONNX Organization: https://huggingface.co/onnx
-* Once approved transfer model from your username to ONNX organization
-* Add a badge for model in model table, see examples in [Models list](https://github.com/onnx/models#models)
-
-## Model
-
-|Model        |Download  | Download (with sample test data)|ONNX version|Opset version|Accuracy |
-|-------------|:--------------|:--------------|:--------------|:--------------|:--------------|
-|Model Name       | Relative link to ONNX Model with size  | tar file containing ONNX model and synthetic test data (in .pb format)|ONNX version used for conversion| Opset version used for conversion|Accuracy values |
-|Example (VGG 19)|    [548.1 MB](classification/vgg/model/vgg19-7.onnx)    |[508.5 MB](classification/vgg/model/vgg19-7.tar.gz)| 1.2.1  |7 | 73.72     |
-
-Please submit new models with Git LFS by committing directly to the repository, and using relative links (i.e. ***model/vgg19-7.onnx***) in the table above. In this file name example, ***vgg19*** is the name of the model and ***7*** is the opset number.
-
-### Source
-Source Framework ==> ONNX model  
-i.e. Caffe2 DenseNet-121 ==> ONNX DenseNet
-
-## Inference
-Step by step instructions on how to use the pretrained model and link to an example notebook/code. This section should ideally contain:
-
-### Input
-Input to network (Example: 224x224 pixels in RGB)
-
-### Preprocessing
-Preprocessing required
-
-### Output
-Output of network
-
-### Postprocessing
-Post processing and meaning of output
-
-## Model Creation
-
-### Dataset (Train and validation)
-This section should discuss datasets and any preparation steps if required.
-
-### Training
-Training details (preprocessing, hyperparameters, resources and environment) along with link to a training notebook (optional).
-
-Also clarify in case the model is not trained from scratch and include the source/process used to obtain the ONNX model.
-
-### Validation accuracy
-Validation script/notebook used to obtain accuracy reported above along with details of how to use it and reproduce accuracy. Details of experiments leading to accuracy from the reference paper.
-
-## Test Data Creation
-
-Creating test data for uploaded models can help CI to verify the uploaded models by ONNXRuntime utilties. Please upload the ONNX model with created test data (`test_data_set_0`) in the .tar.gz.
-
-### Requirement
-```
-pip install onnx onnxruntime numpy
-git clone https://github.com/onnx/models.git
-````
-### Usage
-```
-def create_test_dir(model_path, root_path, test_name,
-                    name_input_map=None, symbolic_dim_values_map=None,
-                    name_output_map=None):
-    """
-    Create a test directory that can be used with onnx_test_runner, onnxruntime_perf_test.
-    Generates random input data for any missing inputs.
-    Saves output from running the model if name_output_map is not provided.
-
-    :param model_path: Path to the onnx model file to use.
-    :param root_path: Root path to create the test directory in.
-    :param test_name: Name for test. Will be added to the root_path to create the test directory name.
-    :param name_input_map: Map of input names to numpy ndarray data for each input.
-    :param symbolic_dim_values_map: Map of symbolic dimension names to values to use for the input data if creating
-                                    using random data.
-    :param name_output_map: Optional map of output names to numpy ndarray expected output data.
-                            If not provided, the model will be run with the input to generate output data to save.
-    :return: None
-    """
-```
-### Example
-The input/output .pb files will be produced under `temp/examples/test1/test_data_set_0`.
-```
-import sys
-sys.path.append('<onnx/models root dir>/workflow_scripts/')
-import ort_test_dir_utils
-import numpy as np
-
-# example model with two float32 inputs called 'input' [batch_size, 1, 224, 224])
-model_path = '<onnx/models root dir>/vision/super_resolution/sub_pixel_cnn_2016/model/super-resolution-10.onnx'
-
-# If there is no symbolic_dim
-ort_test_dir_utils.create_test_dir(model_path, 'temp/examples', 'test1')
-
-# when using the default data generation any symbolic dimension values can be provided
-# otherwise the default value for missing symbolic_vals would be 1
-
-symbolic_vals = {'batch_size': 1} # provide value for symbolic dim named 'batch_size'
-
-# let create_test_dir create random input in the (arbitrary) default range of -10 to 10. 
-# it will create data of the correct type based on the model.
-ort_test_dir_utils.create_test_dir(model_path, 'temp/examples', 'test1', symbolic_dim_values_map=symbolic_vals)
-
-# alternatively some or all input can be provided directly. any missing inputs will have random data generated.
-batch_size = 64
-inputs = {'input': np.random.rand(batch_size, 1, 224, 224).astype(np.float32)}
-
-ort_test_dir_utils.create_test_dir(model_path, 'temp/examples', 'test2', name_input_map=inputs)
-
-```
-
-### More details
-https://github.com/microsoft/onnxruntime/blob/master/tools/python/PythonTools.md
-
-<hr>
-
-### Update ONNX_HUB_MANIFEST.json for ONNX Hub
-If this PR does update/add .onnx or .tar.gz files, please use `python workflow_scripts/generate_onnx_hub_manifest.py --target diff` to update ONNX_HUB_MANIFEST.json with according model information (especially SHA) for ONNX Hub.
-
-### References
-Link to paper or references.
-
-## Contributors
-Contributors' names
-
-## License
-Add license information - on default, Apache 2.0
-<hr>
+-->
+
+### Contribute a new ONNX model with a Python script for reproducing the model
+6 required files under models/python/XXX/ directory:
+- [ ] `XXX.onnx`: the single ONNX model file with the same name as the directory. Please use opset version as model's postfix. For example, `AlexNet-18.onnx` is the ONNX model file for AlexNet model with opset 18.
+- [ ] `model.py`: a Python script named model.py for converting ONNX models from orignal framework, for example, PyTorch, TensorFlow, Huggingface or etc. The model.py should be able to convert the model from source framework to ONNX format with the specified name as above. The script should be able to run with the provided requirements.txt and therefore people can reproduce the ONNX model with the script. Also please include the original inference code in the script for reference.
+- [ ] `requirements.txt`: a text file listing all the required Python packages and their versions.
+- [ ] `test_data_set_0`: a directory containing the test data set. The test data set should be in the format of .pb. The test data set can be generated by the script: [ort_test_dir_utils.py](../workflow_scripts/ort_test_dir_utils.py). For details, please check another tutorials: [CreateTestData.md](../docs/CreateTestData.md).
+- [ ] README.md: a readme file describing the model and how to use the model. See example README.md for model in [ExampleModelREADME.md](../docs/ExampleModelREADME.md).
+- [ ] `LICENSE`: a standalone license file for the model. For instance, MIT License.
+
+<!-- The ONNX file needs to pass onnx.checker with full_check and the test data set needs to be validated by onnxruntime from CI pipelines. Besides, the CI will run mlagility's benchit to test the model. -->
+
+### Required updates for corresponding ONNX model
+- [ ] Update [README.md](../README.md) under the root directory with the new model information. New added model needs to be added to the table as Model Class in the README.md and in that case [generate_onnx_hub_manifest.py](../workflow_scripts/generate_onnx_hub_manifest.py) can detect the new added model.
+- [ ] Update [ONNX_HUB_MANIFEST.json](../ONNX_HUB_MANIFEST.json) for ONNX Hub: If this PR does update/add .onnx, please use `python workflow_scripts/generate_onnx_hub_manifest.py --target diff` to update ONNX_HUB_MANIFEST.json with according model information (especially SHA) for ONNX Hub.
+
+<!-- 
+Contribute a Gradio Demo to ONNX Organization on Hugging Face. See [ContributeGradio.md](../docs/ContributeGradio.md) for details.
+-->
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
@@ -38,7 +38,7 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL

diff --git a/.github/workflows/linux_ci.yml b/.github/workflows/linux_ci.yml
@@ -21,11 +21,11 @@ jobs:
         architecture: ['x64']
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
         name: Checkout repo
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@bd6b4b6205c4dbad673328db7b31b7fab9e241c0 # v4.6.1
         with:
           python-version: ${{ matrix.python-version }}
           architecture: ${{ matrix.architecture }}
@@ -40,13 +40,9 @@ jobs:
       - name: Test updated ONNX_HUB_MANIFEST.json
         run: |
           python -m pip install typepy BeautifulSoup4 markdown pandas
-          python workflow_scripts/generate_onnx_hub_manifest.py --target diff --drop
+          python workflow_scripts/generate_onnx_hub_manifest.py --target diff
           git diff --exit-code -- ONNX_HUB_MANIFEST.json || { echo 'Please use "python workflow_scripts/generate_onnx_hub_manifest.py --target diff" to update ONNX_HUB_MANIFEST.json.' ; exit 1; }
 
-      - name: Test new models by onnx
+      - name: Test new models by onnx and onnxruntime
         run: |
-          python workflow_scripts/test_models.py --target onnx --drop
-
-      - name: Test new models by onnxruntime
-        run: |
-          python workflow_scripts/test_models.py --target onnxruntime --drop
+          python workflow_scripts/test_models.py --target all --drop
diff --git a/.github/workflows/validate_model.yml b/.github/workflows/validate_model.yml
@@ -0,0 +1,42 @@
+name: Validate created ONNX model from mlagility or Python
+
+on:
+  schedule:
+    - cron:  '00 00 * * MON'
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.8']
+
+    steps:
+      - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
+        name: Checkout repo
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniconda-version: "latest"
+          activate-environment: mla
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies and mlagility
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install onnx onnxruntime requests py-cpuinfo
+          # Print CPU info for debugging ONNX Runtime inference difference
+          python -m cpuinfo
+          # or python -m pip install mlagility
+          python -m pip install transformers
+          git clone https://github.com/groq/mlagility.git
+          cd mlagility
+          pip install -r models/requirements.txt
+          pip install -e .
+
+      - name: Validate created ONNX model from Python
+        run: |
+          python workflow_scripts/run_python.py
diff --git a/.github/workflows/windows_ci.yml b/.github/workflows/windows_ci.yml
@@ -21,11 +21,11 @@ jobs:
         architecture: ['x64']
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
         name: Checkout repo
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@bd6b4b6205c4dbad673328db7b31b7fab9e241c0 # v4.6.1
         with:
           python-version: ${{ matrix.python-version }}
           architecture: ${{ matrix.architecture }}
@@ -37,10 +37,6 @@ jobs:
           # Print CPU info for debugging ONNX Runtime inference difference
           python -m cpuinfo
 
-      - name: Test new models by onnx
+      - name: Test new models by onnx and onnxruntime
         run: |
-          python workflow_scripts/test_models.py --target onnx --drop
-
-      - name: Test new models by onnxruntime
-        run: |
-          python workflow_scripts/test_models.py --target onnxruntime --drop
+          python workflow_scripts/test_models.py --target all --drop
diff --git a/ONNX_HUB_MANIFEST.json b/ONNX_HUB_MANIFEST.json
@@ -1,4 +1,97 @@
 [
+    {
+        "model": "alexnet-18",
+        "model_path": "models/python/alexnet-18/alexnet-18.onnx",
+        "onnx_version": "1.14",
+        "opset_version": 18,
+        "metadata": {
+            "model_sha": "3f954fd5efd73ebaf2095770fa69c0c7cd254042f1c3d819c33fc5c8fe5199fa",
+            "model_bytes": 244407346,
+            "tags": [
+                "models",
+                "python",
+                "alexnet-18"
+            ],
+            "io_ports": {
+                "inputs": [
+                    {
+                        "name": "input.1",
+                        "shape": [
+                            1,
+                            3,
+                            224,
+                            224
+                        ],
+                        "type": "tensor(float)"
+                    }
+                ],
+                "outputs": [
+                    {
+                        "name": "36",
+                        "shape": [
+                            1,
+                            1000
+                        ],
+                        "type": "tensor(float)"
+                    }
+                ]
+            }
+        }
+    },
+    {
+        "model": "Bart",
+        "model_path": "models/python/bart-18/bart-18.onnx",
+        "onnx_version": "1.14",
+        "opset_version": 18,
+        "metadata": {
+            "model_sha": "b3a32b95468cde7d9b52cab4879d6bf5b296ef7403c5b7d7a1b03a2c85f34788",
+            "model_bytes": 437579710,
+            "tags": [
+                "models",
+                "python",
+                "bart-18"
+            ],
+            "io_ports": {
+                "inputs": [
+                    {
+                        "name": "input.1",
+                        "shape": [
+                            1,
+                            128
+                        ],
+                        "type": "tensor(int64)"
+                    },
+                    {
+                        "name": "attention_mask",
+                        "shape": [
+                            1,
+                            128
+                        ],
+                        "type": "tensor(float)"
+                    }
+                ],
+                "outputs": [
+                    {
+                        "name": "onnx::Gather_1274",
+                        "shape": [
+                            1,
+                            128,
+                            768
+                        ],
+                        "type": "tensor(float)"
+                    },
+                    {
+                        "name": "1277",
+                        "shape": [
+                            1,
+                            768
+                        ],
+                        "type": "tensor(float)"
+                    }
+                ]
+            }
+        }
+    },
     {
         "model": "BERT-Squad",
         "model_path": "text/machine_comprehension/bert-squad/model/bertsquad-10.onnx",

diff --git a/README.md b/README.md
@@ -161,6 +161,12 @@ There are many interesting deep learning models that do not fit into the categor
 |Autoencoders|[A Hierarchical Neural Autoencoder for Paragraphs and Documents](https://arxiv.org/abs/1506.01057)|An LSTM (long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs.<br>[contribute](contribute.md)|
 <hr>
 
+### Opset 18 models <a name="others"/>
+There are many interesting deep learning models that do not fit into the categories described above. The ONNX team would like to highly encourage users and researchers to [contribute](contribute.md) their models to the growing model zoo.
+
+|Model Class |Reference |Description |
+|-|-|-|
+|<b>[AlexNet](models/python/alexnet-18)</b>|[Krizhevsky et al.](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)|A Deep CNN model (up to 8 layers) where the input is an image and the output is a vector of 1000 numbers. <br> Top-5 error from paper - ~15%| |
 ## Usage <a name="usage-"/>
 
 Every ONNX backend should support running the models out of the box. After downloading and extracting the tarball of each model, you will find:
@@ -233,9 +239,12 @@ outputs = list(sample['outputs'])
 np.testing.assert_almost_equal(outputs, backend.run_model(model, inputs))
 ```
 
+### Usage - Model optimization
+TBD
+
 ### Usage - Model quantization
-You can get quantized ONNX models by using [Intel® Neural Compressor](https://github.com/intel/neural-compressor). It provides web-based UI service to make quantization easier and supports code-based usage for more abundant quantization settings. Refer to [bench document](https://github.com/intel/neural-compressor/blob/master/docs/bench.md) for how to use web-based UI service and [example document](./resource/docs/INC_code.md) for a simple code-based demo.
-![image](./resource/images/INC_GUI.gif)
+You can get quantized ONNX models by using [Intel® Neural Compressor](https://github.com/intel/neural-compressor). It provides web-based UI service to make quantization easier and supports code-based usage for more abundant quantization settings. Refer to [bench document](https://github.com/intel/neural-compressor/blob/master/docs/bench.md) for how to use web-based UI service and [example document](./docs/INCCode.md) for a simple code-based demo.
+![image](./resource/images/INCGUI.gif)
 
 ### Usage - Git LFS <a name="gitlfs-"/>