clone sdk repo in grow up scenario (#3667)

* save

* clone sdk repo in grow up scenario

* lint fix

* doc fix

* clone spec repo in grow up

* add skip-generation param in rerunning task

* update doc

* mute output

* add new param --is-public-repo

* lint

* result change

* fix test

Co-authored-by: Wei Dong <[email protected]>

Author: dw511214992

tools/sdk-generation-pipeline/Dockerfile

@@ -46,7 +46,6 @@ RUN apt-get install -y dotnet-sdk-6.0
 # install git
 RUN add-apt-repository ppa:git-core/ppa -y && apt update && apt upgrade -y && apt install git -y
 RUN git config --global credential.helper store && git config --global core.fileMode false
-
 # install depended packages
 RUN pip3 install --upgrade wheel PyYAML requests
 RUN npm install -g typescript

tools/sdk-generation-pipeline/documents/docker/README.md

@@ -21,7 +21,7 @@ The docker image will be used in different scenarios:
 Command
 
 ```shell
-docker run -it --privileged  -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md docker.image:latest --readme={relative_readme} --sdk={sdk_to_generate}
+docker run -it --privileged  -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme} --sdk={sdk_to_generate}
 ```
 
 Parameter description:
@@ -36,13 +36,13 @@ Parameter description:
 
 Example Command:
 ```shell
-docker run -it --privileged  -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir docker.image:latest --readme="specification/agrifood/resource-manager/readme.md" --sdk=js,java
+docker run -it --privileged  -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md" --sdk=js,java
 ```
 
-After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can open your browser and request `http://127.0.0.1:8080/?folder=/work-dir` for further grow up development.
+After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can [open your local vscode and connect to docker container](./vscode-connect-docker-container.md) for further grow up development.
 If you want to re-generate codes after grow up development or changing swagger, please run command in docker container:
 ```shell
-rerun-tasks -readme={relative_readme} --sdk={sdk_to_generate}
+rerun-tasks --readme={relative_readme} --sdk={sdk_to_generate}
 ```
 rerun-tasks is a script, which invokes task engine to re-run tasks.
 
@@ -53,41 +53,36 @@ There are two scenarios here:
 1. Service team has generated codes locally by using docker image and has exited the docker container. But they want to do grow up development now.
 2. Service team has generated codes by using sdk generation pipeline, and sdk generation pipeline creates a work branch. Service team hope to do grow up based on the work branch.
 
-Compared to scenario 1, scenario 2 needs user to clone and checkout the work branch by themselves. It’s very simple with git:
+Run docker commands to do grow up development:
 ```shell
-cd {local_work_folder}
-git clone -b {work-branch} {repo-url}
+docker run -it --privileged -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme} --spec-link={spec-link} --sdk-work-branch={sdk-work-branch-link}
 ```
 Parameter description:
 
-| Parameter             | Description                                                                             | Example                                       |
-|-----------------------|-----------------------------------------------------------------------------------------|-----------------------------------------------|
-| { local_work_folder } | Required. It's used to point to the work folder, which stores all sdk repositories.     | /home/test/work-dir                           |
-| {work_branch }        | Required. It's used to point the work branch name generated by SDK generation pipeline. | sdkAuto/workbranch                            |
-| {repo_url}            | Required. It's used to point the repository url that work branch is in.                 | https://github.com/Azure/azure-sdk-for-js.git |
+| Parameter                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                       | Example                                                                     |
+|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
+| { local_spec_repo_path }  | Optional. If you want to change the swagger and re-generate codes, you need to mount the swagger repo. If you only want to do grow up development, no need to mount it. If you input a the link by parameter {spec-link}, docker container helps clone it.                                                                                                                                                                                        | /home/test/azure-rest-api-specs                                             |
+| { local_work_folder }     | Required. It's used to point to the work folder, which stores all sdk repositories.                                                                                                                                                                                                                                                                                                                                                               | /home/test/work-dir                                                         |
+| { local_autorest_config } | Optional. When you generate data-plane sdk, and there is no autorest configuration in sdk repository or you want to change the autorest configuration, you can set new autorest config in a file and mount it to the docker container. About the content of file, please refer to [document](https://github.com/Azure/azure-rest-api-specs/blob/dpg-doc/documentation/onboard-dpg-in-sdkautomation/add-autorest-configuration-in-spec-comment.md) | /home/test/autorest.md ([Example file](./autorest-config-file-sample.md))   |
+| { relative_readme }       | Optional. It's used to specify the readme.md file and docker image uses it to start mock server. it's the relative path from {path_to_local_spec_repo}. If not specified, mock server will not start.                                                                                                                                                                                                                                             | specification/agrifood/resource-manager/readme.md                           |
+| { sdk-work-branch-link }  | **Only Required in Scenario 2**. It's used to specify the link to sdk work branch generated by sdk generation pipeline, and docker container will use it to clone sdk repo and checkout the work branch. **In scenario 2, please make sure there is no corresponding sdk repo under {local_work_folder}.**                                                                                                                                        | specification/agrifood/resource-manager/readme.md                           |
+| { spec-link }             | **Only Required in Scenario 2 and no local repo mounted**. It's used to specify the link to spec repo, which can be PR link, repo Link or branch link. Then docker container will use it to clone spec repo and checkout the properly branch. **In scenario 2, please make sure there is no corresponding spec repo with path '/spec-repo' in docker container.**                                                                                 | specification/agrifood/resource-manager/readme.md                           |
 
-Then run docker commands to do grow up development:
+Example Command:
+Scenario 1:
 ```shell
-docker run -it --privileged  -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md docker.image:latest --readme={relative_readme}
+docker run -it --privileged  -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md"
 ```
-Parameter description:
 
-| Parameter                  | Description                                                                                                                                                                                                                                                                                                                                                                                                                                       | Example                                                                     |
-|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
-| { local_spec_repo_path }   | Optional. If you want to change the swagger and re-generate codes, you need to mount the swagger repo. If you only want to do grow up development, no need to mount it.                                                                                                                                                                                                                                                                           | /home/test/azure-rest-api-specs                                             |
-| { local_work_folder }      | Required. It's used to point to the work folder, which stores all sdk repositories.                                                                                                                                                                                                                                                                                                                                                               | /home/test/work-dir                                                         |
-| { local_autorest_config }  | Optional. When you generate data-plane sdk, and there is no autorest configuration in sdk repository or you want to change the autorest configuration, you can set new autorest config in a file and mount it to the docker container. About the content of file, please refer to [document](https://github.com/Azure/azure-rest-api-specs/blob/dpg-doc/documentation/onboard-dpg-in-sdkautomation/add-autorest-configuration-in-spec-comment.md) | /home/test/autorest.md ([Example file](./autorest-config-file-sample.md))   |
-| { relative_readme }        | Optional. It's used to specify the readme.md file and docker image uses it to start mock server. it's the relative path from {path_to_local_spec_repo}. If not specified, mock server will not start.                                                                                                                                                                                                                                             | specification/agrifood/resource-manager/readme.md                           |
-
-Example Command:
+Scenario 2:
 ```shell
-docker run -it --privileged  -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir docker.image:latest
+docker run -it --privileged -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md" --spec-link="https://github.com/Azure/azure-rest-api-specs/pull/19850" --sdk-work-branch="https://github.com/Azure/azure-sdk-for-js/tree/agrifood/dev/branch"
 ```
 
-After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can open your browser and request `http://127.0.0.1:8080/?folder=/work-dir` for further grow up development.
+After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can [open your local vscode and connect to docker container](./vscode-connect-docker-container.md) for further grow up development.
 If you want to re-generate codes after grow up development or changing swagger, please run command in docker container:
 ```shell
-rerun-tasks -readme={relative_readme} --sdk={sdk_to_generate}
+rerun-tasks --readme={relative_readme} --sdk={sdk_to_generate}
 ```
 rerun-tasks is a script, which invokes task engine to re-run tasks.
 
@@ -101,7 +96,7 @@ Before running docker command, pipeline must prepare the spec repo and sdk repo.
 Command:
 
 ```shell
-docker run --privileged  -v {spec_repo_path}:/spec-repo -v {sdk_repo_path}:/sdk-repo -v {local_autorest_config}:/autorest.md -v {output_folder_path}:/tmp/output docker.image:latest --readme={relative_readme}
+docker run --privileged  -v {spec_repo_path}:/spec-repo -v {sdk_repo_path}:/sdk-repo -v {local_autorest_config}:/autorest.md -v {output_folder_path}:/tmp/output sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme}
 ```
 
 Parameter description:
@@ -115,7 +110,7 @@ Parameter description:
 
 Example Command:
 ```shell
-docker run --privileged -v /home/vsts/work/azure-rest-api-specs:/spec-repo -v /home/vsts/work/azure-sdk-for-js:/sdk-repo -v /home/vsts/work/output:/tmp/output docker.image:latest --readme=specification/agrifood/resource-manager/readme.md
+docker run --privileged -v /home/vsts/work/azure-rest-api-specs:/spec-repo -v /home/vsts/work/azure-sdk-for-js:/sdk-repo -v /home/vsts/work/output:/tmp/output sdkgeneration.azurecr.io/sdk-generation:latest --readme=specification/agrifood/resource-manager/readme.md
 ```
 
 After running the command in pipeline, docker will execute tasks automatically. Also, there will be output files generated, which will be used by pipeline's other job, such as upload codes, parsing logs.

tools/sdk-generation-pipeline/documents/task-engine/README.md

@@ -72,7 +72,8 @@ Input file for generate and build script.
   "headRef": "refs/pull/1234/merge",
   "repoHttpsUrl": "https://github.com/Azure/azure-rest-api-specs.git",
   "relatedReadmeMdFile": "compute/resource-manager/readme.md"
-  "serviceType": "resource-manager"
+  "serviceType": "resource-manager",
+  "skipGeneration": "false"
 }
 ```
 

tools/sdk-generation-pipeline/documents/task-engine/schema/GenerateAndBuildInputSchema.json

@@ -27,6 +27,14 @@
       // The type of related swagger: resource-manager or data-plane
       "type": "string",
       "enum": ["resource-manager", "data-plane"]
+    },
+    "autorestConfig": {
+      // The autorest config got from /autorest.md, which is mounted by user
+      "type": "string"
+    },
+    "skipGeneration": {
+      // Whether skip generation
+      "type": "boolean"
     }
   },
   "required": ["specFolder", "headSha", "headRef", "repoHttpsUrl", "relatedReadmeMdFile"]

tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerContext.ts

@@ -17,6 +17,10 @@ export class DockerContext {
     sdkRepo?: string;
     resultOutputFolder?: string;
     autorestConfigFilePath?: string;
+    specLink?: string;
+    sdkWorkBranchLink?: string;
+    skipGeneration: boolean;
+    isPublicRepo: boolean;
     logger: Logger;
 
     /*
@@ -34,22 +38,40 @@ export class DockerContext {
         this.sdkRepo = inputParams.sdkRepo;
         this.resultOutputFolder = inputParams.resultOutputFolder;
         this.autorestConfigFilePath = inputParams.autorestConfigFilePath;
+        this.specLink = inputParams.specLink;
+        this.sdkWorkBranchLink = inputParams.sdkWorkBranchLink;
+        this.skipGeneration = inputParams.skipGeneration;
         this.logger = initializeLogger(path.join(inputParams.resultOutputFolder, inputParams.dockerLogger), 'docker');
 
         if (this.sdkList?.length === 0 && fs.existsSync(this.workDir)) {
             this.logger.info('Preparing environment to do grow up development');
             this.mode = DockerRunningModel.GrowUp;
-            this.validateSpecRepo();
+            this.isPublicRepo = false;
+            if (!this.specLink) {
+                try {
+                    this.validateSpecRepo();
+                } catch (e) {
+                    throw new Error(`Cannot get spec repo link by parameter --spec-link, or get mounted swagger repo.`);
+                }
+            } else {
+                this.validateSpecLink();
+            }
+
             this.validateWorkDir();
+            if (!!this.sdkWorkBranchLink) {
+                this.validateWorkBranchLink();
+            }
         } else if (fs.existsSync(this.workDir)) {
             this.logger.info('Preparing environment to generate codes and do grow up development in local');
             this.mode = DockerRunningModel.CodeGenAndGrowUp;
+            this.isPublicRepo = false;
             this.validateSpecRepo();
             this.validateReadmeMdPath();
             this.validateSdk();
         } else {
             this.logger.info('Preparing environment to generate codes in pipeline');
             this.mode = DockerRunningModel.Pipeline;
+            this.isPublicRepo = inputParams.isPublicRepo;
             this.validateSdkRepo();
             this.validateSpecRepo();
             this.validateReadmeMdPath();
@@ -102,4 +124,18 @@ export class DockerContext {
             throw new Error(`Cannot find ${this.resultOutputFolder}, please mount it to docker container`);
         }
     }
+
+    private validateWorkBranchLink() {
+        const match = this.sdkWorkBranchLink.match(/(https.*\/([^\/]*))\/tree\/(.*)/);
+        if (match?.length !== 4) {
+            throw new Error(`Get invalid sdk work branch link: ${this.sdkWorkBranchLink}`);
+        }
+    }
+
+    private validateSpecLink() {
+        const match = this.specLink.match(/http.*/);
+        if (!match) {
+            throw new Error(`Get invalid sdk work branch link: ${this.specLink}`);
+        }
+    }
 }