Refactor parsing and general refinements to configuration / generated Swagger #67

danielsharvey · 2020-05-15T10:03:01Z

Work in progress:

Replacing return type for `generateAttributeSchema()` with OpenApi.UpdatedSchema causes tsc error, probably due to elements like `allOf`. Proposal is refactor UpdatedSchema more fully based on OpenAPI 3. Also refactor old and unused schema field `common_name`.

Refactor path generation to access model association information (FK attributes + schemas etc) via the original model definition rather than a copy in the route information. As part of this, add association to Sails Model interface.

Includes minor fix to attribute property mapping (Sail --> Swagger).

Includes: - Add handling of `regex` validation as Swagger `pattern`. - Fix handling of `isInteger`. - Fix handling of description; include both model/collection description **and** other provider Sails/Swagger descriptions. - Fix handling over extra 'annotations' (e.g. autoincrement) which overwrote description.

WIP - some of the new interfaces are not utilised / fully utilised. Add `exclude` boolean to annotate Sails actions (Swagger Operations) to be excluded from generated Swagger. Refactor controller file Swagger to include `controller` element containing documentation to be applied to **all** actions in the file e.g. tags. Refactor model file Swagger to include `model` element containing documentation to be applied to **all** blueprint actions for the model e.g. tags. Refactor model attribute Swagger to include `meta.swagger` for applying documentation to individual attributes. Refactor model attribute Swagger to include `meta.swagger.exclude` for excluding individual attributes from the generated Swagger. Add partial Typescript definition for `npm install include-all`.

Includes new proposed model using the `meta.swagger` attribute on actions2 inputs and exits. Includes `outputExample` attribute for actions2 exits.

The `describe()` does not accept async functions; see https://github.com/mochajs/mocha/issues/.

Part of add additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Separate: 1. Parsing of Sails runtime models (`sails.models`), and 2. Loading and parsing of JSDoc from model source files. Add generation of default tags for models and incorporate into Swagger generation. Refactor integration test to compare Swagger JSON object (rather than string) as test output is a far more helpful diff. Correct `parseModels()` test data to be models dictionary rather than array.

Part of add additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Add updated integration test fixture (generated Swagger output).

Part of add additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Add generation of default tags for models and incorporate into Swagger generation. Add updated integration test fixture (generated Swagger output).

danielsharvey · 2020-05-20T01:43:20Z

Some background notes: The need the merge custom routes with bound routes introduced complexity in that there was no single source of truth. Issues matching actions between the two, and handling the flattening that Sails performs in terms of case sensitivity for action identities needed to be handled.

This PR re-examined the Sails 'router:bind' events and the Sails 'controller loading' process. It deals with both to produce a more consistent mechanism with no (less) tricky edge cases.

Part of add additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Refactor SwaggerRouteInfo as the primary 'route' data structure: - Remove some data better looked up as req (models, associations, tags) - Refactor information stored re actions / action types. Refactor from using parsing of `config/routes.js` directly to parsing of only ‘router:bind’ events which contain all the same information and additional routes including blueprints and custom routes added by hooks etc. Replaces the 'withoutSwaggerRoutes' process with loading process based on Sails internal process and merges Swagger elements (including JSDoc). Background: The need the merge custom routes with bound routes introduced complexity in that there was no **single** source of truth. Issues matching actions between the two, and handling the flattening that Sails performs in terms of case sensitivity for action identities needed to be handled. This PR re-examined the Sails 'router:bind' events and the Sails 'controller loading' process. It deals with both to produce a more consistent mechanism with less tricky edge cases.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Refactor loading/parsing of controller files to utilise same mechanism as Sails itself (results in better handling of edges case with case sensitivity of filenames and related). Refactor loading and parsing of controller JSDoc. Refactor merging of controller file and JSDoc `swagger` elements.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing).

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Refactor association route aggregation to consider: 1. Blueprint prefix, REST prefix, and model including any pluralization 2. More complete grouping check including verb, model, and blueprint Update api/controllers and api/models test data to include Sails 'model' and 'collection' associations (used for tests).

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). WIP commit (several back) excluded controllers+controllersJsDoc from mergeComponents() and mergeTags(); re-integrate.

Add more test data including model associations, attribute validations, updated `swagger` / `@swagger` sample documentation and options.

danielsharvey · 2020-05-26T10:19:45Z

Some thoughts on approach:

Documenting Model Schemas

We reference documenting schemas in relation to models (e.g. /User below) but the treatment is inconsistent, in that the description tag is used to populate the model OpenAPI.Schema, but the tags are actually documentation added to all blueprint actions (OpenApi.Operation) for the model.

I suggest:

For @swagger: Use the /User documentation for OpenAPI.Schema properly with a convenience mechanism supporting tags for all blueprint actions as this is typically used to logically group operations in Swagger/OpenAPI renderers.
For the swagger export: Make use of a new modelSchema element alongside the actions element. See below.

/**
 * @swagger
 *
 * /User:
 *   tags:
 *     - Tag Name
 * /findone:
 *   externalDocs:
 *     url: https://docs.com/here
 */
module.exports = {
  attributes: {
    uid: {
      type: 'string',
      example: '012345',
      description: 'A unique identifier',
    }
  },
  swagger: {
    actions: {
      create: { ... },
    },
    modelSchema: {
      externalDocs: { ... },
    },
    tags: [...]
    components: {...}
  }
};

Documenting Controllers

With reference to the convenience mechanism above, used to add a list of tags to all blueprint actions for a model, I propose a special action key named allActions which enables OpenAPI.Operation documentation to be added to all actions contained within the controller file. The obvious example being tags.

For example:

/**
 * @swagger
 *
 * /allActions:
 *   tags:
 *     - Auth
 * tags:
 *   - name: Auth
 *     description: Module description...
 */
module.exports = {
  ...
  swagger: {
    actions: {
      allActions: {
        tags: [ 'Auth' ],
      }
    }
  }
};

Let me know your thoughts @theoomoregbee.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Adjust some warnings from `sails.warn` to `sails.verbose`. Correct shortcut blueprint route processing. Add warning error messages for action documentation for actions which do not exist.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Add warning error messages for action documentation for actions which do not exist.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Tweak Typescript interfaces: 1. Adjust some mandatory OpenAPI elements to be optional as this hook auto-generates. 2. Remove unused interfaces as result of refactor.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Utilise new `swagger.modelSchema` element.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Incorporate transformation phase to merge parsed Swagger/JSDoc from controller files to route actions. Adjust initial creation of final Swagger/OpenAPI to improve ordering of elements to show configured content first.

The `meta` attribute allows extensions to Sails Model attributes; add `meta.swagger` to allow Swagger documentation to be added to the OpenAPI.UpdatedSchema definition for the attribute.

Add support for excluding attributes/actions based upon respective `swagger.exclude` attributes.

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Utilise new `swagger.modelSchema` element.

theoomoregbee · 2020-05-28T02:18:57Z

@danielsharvey nice will go over this over the weekend 👍

index.ts

theoomoregbee · 2020-05-30T21:17:09Z

lib/generators.ts

@@ -258,23 +291,27 @@ export const generatePaths = (routes: SwaggerRouteInfo[], templates: BlueprintAc
        summary: subst(template.summary),
        description: subst(template.description),
        externalDocs: template.externalDocs || undefined,
-        tags: get(route, 'swagger.tags', []) as string[],
+        tags: route.swagger?.tags || route.model.swagger.modelSchema?.tags || route.model.swagger.actions?.allactions?.tags || [route.model.globalId],


theoomoregbee · 2020-05-30T21:44:07Z

lib/transformations.ts

+import { Tag } from "swagger-schema-official";
+import { OpenApi } from "../types/openapi";
+import path from "path";
+


👍 for moving this to transformations.ts

theoomoregbee · 2020-05-30T21:53:19Z

test/fixtures/generatedSwagger.json

@@ -1,719 +1,2509 @@
 {


tried the generatedSwagger.json fixture on http://editor.swagger.io/ and got this

because of the shortcut blueprints. line 628

/pet/create: get: summary: Create Pet * description: |- Create a new **Pet** record. (\*) Note that this is a [Sails blueprint shortcut route](https://sailsjs.com/documentation/concepts/blueprints/blueprint-routes#?shortcut-blueprint-routes) (recommended for **development-mode only**) externalDocs: url: 'https://sailsjs.com/documentation/reference/blueprint-api/create' description: 'See https://sailsjs.com/documentation/reference/blueprint-api/create' tags: - Pet parameters: [] responses: '200': description: Responds with a JSON dictionary representing the newly created **Pet** instance content: application/json: schema: $ref: '#/components/schemas/pet' '404': description: Resource not found '500': description: Internal server error requestBody: description: JSON dictionary representing the Pet instance to create. required: true content: application/json: schema: $ref: '#/components/schemas/pet'

the schema should have been under parameters as query param https://sailsjs.com/documentation/concepts/blueprints/blueprint-routes#?shortcut-blueprint-routes

I uploaded to http://editor.swagger.io/ but without shortcuts. Fixing now.

Done; see new commits

just checked, all good 👍 , but noticed the generateSchemaAsQueryParameters didn't work for one of the shortcut blueprint action

Thanks. Looks to be a side-effect of overriding user/create action https://github.com/theoomoregbee/sails-hook-swagger-generator/blob/refactor-parsing/api/controllers/UserController.js#L76

Will look.

theoomoregbee · 2020-05-30T21:58:56Z

test/transformations.test.ts

+  },
+} as unknown as Sails.Sails;
+
+describe('Transformations', () => {


theoomoregbee

Looks really good 👍

🙌 for the regexps

Just a few concerns

related to convert all generated path methods to lowercase #65 Refactor parsing and general refinements to configuration / generated Swagger #67 (comment)
Generated swagger json: Refactor parsing and general refinements to configuration / generated Swagger #67 (comment)

Correct Typescript interfaces to use Record instead of Map - Map implies implementation of the Map interface. Update schema data type - 'file' not in OpenAPI 3. Update Typescript interfaces for Parameter and Header to OpenAPI 3 Remove incorrect/unneeded EmptyObject references.

Correct shortcut blueprints; shortcuts have requestBody data supplied as query parameters as a convenience during DEV. Fix association foreign key parameter name in blueprint documentation; was documented as `fk`, should be `childid`. Adjust integer to be 64-bit; all numbers in JavaScript 64 bits. Update test data.

OpenAPI 3 specifies the ***Any Type*** by the absence of the `type` property in a schema; this may be achieved by setting a model attribute's `meta.swagger.type` value to `null`.

This *should* be done by Sails bind but to be sure.

danielsharvey · 2020-06-01T07:30:25Z

@theoomoregbee Thanks for the review. I've addressed your concerns and a few more minor commits. See what you think.

Uploaded to https://redoc.ly/developer-portal yielding these issues and fixes. Refactor '$ref' checks using new common code. Fix cross-generated-route data pollution by ensuring source swagger data cloned. Add underscore to path variable names, used to differentiate the PK value used for paths from query variables. Specifically, differentiate the PK value used for shortcut blueprint update routes, which allow for PK update using query parameters. Some validators expect unique names across all parameter types. Sort tag name specified as part of Operation. Update test data and test fixtures.

theoomoregbee

We can treat this as follow up if it won't take ur time:
#67 (comment)

danielsharvey · 2020-06-03T05:41:55Z

See changes correcting handling of blueprint actions overridden in controller. I'd put that test case in specifically but not picked up that the result was not correct :(

theoomoregbee · 2020-06-04T22:20:19Z

🎉 This PR is included in version 3.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

danielsharvey added 5 commits May 15, 2020 17:56

feat: Update Security schema elements to support OpenAPI 3

de5ca2d

refactor: Update Typescript interfaces for Sails objects

b53b9a0

Includes minor fix to attribute property mapping (Sail --> Swagger).

danielsharvey mentioned this pull request May 18, 2020

Parameters not showing up despite actions2 having inputs detailed. #66

Closed

danielsharvey added 8 commits May 19, 2020 08:21

Merge branch 'master' into refactor-parsing

4dc64b3

refactor: Update Typescript interfaces for 'actions2' actions

74bc2d3

Includes new proposed model using the `meta.swagger` attribute on actions2 inputs and exits. Includes `outputExample` attribute for actions2 exits.

build: minor updates to package-lock.json on npm install

c09ad72

test: Correct handling of async code in test (move to before())

4c8ff47

The `describe()` does not accept async functions; see https://github.com/mochajs/mocha/issues/.

refactor(transformations ph): Separate parsing Sails models and JSDoc

bed62fc

Part of add additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Add updated integration test fixture (generated Swagger output).

danielsharvey added 6 commits May 20, 2020 19:03

refactor: Move Sails path -> Swagger conversion to transformation phase

73aed83

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing).

test: WIP Add/update test models, controllers and actions

25430e3

Add more test data including model associations, attribute validations, updated `swagger` / `@swagger` sample documentation and options.

danielsharvey added 8 commits May 26, 2020 19:14

refactor: Refactor JSDoc parsing and improve errors/warnings

d1453d0

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Add warning error messages for action documentation for actions which do not exist.

refactor: Distinguish model OpenAPI.UpdatedSchema vs OpenAPI.Operation

c64547e

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Utilise new `swagger.modelSchema` element.

feat(meta.swagger): Add support for model attribute meta.swagger

4a4b3ca

The `meta` attribute allows extensions to Sails Model attributes; add `meta.swagger` to allow Swagger documentation to be added to the OpenAPI.UpdatedSchema definition for the attribute.

feat(meta.swagger): Add support for swagger.exclude

d6a9526

Add support for excluding attributes/actions based upon respective `swagger.exclude` attributes.

refactor: Refactor for support of new swagger.modelSchema element

b99393e

Part of adding additional ‘transformations’ phase to processing: parse —> transform —> generate (better code structure and testing). Utilise new `swagger.modelSchema` element.

danielsharvey mentioned this pull request May 30, 2020

auto generate tags for routes with model or just controllers #59

Closed

theoomoregbee marked this pull request as ready for review May 30, 2020 19:20

theoomoregbee self-requested a review May 30, 2020 19:20

theoomoregbee changed the title ~~WIP Refactor parsing and general refinements to configuration / generated Swagger~~ Refactor parsing and general refinements to configuration / generated Swagger May 30, 2020

theoomoregbee reviewed May 30, 2020

View reviewed changes

index.ts Show resolved Hide resolved

theoomoregbee reviewed May 30, 2020

View reviewed changes

test/transformations.test.ts

},

} as unknown as Sails.Sails;

describe('Transformations', () => {

Copy link

Owner

theoomoregbee May 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

theoomoregbee reviewed May 30, 2020

View reviewed changes

danielsharvey added 8 commits June 1, 2020 08:48

fix: Correct and improve OpenAPI3 Typescript interface def for Parameter

115a293

feat: Add support for 'Any Type' model attributes.

943a04a

OpenAPI 3 specifies the ***Any Type*** by the absence of the `type` property in a schema; this may be achieved by setting a model attribute's `meta.swagger.type` value to `null`.

fix: Correct and improve OpenAPI3 Typescript interface def for schemas

6a34d4f

refactor: Ensure route verb lowercase.

aea5de5

This *should* be done by Sails bind but to be sure.

fix: Correct find blueprint to return array of models

8c69459

feat: Merge model attribute schemas to enable overrides by user

b04a0ed

danielsharvey added 2 commits June 1, 2020 21:02

chore: Bump sails, swagger-jsdoc

e8104d6

theoomoregbee approved these changes Jun 2, 2020

View reviewed changes

fix: Correct handling of blueprint actions overridden in controller

b835d63

theoomoregbee merged commit a466521 into master Jun 4, 2020

theoomoregbee deleted the refactor-parsing branch June 4, 2020 22:18

theoomoregbee added the released label Jun 4, 2020

danielsharvey linked an issue Jun 5, 2020 that may be closed by this pull request

auto generate tags for routes with model or just controllers #59

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor parsing and general refinements to configuration / generated Swagger #67

Refactor parsing and general refinements to configuration / generated Swagger #67

danielsharvey commented May 15, 2020 •

edited

Loading

danielsharvey commented May 20, 2020

danielsharvey commented May 26, 2020

theoomoregbee commented May 28, 2020

theoomoregbee May 30, 2020

theoomoregbee May 30, 2020

theoomoregbee May 30, 2020

theoomoregbee May 30, 2020

danielsharvey May 31, 2020

danielsharvey Jun 1, 2020

theoomoregbee Jun 2, 2020

danielsharvey Jun 2, 2020

theoomoregbee May 30, 2020

theoomoregbee left a comment

danielsharvey commented Jun 1, 2020

theoomoregbee left a comment

danielsharvey commented Jun 3, 2020

theoomoregbee commented Jun 4, 2020

Refactor parsing and general refinements to configuration / generated Swagger #67

Refactor parsing and general refinements to configuration / generated Swagger #67

Conversation

danielsharvey commented May 15, 2020 • edited Loading

danielsharvey commented May 20, 2020

danielsharvey commented May 26, 2020

Documenting Model Schemas

Documenting Controllers

theoomoregbee commented May 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theoomoregbee left a comment

Choose a reason for hiding this comment

danielsharvey commented Jun 1, 2020

theoomoregbee left a comment

Choose a reason for hiding this comment

danielsharvey commented Jun 3, 2020

theoomoregbee commented Jun 4, 2020

danielsharvey commented May 15, 2020 •

edited

Loading