Merge of CSC fork into main UDB repository (#584)

* Finally able to generate MC100-32 PDF and it looks good.

* Forgot to switch profile_class method of a profile_release to use the updated $ref syntax (matching certificates).

* Forgot to make dest directory in profiles tasks.rake

* Fixing comments

* Fixes to uniq! (returns nil sometimes, not good).

* Improve tasks.rake message

* Found some extra places to rename cfg_arch to design.

* Looks like I found a bug in the csr_field type_cache. It was barfing if a different design was passed to it which seems bizzare since it goes out of its way to support having a different type for different designs. So, I modified it to work like the type cache in ast and do what I think it wanted to do in the first place.

* Rename routines ending in _extensions to _ext_reqs or _ext_vers to make it clearer the return type since I found bugs in the code related to this.

* RVB23 listed RVA as its class erroneously

* Renamed mandatory_extensions to mandatory_ext_reqs in places I shouldn't have (i.e. in @config). Also true for implemented and prohibited.

* Fixed instructions and CSRs in #291 (CRDs list things like instructions, CSRs, and traps that aren't in the configuration). Still need to fix traps and interrupts.

* Working on getting list of exceptions and interrupts correct for the design.

* Created IDesign that unit-level and Design can both use.

* Fixes due to introduction of Design class and removal of ConfiguredArchitecture inheritence from Architecture.

* One more fix

* Fix to bug found after merging in main.

* Change comment to kick off another regression test on GitHub server (seems to have failed incorrectly).

* task for profile is now profile_release_pdf.

* Renaming certificate to CRD in the backend. Also added proc_ prefix to frontend /arch files. And, also igore *.bak files in Git.

* Remove unneccessary concepts of base from architecture and portfolios (especially rakefiles). Now use cfg "_" instead of rv32 or rv64 since there really is no difference in the generated design.

* Adding missing items to .gitignore

* Created portfolio tasks.rake file and moved common Rake code from profile & CRD tasks.rake into it.

* Created portfolio backend with common tasks.rake code between portfolios. Also created include_erb method in PortfolioDesign that invokes ERB partial templates with a standard set of locals.

* Fix updated path to common templates

* Finished

* Portfolio name changes in preparation for CTP.

* One more typo apparently.

* Update profile release name

* Update UDB README to reflect updated Profile/CRD PDF names.

* Clean up adoc links before starting to mock up CTPs

* Added IDL functions to new portfolio appendix in preparation for linking to IDL. Also found more places to use link_to_* and anchor_for_* helpers.

* Found weird typo in backend_helper.rb causing manual regression test to fail.

* Changed anchors and links to use [[anchor]] and <<link>> after learning more about the nuances in adoc and looking at what the ISA manual does.

* Fixed bug I introduced in AntoraUtils (didn't realize that all IDL functions go to the same func.adoc file).

* Improve comment in AntoraUtils and add links to relevant Antora documentation.

* Changed anchors to use #[foo] syntax. Added preliminary CTP information to mul.yaml.

* Generating PDF for CTP with ISA manuals included in it.

* Improve auto update of changes from submodules.

* Started to finally make progress on actual CTP generated doc.

* Added mock instruction to Xmock extension. For testing CTP.

* Changed CTP rake to always sync to latest of csc-riscv-isa-manual.

* Starting to generate CTP coverage points

* Added support for instructions, extensions, CSRs, and CSR fields to have certification test plan information.

* Only add CTP content to CTP docs (was going in CRDs and Profiles too).

* Switch display of test procedures from table to list.

* Fixed regress bug due to incorrect constructor call to ExtensionVersion.

* Add CTP info to extensions, CSRs, and CSR fields

* Fix array in schema

* Improve formating of CTP and add mandatory ID

* Remove comments to work around commit yaml checker problem.

* Create proper adoc for CTP book part 1. Part 2 & 3 (ISA manuals) are giving an invalid part error.

* New version of isa manual (uses level 2 instead of level 1 headings for colophons and now the isa manuals are proper adoc "parts" in the overall "book").

* Improved display of coverage points and test procedures.

* Added Sail operation to instruction appendix.

* Added links from test procedures to coverage points.

* Fix bug in display of all extensions that define a parameter in a CRD.

* Renaming CertLinks to DocLinks since parameters will also use them.

* Renaming of CTP things

* More link doc

* Added priv modes and related specs to CTP by moving it from CRD into templates they can share.

* Update mock.yaml

Just adding a bogus extra line.

Signed-off-by: james-ball-qualcomm <[email protected]>

* Remove blank line

* Creating AC100 CRD from RVB23 profile.

* Forgot these files somehow.

* Added list of CSRs added by extension. Also other misc formating improvements.

* Fix bug in pages.yml that wasn't copying RVI20 correctly. Also reduced TOC levels to 3 since TOC is 200+ pages for AC100-CRD otherwise. And, try adding index.html for Portfolio artifacts.

* Found bug in RVI20U32 that was requiring version 2.2 of C extension but only 2.0 exists. Also, test:nightly was calling task regress but it has to call test:regress. Not sure how this was working in upstream.

* Fixes for version numbers in Yaml files now found by version check added in Ruby.

* Improve index.html to link to all portfolio-created artifacts and copy index.html in pages action.

* Switch to "git clone" for latest CSC fork of ISA manual from trying to use git submodules unreliably.

* Fix bug in bin/setup using braces around ROOT env instead of parenthesis.

* Fixed inst appendix to use adoc description list for coverpoints and test procedures

* Rename coverage points to normative rules to avoid confusion with system verilog coverage points.

* Make M extension optional in MC100 to make it a better pipe cleaner (so we have at least one optional extension).

* Changed cert steps to just passthru adoc from YAML instead of having defined steps. Also, removed name in test procedures since ID is enough. And added optional test file name.

* Add MockProcessor-CTP to list of pages.

* Update pages.yml

fix typo that wasn't copying mc100 CTP properly

* Switched from PDF page size from A4 to larger A3. Also, improved PDF outline and inst appendix headings.

* Remove all non UTF characters added by recent PR to main branch of UDB.

* Added AC200 CRD

* Moved priv modes from cert class to cert model.

* Fix for issue #479 - Certificates not properly processing optional extensions from profiles and S-mode profiles don't inherit U-mode profiles.

* Incorporate fixes to RVB23U64 typos found by Shehroz. Also fixed cert model schema to allow AC100 to be a parent of AC200 (schema was missing \\$parent_of in a couple of places).

* Issue #245 - Creating summary of RISC-V ISA in .xlsx

* Generating .xlsx file with extension name column.

* Made priv/unpriv ext type mandatory in schema and added where missing. Add all columns up to ratification date.

* Added profile presence to riscv summary

* Generating basic dynamic javascript table for riscv_summary backend with fake data.

* Generating javascript with riscv_summary using tabulator-master.

* Add riscv_summary to csc website

* Add -R to riscv_summary page copy

* Move riscv_summary earlier in pages, smoke, & regress since it is very fast.

* Renamed riscv_summary to isa_explorer

* Renamed isa_explorer html to browser.

* Forgot to rename xlsx dir to spreadsheet

* Add column definition to Tabulator to prepare for better formating and sorting.

* Add hyperlink to UDB extension doc to ISA Explorer Extension Name column.

* Improved column formatting, filtering, sorting

* Freeze extension name column

* Added instrucetion table and table height to speed loading in browser.

* Fixed bug in manual inst template

* Create csc_lib and copied upstream lib over original lib

* Accidentally staged lib.tar

* Making progress on merging in Derek's changes. Now hung up on defined_by? missing in DatabaseObject.

* Commiting changes before I start major surgery on my code.

* Remove isa_explorer temporarily from smoke test (since it won't work yet) and fix type checking bug I introduced into symbol_table.rb (now smoke passes).

* Pass config object to ConfiguredArchitecture constructor. Also couple of minor fixes to get smoke to pass.

* Make Config class an abstract base class and create ConfigFromFile child classes in preparation for Config from Portfolio.

* Added ConfigFromPortfolioGroup to satisfy the Config API with a PortfolioGroup object. Unfortunately there is a circular dependency I can't avoid because of additions made in the cpp_hart branch that require a cfg_arch in DatabaseObject base of the new ExtensionRequirementExpression and ConditionalExtensionVersionList classes. I'll need Derek's help to determine a way forward.

* Figured out how to work around cfg_arch circular dependency. Now able to build RVA20 Profile Release. Fixed other bugs I found due to lack of type checking in non-CSC code.

* Passing smoke test now including with ISA Explorer.

* About to copy main "backends" directory over top of mine to merge in changes.

* Finally passing regression test with my changes merged in manually.

* Update regress.yml

Missing empty line

Signed-off-by: james-ball-qualcomm <[email protected]>

* Fix trailing whitespace

* Fixes requested by Derek's review. More to come.

* Added missing classes for CTP caused by manual merge.

* Forgot to change schema of misa.M CTP stuff.

* Fixed schema in mul.yaml for CTP. Also, take advantage of higher priv profiles inheriting everything in lower-priv profiles.

* Making changes request by Derek's review such as adding tabulator-master to .gitmodules (still needs more work to remove copy of code), added CSR table to ISA explorer (actually a new feature), fixes to CTP generation and RVI20 generation, deploy.sh now prints more informative information about its progress and includes all CSC artifacts and the ISA explorer.

* Oops, had wrong max_base logic for portfolio_group.

* obj.rb rename to database_obj.rb wasn't made in this file. Causes regressions to fail right away.

* Fix for instruction appendix backend regression failure (https://github.com/riscv-software-src/riscv-unified-db/actions/runs/14432823958/job/40469667597?pr=584). Use old name for instruction anchor utility routine. Also removed some puts messages about including ERB files in portfolio-based backends that was making the regression log excessively long.

* Instruction appendix backend using old name for another function (convert_monospace_to_links is new name). Also, added regression test to debug launch configurations.

* For isa_explorer backend, remove tabulator-master Javascript code and instead just have HTML load it.

* Forgot to remove tabulator-master from git submodules.

* Forgot to stage some files.

* Rename XLEN to MXLEN in inst/mock.yaml

* Changes requested by 2nd half of Derek's review.

---------

Signed-off-by: james-ball-qualcomm <[email protected]>

Author: james-ball-qualcomm

.github/workflows/pages.yml

@@ -26,6 +26,96 @@ jobs:
         uses: ./.github/actions/singularity-setup
       - name: Create deploy dir
         run: /bin/bash lib/deploy.sh
+      - name: Create _site/isa_explorer
+        run: mkdir -p _site/isa_explorer
+      - name: Create isa_explorer_browser
+        run: ./do gen:isa_explorer_browser
+      - name: Copy isa_explorer_browser
+        run: cp -R gen/isa_explorer/browser _site/isa_explorer
+      - name: Create isa_explorer_spreadsheet
+        run: ./do gen:isa_explorer_spreadsheet
+      - name: Copy isa_explorer_spreadsheet
+        run: cp -R gen/isa_explorer/spreadsheet _site/isa_explorer
+      - name: Build manual
+        run: ./do gen:html_manual MANUAL_NAME=isa VERSIONS=all
+      - name: Build html documentation for generic_rv64
+        run: ./do gen:html[generic_rv64]
+      - name: Generate YARD docs
+        run: ./do gen:tool_doc
+      - name: Create _site/example_cfg
+        run: mkdir -p _site/example_cfg
+      - name: Create _site/manual
+        run: mkdir -p _site/manual
+      - name: Create _site/pdfs
+        run: mkdir -p _site/pdfs
+      - name: Create _site/htmls
+        run: mkdir -p _site/htmls
+      - name: Copy cfg html
+        run: cp -R gen/cfg_html_doc/generic_rv64/html _site/example_cfg
+      - name: Copy top-level index.html with links to portfolio artifacts
+        run: cp index.html _site
+      - name: Create RVI20 Profile Release PDF Spec
+        run: ./do gen:profile_release_pdf[RVI20]
+      - name: Copy RVI20 Profile Release PDF
+        run: cp gen/profile/pdf/RVI20ProfileRelease.pdf _site/pdfs
+      - name: Create RVA20 Profile Release PDF Spec
+        run: ./do gen:profile_release_pdf[RVA20]
+      - name: Copy RVA20 Profile Release PDF
+        run: cp gen/profile/pdf/RVA20ProfileRelease.pdf _site/pdfs
+      - name: Create RVA22 Profile Release PDF Spec
+        run: ./do gen:profile_release_pdf[RVA22]
+      - name: Copy RVA22 Profile Release PDF
+        run: cp gen/profile/pdf/RVA22ProfileRelease.pdf _site/pdfs
+      - name: Create RVA23 Profile Release PDF Spec
+        run: ./do gen:profile_release_pdf[RVA23]
+      - name: Copy RVA23 Profile Release PDF
+        run: cp gen/profile/pdf/RVA23ProfileRelease.pdf _site/pdfs
+      - name: Create RVB23 Profile Release PDF Spec
+        run: ./do gen:profile_release_pdf[RVB23]
+      - name: Copy RVB23 Profile Release PDF
+        run: cp gen/profile/pdf/RVB23ProfileRelease.pdf _site/pdfs
+      - name: Create AC100-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[AC100]
+      - name: Copy AC100-CRD PDF
+        run: cp gen/proc_crd/pdf/AC100-CRD.pdf _site/pdfs
+      - name: Create AC200-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[AC200]
+      - name: Copy AC200-CRD PDF
+        run: cp gen/proc_crd/pdf/AC200-CRD.pdf _site/pdfs
+      - name: Create MC100-32-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC100-32]
+      - name: Copy MC100-32-CRD PDF
+        run: cp gen/proc_crd/pdf/MC100-32-CRD.pdf _site/pdfs
+      - name: Create MC100-64-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC100-64]
+      - name: Copy MC100-64-CRD PDF
+        run: cp gen/proc_crd/pdf/MC100-64-CRD.pdf _site/pdfs
+      - name: Create MC200-32-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC200-32]
+      - name: Copy MC200-32-CRD PDF
+        run: cp gen/proc_crd/pdf/MC200-32-CRD.pdf _site/pdfs
+      - name: Create MC200-64-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC200-64]
+      - name: Copy MC200-64-CRD PDF
+        run: cp gen/proc_crd/pdf/MC200-64-CRD.pdf _site/pdfs
+      - name: Create MC300-32-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC300-32]
+      - name: Copy MC300-32-CRD PDF
+        run: cp gen/proc_crd/pdf/MC300-32-CRD.pdf _site/pdfs
+      - name: Create MC300-64-CRD PDF Spec
+        run: ./do gen:proc_crd_pdf[MC300-64]
+      - name: Copy MC300-64-CRD PDF
+        run: cp gen/proc_crd/pdf/MC300-64-CRD.pdf _site/pdfs
+      - name: Create MC100-32-CTP PDF Spec
+        run: ./do gen:proc_ctp_pdf[MC100-32]
+      - name: Copy MC100-32-CTP PDF
+        run: cp gen/proc_ctp/pdf/MC100-32-CTP.pdf _site/pdfs
+      - name: Create MockProcessor-CTP PDF Spec
+        run: ./do gen:proc_ctp_pdf[MockProcessor]
+      - name: Copy MockProcessor-CTP PDF
+        run: cp gen/proc_ctp/pdf/MockProcessor-CTP.pdf _site/pdfs
+      - name: Copy manual html
+        run: cp -R gen/manual/isa/top/all/html _site/manual
       - name: Setup Pages
         uses: actions/configure-pages@v5
       - name: Upload artifact

.github/workflows/regress.yml

@@ -86,7 +86,7 @@ jobs:
       - name: singularity setup
         uses: ./.github/actions/singularity-setup
       - name: Generate extension PDF
-        run: ./do gen:cert_model_pdf[MockCertificateModel]
+        run: ./do gen:proc_crd_pdf[MockProcessor]
   regress-gen-profile:
     runs-on: ubuntu-latest
     env:
@@ -97,7 +97,7 @@ jobs:
       - name: singularity setup
         uses: ./.github/actions/singularity-setup
       - name: Generate extension PDF
-        run: ./do gen:profile[MockProfileRelease]
+        run: ./do gen:profile_release_pdf[Mock]
   regress-gen-go:
     runs-on: ubuntu-latest
     env:

.gitignore

@@ -16,4 +16,5 @@ gen
 node_modules
 _site
 images
+*.bak
 *.log

.pre-commit-config.yaml

@@ -46,12 +46,12 @@ repos:
         args: ["--schemafile", "schemas/ext_schema.json"]
       - id: check-jsonschema
         alias: check-jsonschema-cert-model
-        files: ^arch/certificate_model/.*\.(yaml|yml)$
-        args: ["--schemafile", "schemas/cert_model_schema.json"]
+        files: ^arch/proc_cert_model/.*\.(yaml|yml)$
+        args: ["--schemafile", "schemas/proc_cert_model_schema.json"]
       - id: check-jsonschema
         alias: check-jsonschema-cert-class
-        files: ^arch/certificate_class/.*\.(yaml|yml)$
-        args: ["--schemafile", "schemas/cert_class_schema.json"]
+        files: ^arch/proc_cert_class/.*\.(yaml|yml)$
+        args: ["--schemafile", "schemas/proc_cert_class_schema.json"]
       # Commenting because throwing errors and not sure this is complete yet
       # - id: check-jsonschema
       #   alias: check-jsonschema-manual-version

.vscode/launch.json

@@ -3,19 +3,109 @@
   "configurations": [
     {
       "type": "rdbg",
-      "name": "MC100-32",
+      "name": "Smoke test",
       "request": "launch",
       "command": "bundle exec rake",
-      "script": "gen:cert_model_pdf[MC100-32]",
+      "script": "test:smoke",
       "args": [],
       "askParameters": false
     },
     {
       "type": "rdbg",
-      "name": "MC200-32",
+      "name": "Regression test",
       "request": "launch",
       "command": "bundle exec rake",
-      "script": "gen:cert_model_pdf[MC200-32]",
+      "script": "test:regress",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "Manual",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:html_manual MANUAL_NAME=isa VERSIONS=all",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MC100-32-CRD",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_crd_pdf[MC100-32]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MC100-32-CTP",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_ctp_pdf[MC100-32]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MC200-32-CRD",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_crd_pdf[MC200-32]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "AC100-CRD",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_crd_pdf[AC100]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "AC200-CRD",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_crd_pdf[AC200]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MockCRD",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_crd_pdf[MockProcessor]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MockCTP",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:proc_ctp_pdf[MockProcessor]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "portfolios",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "portfolios",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "MockProfile",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:profile_release_pdf[Mock]",
       "args": [],
       "askParameters": false
     },
@@ -24,7 +114,43 @@
       "name": "RVA20",
       "request": "launch",
       "command": "bundle exec rake",
-      "script": "gen:profile[RVA20]",
+      "script": "gen:profile_release_pdf[RVA20]",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "ISA Explorer Browser Extension Generator",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:isa_explorer_browser_ext",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "ISA Explorer Browser Instruction Generator",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:isa_explorer_browser_inst",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "ISA Explorer Browser CSR Generator",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:isa_explorer_browser_csr",
+      "args": [],
+      "askParameters": false
+    },
+    {
+      "type": "rdbg",
+      "name": "ISA Explorer Spreadsheet Generator",
+      "request": "launch",
+      "command": "bundle exec rake",
+      "script": "gen:isa_explorer_spreadsheet",
       "args": [],
       "askParameters": false
     },

Gemfile

@@ -19,6 +19,7 @@ gem "ruby-progressbar", "~> 1.13"
 gem "treetop", "1.6.12"
 gem "ttfunk", "1.7" # needed to avoid having asciidoctor-pdf dependencies pulling in a buggy version of ttunk (1.8)
 gem "webrick"
+gem "write_xlsx"
 gem "yard"
 
 group :development do

Gemfile.lock

@@ -89,6 +89,7 @@ GEM
     logger (1.6.6)
     matrix (0.4.2)
     minitest (5.25.5)
+    nkf (0.2.0)
     nokogiri (1.18.6-aarch64-linux-gnu)
       racc (~> 1.4)
     nokogiri (1.18.6-x86_64-linux-gnu)
@@ -166,6 +167,7 @@ GEM
     ruby-prof (1.7.1)
     ruby-progressbar (1.13.0)
     ruby-rc4 (0.1.5)
+    rubyzip (2.4.1)
     securerandom (0.4.1)
     simpleidn (0.2.3)
     solargraph (0.52.0)
@@ -200,6 +202,9 @@ GEM
     unicode-emoji (4.0.4)
     uri (1.0.3)
     webrick (1.9.1)
+    write_xlsx (1.12.1)
+      nkf
+      rubyzip (>= 1.0.0)
     yard (0.9.37)
     yard-solargraph (0.1.0)
       yard (~> 0.9)
@@ -232,6 +237,7 @@ DEPENDENCIES
   treetop (= 1.6.12)
   ttfunk (= 1.7)
   webrick
+  write_xlsx
   yard
 
 RUBY VERSION