Provide option for more relaxed fenced code headers (#2649)

* Provide option for more relaxed fenced code headers

Resolves #2544

* Add more tests

* Add changelog entry and fix spelling

Author: facelessuser

docs/src/markdown/about/changelog.md

@@ -1,7 +1,10 @@
 # Changelog
 
-## Unreleased
+## 10.15.0
 
+-   **NEW**: SuperFences: Add `relaxed_headers` option which can tolerate bad content in the fenced code header. When
+    enabled, code blocks with bad content in the header will likely still convert into code blocks, often respecting
+    the specified language.
 -   **FIX**: Blocks: Fix some corner cases of nested blocks with lists.
 -   **FIX**: Tab and Tabbed: Fix a case where tabs could fail if `combine_header_slug` was enabled and there was no
     header.

docs/src/markdown/extensions/superfences.md

@@ -611,6 +611,54 @@ import foo
 ```
 ///
 
+## Relaxed Headers
+
+/// new | New in 10.15
+///
+
+The `relaxed_headers` [option][#options] is a feature that can be enabled which will cause SuperFences to treat fenced
+code headers in a more relaxed manner.
+
+The default approach of SuperFences is to bail on processing a fenced code if the there is invalid content in a fenced
+header. This may be because you are processing fenced code blocks from other, incompatible Markdown parsers. Having a
+way to treat incompatible code blocks as code blocks, even if they have invalid syntax may be desirable.
+
+As an example, some Markdown parsers may allow for specifying line highlights with `[number]`. This is not supported in
+SuperFences, but if we were processing such a block, with `relaxed_headers` enabled, we would treat it as an actual
+code block ignoring the invalid syntax instead of assuming the entire code block was invalid.
+
+````text title="Bad Code Header Syntax"
+```python [3]
+import code
+
+code.do_stuff()
+```
+````
+
+/// html | div.result
+````md-render
+---
+extensions:
+- pymdownx.highlight
+- pymdownx.superfences
+
+extension_configs:
+  pymdownx.superfences:
+    relaxed_headers: true
+---
+```python [3]
+import code
+
+code.do_stuff()
+```
+````
+///
+
+It should be noted that this mode does not always guarantee that your code language will be picked up or that relevant
+classes, IDs, attributes, or options will be recognized if invalid content is present, only that it will try and process
+the block as a code block. Additionally, a custom block that explicitly throws a `SuperFencesException` will still have
+that exception raised which will terminate Markdown processing.
+
 ## Custom Fences
 
 SuperFences allows defining custom fences for special purposes. For instance, we could create special fences for
@@ -911,3 +959,4 @@ Option                         | Type         | Default       | Description
 `disable_indented_code_blocks` | bool         | `#!py3 False` | Disables Python Markdown's indented code block parsing.  This is nice if you only ever use fenced blocks.
 `custom_fences`                | [dictionary] | `#!py3 []`    | Custom fences.
 `preserve_tabs`                | bool         | `#!py3 False` | Experimental feature that preserves tabs in fenced code blocks.
+`relaxed_headers`              | bool         | `#!py3 False` | Enable relaxed fenced code headers that are more forgiving of invalid content in the fenced header.

docs/src/mkdocs.yml

@@ -108,6 +108,7 @@ markdown_extensions:
   - markdown.extensions.md_in_html:
   - pymdownx.superfences:
       preserve_tabs: true
+      relaxed_headers: true
       custom_fences:
         # Mermaid diagrams
         - name: diagram

mkdocs.yml

@@ -108,6 +108,7 @@ markdown_extensions:
   - markdown.extensions.md_in_html:
   - pymdownx.superfences:
       preserve_tabs: true
+      relaxed_headers: true
       custom_fences:
         # Mermaid diagrams
         - name: diagram

pymdownx/superfences.py

@@ -27,7 +27,6 @@
 
 License: [BSD](http://www.opensource.org/licenses/bsd-license.php)
 """
-
 from markdown.extensions import Extension
 from markdown.preprocessors import Preprocessor
 from markdown.blockprocessors import CodeBlockProcessor
@@ -51,6 +50,9 @@
             (?:
                 (?:[ \t]*[a-zA-Z][a-zA-Z0-9_]*(?:=(?P<quot>"|').*?(?P=quot))?)(?=[\t ]|$)  # Options
             )+
+        ) |
+        (?P<unrecognized>
+            (?:([ \t]*[^\s]+)(?=[\t ]|$))+
         )
     )?[ \t]*$
     '''
@@ -231,7 +233,8 @@ def __init__(self, *args, **kwargs):
                 "if nothing is set. - "
                 "Default: ''"
             ],
-            'preserve_tabs': [False, "Preserve tabs in fences - Default: False"]
+            'preserve_tabs': [False, "Preserve tabs in fences - Default: False"],
+            'relaxed_headers': [False, "Relaxed fenced code headers - Default: False"]
         }
         super().__init__(*args, **kwargs)
 
@@ -376,6 +379,7 @@ def get_hl_settings(self):
             css_class = self.config['css_class']
             self.css_class = css_class if css_class else config['css_class']
 
+            self.relaxed_headers = self.config.get('relaxed_headers', False)
             self.extend_pygments_lang = config.get('extend_pygments_lang', None)
             self.guess_lang = config['guess_lang']
             self.pygments_style = config['pygments_style']
@@ -594,9 +598,9 @@ def parse_options(self, m):
         self.formatter = None
         values = {}
         if string:
-            for m in RE_OPTIONS.finditer(string):
-                key = m.group('key')
-                value = m.group('value')
+            for m2 in RE_OPTIONS.finditer(string):
+                key = m2.group('key')
+                value = m2.group('value')
                 if value is None:
                     value = key
                 values[key] = value
@@ -620,8 +624,48 @@ def parse_options(self, m):
                     self.options = options
                     break
 
+        if not okay and self.relaxed_headers:
+            return self.handle_unrecognized(m)
+
         return okay
 
+    def handle_unrecognized(self, m):
+        """Handle unrecognized code headers."""
+
+        okay = False
+        if not self.relaxed_headers:
+            return okay
+
+        if m.group('lang'):
+            self.lang = m.group('lang')
+
+        self.options = {}
+        self.attrs = {}
+        self.formatter = None
+
+        # Run per language validator
+        for entry in reversed(self.extension.superfences):
+            if entry["test"](self.lang):
+                options = {}
+                attrs = {}
+                validator = entry.get("validator", functools.partial(_validator, validator=default_validator))
+                try:
+                    okay = validator(self.lang, {}, options, attrs, self.md)
+                except SuperFencesException:
+                    raise
+                except Exception:
+                    pass
+                if okay:
+                    self.formatter = entry.get("formatter")
+                    self.options = options
+                    if self.attr_list:
+                        self.attrs = attrs
+                    break
+
+        if not okay:
+            self.lang = None  # pragma: no cover
+        return True
+
     def handle_attrs(self, m):
         """Handle attribute list."""
 
@@ -664,6 +708,9 @@ def handle_attrs(self, m):
                         self.attrs = attrs
                     break
 
+        if not okay and self.relaxed_headers:
+            return self.handle_unrecognized(m)  # pragma: no cover
+
         return okay
 
     def search_nested(self, lines):
@@ -683,7 +730,9 @@ def search_nested(self, lines):
                 if m is not None:
 
                     # Parse options
-                    if m.group('attrs'):
+                    if m.group('unrecognized'):
+                        okay = self.handle_unrecognized(m)
+                    elif m.group('attrs'):
                         okay = self.handle_attrs(m)
                     else:
                         okay = self.parse_options(m)

tests/test_extensions/test_superfences.py

@@ -1634,3 +1634,154 @@ def test_broken_brace(self):
             ''',
             True
         )
+
+
+class TestHighlightRelaxedHeaders(util.MdCase):
+    """Test relaxed header cases."""
+
+    extension = ['pymdownx.highlight', 'pymdownx.superfences', 'attr_list']
+    extension_configs = {
+        'pymdownx.superfences': {
+            'relaxed_headers': True,
+            'custom_fences': [
+                {
+                    'name': 'test',
+                    'class': 'test',
+                    'format': custom_format,
+                    'validator': custom_validator_except
+                },
+                {
+                    'name': 'test2',
+                    'class': 'test',
+                    'format': custom_format,
+                    'validator': custom_validator_exploder
+                }
+            ]
+        }
+    }
+
+    def test_custom_fail_exception(self):
+        """Test custom fences forced exception."""
+
+        with self.assertRaises(SuperFencesException):
+            self.check_markdown(
+                r'''
+                ```test [1]
+                test
+                ```
+                ''',
+                '',
+                True
+            )
+
+    def test_custom_fail_exception_relaxed(self):
+        """Test custom fences relaxed forced exception."""
+
+        self.check_markdown(
+            r'''
+            ```test2 [1]
+            test
+            ```
+            ''',
+            '''
+            <div class="highlight"><pre><span></span><code>test
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_bad_lang(self):
+        """Test bad language."""
+
+        self.check_markdown(
+            r'''
+            ```bad
+            test
+            ```
+            ''',
+            '''
+            <div class="highlight"><pre><span></span><code>test
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_bad_option(self):
+        """Test bad options."""
+
+        self.check_markdown(
+            r'''
+            ```pycon bad="My title"
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_attribute(self):
+        """Test bad attribute."""
+
+        self.check_markdown(
+            r'''
+            ```pycon {.class <nonsense>!!!}
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="class highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_attribute_full(self):
+        """Test bad attribute full."""
+
+        self.check_markdown(
+            r'''
+            ```{.pycon .class <nonsense>!!!}
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="class highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_header(self):
+        """Test bad header."""
+
+        self.check_markdown(
+            r'''
+            ```pycon [nonsense]
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_nonsense(self):
+        """Test only nonsense."""
+
+        self.check_markdown(
+            r'''
+            ```[nonsense]
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code>&gt;&gt;&gt; import test
+            </code></pre></div>
+            ''',
+            True
+        )