How to use a custom parser with Python's Beautiful Soup?

Question

I am using Beautiful Soup 4 to parse and modify a set of HTML files. The HTML files are Angular templates, which means their markup is somehow different than the one in regular HTML documents (mixed case attributes, directives, input/output bindings, etc.).

None of the parsers listed in Beautiful Soups documentation (html.parser, lxml, html5lib) totally fit my needs. The closest to it is Python’s built in html.parser, but I had to make a few adjustments to it.

Is it possible to use a custom parser class with Beautiful Soup? If yes, how could this be implemented?

Edit: Below is an example of how Beautiful Soup parses a template. The main issue is that all attributes from the result are lowercase (*ngIf -> *ngif) and the directives (appAutoFocus, appKeepFocusInside, etc.) get an empty string value (e.g. appautofocus="").

from bs4 import BeautifulSoup

test_html = """
<kendo-dialog *ngIf="dialogOpened" appAutoFocus appKeepFocusInside (close)="closeDialog()" width="1000">
   <kendo-dialog-titlebar>A title</kendo-dialog-titlebar>
   <div *ngIf="model.showValidationFailedMessage" class="alert alert-danger alert-dismissible">
      <button type="button" class="close" aria-label="Close" (click)="model.closeValidationAlert()"><span
         aria-hidden="true">&amp;times;</span></button>
      Validation failed
   </div>

   <kendo-tabstrip [keepTabContent]="true">
      <kendo-tabstrip-tab title="Tab title 1" [selected]="true">
         <ng-template kendoTabContent>
            <div>Tab content</div>
         </ng-template>
      </kendo-tabstrip-tab>
   </kendo-tabstrip>
</kendo-dialog>
"""

document = BeautifulSoup(test_html, "html.parser")
print(document.prettify())

Result:

<kendo-dialog (close)="closeDialog()" *ngif="dialogOpened" appautofocus="" appkeepfocusinside="" width="1000">
 <kendo-dialog-titlebar>
  A title
 </kendo-dialog-titlebar>
 <div *ngif="model.showValidationFailedMessage" class="alert alert-danger alert-dismissible">
  <button (click)="model.closeValidationAlert()" aria-label="Close" class="close" type="button">
   <span aria-hidden="true">
    &amp;times;
   </span>
  </button>
  Validation failed
 </div>
 <kendo-tabstrip [keeptabcontent]="true">
  <kendo-tabstrip-tab [selected]="true" title="Tab title 1">
   <ng-template kendotabcontent="">
    <div>
     Tab content
    </div>
   </ng-template>
  </kendo-tabstrip-tab>
 </kendo-tabstrip>
</kendo-dialog>

Answer 1

您可以使用bs4.builder.register_treebuilders_from。内置构建器是bs4.builder包中的模块。例如，bs4.builder._lxml。

这是register_treebuilders_from。

def register_treebuilders_from(module):
    """Copy TreeBuilders from the given module into this module."""
    # I'm fairly sure this is not the best way to do this.
    this_module = sys.modules['bs4.builder']
    for name in module.__all__:
        obj = getattr(module, name)

        if issubclass(obj, TreeBuilder):
            setattr(this_module, name, obj)
            this_module.__all__.append(name)
            # Register the builder while we're at it.
            this_module.builder_registry.register(obj)

包的根模块的尾部。

# Builders are registered in reverse order of priority, so that custom
# builder registrations will take precedence. In general, we want lxml
# to take precedence over html5lib, because it's faster. And we only
# want to use HTMLParser as a last result.
from . import _htmlparser
register_treebuilders_from(_htmlparser)
try:
    from . import _html5lib
    register_treebuilders_from(_html5lib)
except ImportError:
    # They don't have html5lib installed.
    pass
try:
    from . import _lxml
    register_treebuilders_from(_lxml)
except ImportError:
    # They don't have lxml installed.
    pass

因此，您需要创建一个子类为bs4.builder.TreeBuilder的模块，并在模块的__all__中注册它。然后将模块传递给register_treebuilders_from。