English | 简体中文

Changelog

• 2024/11/25: Project Initialization

Table of Contents

1. llm-web-kit
   • Project Introduction
   • Key Features
   • Quick Start
     • Online Demo
     • Quick CPU Demo
     • Using GPU
   • Usage
     • Command Line
     • API
     • Deploy Derived Projects
     • Development Guide
2. TODO
3. Known Issues
4. FAQ
5. All Thanks To Our Contributors
6. License Information
7. Acknowledgments
8. Citation
9. Star History
10. Links

llm-web-kit

Project Introduction

llm-web-kit is a python library that ..

Key Features

• Remove headers, footers, footnotes, page numbers, etc., to ensure semantic coherence.
• Output text in human-readable order, suitable for single-column, multi-column, and complex layouts.

Quick Start

extract by magic_html+recognize

from llm_web_kit.simple import extract_html_to_md, extract_html_to_mm_md
import traceback
from loguru import logger

def extract(url:str, html:str) -> str:
    try:
        nlp_md = extract_html_to_md(url, html)
        # or mm_nlp_md = extract_html_to_mm_md(url, html)
        return nlp_md
    except Exception as e:
        logger.exception(e)
    return None

if __name__=="__main__":
    url = ""
    html = ""
    markdown = extract(url, html)

only extract by recognize

from llm_web_kit.simple import extract_html_to_md, extract_html_to_mm_md
import traceback
from loguru import logger

def extract(url:str, raw_html:str) -> str:
    try:
        nlp_md = extract_html_to_md(url, raw_html, clip_html=False)
        # or mm_nlp_md = extract_html_to_mm_md(url, raw_html, clip_html=False)
        return nlp_md
    except Exception as e:
        logger.exception(e)
    return None

if __name__=="__main__":
    url = ""
    html = ""
    markdown = extract(url, html)

only extract main_html by magic-html

from llm_web_kit.simple import extract_main_html_by_maigic_html
import traceback
from loguru import logger

def extract(url:str, html:str) -> str:
    try:
        main_html = extract_main_html_by_maigic_html(url, html)
        # or mm_main_html = extract_pure_html_to_mm_md(url, html)
        return main_html
    except Exception as e:
        logger.exception(e)
    return None

if __name__=="__main__":
    url = ""
    html = ""
    main_html = extract(url, html)

extract main_html by model response

import traceback
from loguru import logger
from llm_web_kit.main_html_parser.simplify_html.simplify_html import simplify_html
from llm_web_kit.input.pre_data_json import PreDataJson, PreDataJsonKey
from llm_web_kit.main_html_parser.parser.tag_mapping import MapItemToHtmlTagsParser

def extract(response_json: dict, html:str) -> str:
    try:
        _, typical_raw_tag_html, _ = simplify_html(html)
        pre_data = PreDataJson({})
        pre_data[PreDataJsonKey.TYPICAL_RAW_TAG_HTML] = typical_raw_tag_html
        pre_data[PreDataJsonKey.TYPICAL_RAW_HTML] = html
        pre_data['success_label_enable'] = True
        pre_data[PreDataJsonKey.LLM_RESPONSE] = response_json
        parser = MapItemToHtmlTagsParser({})
        pre_data = parser.parse_single(pre_data)
        main_html = pre_data[PreDataJsonKey.TYPICAL_MAIN_HTML]
        is_success = pre_data[PreDataJsonKey.TYPICAL_MAIN_HTML_SUCCESS]
        return main_html, is_success
    except Exception as e:
        logger.exception(e)
    return None

if __name__=="__main__":
    response_json =  {'item_id 1': 0, 'item_id 2': 1, 'item_id 3': 1}
    html = ""
    main_html, is_success = extract(response_json, html)

Pipeline

1. HTML pre-dedup
2. domain clustering
3. layout clustering
4. typical layout node selection
5. HTML node select by LLM
6. html parse layout by layout

Usage

TODO

Known Issues

FAQ

contributors

License Information

Acknowledgments

Citation

Star History

links