菜单 搜索
首页 前端知识 LangChain-21 Text Splitters 内容切分器 支持多种格式 HTML JSON md Code(JS/Py/TS/etc) 进行切分并输出 方便将数据进行结构化后检索

LangChain-21 Text Splitters 内容切分器 支持多种格式 HTML JSON md Code(JS/Py/TS/etc) 进行切分并输出 方便将数据进行结构化后检索

2024-08-25 23:08:45 前端知识 前端哥 833 931 我要收藏

请添加图片描述

背景介绍

LangChain提供了多种类型的Text Splitters，以满足不同的需求：

  • RecursiveCharacterTextSplitter：基于字符将文本划分，从第一个字符开始。如果结果片段太大，则继续划分下一个字符。这种方式提供了定义划分字符和片段大小的灵活性。
  • CharacterTextSplitter：类似于RecursiveCharacterTextSplitter，但能够指定自定义分隔符以实现更具体的划分。默认情况下，它尝试在如“\n\n”、“\n”和空格等字符上进行分割。
  • RecursiveTextSplitter：与前两种类型不同，RecursiveTextSplitter基于单词或令牌而不是字符来划分文本。这种方法提供了更多的语义视角，使其成为内容分析的理想选择。
  • TokenTextSplitter：利用OpenAI的语言模型基于令牌划分文本。这使得分割过程极其精确和具有上下文相关性，成为高级自然语言处理应用中不可或缺的工具。

安装依赖

pip install -qU langchain-text-splitters 
复制

HTML Splitter

编写代码

from langchain_text_splitters import HTMLHeaderTextSplitter
 
html_string = """
<!DOCTYPE html>
<html>
<body>
    <div>
        <h1>Foo</h1>
        <p>Some intro text about Foo.</p>
        <div>
            <h2>Bar main section</h2>
            <p>Some intro text about Bar.</p>
            <h3>Bar subsection 1</h3>
            <p>Some text about the first subtopic of Bar.</p>
            <h3>Bar subsection 2</h3>
            <p>Some text about the second subtopic of Bar.</p>
        </div>
        <div>
            <h2>Baz</h2>
            <p>Some text about Baz</p>
        </div>
        <br>
        <p>Some concluding text about Foo</p>
    </div>
</body>
</html>
"""
 
headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
]
 
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits
复制

运行结果

➜ python3 test21.py
[Document(page_content='Foo'), Document(page_content='Some intro text about Foo.  \nBar main section Bar subsection 1 Bar subsection 2', metadata={'Header 1': 'Foo'}), Document(page_content='Some intro text about Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section'}), Document(page_content='Some text about the first subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 1'}), Document(page_content='Some text about the second subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 2'}), Document(page_content='Baz', metadata={'Header 1': 'Foo'}), Document(page_content='Some text about Baz', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}), Document(page_content='Some concluding text about Foo', metadata={'Header 1': 'Foo'})]
复制

WebHTML Splitter

编写代码

from langchain_text_splitters import RecursiveCharacterTextSplitter
 
url = "https://plato.stanford.edu/entries/goedel/"
 
headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
    ("h4", "Header 4"),
]
 
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
 
# for local file use html_splitter.split_text_from_file(<path_to_file>)
html_header_splits = html_splitter.split_text_from_url(url)
 
chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)
 
# Split
splits = text_splitter.split_documents(html_header_splits)
print(splits[80:85])
复制

Character Splitter

编写代码

from langchain_text_splitters import CharacterTextSplitter
 
 
with open("../../state_of_the_union.txt") as f:
    state_of_the_union = f.read()
 
text_splitter = CharacterTextSplitter(
    separator="\n\n",
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    is_separator_regex=False,
)
 
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
复制

Code Splitter

编写代码

from langchain_text_splitters import (
    Language,
    RecursiveCharacterTextSplitter,
)
 
# Full list of supported languages
[e.value for e in Language]
 
# You can also see the separators used for a given language
RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)
复制

Python Code Splitter

编写代码

PYTHON_CODE = """
def hello_world():
    print("Hello, World!")
 
# Call the function
hello_world()
"""
python_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=50, chunk_overlap=0
)
python_docs = python_splitter.create_documents([PYTHON_CODE])
print(python_docs)
复制

JavaScript Code Splitter

编写代码

JS_CODE = """
function helloWorld() {
  console.log("Hello, World!");
}
 
// Call the function
helloWorld();
"""
 
js_splitter = RecursiveCharacterTextSplitter.from_language(
  language=Language.JS, chunk_size=60, chunk_overlap=0
)
js_docs = js_splitter.create_documents([JS_CODE])
复制

TypeScript Code Splitter

编写代码

TS_CODE = """
function helloWorld(): void {
  console.log("Hello, World!");
}
 
// Call the function
helloWorld();
"""
 
ts_splitter = RecursiveCharacterTextSplitter.from_language(
  language=Language.TS, chunk_size=60, chunk_overlap=0
)
ts_docs = ts_splitter.create_documents([TS_CODE])
print(ts_docs)
复制

Markdown Splitter

编写代码

文本内容：

markdown_text = """
# 🦜️🔗 LangChain
 
⚡ Building applications with LLMs through composability ⚡
 
## Quick Install
 
``bash
 Hopefully this code block isn't split
pip install langchain
``
 
As an open-source project in a rapidly developing field, we are extremely open to contributions.
"""
 
复制

代码解析：

md_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0
)
md_docs = md_splitter.create_documents([markdown_text])
print(md_docs)
复制

Markdown Header Splitter

编写代码

文本内容：

# Foo\n\n ## Bar\n\nHi this is Jim  \nHi this is Joe\n\n ## Baz\n\n Hi this is Molly
复制

代码解析：

from langchain_text_splitters import MarkdownHeaderTextSplitter
 
markdown_document = "# Foo\n\n    ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"
 
headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]
 
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
md_header_splits = markdown_splitter.split_text(markdown_document)
print(md_header_splits)
复制

JSON Splitter

import json
import requests
from langchain_text_splitters import RecursiveJsonSplitter
 
 
json_data = requests.get("https://api.smith.langchain.com/openapi.json").json()
splitter = RecursiveJsonSplitter(max_chunk_size=300)
# Recursively split json data - If you need to access/manipulate the smaller json chunks
json_chunks = splitter.split_json(json_data=json_data)
# The splitter can also output documents
docs = splitter.create_documents(texts=[json_data])
 
# or a list of strings
texts = splitter.split_text(json_data=json_data)
 
print(texts[0])
print(texts[1])
复制
转载请注明出处或者链接地址：https://www.qianduange.cn//article/16849.html
标签
chatgptlangchain
上一篇：HTML基本结构及基本标签
评论
相关文章

前端哥

前端哥

广告位招租：QQ85750956
发布的文章

C#解析JSON的常用库--Newtonsoft.Json

2024-11-30 11:11:42

jsonfield 项目常见问题解决方案

2024-11-30 11:11:42

【SpringMVC】_SpringMVC项目返回HTML与JSON

2024-11-30 11:11:41

BugJson因为json格式问题OOM怎么办

2024-11-30 11:11:41

python 解读JSON文件，一文搞懂！

2024-11-30 11:11:40

Redisson同时使用jackson、fastjson、kryo、protostuff序列化（含效率对比）

2024-11-30 11:11:40

开源项目“Pretty JSON”安装与配置完全指南

2024-11-30 11:11:39

2024年前端最新Nodejs基础之包管理工具npm(二)(2)，微软面试题及答案

2024-11-30 11:11:39

解决全局安装pnpm后无法使用的问题

2024-11-30 11:11:39

安装Nodejs后，npm无法使用

2024-11-30 11:11:38

广告位招租：QQ85750956
大家推荐的文章
1

【Echarts系列】—— 实现电池图、3D立体圆形柱状图

2024-03-03 11:03:011001
2

CSS常用属性（文本属性）

2024-11-04 09:11:111000
3

TypeScript 中的 Number 类型，Number 类型的特性、常见操作和注意事项

2024-09-30 23:09:061000
4

CSS写代码使页面划分为左右两个区域

2024-09-09 00:09:071000
5

vue使用datav echarts

2024-09-06 00:09:381000
6

使用TweenMax.js和CSS3创建冰球运动员动画效果教程

2024-09-04 23:09:411000
7

使用CDN提高jQuery加载速度

2024-08-24 23:08:211000
8

小兔鲜儿网页首页制作 黑马程序员前端基础项目自学笔记

2024-08-19 22:08:161000
9

《Vue》你的弹窗能拖动吗？Vue自定义指令实现可拖动弹窗

2024-08-19 22:08:121000
10

npm的使用

2024-08-18 00:08:131000

rss订阅 百度xml 谷歌xml 搜狗xml soxml 神马搜索xml 网站地图

Copyright © 2018-2022 前端哥 陕ICP备2023000550号-1 前端哥

讲个笑话：程序员心理活动：看别人的代码，这都写得什么玩意儿，垃圾！！看自己一个月前的代码：这TM谁写得代码，垃圾！再一看，这代码怎么这么眼熟？我自己写得？仔细一看，卧槽！还TM的真是我自己写的！！

前端技术交流源码工具分享网站

会员中心 联系我 留言建议 回顶部
复制成功！