0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-08 02:45:22 +03:00

Files

T

Kevin Huang e4cfaa5680 Introduces SeleniumURLLoader for JavaScript-Dependent Web Page Data Retrieval (#2291 )

### Summary
This PR introduces a `SeleniumURLLoader` which, similar to
`UnstructuredURLLoader`, loads data from URLs. However, it utilizes
`selenium` to fetch page content, enabling it to work with
JavaScript-rendered pages. The `unstructured` library is also employed
for loading the HTML content.

### Testing
```bash
pip install selenium
pip install unstructured
```

```python
from langchain.document_loaders import SeleniumURLLoader

urls = [
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "https://goo.gl/maps/NDSHwePEyaHMFGwh8"
]

loader = SeleniumURLLoader(urls=urls)
data = loader.load()
```

2023-04-02 14:05:00 -07:00

2.9 KiB

Raw Permalink Blame History

URL

This covers how to load HTML documents from a list of URLs into a document format that we can use downstream.

In [1]:

 from langchain.document_loaders import UnstructuredURLLoader

In [2]:

urls = [
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023",
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023"
]

In [3]:

loader = UnstructuredURLLoader(urls=urls)

In [4]:

data = loader.load()

Selenium URL Loader

This covers how to load HTML documents from a list of URLs using the SeleniumURLLoader.

Using selenium allows us to load pages that require JavaScript to render.

Setup

To use the SeleniumURLLoader, you will need to install selenium and unstructured.

In [ ]:

from langchain.document_loaders import SeleniumURLLoader

In [ ]:

urls = [
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "https://goo.gl/maps/NDSHwePEyaHMFGwh8"
]

In [ ]:

loader = SeleniumURLLoader(urls=urls)

In [ ]:

data = loader.load()

2.9 KiB Raw Permalink Blame History

URL

Selenium URL Loader

Setup

2.9 KiB

Raw Permalink Blame History