Skip to content

Commit 89322fd

Browse files
nberlettenberlette
committedJul 12, 2022
📝 update readme.md
·
v0.0.6
1 parent 9d34b2c commit 89322fd

File tree

1 file changed

+105
-90
lines changed

1 file changed

+105
-90
lines changed
 

‎README.md

+105-90
Original file line numberDiff line numberDiff line change
@@ -1,155 +1,170 @@
11
<div align="center">
22

3-
# [DQL](https://deno.land/x/dql)
3+
# [🦕 DQL](https://deno.land/x/dql)
44

55
### _**Web Scraping with Deno  –  DOM + GraphQL**_
66

77
</div>
88

99
---
1010

11-
**`DQL`** lets you use GraphQL queries to extract data from the DOM of a web page or HTML fragment (for sandboxing or use cases without network access). It accepts [**GraphQL Queries**](https://graphql.org/learn/queries) as input, and returns formatted JSON data as output.
12-
13-
> - [**Try out a real-world example of `useQuery` in the `Deno Playground`**](https://dash.deno.com/playground/dql)
14-
> - [**View the example's JSON endpoint at `dql.deno.dev`**](https://dql.deno.dev)
15-
16-
## Summary
17-
18-
This is a fork of [**DenoQL**](https://deno.land/x/denoql) with some heavy refactoring and some additional features:
11+
**`DQL`** is a web scraping module for Deno and Deno Deploy that integrates the power of [**GraphQL Queries**](https://graphql.org/learn/queries) with the DOM tree of a remote webpage or HTML document fragment. This is a fork of [**DenoQL**](https://deno.land/x/denoql) with some heavy refactoring and some additional features:
1912

2013
- [x] Compatibility with the [**Deno Deploy**](https://deno.com/deploy) architecture
2114
- [x] Ability to pass variables alongside all queries
2215
- [x] New state-management class with additional methods
2316
- [x] Modular project structure (as opposed to a mostly single-file design)
2417
- [x] Improved types and schema structure
25-
- [ ] **This is a work-in-progress and there is still much to be done.** *
2618

27-
## Usage
19+
> **Note**: _This is a work-in-progress and there is still a lot to be done._
2820
29-
The primary function exported by the module is the workhorse named `useQuery`:
21+
### 🛝  [**`GraphQL Playground`**](https://dql.deno.dev)
3022

31-
```ts
32-
import { useQuery } from "https://deno.land/x/dql/mod.ts";
23+
### 📝  [**`HackerNews Scraper`**](https://dash.deno.com/playground/dql-hn)
3324

34-
const data = await useQuery(`query { ... }`);
35-
```
25+
### 🚛  [**`Junkyard Scraper`**](https://dash.deno.com/playground/dirty-sparrow-69)
3626

37-
### Query Options
27+
---
3828

39-
You can also provide an options object for the second argument of `useQuery`:
29+
## `useQuery`
30+
31+
The primary function exported by the module is the workhorse named `useQuery`:
4032

4133
```ts
42-
const data = await useQuery(`query { ... }`, {
43-
concurrency: 8,
44-
fetch_options: {
45-
// passed as the second param to fetch()
46-
},
47-
variables: {
48-
// any variables used in your queries go here
49-
},
50-
});
34+
import { useQuery } from "https://deno.land/x/dql/mod.ts";
35+
36+
const data = await useQuery(`query { ... }`);
5137
```
5238

53-
### Authenticated Requests
39+
### `QueryOptions`
5440

55-
To authenticate your requests, you can add an `Authorization` header like so:
41+
You can also provide a `QueryOptions` object as the second argument of `useQuery`, to further control the behavior of your query requests. All properties are optional.
5642

5743
```ts
5844
const data = await useQuery(`query { ... }`, {
59-
fetch_options: {
45+
concurrency: 8, // passed directly to PQueue initializer
46+
fetch_options: { // passed directly to Fetch API requests
6047
headers: {
6148
"Authorization": "Bearer ghp_a5025a80a24defd0a7d06b4fc215bb5635a167c6",
6249
},
6350
},
51+
variables: {}, // variables defined in your queries
52+
operationName: "", // when using multiple queries
6453
});
6554
```
6655

67-
## GraphQL Playground
68-
69-
### Deno Deploy
56+
## `createServer`
7057

71-
With [**Deno Deploy**](https://dash.deno.com/new), you can deploy **`DQL`** with a GraphQL Playground in **only 2 LOC**:
58+
With [**Deno Deploy**](https://dash.deno.com/new), you can deploy **`DQL`** with a GraphQL Playground in **only 2 lines of code**:
7259

7360
```ts
7461
import { createServer } from "https://deno.land/x/dql/mod.ts";
7562

76-
// change the endpoint to your unique URL ([...].deno.dev)
77-
createServer(80, { endpoint: "https://dirty-sparrow-69.deno.dev" });
63+
createServer(80, { endpoint: "https://dql.deno.dev" });
7864
```
7965

80-
> - [**Try it out at `dirty-sparrow-69.deno.dev`**](https://dirty-sparrow-69.deno.dev)
81-
> - [**View the public code in the `Deno Playground`**](https://dash.deno.com/playground/dirty-sparrow-69)
66+
`🛝` [Try the **GraphQL Playground** at **`dql.deno.dev`**](https://dql.deno.dev)\
67+
`🦕` [View the source code in the **`Deno Playground`**](https://dash.deno.com/playground/dql)
8268

83-
### Command Line Usage (CLI)
69+
## Command Line Usage (CLI)
8470

8571
```bash
86-
# spin up a playground on port 8080
8772
deno run -A --unstable https://deno.land/x/dql/serve.ts
8873
```
8974

75+
#### Custom port (default is `8080`)
76+
9077
```bash
91-
# ... or using a custom port
92-
deno run -A --unstable https://deno.land/x/dql/serve.ts --port 3000
78+
deno run -A https://deno.land/x/dql/serve.ts --port 3000
9379
```
9480

95-
> **Note**: you need to have the [**Deno CLI**](https://deno.land) installed for CLI usage.
81+
> **Warning**: you need to have the [**Deno CLI**](https://deno.land) installed first.
9682
97-
### Programmatic Usage
83+
---
9884

99-
```ts
100-
import { createServer } from "https://deno.land/x/dql/mod.ts";
85+
## 💻 Examples
10186

102-
// start a playground on port 8080
103-
createServer();
87+
### `🚛` Junkyard Scraper · [**`Deno Playground 🦕`**](https://dash.deno.com/playground/dirty-sparrow-69)
10488

105-
// or using a custom port
106-
createServer(3000);
89+
```ts
90+
import { useQuery } from "https://deno.land/x/dql/mod.ts";
91+
import { serve } from "https://deno.land/std@0.147.0/http/server.ts";
92+
93+
serve(async (res: Request) =>
94+
await useQuery(
95+
`
96+
query Junkyard (
97+
$url: String
98+
$itemSelector: String = "table > tbody > tr"
99+
) {
100+
vehicles: page(url: $url) {
101+
totalCount: count(selector: $itemSelector)
102+
nodes: queryAll(selector: $itemSelector) {
103+
id: index
104+
vin: text(selector: "td:nth-child(7)", trim: true)
105+
sku: text(selector: "td:nth-child(6)", trim: true)
106+
year: text(selector: "td:nth-child(1)", trim: true)
107+
model: text(selector: "td:nth-child(2) > .notranslate", trim: true)
108+
aisle: text(selector: "td:nth-child(3)", trim: true)
109+
store: text(selector: "td:nth-child(4)", trim: true)
110+
color: text(selector: "td:nth-child(5)", trim: true)
111+
date: attr(selector: "td:nth-child(8)", name: "data-value")
112+
image: src(selector: "td > a > img")
113+
}
114+
}
115+
}`,
116+
{
117+
variables: {
118+
"url": "http://nvpap.deno.dev/action=getVehicles&makes=BMW",
119+
},
120+
},
121+
)
122+
.then((data) => JSON.stringify(data, null, 2))
123+
.then((json) =>
124+
new Response(json, {
125+
headers: { "content-type": "application/json;charset=utf-8" },
126+
})
127+
)
128+
);
107129
```
108130

109-
## Examples
110-
111-
### Junkyard Inventory Scraper
112-
113-
> - [**Try it for yourself in the `Deno Playground`**](https://dash.deno.com/playground/dql)
114-
> - [**View the JSON endpoint at `dql.deno.dev`**](https://dql.deno.dev)
131+
### 📝 HackerNews Scraper · [**`Deno Playground 🦕`**](https://dash.deno.com/playground/dql-hn)
115132

116133
```ts
117134
import { useQuery } from "https://deno.land/x/dql/mod.ts";
118-
119-
const query = `query Junkyard ($url: String, $itemSelector: String) {
120-
vehicles: page(url: $url) {
121-
totalCount: count(selector: $itemSelector)
122-
items: queryAll(selector: $itemSelector) {
123-
id: index
124-
vin: text(selector: "td:nth-child(7)", trim: true)
125-
sku: text(selector: "td:nth-child(6)", trim: true)
126-
year: text(selector: "td:nth-child(1)", trim: true)
127-
model: text(selector: "td:nth-child(2) > .notranslate", trim: true)
128-
aisle: text(selector: "td:nth-child(3)", trim: true)
129-
store: text(selector: "td:nth-child(4)", trim: true)
130-
color: text(selector: "td:nth-child(5)", trim: true)
131-
date: attr(selector: "td:nth-child(8)", name: "data-value")
132-
image: src(selector: "td > a > img")
135+
import { serve } from "https://deno.land/std@0.147.0/http/server.ts";
136+
137+
serve(async (res: Request) =>
138+
await useQuery(`
139+
query HackerNews (
140+
$url: String = "http://news.ycombinator.com"
141+
$rowSelector: String = "tr.athing"
142+
) {
143+
page(url: $url) {
144+
title
145+
totalCount: count(selector: $rowSelector)
146+
nodes: queryAll(selector: $rowSelector) {
147+
rank: text(selector: "td span.rank", trim: true)
148+
title: text(selector: "td.title a", trim: true)
149+
site: text(selector: "span.sitestr", trim: true)
150+
url: href(selector: "td.title a")
151+
attrs: next {
152+
score: text(selector: "span.score", trim: true)
153+
user: text(selector: "a.hnuser", trim: true)
154+
date: attr(selector: "span.age", name: "title")
155+
}
156+
}
133157
}
134-
}
135-
}`;
136-
137-
// pass any variables using the 'variables' key
138-
const response = await useQuery(query, {
139-
variables: {
140-
"url": "http://nvpap.deno.dev/action=getVehicles&makes=BMW",
141-
"itemSelector": "table > tbody > tr",
142-
},
143-
});
144-
145-
// do something with response (Object)
146-
console.log(response);
158+
}`)
159+
.then((data) => JSON.stringify(data, null, 2))
160+
.then((json) =>
161+
new Response(json, {
162+
headers: { "content-type": "application/json;charset=utf-8" },
163+
})
164+
)
165+
);
147166
```
148167

149-
---
168+
## License
150169

151-
<div align="center">
152-
153-
MIT © [Nicholas Berlette](https://github.com/nberlette) • based on [DenoQL](https://deno.land/x/denoql) by [nyancodeid](https://github.com/nyancodeid)
154-
155-
</div>
170+
MIT © [**Nicholas Berlette**](https://github.com/nberlette), based on [DenoQL](https://deno.land/x/denoql).

0 commit comments

Comments
 (0)
Please sign in to comment.