docs: add Jsdom approach blog #2687

souravjain540 · 2024-09-30T16:01:35Z

added blog(approved from engg+marketing)

B4nan

left a few nitpicks about formatting for now.

i haven't read the whole thing, but i see there is not a single mention of the JSDOMCrawler we have in crawlee, which feels weird given it should be published here.

website/blog/2024/09-12-finding-students-accommodations/index.md

website/blog/2024/09-30-jsdom-based-scraping/index.md

barjin

I also skimmed through it and left some (incomplete) grammar suggestions.

I don't know how the new rules work, but you can try asking the marketing team for a Grammarly license. I'm dreading the day when my academic pass expires (and I'll likely continue paying for it). It really pays off :)

website/blog/2024/09-30-jsdom-based-scraping/index.md

Co-authored-by: Martin Adámek <banan23@gmail.com> Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>

souravjain540 · 2024-10-03T05:23:10Z

@B4nan @barjin, thanks for the review. Suggestions applied.

I will take care of US English next time :)

also, according to Alexey, The JSDOM crawler waits till the page is loaded, and there is no option to customize the process, so the approach does not apply to the current version of the crawler. That's why we didn't use it.

However, this article still uses Crawlee and is for experienced developers, so it might be a nice way to reach that audience and introduce them to the crawler.

B4nan · 2024-10-03T08:55:17Z

I don't understand what he means, but either way, this is crawlee blog, we need to mention it, otherwise publishing such article here makes little to no sense if you ask me. Its like talking about a feature you have, but you know you did it wrong, so you don't even mention it. This is bad marketing for me.

Btw this is the first time hearing about some issues about jsdom in crawlee, he should have reached out to us instead so we could fix things. I wouldn't merge this unless we rewrite it to use the crawler, not just mention it.

souravjain540 · 2024-10-03T08:57:21Z

okay, i will talk to him :)

B4nan · 2024-10-03T09:02:46Z

Yeah tell him to reach out to us on slack and we will try to sort things out with priority to unblock this article, I can imagine it won't be much work.

mnmkng · 2024-10-03T09:58:12Z

@B4nan @barjin My process is to not bother with grammar too much, because the blogs should go through Content editing at the end.

@souravjain540 can you confirm that it does and that the guys can skip grammar fixes in future reviews?

EDIT: Oh, it says approved from Marketing in the initial post. How could it be approved by Marketing with so many errors?

website/blog/2024/09-30-jsdom-based-scraping/index.md

Co-authored-by: Ondra Urban <23726914+mnmkng@users.noreply.github.com>

B4nan

left few more comments.

i really think we shold simplify the example and remove the SDK/apify related parts like the input handling or proxy configuration

i also dislike the overuse of object destructing, and arrow functions, but maybe that's just my pet peeve again.

website/blog/2024/09-30-jsdom-based-scraping/index.md

B4nan · 2024-10-04T08:38:41Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+        const items = data.list;
+        const counter = itemsCounter + items.length;
+        const dataItems = items.slice(0, resultsLimit && counter > resultsLimit ? resultsLimit - itemsCounter : undefined);
+        await Actor.pushData(dataItems);


we should use the context helper here instead of the global API. also the use of apify SDK here feels a bit weird, its an article about crawlee, it doesn't mention the apify platform anyhow, yet in the code you can find those pieces. it could be confusing.

@B4nan it was a POC of building an Actor using Crawlee, I can use context helper, but want it to remain as an Actor, will add few lines to explain.

Also, there are not many articles that link Actor building and crawlee. i will add explanation of building Actor and then let's if it makes sense.

Co-authored-by: Martin Adámek <banan23@gmail.com>

souravjain540 · 2024-10-07T08:28:41Z

@B4nan made these changes:

formatted code
changed title
added a note section to mention JSDOM Crawler
added link to explain the SDK in between code.

website/blog/2024/09-30-jsdom-based-scraping/index.md

B4nan · 2024-10-07T08:44:00Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+Let’s see how we are going to create the `getApiUrlWithVerificationToken` function:
+
+```js
+const getApiUrlWithVerificationToken = async (body, url) => {


and this is pretty much the same, the formatting is completely off on many places

B4nan · 2024-10-07T08:45:13Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+Now coming to our main code, we will use CheerioCrawler and will use `prenavigationHooks` to inject the headers that we got from the earlier function into the `requestHandler`.
+
+```js
+const crawler = new CheerioCrawler({


ok, i see you broke all the code blocks actually :D this one is also misformatted.

Grammarly it is 🤦 updating

ouch, formatting code blocks with grammarly, that's bold :D

i can format this myself if you'd struggle with that

was editing english with it :)

done using prettier now.

B4nan

better, but still not there. also please use 4 spaces instead of 2 for indenting

B4nan · 2024-10-07T10:02:21Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+    if (
+      ["static", "scontent"].find((x) => urlToOpen.startsWith(`https://${x}`))
+    ) {
+    }


this is not doing anything

B4nan · 2024-10-07T10:02:35Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+        createSessionFunction: async (sessionPool) => createSessionFunction(sessionPool, proxyConfiguration),
+ },
+    preNavigationHooks: [
+ (crawlingContext) => {
+            const { request, session } = crawlingContext;
+            request.headers = {
+ ...request.headers,
+ ...session.userData?.headers,
+
+ };
+ },
+ ],


this is still misformatted

souravjain540 · 2024-10-08T05:34:16Z

better, but still not there. also please use 4 spaces instead of 2 for indenting

done 👍

B4nan · 2024-10-08T07:23:34Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+Before making calls to this API, we will need few required headers (auth data, so we will first make the call to `https://ads.tiktok.com/business/creativecenter/inspiration/popular/hashtag/pad/en`
+We will start this approach by creating a function that will create the URL for the API call for us and, make the call and get the data.


you are missing a closing ) here (and a dot I guess), and remove the line break (or add a blank line if this was supposed to be two paragraphs)

B4nan · 2024-10-08T07:23:50Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+```js
+export const createStartUrls = (input) => {
+    const {
+        days = "7",


we use single quotes everywhere

B4nan · 2024-10-08T07:25:06Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+
+In the above function, we create the start url for the API call that include various parameters as we talked about earlier. After creating the URL according to the parameters it will call the `creative_radar_api` and fetch all the results.
+
+But it won’t work until we get the headers. So, let’s create a function that will first create a session using sessionPool and proxyConfiguration.


Suggested change

But it won’t work until we get the headers. So, let’s create a function that will first create a session using sessionPool and proxyConfiguration.

But it won’t work until we get the headers. So, let’s create a function that will first create a session using `sessionPool` and `proxyConfiguration`.

souravjain540 · 2024-10-08T07:34:29Z

done @B4nan

B4nan · 2024-10-08T07:41:33Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+To make things more clear, here is how code flow looks:
+![code flow](./img/code-flow.webp)


Suggested change

To make things more clear, here is how code flow looks:

![code flow](./img/code-flow.webp)

To make things more clear, here is how code flow looks:

![code flow](./img/code-flow.webp)

B4nan · 2024-10-08T07:43:00Z

website/blog/2024/09-30-jsdom-based-scraping/index.md

+        const { itemsCounter = 0, resultsLimit = 0 } = userData;
+        if (!json.data) {
+            throw new Error('BLOCKED');
+        }
+        const { data } = json;
+        const items = data.list;
+        const counter = itemsCounter + items.length;
+        const dataItems = items.slice(
+            0,
+            resultsLimit && counter > resultsLimit
+                ? resultsLimit - itemsCounter
+                : undefined,
+        );
+        await context.pushData(dataItems);
+        const {
+            pagination: { page, total },
+        } = data;
+        log.info(


this is still misindented, 8 spaces instead of 4

souravjain540 added 2 commits September 30, 2024 21:29

add blog

2d4ce25

minor changes

b50f064

souravjain540 requested review from B4nan and barjin September 30, 2024 16:01

B4nan reviewed Sep 30, 2024

View reviewed changes

barjin requested changes Oct 1, 2024

View reviewed changes

fnesveda added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 2, 2024

Apply suggestions from code review

0308df0

Co-authored-by: Martin Adámek <banan23@gmail.com> Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>

souravjain540 requested review from B4nan and barjin October 3, 2024 05:23

mnmkng requested changes Oct 3, 2024

View reviewed changes

website/blog/2024/09-30-jsdom-based-scraping/index.md Outdated Show resolved Hide resolved

Update website/blog/2024/09-30-jsdom-based-scraping/index.md

eade782

Co-authored-by: Ondra Urban <23726914+mnmkng@users.noreply.github.com>

B4nan requested changes Oct 4, 2024

View reviewed changes

souravjain540 and others added 3 commits October 7, 2024 10:51

Apply suggestions from code review

667eaf3

Co-authored-by: Martin Adámek <banan23@gmail.com>

changes

93a1120

changes

82310e6

souravjain540 requested a review from B4nan October 7, 2024 08:28

B4nan reviewed Oct 7, 2024

View reviewed changes

website/blog/2024/09-30-jsdom-based-scraping/index.md Show resolved Hide resolved

B4nan reviewed Oct 7, 2024

View reviewed changes

changes

9e51c07

B4nan requested changes Oct 7, 2024

View reviewed changes

correct indent

d75a8c9

B4nan reviewed Oct 8, 2024

View reviewed changes

B4nan requested changes Oct 8, 2024

View reviewed changes

souravjain540 added 2 commits October 8, 2024 13:03

quotes

2a815ea

space

6683a85

B4nan requested changes Oct 8, 2024

View reviewed changes

indent

1120c6a

B4nan approved these changes Oct 8, 2024

View reviewed changes

souravjain540 merged commit 1c5cb67 into master Oct 8, 2024
9 checks passed

souravjain540 deleted the jsdom branch October 8, 2024 08:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add Jsdom approach blog #2687

docs: add Jsdom approach blog #2687

souravjain540 commented Sep 30, 2024

B4nan left a comment •

edited

Loading

barjin left a comment

souravjain540 commented Oct 3, 2024

B4nan commented Oct 3, 2024

souravjain540 commented Oct 3, 2024

B4nan commented Oct 3, 2024

mnmkng commented Oct 3, 2024 •

edited

Loading

B4nan left a comment

B4nan Oct 4, 2024

souravjain540 Oct 7, 2024

souravjain540 Oct 7, 2024

souravjain540 commented Oct 7, 2024

B4nan Oct 7, 2024

B4nan Oct 7, 2024

souravjain540 Oct 7, 2024

B4nan Oct 7, 2024

souravjain540 Oct 7, 2024

souravjain540 Oct 7, 2024

B4nan left a comment

B4nan Oct 7, 2024

B4nan Oct 7, 2024

souravjain540 commented Oct 8, 2024

B4nan Oct 8, 2024 •

edited

Loading

B4nan Oct 8, 2024

B4nan Oct 8, 2024

souravjain540 commented Oct 8, 2024

B4nan Oct 8, 2024

B4nan Oct 8, 2024

souravjain540 Oct 8, 2024

		Before making calls to this API, we will need few required headers (auth data, so we will first make the call to `https://ads.tiktok.com/business/creativecenter/inspiration/popular/hashtag/pad/en`
		We will start this approach by creating a function that will create the URL for the API call for us and, make the call and get the data.


		In the above function, we create the start url for the API call that include various parameters as we talked about earlier. After creating the URL according to the parameters it will call the `creative_radar_api` and fetch all the results.

		But it won’t work until we get the headers. So, let’s create a function that will first create a session using sessionPool and proxyConfiguration.

		To make things more clear, here is how code flow looks:
		![code flow](./img/code-flow.webp)

docs: add Jsdom approach blog #2687

docs: add Jsdom approach blog #2687

Conversation

souravjain540 commented Sep 30, 2024

B4nan left a comment • edited Loading

Choose a reason for hiding this comment

barjin left a comment

Choose a reason for hiding this comment

souravjain540 commented Oct 3, 2024

B4nan commented Oct 3, 2024

souravjain540 commented Oct 3, 2024

B4nan commented Oct 3, 2024

mnmkng commented Oct 3, 2024 • edited Loading

B4nan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

souravjain540 commented Oct 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

B4nan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

souravjain540 commented Oct 8, 2024

B4nan Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

souravjain540 commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

B4nan left a comment •

edited

Loading

mnmkng commented Oct 3, 2024 •

edited

Loading

B4nan Oct 8, 2024 •

edited

Loading