Skip to content

Commit

Permalink
Build docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mnmkng committed Sep 30, 2019
1 parent 14418f6 commit e1e1ab5
Show file tree
Hide file tree
Showing 8 changed files with 20 additions and 37 deletions.
6 changes: 2 additions & 4 deletions docs/api/BasicCrawler.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,7 @@ await crawler.run();
<pre><code>{
request: Request,
autoscaledPool: AutoscaledPool
}
</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
<p> The function must return a promise, which is then awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
Expand Down Expand Up @@ -121,8 +120,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
}
</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
Expand Down
12 changes: 4 additions & 8 deletions docs/api/CheerioCrawler.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,7 @@ await crawler.run();
request: Request,
response: Object // An instance of Node&#39;s http.IncomingMessage object,
autoscaledPool: AutoscaledPool
}
</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
<p> If the function returns a promise, it is awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
Expand Down Expand Up @@ -147,8 +146,7 @@ await crawler.run();
payload, // Provided by RequestList and/or RequestQueue
strictSSL, // Use options.ignoreSslErrors
proxy, // Use options.useApifyProxy or options.proxyUrls
}
</code></pre></td></tr><tr>
}</code></pre></td></tr><tr>
<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
Expand All @@ -157,8 +155,7 @@ await crawler.run();
<p> The function receives the following object as an argument:</p>
<pre><code>{
request: Request
}
</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
in place because there are already earlier references to it. Making a copy and returning it from
this function is therefore not supported, because it would create inconsistencies where
Expand Down Expand Up @@ -219,8 +216,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
}
</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
for the default implementation of this function.</p>
Expand Down
6 changes: 2 additions & 4 deletions docs/api/PuppeteerCrawler.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,7 @@ await crawler.run();
page: Page,
puppeteerPool: PuppeteerPool,
autoscaledPool: AutoscaledPool
}
</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
<code>response</code> is an instance of the <code>Puppeteer</code>
<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
<code>page</code> is an instance of the <code>Puppeteer</code>
Expand Down Expand Up @@ -156,8 +155,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
}
</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
Expand Down
3 changes: 1 addition & 2 deletions docs/api/Request.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,7 @@ to override the default behavior and specify which URLs shall be considered equa
<pre><code> {
Accept: &#39;text/html&#39;,
&#39;Content-Type&#39;: &#39;application/json&#39;
}
</code></pre></td></tr><tr>
}</code></pre></td></tr><tr>
<td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
</tr>
<tr>
Expand Down
6 changes: 2 additions & 4 deletions docs/api/RequestList.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,7 @@ await requestList.reclaimRequest(request2);
{ method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
// Batch import of URLs from a file hosted on the web
{ method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
]
</code></pre></td></tr><tr>
]</code></pre></td></tr><tr>
<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
Expand Down Expand Up @@ -133,8 +132,7 @@ await requestList.reclaimRequest(request2);
&#39;unique-key-1&#39;: true,
&#39;unique-key-4&#39;: true,
},
}
</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
is to use the <code>stateKeyPrefix</code> parameter instead.</p>
</td></tr><tr>
<td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
Expand Down
5 changes: 2 additions & 3 deletions docs/api/puppeteer.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,14 +135,13 @@ objects.
a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits
(POST requests).</p>
<p> <strong>Example:</strong></p>
<pre><code class="lang-javascript"> {
<pre><code class="language-javascript"> {
transformRequestFunction: (request) =&gt; {
request.userData.foo = &#39;bar&#39;;
request.useExtendedUniqueKey = true;
return request;
}
}
</code></pre>
}</code></pre>
</td></tr><tr>
<td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td>
</tr>
Expand Down
7 changes: 3 additions & 4 deletions docs/api/utils.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,13 @@ objects.
<p> For example: by adding <code>keepUrlFragment: true</code> to the <code>request</code> object, URL fragments will not be removed
when <code>uniqueKey</code> is computed.</p>
<p> <strong>Example:</strong></p>
<pre><code class="lang-javascript"> {
<pre><code class="language-javascript"> {
transformRequestFunction: (request) =&gt; {
request.userData.foo = &#39;bar&#39;;
request.keepUrlFragment = true;
return request;
}
}
</code></pre>
}</code></pre>
</td></tr></tbody>
</table>
<a name="utils.requestAsBrowser"></a>
Expand Down Expand Up @@ -197,7 +196,7 @@ request. All `options` not recognized by this function are passed to it, so see
<tr>
<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.
If function returns true request gets aborted. This function is passed to the
(@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request]">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
(@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
</td></tr></tbody>
</table>
<a name="utils.sleep"></a>
Expand Down
12 changes: 4 additions & 8 deletions docs/typedefs/ActorRun.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;origin&quot;: &quot;API&quot;,
&quot;clientIp&quot;: &quot;1.2.3.4&quot;,
&quot;userAgent&quot;: &quot;ApifyClient/0.2.13 (Linux; Node/v8.11.3)&quot;
}
</code></pre></td></tr><tr>
}</code></pre></td></tr><tr>
<td><code>stats</code></td><td><code>Object</code></td>
</tr>
<tr>
Expand All @@ -65,8 +64,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;inputBodyLen&quot;: 22,
&quot;restartCount&quot;: 0,
&quot;workersUsed&quot;: 1,
}
</code></pre><p> Beware that object fields might change in future releases.</p>
}</code></pre><p> Beware that object fields might change in future releases.</p>
</td></tr><tr>
<td><code>options</code></td><td><code>Object</code></td>
</tr>
Expand All @@ -77,8 +75,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;waitSecs&quot;: 0,
&quot;memoryMbytes&quot;: 256,
&quot;diskMbytes&quot;: 512
}
</code></pre></td></tr><tr>
}</code></pre></td></tr><tr>
<td><code>buildId</code></td><td><code>String</code></td>
</tr>
<tr>
Expand Down Expand Up @@ -130,6 +127,5 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;body&quot;: {
&quot;message&quot;: &quot;Hello world!&quot;
}
}
</code></pre></td></tr></tbody>
}</code></pre></td></tr></tbody>
</table>

0 comments on commit e1e1ab5

Please sign in to comment.