Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lint 404s #10814

Merged
merged 22 commits into from
Jul 25, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/lint-404s.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Lint Docs for 404s

on:
push:
branches: [lint-404s]
pull_request:
branches: [lint-404s]

jobs:
index:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: getsentry/action-setup-volta@c52be2ea13cfdc084edb806e81958c13e445941e # v1.2.0
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
docs:
- 'docs/**'
- 'includes/**'
- 'platform-includes/**'
dev-docs:
- 'develop-docs/**'
- uses: oven-sh/setup-bun@v1
with:
bun-version: latest

- uses: actions/cache@v4
id: cache
with:
path: ${{ github.workspace }}/node_modules
key: node-${{ runner.os }}-${{ hashFiles('**/yarn.lock') }}

- run: yarn install --frozen-lockfile
if: steps.cache.outputs.cache-hit != 'true'

# Remove the changelog directory to avoid a build error due to missing `DATABASE_URL`
# and save some build time.
- run: rm -r app/changelog

- run: yarn build
if: steps.filter.outputs.docs == 'true'

- run: yarn build:developer-docs
if: steps.filter.outputs.dev-docs == 'true'

- name: Start Http Server
run: yarn start &
if: steps.filter.outputs.docs == 'true' || steps.filter.outputs.dev-docs == 'true'

- name: Lint 404s
run: bun ./scripts/lint-404s/main.ts
if: steps.filter.outputs.docs == 'true' || steps.filter.outputs.dev-docs == 'true'

- name: Kill Http Server
run: kill $(lsof -t -i:3000) || true
if: steps.filter.outputs.docs == 'true' || steps.filter.outputs.dev-docs == 'true'
continue-on-error: true
2 changes: 1 addition & 1 deletion docs/account/index.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Account Settings
title: Account Settings.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title: Account Settings.
title: Account Settings

sidebar_order: 400
description: "Learn about Sentry's user settings and auth tokens."
---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ description: "Learn about the experimental features available for Sentry's Apple
Do you want to try some new experimental features? On the latest version of the Apple SDK, you can:

- Enable <PlatformLink to="/tracing/instrumentation/automatic-instrumentation/#time-to-full-display">Time to Full Display (TTFD)</PlatformLink> to gain insight into how long it takes your view controller to launch and load all of its content.
<PlatformSection notSupported={["apple.tvos", "apple.watchos", "apple.visionos"]}>
- Enable <PlatformLink to="/profiling/#enable-launch-profiling">App Launch Profiling</PlatformLink> to get detailed profiles for your app launches.
</PlatformSection>
- If you use Swift concurrency, stitch together stack traces of your async code with the `swiftAsyncStacktraces` option. Note that you can enable this in your Objective-C project, but only async code written in Swift will be stitched together.


<Note>
Experimental features are still a work-in-progress and may have bugs. We recognize the irony.
Experimental features are still a work-in-progress and may have bugs. We
recognize the irony.
</Note>

```swift {tabTitle:Swift}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,72 @@ tracer.startActiveSpan("span name", () => {
```
You can also use any other tracer; all OpenTelemetry spans will be picked up by Sentry automatically.
## Using a Custom Sampler
We recommend using `SentrySampler` as this will ensure the correct subset of traces is sent to Sentry depending on your `tracesSampleRate`, as well as that all other Sentry features like trace propagation work as expected.
If you however need to use your own sampler then make sure to wrap your `SamplingResult` with our `wrapSamplingDecision` method:
```js {filename: custom-sampler.js}
const { wrapSamplingDecision } = require("@sentry/opentelemetry");

// implements Sampler from "@opentelemetry/sdk-trace-node"
class CustomSampler {
shouldSample(
context,
_traceId,
_spanName,
_spanKind,
attributes,
_links
) {
const decision = yourDecisionLogic();

// wrap the result
return wrapSamplingDecision({
decision,
context,
spanAttributes: attributes,
});
}

toString() {
return CustomSampler.name;
}
}

module.exports = CustomSampler;

```
Now use your sampler in your `TraceProvider`:
```js {filename: instrument.js}
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const Sentry = require("@sentry/node");
const {
SentrySpanProcessor,
SentryPropagator,
SentrySampler,
} = require("@sentry/opentelemetry");
const CustomSampler = require("./custom-sampler");

const sentryClient = Sentry.init({
dsn: "__DSN__",
skipOpenTelemetrySetup: true,

// By defining any sample rate,
// tracing intergations will be added by default
tracesSampleRate: 0
});


const provider = new NodeTracerProvider({
sampler: new CustomSampler(sentryClient),
});

// ...rest of your setup

// Validate that the setup is correct
Sentry.validateOpenTelemetrySetup();
```
2 changes: 1 addition & 1 deletion docs/platforms/javascript/guides/aws-lambda/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Depending on your setup, there are different ways to install and use Sentry in y
- [Install the Sentry AWS Lambda Layer](./install/cjs-layer) if your Lambda functions are written in CommonJS (CJS) using `require` syntax.
- [Install the Sentry AWS NPM package](./install/esm-npm) if your Lambda functions are running in EcmaScript Modules (ESM) using `import` syntax.

If you're not sure which installation method to use or want an overview of all available options to use Sentry in your Lambda functions, read the [installation methods overview](/guides/aws-lambda/install).
If you're not sure which installation method to use or want an overview of all available options to use Sentry in your Lambda functions, read the [installation methods overview](./install).

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Learn how to add the Sentry Node Lambda Layer to use Sentry in you
sidebar_order: 1
---

The easiest way to get started with Sentry is to use the Sentry [Lambda Layer](https://docs.aws.amazon.com/Lambda/latest/dg/configuration-layers.html) instead of adding `@sentry/aws-serverless` with `npm` or `yarn` [manually](../cjs-manual).
The easiest way to get started with Sentry is to use the Sentry [Lambda Layer](https://docs.aws.amazon.com/Lambda/latest/dg/configuration-layers.html) instead of adding `@sentry/aws-serverless` with `npm` or `yarn` [manually](../cjs-npm).
If you follow this guide, you don't have to worry about deploying Sentry dependencies alongside your function code.
To actually start the SDK, you can decide between setting up the SDK using environment variables or in your Lambda function code. We recommend using environment variables as it's the easiest way to get started. [Initializing the SDK in code](#alternative-initialize-the-sdk-in-code) instead of setting environment variables gives you more control over the SDK setup if you need it.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ For detailed information about which data can be set, see the [Cache Module deve

## Custom Instrumentation

If you're using anything other than our <PlatformLink to="/guides/laravel/">Laravel SDK</PlatformLink>, you'll need to manually instrument the [Cache Module](https://sentry.io/orgredirect/organizations/:orgslug/performance/caches/) by following the steps below.
If you're using anything other than our [Laravel SDK](/platforms/php/guides/laravel/), you'll need to manually instrument the [Cache Module](https://sentry.io/orgredirect/organizations/:orgslug/performance/caches/) by following the steps below.

### Add Span When Putting Data Into the Cache

Expand Down
2 changes: 1 addition & 1 deletion docs/product/issues/issue-details/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ It's the most important piece of information that the Sentry grouping algorithm

You can set your own <PlatformLink to="/enriching-events/breadcrumbs/">breadcrumbs</PlatformLink> to make them more useful for debugging.

If you’ve enabled [Session Replay](/product/explore/session-replay/), you’ll see a replay preview under Breadcrumbs if there’s one associated with the event you’re viewing. Replays can be associated with both frontend and [backend errors](/product/explore/session-replay/getting-started#replays-for-backend-errors) (as long as distrubted tracing is set up). Clicking on the replay preview will lead you to the [Replay Details](/product/explore/session-replay/replay-details/) page.
If you’ve enabled [Session Replay](/product/explore/session-replay/), you’ll see a replay preview under Breadcrumbs if there’s one associated with the event you’re viewing. Replays can be associated with both frontend and [backend errors](/product/explore/session-replay/getting-started#replays-for-backend-errors) (as long as distrubted tracing is set up). Clicking on the replay preview will lead you to the [Replay Details](/product/explore/session-replay/web/replay-details/) page.

## Tags

Expand Down
2 changes: 1 addition & 1 deletion docs/product/performance/transaction-summary.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ Spans with the same operation and description are grouped together into a single

### Replays

The Replays tab displays a list of replays where the transaction you’re viewing had occurred. Go directly to [Replay Details](/product/explore/session-replay/replay-details/) for any replay and see how a slow transaction impacted the user experience. Note: you must have [Session Replay](/product/explore/session-replay/) enabled to see this tab.
The Replays tab displays a list of replays where the transaction you’re viewing had occurred. Go directly to [Replay Details](/product/explore/session-replay/web/replay-details/) for any replay and see how a slow transaction impacted the user experience. Note: you must have [Session Replay](/product/explore/session-replay/) enabled to see this tab.

## Additional Actions

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@
"@types/ws": "^8.5.10",
"@untitaker/quicktype-core-with-markdown": "^6.0.71",
"autoprefixer": "^10.4.17",
"cli-progress": "^3.12.0",
"concurrently": "^8.2.2",
"dotenv-cli": "^7.4.1",
"eslint": "^8",
Expand Down
2 changes: 2 additions & 0 deletions scripts/lint-404s/ignore-list.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/
/changelog/
155 changes: 155 additions & 0 deletions scripts/lint-404s/main.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
/* eslint-disable no-console */

import {readFileSync} from 'fs';
import path, {dirname} from 'path';
import {fileURLToPath} from 'url';

const baseURL = 'http://localhost:3000/';
type Link = {href: string; innerText: string};

const trimSlashes = (s: string) => s.replace(/(^\/|\/$)/g, '');

// @ts-ignore
const ignoreListFile = path.join(dirname(import.meta.url), './ignore-list.txt');

const showProgress = process.argv.includes('--progress');

// Paths to skip
const ignoreList: string[] = readFileSync(fileURLToPath(ignoreListFile), 'utf8')
.split('\n')
.map(trimSlashes)
.filter(Boolean);

async function fetchWithFollow(url: URL | string): Promise<Response> {
const r = await fetch(url);
if (r.status >= 300 && r.status < 400 && r.headers.has('location')) {
return fetchWithFollow(r.headers.get('location')!);
}
return r;
}

async function main() {
const sitemap = await fetch(`${baseURL}sitemap.xml`).then(r => r.text());

const slugs = [...sitemap.matchAll(/<loc>([^<]*)<\/loc>/g)]
.map(l => l[1])
.map(url => trimSlashes(new URL(url).pathname))
.filter(Boolean);
const allSlugsSet = new Set(slugs);

console.log('Checking 404s on %d pages', slugs.length);

const all404s: {page404s: Link[]; slug: string}[] = [];

// check if the slug equivalent of the href is in the sitemap
const isInSitemap = (href: string) => {
// remove hash
const pathnameSlug = trimSlashes(href.replace(/#.*$/, ''));

// some #hash links result in empty slugs when stripped
return pathnameSlug === '' || allSlugsSet.has(pathnameSlug);
};

function shoudlSkipLink(href: string) {
const isExternal = (href_: string) =>
href_.startsWith('http') || href_.startsWith('mailto:');
const isLocalhost = (href_: string) =>
href_.startsWith('http') && new URL(href_).hostname === 'localhost';
const isIp = (href_: string) => /(\d{1,3}\.){3}\d{1,3}/.test(href_);
const isImage = (href_: string) => /\.(png|jpg|jpeg|gif|svg|webp)$/.test(href_);

return [
isExternal,
(s = '') => ignoreList.includes(trimSlashes(s)),
isImage,
isLocalhost,
isIp,
].some(fn => fn(href));
}

async function is404(link: Link, pageUrl: URL): Promise<boolean> {
if (shoudlSkipLink(link.href)) {
return false;
}

const fullPath = link.href.startsWith('/')
? trimSlashes(link.href)
: // relative path
trimSlashes(new URL(pageUrl.pathname + '/' + link.href, baseURL).pathname);

if (isInSitemap(fullPath)) {
return false;
}
const fullUrl = new URL(fullPath, baseURL);
const resp = await fetchWithFollow(fullUrl);
if (resp.status === 404) {
return true;
}
return false;
}

for (const slug of slugs) {
const pageUrl = new URL(slug, baseURL);
const now = performance.now();
const html = await fetchWithFollow(pageUrl.href).then(r => r.text());

const linkRegex = /<a[^>]*href="([^"]*)"[^>]*>([^<]*)<\/a>/g;
const links = Array.from(html.matchAll(linkRegex)).map(m => {
const [, href, innerText] = m;
return {href, innerText};
});
const page404s = (
await Promise.all(
links.map(async link => {
const is404_ = await is404(link, pageUrl);
return [link, is404_] as [Link, boolean];
})
)
)
.filter(([_, is404_]) => is404_)
.map(([link]) => link);

if (page404s.length) {
all404s.push({slug, page404s});
}

if (showProgress) {
console.log(
page404s.length ? '❌' : '✅',
`in ${(performance.now() - now).toFixed(1).padStart(4, '0')} ms | ${slug}`
);
}
}

if (all404s.length === 0) {
console.log('\n\n🎉 No 404s found');
return false;
}
const numberOf404s = all404s.map(x => x.page404s.length).reduce((a, b) => a + b, 0);
console.log(
'\n❌ Found %d %s across %d %s',
numberOf404s,
numberOf404s === 1 ? '404' : '404s',
all404s.length,
all404s.length === 1 ? 'page' : 'pages'
);
for (const {slug, page404s} of all404s) {
console.log('\n🌐', baseURL + slug);
for (const link of page404s) {
console.log(` - [${link.innerText}](${link.href})`);
}
}

console.log(
'\n👉 Note: the markdown syntax is not necessarily present on the source files, but the links do exist on the final pages'
);
// signal error
return true;
}
const now = performance.now();
main().then(has404s => {
console.log(`\n Done in ${(performance.now() - now).toFixed(1)} ms`);
process.exit(has404s ? 1 : 0);
});

export {};
7 changes: 7 additions & 0 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -4206,6 +4206,13 @@ clean-css@^5.0.0:
dependencies:
source-map "~0.6.0"

cli-progress@^3.12.0:
version "3.12.0"
resolved "https://registry.yarnpkg.com/cli-progress/-/cli-progress-3.12.0.tgz#807ee14b66bcc086258e444ad0f19e7d42577942"
integrity sha512-tRkV3HJ1ASwm19THiiLIXLO7Im7wlTuKnvkYaTkyoAPefqjNg7W7DHKUlGRxy9vxDvbyCYQkQozvptuMkGCg8A==
dependencies:
string-width "^4.2.3"

client-only@0.0.1:
version "0.0.1"
resolved "https://registry.yarnpkg.com/client-only/-/client-only-0.0.1.tgz#38bba5d403c41ab150bff64a95c85013cf73bca1"
Expand Down
Loading