Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update to ipfs-geoip v9 #2061

Merged
merged 15 commits into from
Nov 9, 2022
Merged

feat: update to ipfs-geoip v9 #2061

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e0f482f
Updating to geoip@9
whizzzkid Nov 2, 2022
ff8719a
Merge branch 'main' into feature/ipfs-geoip-v9
whizzzkid Nov 2, 2022
2c650fb
Merge branch 'main' into feature/ipfs-geoip-v9
whizzzkid Nov 8, 2022
a2a36c0
upgrading to lates
whizzzkid Nov 8, 2022
056195f
Only upgrading geoip
whizzzkid Nov 8, 2022
a40d8d7
Undo package install
whizzzkid Nov 8, 2022
dfba157
Upgrading geoip
whizzzkid Nov 9, 2022
780be59
Importing only lookup function
whizzzkid Nov 9, 2022
ef1f699
Skipping babel transpilation for Pure ESM
whizzzkid Nov 9, 2022
98312a2
Fixing unit tests
whizzzkid Nov 9, 2022
39e704e
comments
whizzzkid Nov 9, 2022
109ccfd
syncing package-files
whizzzkid Nov 9, 2022
ab9bb6f
Fixing tests
whizzzkid Nov 9, 2022
d527194
Refactoring logic
whizzzkid Nov 9, 2022
7d24f15
Removing unreachable code.
whizzzkid Nov 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 54 additions & 29 deletions config-overrides.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,46 +5,69 @@
* @see https://alchemy.com/blog/how-to-polyfill-node-core-modules-in-webpack-5
*/
const webpack = require('webpack')
const PURE_ESM_MODULES = [
'ipfs-geoip'
]

/**
* This function goes through the loader rules and applies modifier function to the said rule.
* Validation can be set within the modifier function.
*
* @param {import('webpack').RuleSetRule[]} rules
* @param {function} modifier defaults to identity function
* @returns {import('webpack').RuleSetRule[]}
*/
function modifyBabelLoaderRule (rules, root = true) {
const foundRules = []
rules.forEach((rule, i) => {
if (rule.loader != null) {
if (rule.loader.includes('babel-loader')) {
foundRules.push(rule)
}
} else if (rule.use?.loader != null) {
if (typeof rule.use.loader !== 'string') {
if (rule.use.loader.find(loader => loader.indexOf('babel-loader') >= 0)) {
foundRules.push(rule)
}
} else if (rule.use.loader.indexOf('babel-loader') >= 0) {
foundRules.push(rule)
}
} else if (rule.oneOf) {
const nestedRules = modifyBabelLoaderRule(rule.oneOf, false)
foundRules.push(...nestedRules)
function modifyBabelLoaderRules (rules, modifier = r => r) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored the logic to make this painless and D.R.Y.

return rules.map(rule => {
if (rule.oneOf) {
rule.oneOf = modifyBabelLoaderRules(rule.oneOf, modifier)
} else if (rule.use) {
rule.use = modifyBabelLoaderRules(rule.use, modifier)
} else {
rule = modifier(rule)
}
return rule
})
}

if (root) {
foundRules.forEach((rule, index) => {
if (rule.include?.indexOf('src') >= 0) {
console.log('Found CRA babel-loader rule for source files. Modifying it to instrument for code coverage.')
console.log('rule: ', rule)
rule.options.plugins.push('istanbul')
/**
* Adds exclude rules for pure ESM Modules.
*
* @param {import('webpack').RuleSetRule[]} rules
* @returns {import('webpack').RuleSetRule[]}
*/
function modifyBabelLoaderRuleForBuild (rules) {
return modifyBabelLoaderRules(rules, rule => {
if (rule.loader && rule.loader.includes('babel-loader')) {
if ('exclude' in rule) {
if (!Array.isArray(rule.exclude)) {
rule.exclude = [rule.exclude]
}
PURE_ESM_MODULES.forEach(module => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since pure ESM modules like ipfs-geoip are already built, we need to let babel know that this can be skipped for transpilation.

rule.exclude.push(new RegExp(`node_modules/${module}`))
})
}
})
}
}
return rule
})
}

return foundRules
/**
* Adds instrumentation plugin for code coverage in test mode.
*
* @param {import('webpack').RuleSetRule[]} rules
* @returns {import('webpack').RuleSetRule[]}
*/
function modifyBabelLoaderRuleForTest (rules) {
return modifyBabelLoaderRules(rules, rule => {
if (rule.options && rule.options.plugins) {
rule.options.plugins.push('istanbul')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this also gives us an opportunity to inject instrumentation in a much more cleaner way.

}
return rule
})
}

function webpackOverride (config) {
function webpackOverride(config) {
const fallback = config.resolve.fallback || {}

Object.assign(fallback, {
Expand All @@ -63,10 +86,12 @@ function webpackOverride (config) {
})
])

config.module.rules = modifyBabelLoaderRuleForBuild(config.module.rules)

// Instrument for code coverage in development mode
const REACT_APP_ENV = process.env.REACT_APP_ENV ?? process.env.NODE_ENV ?? 'production'
if (REACT_APP_ENV === 'test') {
modifyBabelLoaderRule(config.module.rules)
config.module.rules = modifyBabelLoaderRuleForTest(config.module.rules)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

explicit assignment instead of in place modification on pass by reference.

}

return config
Expand Down
50 changes: 9 additions & 41 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
"intl-messageformat": "^9.11.3",
"ip": "^1.1.5",
"ipfs-css": "^1.4.0",
"ipfs-geoip": "^8.0.0",
"ipfs-geoip": "9.0.1",
"ipfs-http-client": "49.0.2",
"ipfs-provider": "^2.1.0",
"ipld-explorer-components": "^2.4.1",
Expand Down
12 changes: 5 additions & 7 deletions src/bundles/peer-locations.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { createAsyncResourceBundle, createSelector } from 'redux-bundler'
import { getConfiguredCache } from 'money-clip'
import geoip from 'ipfs-geoip'
import { lookup } from 'ipfs-geoip'
import PQueue from 'p-queue'
import HLRU from 'hashlru'
import Multiaddr from 'multiaddr'
Expand Down Expand Up @@ -32,10 +32,8 @@ function createPeersLocations (opts) {
const bundle = createAsyncResourceBundle({
name: 'peerLocations',
actionBaseType: 'PEER_LOCATIONS',
getPromise: async ({ store, getIpfs }) => {
const peers = store.selectPeers()
return peerLocResolver.findLocations(peers, getIpfs)
},
getPromise: ({ store }) => peerLocResolver.findLocations(
store.selectAvailableGatewayUrl(), store.selectPeers()),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now we can just pull the list the gateway urls active and pass the peers.

staleAfter: UPDATE_EVERY,
retryAfter: UPDATE_EVERY,
persist: false,
Expand Down Expand Up @@ -227,7 +225,7 @@ class PeerLocationResolver {
this.pass = 0
}

async findLocations (peers, getIpfs) {
async findLocations (gatewayUrls, peers) {
const res = {}

for (const p of this.optimizedPeerSet(peers)) {
Expand Down Expand Up @@ -257,7 +255,7 @@ class PeerLocationResolver {

this.geoipLookupPromises.set(ipv4Addr, this.queue.add(async () => {
try {
const data = await geoip.lookup(getIpfs(), ipv4Addr)
const data = await lookup(gatewayUrls, ipv4Addr)
await this.geoipCache.set(ipv4Addr, data)
} catch (e) {
// mark this one as failed so we don't retry again
Expand Down
121 changes: 61 additions & 60 deletions src/bundles/peer-locations.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,55 +274,58 @@ describe('PeerLocationResolver', () => {

const result = await getPromise({
store: {
selectPeers: () => [{
peer: '1aaa1',
latency: 'n/a',
addr: {
stringTuples: () => [[4, '123.123.123.123']]
selectPeers: () => [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reformatted for readability. tab +1

{
peer: '1aaa1',
latency: 'n/a',
addr: {
stringTuples: () => [[4, '123.123.123.123']]
}
},
{
peer: '1b1',
latency: '1ms',
addr: {
stringTuples: () => [[4, '127.0.0.1']]
}
},
{
peer: '3b3',
latency: '1ms',
addr: {
stringTuples: () => [[4, '16.19.16.19']]
}
},
{
peer: '44asd',
latency: '1ms',
addr: {
stringTuples: () => [[4, '4.4.4.4']]
}
},
{
peer: '4sameAs4',
latency: '1ms',
addr: {
stringTuples: () => [[4, '4.4.4.4']]
}
},
{
peer: 'newPeerThatShouldThrow',
latency: '100s',
addr: {
stringTuples: () => [[4, '5.5.5.5']]
}
},
{
peer: 'sameIpAs1',
latency: 'n/a',
addr: {
stringTuples: () => [[4, '123.123.123.123']]
}
}
},
{
peer: '1b1',
latency: '1ms',
addr: {
stringTuples: () => [[4, '127.0.0.1']]
}
},
{
peer: '3b3',
latency: '1ms',
addr: {
stringTuples: () => [[4, '16.19.16.19']]
}
},
{
peer: '44asd',
latency: '1ms',
addr: {
stringTuples: () => [[4, '4.4.4.4']]
}
},
{
peer: '4sameAs4',
latency: '1ms',
addr: {
stringTuples: () => [[4, '4.4.4.4']]
}
},
{
peer: 'newPeerThatShouldThrow',
latency: '100s',
addr: {
stringTuples: () => [[4, '5.5.5.5']]
}
},
{
peer: 'sameIpAs1',
latency: 'n/a',
addr: {
stringTuples: () => [[4, '123.123.123.123']]
}
}]
],
selectAvailableGatewayUrl: () => 'https://ipfs.io'
},
getIpfs: () => 'smth'
})
Expand All @@ -346,10 +349,13 @@ describe('PeerLocationResolver', () => {
}
}))

const mockStore = {
selectAvailableGatewayUrl: () => 'https://ipfs.io',
selectPeers: () => peers
}

const result = await getPromise({
store: {
selectPeers: () => peers
}
store: mockStore
})

expect(result).toEqual({
Expand All @@ -368,10 +374,9 @@ describe('PeerLocationResolver', () => {
// ==== 100 ====

const result100 = await getPromise({
store: {
selectPeers: () => peers
}
store: mockStore
})

const expect100 = new Array(100).fill().reduce((prev, _, index) => ({
...prev,
[`${index}aa`]: 'location-cached'
Expand All @@ -382,9 +387,7 @@ describe('PeerLocationResolver', () => {
// ==== 200 ====

const result200 = await getPromise({
store: {
selectPeers: () => peers
}
store: mockStore
})
const expect200 = new Array(200).fill().reduce((prev, _, index) => ({
...prev,
Expand All @@ -396,9 +399,7 @@ describe('PeerLocationResolver', () => {
// ==== Over 200 ====

const resultMore = await getPromise({
store: {
selectPeers: () => peers
}
store: mockStore
})

const expectMore = new Array(1000).fill().reduce((prev, _, index) => ({
Expand Down
2 changes: 1 addition & 1 deletion src/peers/PeersPage.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import AddConnection from './AddConnection/AddConnection'
import CliTutorMode from '../components/cli-tutor-mode/CliTutorMode'
import { cliCmdKeys, cliCommandList } from '../bundles/files/consts'

const PeersPage = ({ t, toursEnabled, handleJoyrideCallback, isCliTutorModeEnabled }) => (
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused assignment.

const PeersPage = ({ t, toursEnabled, handleJoyrideCallback }) => (
<div data-id='PeersPage' className='overflow-hidden'>
<Helmet>
<title>{t('title')} | IPFS</title>
Expand Down