Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: force scrap #5

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: force scrap #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 26 additions & 5 deletions client/src/App.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,26 @@ function App() {
const [isLoading, setIsLoading] = useState(false);
const [ready, setReady] = useState(false);


const forceScrap = async () => {
setIsLoading(true);
const url = `${process.env.REACT_APP_API_URL}/api/force-scrap`;
try {
const res = await axios.post(url);

const { ready } = res.data;
setReady(ready);

setIsLoading(false);
alert('All is Ready Now');

} catch (error) {
alert('An error occurred. \n' + error);
setIsLoading(false);
}

}

const onSubmit = async (data: any) => {
setIsLoading(true);

Expand Down Expand Up @@ -44,15 +64,16 @@ function App() {
}
};


return (
<div className="App">
<header className="header">
<body>
{results === undefined ? (
<InputForm isLoading={isLoading} onSubmit={onSubmit} />
) : (
<ResultsTable results={results} />
)}
{
(results === undefined ?
<InputForm isLoading={isLoading} onSubmit={onSubmit} forceScrap={forceScrap} />
: <ResultsTable results={results} />)
}
</body>
</header>
</div>
Expand Down
67 changes: 40 additions & 27 deletions client/src/InputForm.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import React, { useState } from 'react';

function InputForm({ onSubmit, isLoading }) {
function InputForm({ onSubmit, isLoading ,forceScrap}) {

const [numOfPolygonPages, setNumOfPolygonPages] = useState('');
const [problemsId, setProblemsId] = useState('');
Expand All @@ -18,38 +18,51 @@ function InputForm({ onSubmit, isLoading }) {
setter(event.target.value);
};



return (
<form classname='form' onSubmit={submitHandler}>
<div>
<input
className="input form-entry"
id="numOfPolygonPages"
placeholder="Enter Number of Polygon Pages"
value={numOfPolygonPages}
onChange={createOnChangeHandler(setNumOfPolygonPages)}
/>
</div>

<div>
<textarea
className="comma-separated-input form-entry"
id="problem-id"
placeholder="Enter Problem ID"
value={problemsId}
onChange={createOnChangeHandler(setProblemsId)}
/>
</div>

<button type="submit" className="submit form-entry" disabled={isLoading}>
<div>
<form classname='form' onSubmit={submitHandler}>
<div>
<input
className="input form-entry"
id="numOfPolygonPages"
placeholder="Enter Number of Polygon Pages"
value={numOfPolygonPages}
onChange={createOnChangeHandler(setNumOfPolygonPages)}
/>
</div>

<div>
<textarea
className="comma-separated-input form-entry"
id="problem-id"
placeholder="Enter Problem ID"
value={problemsId}
onChange={createOnChangeHandler(setProblemsId)}
/>
</div>

<button type="submit" className="submit form-entry" disabled={isLoading}>
{isLoading && (
<div class="spinner-border spinner-grow-sm mr-1" role="status">
<span class="sr-only">Loading...</span>
</div>
)}
Submit

</button>
</form>
<button className="submit form-entry" disabled={isLoading} onClick={forceScrap}>
{isLoading && (
<div class="spinner-border spinner-grow-sm mr-1" role="status">
<span class="sr-only">Loading...</span>
<div className="spinner-border spinner-grow-sm mr-1">
<span className="sr-only">Loading...</span>
</div>
)}
Submit
Force Scrap

</button>
</form>
</div>
);
}

Expand Down
8 changes: 8 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"cors": "^2.8.5",
"dotenv": "^8.2.0",
"express": "^4.17.1",
"express-basic-auth": "^1.2.0",
"http-delayed-response": "0.0.4",
"morgan": "^1.10.0",
"path": "^0.12.7",
Expand Down
21 changes: 21 additions & 0 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ import path from 'path';
import morgan from 'morgan';
import cors from 'cors';
import { Scraper } from './scraper';
import basicAuth from 'express-basic-auth';

const fs = require('fs');
const app = express();



let problemsScraper = undefined;
let ready = false;

Expand All @@ -26,8 +29,16 @@ const logFile = fs.createWriteStream(path.join(__dirname, 'logs.log'), {
app.use(express.json());
app.use(cors());
app.use(morgan('dev', { stream: logFile }));
app.use(
basicAuth({
users: { [process.env.BASIC_AUTH_USERNAME as string]: process.env.BASIC_AUTH_PASSWORD as string },
challenge: true,
}),
);

app.use(express.static(path.join(__dirname, '../client/build')));


const startScraper = async () => {
if (!ready) {
problemsScraper = new Scraper();
Expand Down Expand Up @@ -61,6 +72,16 @@ app.post('/api/cf-problems-matching', async (req, res) => {
res.end();
});


app.post('/api/force-scrap', async (req, res) => {
req.setTimeout(1000 * 60 * 10); // 10 Minutes
ready = false;
await startScraper();

res.send({ ready });
res.end();
});

app.get('*', (req, res) => {
res.sendFile(path.join(__dirname, '../client/build/index.html'));
});
Expand Down