Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database #4

Merged
merged 62 commits into from
Jun 11, 2024
Merged
Show file tree
Hide file tree
Changes from 61 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
3df0492
Updated structure of project and created separate router for projects…
IvanC112 May 29, 2024
cb2fc29
Added router when using database instead of local JSON and attempt to…
IvanC112 May 30, 2024
684bae8
Updated imports for bun environment
IvanC112 Jun 1, 2024
9c36e05
updated import path to work with bun
LordFarquaadtheCreator Jun 1, 2024
c17c790
Handled error for multiple attempts of connecting to DB and allowed f…
IvanC112 Jun 1, 2024
58d052c
Updated previous mistake on client, can now use connected client for …
IvanC112 Jun 1, 2024
85d7193
removing common js
LordFarquaadtheCreator Jun 3, 2024
58f9152
removed all commonjs imports
dddictionary Jun 3, 2024
c34fed0
made router constant
dddictionary Jun 3, 2024
d985a35
added logging
LordFarquaadtheCreator Jun 3, 2024
e3178d9
Removed commonJS
IvanC112 Jun 3, 2024
e2b1210
moved middleware to projectsDB file & handled multiple connection error
LordFarquaadtheCreator Jun 3, 2024
eed74a0
Create and read operation to the database
IvanC112 Jun 4, 2024
cc47ac2
added filter to return an error message for non-existant routes
dddictionary Jun 4, 2024
c326f08
implemented filters for /get route.
dddictionary Jun 4, 2024
0613352
adding database function abstraction
LordFarquaadtheCreator Jun 4, 2024
53a3ace
adding import
LordFarquaadtheCreator Jun 4, 2024
282b8f5
formatting
LordFarquaadtheCreator Jun 4, 2024
66a92e7
added error handling
LordFarquaadtheCreator Jun 4, 2024
a26406e
finished add endpoint
LordFarquaadtheCreator Jun 4, 2024
a121f67
added doc
LordFarquaadtheCreator Jun 4, 2024
91fd251
added null fn
LordFarquaadtheCreator Jun 4, 2024
b5d6bce
Boolean switch for database that is already connected
IvanC112 Jun 4, 2024
98b39b5
Updated connected db issue in db.ts
IvanC112 Jun 4, 2024
3822fb3
query database refactor
dddictionary Jun 4, 2024
d3e25ce
Removed extra code in connectDB causing connection issues
IvanC112 Jun 4, 2024
6abf3b3
implemented update endpoint and database function
dddictionary Jun 4, 2024
e09dec8
Updated README.md for db actions and local JSON file queries
IvanC112 Jun 5, 2024
505fca4
Merge branch 'main' into Database
dddictionary Jun 5, 2024
27ec86c
updated readme with more info
dddictionary Jun 5, 2024
27b2e1b
created universal query function to query database
dddictionary Jun 5, 2024
7cb85ba
Validate middleware before inserting new data into DB
IvanC112 Jun 6, 2024
3efa716
small tweaks to responses
dddictionary Jun 6, 2024
7d053b8
Merge branch 'Database' of https://github.com/BYTE-Club-CCNY/byte-ser…
dddictionary Jun 6, 2024
2c4f6f5
fixed incorrect logging issue for update
dddictionary Jun 6, 2024
38e4415
Significantly reduced lines of code while retaining the logic of vali…
IvanC112 Jun 7, 2024
dfdc050
adding initial frakework for parallel processing the database check
LordFarquaadtheCreator Jun 8, 2024
bcb3a46
fn dbCheck.ts will try to connect to db with a given timeout
LordFarquaadtheCreator Jun 8, 2024
b238251
updating time settings on timeout and interval
LordFarquaadtheCreator Jun 8, 2024
07eb0b6
checkDB now runs parallel to server
LordFarquaadtheCreator Jun 9, 2024
300adc7
adding logging
LordFarquaadtheCreator Jun 9, 2024
24cb6d9
finishing up merging with main and adding db connection to projectsDa…
LordFarquaadtheCreator Jun 9, 2024
0434d7c
removed app.use(express.json()) since it caused errors
LordFarquaadtheCreator Jun 9, 2024
5e8f0ec
changed db checker interval from 5 sec to 1 hr
LordFarquaadtheCreator Jun 9, 2024
42eb3c7
formatting validation middleware
LordFarquaadtheCreator Jun 9, 2024
ae4e545
Delete app.ts
LordFarquaadtheCreator Jun 9, 2024
c0f85a8
Removed redundant connectDB.tsx file and updated the typing for valid…
IvanC112 Jun 9, 2024
f87ed21
adding http and https to server
LordFarquaadtheCreator Jun 10, 2024
4128c12
removed duplicate listner
LordFarquaadtheCreator Jun 10, 2024
3971498
im gonna touch abrar
LordFarquaadtheCreator Jun 11, 2024
feab096
removed the app.listen when we have a httpserver doing that job
dddictionary Jun 11, 2024
8aa04c0
moved all routes under a single async function so we can use the client.
dddictionary Jun 11, 2024
4b7631d
switched express.json() to projectsDB so no issues are caused in serv…
IvanC112 Jun 11, 2024
366c161
misc
LordFarquaadtheCreator Jun 11, 2024
be99702
formatting validation middleware
LordFarquaadtheCreator Jun 9, 2024
e064eed
Removed redundant connectDB.tsx file and updated the typing for valid…
IvanC112 Jun 9, 2024
1e38b02
Merge remote-tracking branch 'origin/Database' into parallel
LordFarquaadtheCreator Jun 11, 2024
d79d7a8
Merge pull request #3 from BYTE-Club-CCNY/parallel
LordFarquaadtheCreator Jun 11, 2024
39d73ac
Updated keys from body to be case sensitive and updated typing of val…
IvanC112 Jun 11, 2024
02c5a4e
Added UUID to each project
IvanC112 Jun 11, 2024
ff79c04
Changed timeout and interval back to its original values
IvanC112 Jun 11, 2024
ca6093d
Merge branch 'main' into Database
LordFarquaadtheCreator Jun 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 84 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,88 @@
# byte-server
# Byte-server
Server for byte

# Description
Server is built with Node.js and Express. Provides endpoints for managing a collection of projects, including creating, reading and updating. Exists both a json file and database backup for storing and quering projects
## Description
Server is built with Node.js and Express. Provides endpoints for managing a collection of projects, including creating, reading and updating. Exists both database and a local JSON file backup for storing and quering projects.

## How to use
Creating, Reading and Updating can be done from the database (default approach) endpoints. Only reading can be done for the local JSON file.

## Endpoints for DB

### Get projects
`/projects/get`

Retrieves all projects in the database and supports additional query params for filtering.

Filters include:
- **team**: TYPE string(s)
- **cohort**: TYPE string(s)
- **name**: TYPE string(s)

Filters are stackable, meaning you can do something similar to:
`/projects/get?team=John&cohort=1&team=Jane`
This will return all projects that have John **AND** Jane in their team and are in cohort 1. You can stack as many filters as you want but be aware that the more filters you add, the more specific the query will be, meaning it might return no results.

### Post new projects
`/projects/add`

**Please Fill Out All Fields!!!**
These fields must be provided as JSON in the body of the request.

The schema for projects is defined as follows:
- **name**: TYPE string(s)
- **"short-desc"**: TYPE string(s)
- **"long-desc"**: TYPE string(s)
- **team**: TYPE array of strings
- **link**: TYPE string(s)
- **Image**: TYPE string(s)
- **"tech-stack"**: TYPE array of strings
- **cohort**: TYPE string(s)
- **topic**: TYPE array of strings

### UPDATE projects
`/projects/update`

You **MUST** provide the project name you want to update as a query parameter like so:
`/projects/update?name={project_name}`

You can provide any key value pair you want to update in the body of the request, but it **HAS** to be JSON. This doesn't have to be all the fields, only the ones you want to update.

Reminder that the schema for projects is defined as follows:
- **name**: TYPE string(s)
- **"short-desc"**: TYPE string(s)
- **"long-desc"**: TYPE string(s)
- **team**: TYPE array of strings
- **link**: TYPE string(s)
- **Image**: TYPE string(s)
- **"tech-stack"**: TYPE array of strings
- **cohort**: TYPE string(s)
- **topic**: TYPE array of strings

Any mismatch of the schema types (i.e. providing a string when we expect an array) will return an error.

## Endpoints for Local File

### Get all projects
`/projects`

Retrieves all projects from the local JSON file and also supports additional query params for filtering.

### Get projects based on team members
`/projects?team={member_name}`

Replace `{member_name}` with a member that is in the project you would like to query. This filter is stackable, meaning you can do something similar to:
`/projects?team=John&team=Jane`

### Get projects based on cohort
`/projects?cohort={cohort}`

Replace `{cohort}` with the desired cohort to filter projects. This filter is stackable, meaning you can do something similar to:
`/projects?cohort=1&cohort=2`

### Get projects based on name
`/projects?name={project_name}`

Replace `{project_name}` with the desired project name to filter projects.
This filter is stackable, meaning you can do something similar to:
`/projects?name=Website&name=Server`
18 changes: 12 additions & 6 deletions data.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
"topic": [
"AI",
"ML"
]
],
"uuid": "80eb8a48-9172-4f6f-87ca-4ae89de1938c"
},
{
"name": "Stockbros",
Expand All @@ -40,7 +41,8 @@
"topic": [
"Data Science",
"Finance"
]
],
"uuid": "5dca7d6e-5131-434f-9eb9-5670a820a260"
},
{
"name": "TrackLeet",
Expand All @@ -65,7 +67,8 @@
"Chrome Extension",
"Web Development",
"Authentication"
]
],
"uuid": "e00bfd77-58a4-4d2f-a56e-6fc5b8280454"
},
{
"name": "Don't Be Alarmed",
Expand All @@ -87,7 +90,8 @@
"cohort": "Spring 2024",
"topic": [
"Android App Development"
]
],
"uuid": "9fe8f025-d0a4-4a88-abb5-184da85db827"
},
{
"name": "BYTE Website",
Expand All @@ -111,7 +115,8 @@
"cohort": "Summer 2024",
"topic": [
"Web Development"
]
],
"uuid": "d78df4b2-4ad6-4c0e-82b1-31473c4ef9f2"
},
{
"name": "BYTE Server",
Expand Down Expand Up @@ -140,6 +145,7 @@
"Server Development",
"API",
"Postgres"
]
],
"uuid": "52a901bf-685a-4320-9336-5ce7785e4f18"
}
]
13 changes: 10 additions & 3 deletions db.config.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,20 @@
import dotenv from 'dotenv';
import {Client} from 'pg';
import dotenv from "dotenv";
import { Client } from "pg";
dotenv.config();

const client = new Client({
host: process.env.POSTGRESQL_DB_HOST,
user: process.env.POSTGRESQL_DB_USER,
password: process.env.POSTGRESQL_DB_PASSWORD,
database: process.env.POSTGRESQL_DB,
port: process.env.POSTGRESQL_DB_PORT ? parseInt(process.env.POSTGRESQL_DB_PORT) : 5432
port: process.env.POSTGRESQL_DB_PORT
? parseInt(process.env.POSTGRESQL_DB_PORT)
: 5432,
});

client.on("end", () => console.log("Client has disconnected"));
client.on("error", (err) =>
console.error("Unexpected error on idle client", err),
);

export default client;
32 changes: 18 additions & 14 deletions db.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
import client from './db.config';
import client from './db.config';

const activateDb = async () => {
console.log("Connecting to Database ...");
let isActive = false;
const getDB = async () => {
console.log("Connecting to Database ...");
if (isActive) {
console.log("Database already connected");
return client;
}
try {
await client.connect();
console.log("Database connected");
isActive = true;
return client;
} catch (err: any) {
return client;
}
}

try {
await client.connect();
console.log("Database connected");

return client;
} catch (err: any) {
throw new Error(`Database connection error\n ${err.message}`);
}
}

export default activateDb;
export default getDB;
22 changes: 22 additions & 0 deletions dbCheck.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import client from "./db.config";

async function getDB() {
await client.connect();
}

async function checkDB(timeout: number) {
setTimeout(async () => {
try {
await getDB();
process.exit(0);
} catch (e: any) {
console.error("Error Connecting to DB:", e.message);
process.exit(1);
}
}, timeout);
}

const args: string[] = process.argv;
const timeout: number = parseInt(args[2]);

await checkDB(timeout);
34 changes: 34 additions & 0 deletions dbChecker.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import { spawn } from "child_process";

export function secondsToMs(d: number) {
return d * 1000;
}

// threading should happen at top level of server "setInterval"
async function checkDB(TIMEOUT: number): Promise<boolean> {
return new Promise((resolve, reject) => {
let dbAval: boolean = false;

const database = spawn("bun", ["dbCheck.ts", TIMEOUT.toString()]);

database.stdout.on("data", (data) => {
console.log("Output from dbCheck.ts:", data.toString());
});

database.on("exit", (code) => {
if (code === 0) {
dbAval = true;
} else {
dbAval = false;
}
resolve(dbAval);
});

database.on("error", (error) => {
console.error(error);
reject(error);
});
});
}

export default checkDB;
14 changes: 14 additions & 0 deletions middlewares/connectDB.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import getDB from "../db";
import logger from "../utils/logger";

const connectDB = async (req: any, res: any, next: any) => {
try {
req.client = await getDB();
next();
} catch (err: any) {
logger.info(err.message);
next("route");
}
}

export default connectDB;
81 changes: 81 additions & 0 deletions middlewares/validate.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
//validate all fields and their types
function validating(
keys: string[],
values: any,
requiredFields: { [key: string]: string },
res: any,
) {
for (let i = 0; i < values.length; i++) {
//initial check if the field is a required field
if (!(keys[i] in requiredFields)) {
return res
.status(400)
.json({
message: `Please insert a valid field name, ${keys[i]} is invalid`,
});
}
//check for the correct typing
if (typeof values[i] !== requiredFields[keys[i]]) {
return res
.status(400)
.json({
message: `Please insert the correct typing for ${keys[i]}, it should be a ${requiredFields[keys[i]] === "object" ? "array of strings" : requiredFields[keys[i]]}!`,
});
}
}
// if no response is sent meaning all validations passed, return false at the end of the function
return false;
}

const validate = (req: any, res: any, next: any) => {
const requiredFields = {
name: "string",
"short-desc": "string",
"long-desc": "string",
team: "object",
link: "string",
image: "string",
"tech-stack": "object",
cohort: "string",
topic: "object",
};
const values = Object.values(req.body);
const keys = Object.keys(req.body);

if (req.method === "POST") {
//initial check for empty request body
if (Object.keys(req.body).length === 0) {
return res
.status(400)
.json({
message: "Please insert a object with all required fields!",
});
}
if (keys.length !== 9) {
return res
.status(400)
.json({
message:
"Please insert all required fields, you are missing some fields!",
});
} else {
if (validating(keys, values, requiredFields, res)) {
return;
}
}
} else {
//initial check for empty request body
if (Object.keys(req.body).length === 0) {
return res
.status(400)
.json({ message: "Please insert a object to update!" });
} else {
if (validating(keys, values, requiredFields, res)) {
return;
}
}
}
next();
};

export default validate;
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@
"dependencies": {
"@types/chai": "^4.3.16",
"@types/chai-http": "^4.2.0",
"@types/cors": "^2.8.17",
"@types/dotenv": "^8.2.0",
"@types/express": "^4.17.21",
"@types/mocha": "^10.0.6",
"@types/pg": "^8.11.6",
"chai": "^5.1.1",
"chai-http": "^4.4.0",
"cors": "^2.8.5",
"dotenv": "^16.4.5",
"express": "^4.19.2",
"nodemon": "^3.1.1",
Expand Down
Loading