Projects overview

[sirinoks]

This is a discussion designed to figure out what kinds of projects Telescope can be divided into and assigning our issues into those projects. This discussion will be useful to follow for those who would like to see a better picture of Telescope and discover other parts of it. Perhaps you will get interested in something you didn't know existed!

Project keepers:
@sirinoks
@JerryHue

Our task is to complete and maintain GitHub projects. If you are already working on something, feel free to explain what is the larger structure you're contributing to and how it works within Telescope.

Since we are resurrecting GitHub projects, we will now switch to Projects Beta. A structure for making projects is being currently developed.

For now as I'm discovering different areas, I will keep them in here in text format before I start outlining a large structure. This way it's easy to change things for now. Thanks to the meeting we had, I have a picture of different areas. However, many of the topics described here might be outdated or/and should be investigated further.

If you see a topic with a ? - it means it has to be investigated and explained.

Structure:

Back end:

• 1.0 chunk - src/backend/*, pnpm, Bull
• Microservices (docker containers):

• Auth src/api/auth - Disabled ?
• Feed-Discovery src/api/feed-discovery
• Image src/api/image
• Planet src/api/planet - Still used ?
• Posts src/api/posts - Redis
• Search src/api/search - Elasticsearch
• Users src/api/users - Firebase - Still Firebase ?
• Parser src/api/parser - Still used ?

• Satelite - Still used?

• Authentication - SAML-2 with Seneca auth

• Server - tools/autodeployment/* - Github webhooks

• Automation, tests:

• Tools:

• pnpm - pnpm-workspace.yaml
• jest - test/*
• eslint - .eslintrc.js
• prettier - .prettierrc, .husky/
• vscode - .vscode/
• github actions - .vscode/
• renovatebot - dependencies
• gitpod - gitpod.Dockerfile - cloud

• Runtime tools:

• ngnix - hosting
• certbot - SSL certificates
• logrotate - log files
• portainer - containers

Website areas:

• Blogs
• Dashboard - Note, no React involved
• Search

Other things:

• Docker
• Cloud
• Documentation

Existing areas

We already have some projects created, not all of them make sense to me. It will be our task to discover what they do, what they mean and where they belong in the project's architecture.
Projects that are complete, or don't make sense will be closed. Broad names will be changed. Descriptions will be added.

• Depencency visualization - new project, additional, requires Back end, Front end and Documentation.
• Microservice Architecture Port - ? - connected to Bull, search, microservices?
• Next.js - Old project transition. Transition has been made and is now not relevant. Project closed.
• Documentation - Stuff in /docs/*. Outdated name and project structure. Project closed.
• Test - Outdated name and project structure. Project closed.
• CI/CD - Continuous integration/deployment - Outdated name and project structure. Project closed.
• Backend - Outdated name and project structure. Project closed.
• UI and Frontend - Outdated name and project structure. Project closed.

Using project correctly

What qualifies as a Project?

Keep in mind that an issue/PR can only belong to one project. Therefore:
A project is a larger task which involves multiple areas (front/back end, UI). A clearly new idea is a new project. An idea/enhancement that involves more than a single issue and PR is a project. A project is closed once all the features of it are working and stable. Some issues on non-vital improvements can be ignored for closing the project.

Bad project usage:

• Back end: something that is broad and already is a label
• Bugfixing: too general, it doesn't have an end goal, too broad
• Feed-discovery tests: it doesn't have an end goal (feed-discovery already has tests), too broad
• Parser tests: might be too broad (since the parser service doesn't have tests, it might be a Project, but we need to denote when this project is finished. Should a certain level of test coverage be accomplished?)

Good project usage:

• Microservice Port: it has an end goal (rewrite the backend into a group of microservices).
• React Native Front End: while broad, it has a specific end goal (a finalized React Native mobile app).
• Initial Documentation: broad, the goal is definite (document all of Telescope)
• Multimedia Integration: broad, the goal is specific to a feature (videos and livestreams)

---

Structure

What is currenlty lacking in projects is a clear description and instruction. There is a name and description built in, however what I mean is a more detailed explanation. What I propose is to always have a note/column/card that would lead to the main page where the project idea is outlined.
Projects can be in different stages of life. Here is how they should be documented based on it:

• Birth of an idea - A note with a discussion. - Add a note with a linked discussion. After the projects gains some structure, it can be turned into an issue with a single click.
• Deveoplment stage - An issue - A parent issue that would have instructions, checklists to other issues, PRs, any other things needed for a project to be completed.
• Completed retirement - A wiki page - After the project has been completed, it becomes a constant piece of our Telescope puzzle. Having an issue already, we will already have most of the explaining done. Some formatting and linking will be required at this stage. Eventually we will have all areas of Telescope documented.

There is a need for a template/instruction on how to create and maintain a project. Here are some sketches on what should be in there:
Using a descriptive name
How to run the environment
Where in the code is it
Documentation to the major technology (if we're integrating a new one)

Check out project board instructions

[cindyorangis]

Good Projects candidates would be:

• Dependency Visualization
• React Native
• 3.0 Docker Build Improvements

Telescope 1.0 team had a Project board for each Release, but of course, you guys don't have to do it that way. Each cohort function different and you'll figure out what works best for you guys :)

[TueeNguyen]

I'm trying to write some tests for Parser Service, there should be some extra steps before we could switch to Parser Service (currently is src/backend) completely. Definitely:

• Enabling Parser Service and removing src/backend

[cindyorangis]

• Next.js - ? - connected to React.

What I don't like in the current system is that we have separate project areas (such as a concrete framework next.js) along with broad categories such as Back end. We already have labels for such categories, defined as areas. Do we need to also have them as projects, could we mix them along with things like Dependency visualization, or should we keep them separate? @JerryHue I would like your input

Telescope 1.0's frontend was built on Gatsby.js so the Next.js Project board was to help the Telescope 2.0 team keep track of the progress for porting the frontend to Next.js. Unused boards should be closed. If there's a way to add a description to these boards, that would be nice because you're right, it's hard to know what the board was used for.

[sirinoks]

There is a description. It's just empty right now.
So just to confirm, next.js was a part of backend to implement for 2.0, and right now is an implemented and a complete chunk of code.
Therefore, it is not a good category description to use, since it's not a project, nor an area under which future issues could be assigned?

[cindyorangis]

So when we were planning the MVP for Telescope, we talked about different frontend frameworks that could work. Some of our team members contributed to Gatsby.js in OSD600/DPS909 and Gatsby.js was pretty hot at the time so we decided to try it out. However, the build time for Gatsby became horrendous so we had to look elsewhere. Next.js was a better fit for Telescope because we can use SSR, unlike Gatsby.js where it was mostly used for SSG. You can find the code in the /src/web directory

Anyway, the Telescope 2.0 team has finished porting Telescope from Gatsby.js to Next.js so the board can be closed.

[cindyorangis]

• Documentation - ?- does it mean commenting the code? The wiki page? A completely different web page?

The development for Telescope is always ongoing so the documentation often falls behind and is hard to keep up with. The Documentation board was an attempt to keep the docs in /docs up to date. We can totally make it a webpage just for docs inside of Telescope as filed in #290

[cindyorangis]

• Microservice Architecture Port - ? - connected to Bull, search, microservices?

The backend used to be in one directory /src/backend and most of it has been broken up into microservices. These microservices live in /src/api. However, there is still a good chunk of the backend in the /src/backend directory. Our goal is to completely break up the backend into microservices. The Microservices board should still be open until the backend is completely broken up into microservices.

[sirinoks]

Do we want to try out beta projects by the way? Since we are re-doing it all

[JerryHue]

We would need to check whether the features offered in the Projects Beta are useful. Since they are on beta stage, several features might change. That means that we are willing to keep track of whatever changes the Projects Beta goes through.

[sirinoks]

It seems that it goes through the organization (Seneca-CDOT). I'm not sure I should touch this. Can I @humphd ? I want to test it out.

[JerryHue]

What I don't like in the current system is that we have separate project areas (such as a concrete framework next.js) along with broad categories such as Back end. We already have labels for such categories, defined as areas. Do we need to also have them as projects, could we mix them along with things like Dependency visualization, or should we keep them separate?

This is a tough one... I admit I opened the Dependency Visualization project as a way to start up, but I think I need a better approach on how projects should be used. Here are some of my ideas and reasoning:

• Since issues and PRs can only belong to a single GitHub Project, that gives an idea on how we should create the Projects themselves. Projects like Backend or Frontend are not that useful, as they are too broad. One issue in a specific microservice technically belongs to the Backend project, but it doesn't help us to organize it that much. Such projects might work as simply labels, just to categorize them, and thus it will be better off by simply closing the projects and using the labels to categorize them (already being done when the issue is filed).
• The way I view a GitHub Project is essentially wanting to correlate issues and PRs that might not be directly related by the nature of the work (work on the backend or frontend, work on testing, work on documentation), but instead by what they are trying to accomplish when finished. For example, the YouTube integration work that happened in PRs Add backend and API functionality for uploaded videos #2581 and Show videos in the main timeline #2596 would have belonged into a single GitHub project labelled "YouTube Integration" or a broader one such as "Multimedia Integration".
• The main idea I want to highlight is that we should use GitHub Projects as a way to track that we have finished a set issues/PRs that can be described into something that can be considered "finished". You can never finish working on the "backend" or the "frontend", but you can finish working on "YouTube Integration" until you consider it a stable feature that is usable.
• So, while that means that the current Documentation project is not a good name, it helps to think of a more focused one. Let us say that our current objective is to document all of the current Telescope project. Thus, our GitHub Project may be named like "Initial Documentation" or something along those lines, and its extended description can be something like "build the initial set of technical documentation for the entirety of the Telescope project". While this project might be ongoing for a while, it will be finished when all of Telescope has been documented.

I hope that the wall of text above didn't bore you 😅 and that it will help us to determine how to efficiently use GitHub Projects in Telescope.

[sirinoks]

Having issues and PRs being able to only belong to one Project makes it clear. We will use labels for areas, as we were. Projects then should be larger chunks of work that would belong in multiple areas. Your Dependency visualization project fits the criteria.

I completely agree with your view on how to use them.

[sirinoks]

Also, this might be off-topic, but if we want to improve the labeling of the issues, we can always create more issue templates that are more specific. Currently, we have two: bug report, and feature request.

I agree, but I currently don't have any in mind to add. Do you have any ideas?

[JerryHue]

I agree, but I currently don't have any in mind to add. Do you have any ideas?

For example, we could add a template to file visual bugs. We can also have a template for improving tests. Another template would be changing the documentation (a really important we need to add if we want to carry through the documentation project fully).

[sirinoks]

For example, we could add a template to file visual bugs. We can also have a template for improving tests. Another template would be changing the documentation (a really important we need to add if we want to carry through the documentation project fully).

I see, those would be useful for sure. Perhaps you should add a separate issue for those?

I have updated the post. I came up with an idea of a "core" page for each project. Not sure what it should be, an issue/discussion/note. If it will be an issue, we can have specific template for creating a new project. I can already imagine what it would look like!

[JerryHue]

In my opinion, issues are the best, as they show a sense of history for the project. The only problem with discussions is that they are very conversational, and there are various threads to keep track of. Issues are a single thread. The only problem is that if the project is closed, the issue won't be closed, and the opposite does not happen either (if issue is closed, the project won't close automatically).

[JerryHue]

Bad project usage:

* `Back end` - something that is broad and already is a label.

* Pls add more I ran out of ideas

Good project usage:

* `Depencency visualization` - A new idea which involves multiple areas of development.

Bad project usage:

• Bugfixing: too general, it doesn't have an end goal, too broad
• Feed-discovery tests: it doesn't have an end goal (feed-discovery already has tests), too broad
• Parser tests: might be too broad (since the parser service doesn't have tests, it might be a Project, but we need to denote when this project is finished. Should a certain level of test coverage be accomplished?)

Good project usage:

• Microservice Port: it has an end goal (rewrite the backend into a group of microservices).
• React Native Front End: while broad, it has a specific end goal (a finalized React Native mobile app).
• Initial Documentation: broad, the goal is definite (document all of Telescope)
• Multimedia Integration: broad, the goal is specific to a feature (videos and livestreams)

I will try to think of more if I can

[sirinoks]

It was decided. We are using beta. Beta projects have all the functionality of the old projects, + other things. No reason to not use it.

My current focus is to start filing our newer ideas as new projects. I will develop a structure for creating projects and a procedure people should follow managing projects. Please note the main post gets updated with new information!