General: Add cleanup service for admins

[coolchock]

Checklist

General

• I tested all changes and their related features with all corresponding user types on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Server

• Important: I implemented the changes with a very good performance and prevented too many (unnecessary) and too complex database calls.
• I strictly followed the principle of data economy for all database calls.
• I strictly followed the server coding and design guidelines.
• I added multiple integration tests (Spring) related to the features (with a high test coverage).
• I added pre-authorization annotations according to the guidelines and checked the course groups for all new REST Calls (security).
• I documented the Java code using JavaDoc style.

Client

• Important: I implemented the changes with a very good performance, prevented too many (unnecessary) REST calls and made sure the UI is responsive, even with large data (e.g. using paging).
• I strictly followed the principle of data economy for all client-server REST calls.
• I strictly followed the client coding and design guidelines.
• Following the theming guidelines, I specified colors only in the theming variable files and checked that the changes look consistent in both the light and the dark theme.
• I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
• I added authorities to all new routes and checked the course groups for displaying navigation elements (links, buttons).
• I documented the TypeScript code using JSDoc style.
• I added multiple screenshots/screencasts of my UI changes.
• I translated all newly inserted strings into English and German.

Motivation and Context

Artemis stores a large amount of information, some of which can become corrupted or irrelevant over time. These records occupy significant database space, which can be reclaimed by deleting unused data. This includes old results, unresolved plagiarism cases, and unused feedback.

Currently, there are scripts for cleaning up results and feedback, but no automated process exists for clearing unresolved plagiarism cases. Additionally, orphaned entities, such as results without associated submissions or participation, remain unaddressed, leading to inefficient database usage. Furthermore, these cleanup scripts must be run manually by administrators, which is not ideal. To address these issues, a dedicated cleanup service for administrators was implemented to streamline the process.

Description

A cleanup service has been added to the admin panel, allowing administrators to select specific cleanup operations to perform on the database. They can specify a time period for which the cleanup should be executed, providing flexibility in managing old or unused data. In future iterations, a scheduled job is planned to be introduced, which will automatically execute the cleanup operations periodically, reducing the need for manual intervention and ensuring ongoing database optimization.

Steps for Testing

Prerequisites:

• 1 Administrator
• 1 Test Server where you have access rights
• Test data generated using large_course_main.py script. See README in the folder quick-course-setup for more details.

1. Log in to Artemis.
2. Navigate to course management and select the course created by the script.
3. Set the start and end dates of the course within the period for which you will delete data.
4. Open the programming exercise created by the script and go to the Plagiarism section.
5. Set the threshold to 0 and execute plagiarism detection. You should receive a few results.
6. In Server Administration -> Cleanup Service, execute "Delete plagiarism comparisons."
7. Verify that the plagiarism comparisons were deleted.
8. Execute the "Delete old rated results" operation.
9. Verify that only the last student submission has a linked result.
10. Set the end date for the programming exercise to a past date and start a practice mode participation.
11. In the cleanup service, execute "Delete non-rated results."
12. Verify that the practice mode submission no longer has a linked result.

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.

Review Progress

Performance Review

• I (as a reviewer) confirm that the client changes (in particular related to REST calls and UI responsiveness) are implemented with a very good performance even for very large courses with more than 2000 students.
• I (as a reviewer) confirm that the server changes (in particular related to database calls) are implemented with a very good performance even for very large courses with more than 2000 students.

Code Review

• Code Review 1
• Code Review 2

Manual Tests

• Test 1
• Test 2

Performance Tests

• Test 1
• Test 2

Test Coverage

Info: ✅ ❌ in Confirmation (assert/expect) have to be adjusted manually, also delete trivial files!

Client

Class/File	Line Coverage	Confirmation (assert/expect)
admin.route.ts	not found (modified)	❌
cleanup-operation.model.ts	0%	✅ ❌
cleanup-service.component.ts	82.5%	✅ ❌
data-cleanup.service.ts.ts	100%	✅
date-time-picker.component.ts	96.66%	✅ ❌
navbar.component.ts	87.76%	✅ ❌

Server

Class/File	Line Coverage	Confirmation (assert/expect)
FeedbackRepository.java	86%	✅ ❌
LongFeedbackTextRepository.java	100%	✅
ParticipantScoreRepository.java	81%	✅ ❌
RatingRepository.java	not found (modified)	❌
ResultRepository.java	96%	✅ ❌
StudentScoreRepository.java	not found (modified)	❌
TeamScoreRepository.java	not found (modified)	❌
TextBlockRepository.java	not found (modified)	❌
CleanupJobExecution.java	80%	✅ ❌
CleanupJobType.java	100%	✅
CleanupServiceExecutionRecordDTO.java	100%	✅
CleanupJobExecutionRepository.java	not found (added)	❌
DataCleanupService.java	100%	✅
AdminOldDataCleanupResource.java	100%	✅
PlagiarismComparisonRepository.java	100%	✅

Screenshots

View from the navigation bar

Cleanup service

Confirmation dialog

Summary by CodeRabbit

• New Features

  • Introduced a new CleanupServiceComponent for managing data cleanup operations in the admin interface.
  • Added a new route for the CleanupServiceComponent in the admin module.
  • Enhanced the navbar with a new cleanup icon and breadcrumb translation for the cleanup service.
  • Added multiple new cleanup repository interfaces and methods for managing orphaned and outdated data across various entities.
  • Added new translations for the cleanup service in both German and English.
  • Implemented a DataCleanupService to handle HTTP requests for various cleanup operations.

• Bug Fixes

  • Clarified transactional behavior in various repository methods to improve code readability.

• Documentation

  • Added new documentation for the Cleanup Service, detailing data deletion operations and types.
  • Updated the index to include the new cleanup service documentation.

• Tests

  • Added comprehensive integration tests for various cleanup operations to ensure data integrity.
  • Introduced unit tests for the CleanupServiceComponent and DataCleanupService to validate functionality and HTTP requests.

[coderabbitai]

Actionable comments posted: 8

🧹 Outside diff range and nitpick comments (9)

src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.service.spec.ts (1)

11-12: Consider making the mock data more explicit.

The mock date could be more explicit to avoid potential timezone issues in tests.

-const mockDate = dayjs();
+const mockDate = dayjs('2024-03-08T12:00:00.000Z');

src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.component.spec.ts (4)

1-10: Optimize imports for better performance and maintainability.

1. Replace the full dayjs import with a specific import:

-import dayjs from 'dayjs/esm';
+import { Dayjs } from 'dayjs/esm';

2. Consider mocking dayjs functionality instead of using the actual implementation to improve test performance.

---

17-38: Consider using async/await for cleaner test setup.

The promise chain can be simplified:

-    beforeEach(() => {
+    beforeEach(async () => {
         const mockCleanupService = {
             getLastExecutions: jest.fn(),
             deleteOrphans: jest.fn(),
             deletePlagiarismComparisons: jest.fn(),
             deleteNonRatedResults: jest.fn(),
             deleteOldRatedResults: jest.fn(),
             deleteOldSubmissionVersions: jest.fn(),
             deleteOldFeedback: jest.fn(),
         };
 
-        TestBed.configureTestingModule({
+        await TestBed.configureTestingModule({
             imports: [ArtemisTestModule, CleanupServiceComponent],
             providers: [{ provide: DataCleanupService, useValue: mockCleanupService }],
-        })
-            .compileComponents()
-            .then(() => {
-                fixture = TestBed.createComponent(CleanupServiceComponent);
-                comp = fixture.componentInstance;
-                cleanupService = TestBed.inject(DataCleanupService);
-            });
+        }).compileComponents();
+
+        fixture = TestBed.createComponent(CleanupServiceComponent);
+        comp = fixture.componentInstance;
+        cleanupService = TestBed.inject(DataCleanupService);
     });

---

40-52: Enhance date comparison assertion specificity.

Replace the generic equality check with a more specific date comparison:

-expect(comp.cleanupOperations[0].lastExecuted).toEqual(dayjs(executionRecord[0].executionDate));
+expect(comp.cleanupOperations[0].lastExecuted?.isSame(executionRecord[0].executionDate)).toBeTrue();

---

90-112: Enhance date validation test coverage.

While the current test covers basic valid/invalid cases, consider adding tests for:

1. Edge cases (same date for from/to)
2. Null/undefined dates
3. Extreme date ranges

Also, consider extracting the test data setup:

const createOperation = (from: Dayjs, to: Dayjs): CleanupOperation => ({
    name: 'deleteOrphans',
    deleteFrom: from,
    deleteTo: to,
    lastExecuted: undefined,
    datesValid: signal(true),
});

src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/LongFeedbackTextCleanupRepository.java (2)

34-42: SQL keywords should be uppercase for consistency.

According to the coding guidelines (sql: uppercase), SQL keywords should be uppercase for better readability.

-            delete from LongFeedbackText lft
-            where lft.feedback.id in (
-                select f.id
-                from Feedback f
-                where f.result.participation is null
-                    and f.result.submission is null
+            DELETE FROM LongFeedbackText lft
+            WHERE lft.feedback.id IN (
+                SELECT f.id
+                FROM Feedback f
+                WHERE f.result.participation IS NULL
+                    AND f.result.submission IS NULL

---

1-125: Consider adding safeguards for large-scale deletions.

Since this repository handles cleanup operations that could affect a large number of records:

1. Consider adding batch processing to prevent memory issues and reduce database load
2. Add monitoring/logging to track the number of records deleted in each operation
3. Consider implementing rate limiting to prevent database overload

This is especially important for the date-range based methods that could potentially match a large number of records.

src/main/webapp/app/admin/cleanup-service/data-cleanup.service.ts (1)

33-34: Refactor duplicated date conversion logic into a helper method

The date conversion code in the deletePlagiarismComparisons, deleteNonRatedResults, and deleteOldRatedResults methods is duplicated. To improve code reuse and maintainability, consider extracting this logic into a private helper method.

Apply this diff to refactor the code:

+    private convertDeleteDates(deleteFrom: dayjs.Dayjs, deleteTo: dayjs.Dayjs): { deleteFromString: string; deleteToString: string } {
+        return {
+            deleteFromString: convertDateFromClient(deleteFrom)!,
+            deleteToString: convertDateFromClient(deleteTo)!,
+        };
+    }

     deletePlagiarismComparisons(deleteFrom: dayjs.Dayjs, deleteTo: dayjs.Dayjs): Observable<HttpResponse<CleanupServiceExecutionRecordDTO>> {
-        const deleteFromString = convertDateFromClient(deleteFrom)!;
-        const deleteToString = convertDateFromClient(deleteTo)!;
+        const { deleteFromString, deleteToString } = this.convertDeleteDates(deleteFrom, deleteTo);
         return this.http.delete<CleanupServiceExecutionRecordDTO>(`${this.adminResourceUrl}/plagiarism-comparisons`, {
             params: { deleteFrom: deleteFromString, deleteTo: deleteToString },
             observe: 'response',
         });
     }

     deleteNonRatedResults(deleteFrom: dayjs.Dayjs, deleteTo: dayjs.Dayjs): Observable<HttpResponse<CleanupServiceExecutionRecordDTO>> {
-        const deleteFromString = convertDateFromClient(deleteFrom)!;
-        const deleteToString = convertDateFromClient(deleteTo)!;
+        const { deleteFromString, deleteToString } = this.convertDeleteDates(deleteFrom, deleteTo);
         return this.http.delete<CleanupServiceExecutionRecordDTO>(`${this.adminResourceUrl}/non-rated-results`, {
             params: { deleteFrom: deleteFromString, deleteTo: deleteToString },
             observe: 'response',
         });
     }

     deleteOldRatedResults(deleteFrom: dayjs.Dayjs, deleteTo: dayjs.Dayjs): Observable<HttpResponse<CleanupServiceExecutionRecordDTO>> {
-        const deleteFromString = convertDateFromClient(deleteFrom)!;
-        const deleteToString = convertDateFromClient(deleteTo)!;
+        const { deleteFromString, deleteToString } = this.convertDeleteDates(deleteFrom, deleteTo);
         return this.http.delete<CleanupServiceExecutionRecordDTO>(`${this.adminResourceUrl}/old-rated-results`, {
             params: { deleteFrom: deleteFromString, deleteTo: deleteToString },
             observe: 'response',
         });
     }

Also applies to: 47-48, 61-62

src/main/java/de/tum/cit/aet/artemis/core/web/admin/AdminCleanupResource.java (1)

28-28: Remove trailing slash from @RequestMapping path

The trailing slash in the @RequestMapping("api/admin/cleanup/") may cause inconsistent URL mappings or unnecessary complexity. It's recommended to remove the trailing slash to standardize the endpoints and prevent potential issues with request handling.

Apply this diff:

-@RequestMapping("api/admin/cleanup/")
+@RequestMapping("api/admin/cleanup")

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between bfffbe1 and 118d5b4.

📒 Files selected for processing (8)

• src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/FeedbackCleanupRepository.java (1 hunks)
• src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/LongFeedbackTextCleanupRepository.java (1 hunks)
• src/main/java/de/tum/cit/aet/artemis/core/web/admin/AdminCleanupResource.java (1 hunks)
• src/main/webapp/app/admin/cleanup-service/data-cleanup.service.ts (1 hunks)
• src/main/webapp/i18n/en/cleanupService.json (1 hunks)
• src/test/java/de/tum/cit/aet/artemis/core/CleanupIntegrationTest.java (1 hunks)
• src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.component.spec.ts (1 hunks)
• src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.service.spec.ts (1 hunks)

✅ Files skipped from review due to trivial changes (1)

• src/main/webapp/i18n/en/cleanupService.json

🚧 Files skipped from review as they are similar to previous changes (2)

• src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/FeedbackCleanupRepository.java
• src/test/java/de/tum/cit/aet/artemis/core/CleanupIntegrationTest.java

🧰 Additional context used

📓 Path-based instructions (5)

src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/LongFeedbackTextCleanupRepository.java (1)

Pattern src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_imports

src/main/java/de/tum/cit/aet/artemis/core/web/admin/AdminCleanupResource.java (1)

Pattern src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_imports

src/main/webapp/app/admin/cleanup-service/data-cleanup.service.ts (1)

Pattern src/main/webapp/**/*.ts: angular_style:https://angular.io/guide/styleguide;methods_in_html:false;lazy_loading:true;code_reuse:true;tests:meaningful;types:PascalCase;enums:PascalCase;funcs:camelCase;props:camelCase;no_priv_prefix:true;strings:single_quotes;localize:true;btns:functionality;links:navigation;icons_text:newline;labels:associate;code_style:arrow_funcs,curly_braces,open_braces_same_line,indent_4;memory_leak_prevention:true;routes:naming_schema;chart_framework:ngx-charts;responsive_layout:true

src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.component.spec.ts (1)

Pattern src/test/javascript/spec/**/*.ts: jest: true; mock: NgMocks; bad_practices: avoid_full_module_import; perf_improvements: mock_irrelevant_deps; service_testing: mock_http_for_logic; no_schema: avoid_NO_ERRORS_SCHEMA; expectation_specificity: true; solutions: {boolean: toBeTrue/False, reference: toBe, existence: toBeNull/NotNull, undefined: toBeUndefined, class_obj: toContainEntries/toEqual, spy_calls: {not_called: not.toHaveBeenCalled, once: toHaveBeenCalledOnce, with_value: toHaveBeenCalledWith|toHaveBeenCalledExactlyOnceWith}}

src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.service.spec.ts (1)

Pattern src/test/javascript/spec/**/*.ts: jest: true; mock: NgMocks; bad_practices: avoid_full_module_import; perf_improvements: mock_irrelevant_deps; service_testing: mock_http_for_logic; no_schema: avoid_NO_ERRORS_SCHEMA; expectation_specificity: true; solutions: {boolean: toBeTrue/False, reference: toBe, existence: toBeNull/NotNull, undefined: toBeUndefined, class_obj: toContainEntries/toEqual, spy_calls: {not_called: not.toHaveBeenCalled, once: toHaveBeenCalledOnce, with_value: toHaveBeenCalledWith|toHaveBeenCalledExactlyOnceWith}}

🔇 Additional comments (2)

src/test/javascript/spec/component/admin/cleanup-service/cleanup-service.service.spec.ts (1)

1-6: LGTM! Modern Angular testing setup.

The imports are minimal and follow best practices by using the new provider-based HTTP testing setup.

src/main/java/de/tum/cit/aet/artemis/assessment/repository/cleanup/LongFeedbackTextCleanupRepository.java (1)

1-26: LGTM! Repository declaration follows best practices.

The repository is well-structured with appropriate annotations, clear documentation, and follows coding guidelines for imports and dependency injection.

[coderabbitai]

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (2)

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.html (1)

Line range hint 17-45: Consider improving button positioning for better responsiveness.

While the implementation is functional, the absolute positioning of buttons might cause layout issues on different screen sizes.

Consider using a more flexible layout approach:

-    <button [owlDateTimeTrigger]="dt" class="btn position-absolute" type="button">
+    <button [owlDateTimeTrigger]="dt" class="btn" type="button">
         <fa-icon [icon]="faCalendarAlt" />
     </button>
-    <div class="position-relative">
-        <button class="btn position-absolute" type="button" style="right: 0" (click)="clearDate()">
+    <div class="ms-auto">
+        <button class="btn" type="button" (click)="clearDate()">
             <fa-icon [icon]="faCircleXmark" />
         </button>
     </div>

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.ts (1)

28-28: Add JSDoc documentation for the new input property.

While the implementation is correct and follows Angular's new input() API, adding JSDoc documentation would improve maintainability by clearly describing the property's purpose and expected values.

+    /** Controls the visibility of the label. When true, the label is hidden. */
     hideLabelName = input<boolean>(false);

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 118d5b4 and 17b2af8.

⛔ Files ignored due to path filters (1)

• docs/admin/cleanup/cleanup-menu.png is excluded by !**/*.png, !**/*.png

📒 Files selected for processing (3)

• docs/admin/cleanup-service.rst (1 hunks)
• src/main/webapp/app/shared/date-time-picker/date-time-picker.component.html (1 hunks)
• src/main/webapp/app/shared/date-time-picker/date-time-picker.component.ts (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/admin/cleanup-service.rst

🧰 Additional context used

📓 Path-based instructions (2)

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.html (1)

Pattern src/main/webapp/**/*.html: @if and @for are new and valid Angular syntax replacing *ngIf and *ngFor. They should always be used over the old style.

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.ts (1)

Pattern src/main/webapp/**/*.ts: angular_style:https://angular.io/guide/styleguide;methods_in_html:false;lazy_loading:true;code_reuse:true;tests:meaningful;types:PascalCase;enums:PascalCase;funcs:camelCase;props:camelCase;no_priv_prefix:true;strings:single_quotes;localize:true;btns:functionality;links:navigation;icons_text:newline;labels:associate;code_style:arrow_funcs,curly_braces,open_braces_same_line,indent_4;memory_leak_prevention:true;routes:naming_schema;chart_framework:ngx-charts;responsive_layout:true

🔇 Additional comments (5)

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.html (4)

Line range hint 1-5: LGTM! Proper usage of @if directive.

The label visibility logic is well-implemented and follows the new Angular syntax guidelines by using @if instead of *ngIf.

---

Line range hint 6-10: LGTM! Clean tooltip implementation.

The tooltip rendering follows best practices with proper icon usage and accessibility considerations.

---

Line range hint 11-16: LGTM! Well-structured timezone warning.

The timezone warning implementation properly handles internationalization with parameter interpolation.

---

Line range hint 46-55: LGTM! Comprehensive validation feedback.

The validation messages are well-structured with proper internationalization and warning states.

src/main/webapp/app/shared/date-time-picker/date-time-picker.component.ts (1)

28-28: Verify template usage of hideLabelName.

The implementation looks good. Let's verify that the template is correctly using this property for conditional rendering.

✅ Verification successful

Let me gather more information to verify the template usage.

---

Template correctly uses hideLabelName for conditional rendering

The template properly uses hideLabelName() in a conditional statement to control the visibility of the label element. The implementation is correct as it shows the label only when labelName() is truthy and hideLabelName() is falsy:

@if (labelName() && !hideLabelName()) {
    <label for="date-input-field" class="form-control-label col">
        {{ labelName() }}
    </label>
}

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the template is correctly using hideLabelName for conditional rendering

# Test: Check if the template file exists and contains the expected condition
rg -l "date-time-picker.component.html" | xargs rg "hideLabelName"

Length of output: 107

---

Script:

#!/bin/bash
# Search for the template file and show its content
fd "date-time-picker.component.html" --type f --exec cat {}

Length of output: 2156

[BBesrour]

re-approve

[pzdr7]

Code

[SimonEntholzer]

re-approve