Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

N21-2103 update media activations #5364

Open
wants to merge 81 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
dd8bdff
N21-2103 WIP
mrikallab Nov 27, 2024
f250d37
init add vidis sync strategy
sdinkov Nov 29, 2024
07ecc98
add vidis console job script
sdinkov Nov 29, 2024
9490543
init add ansible role for vidis sync cron job
sdinkov Nov 29, 2024
1f113ce
N21-2103 mediasource module
mrikallab Dec 3, 2024
ba85962
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
mrikallab Dec 3, 2024
69f3295
rename role
sdinkov Dec 4, 2024
3a72046
update job definition
sdinkov Dec 4, 2024
80f9e23
Merge branch 'main' into N21-2103-update-media-activations
sdinkov Dec 4, 2024
f79a88b
N21-2103 mediasource module imports
mrikallab Dec 4, 2024
186ce64
Merge branch 'main' into N21-2103-update-media-activations
mrikallab Dec 4, 2024
ade04ee
update job scheduling
sdinkov Dec 4, 2024
3638343
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
mrikallab Dec 4, 2024
cf0143b
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
mrikallab Dec 4, 2024
82e5614
test scheduling
sdinkov Dec 4, 2024
3dc4521
format job
sdinkov Dec 4, 2024
d468f02
update job
sdinkov Dec 4, 2024
a9643b8
update job formatting
sdinkov Dec 4, 2024
2bf9f64
update job: add deployment and service
sdinkov Dec 5, 2024
294696c
update vidis sync job
sdinkov Dec 5, 2024
2933de5
update vidis job
sdinkov Dec 5, 2024
c693c61
update cronjob
sdinkov Dec 5, 2024
23bfe2d
update vidis job
sdinkov Dec 5, 2024
2ee912e
update vidis cronjob
sdinkov Dec 5, 2024
bf32018
update cronjob vidis
sdinkov Dec 5, 2024
cd9a6f5
Merge branch 'main' into N21-2103-update-media-activations
sdinkov Dec 5, 2024
4b8ffc2
update cronjob
sdinkov Dec 5, 2024
96e4277
Merge branch 'N21-2103-update-media-activations' of https://github.co…
sdinkov Dec 5, 2024
a5f9ea1
update job
sdinkov Dec 5, 2024
280c5c7
update vidis job
sdinkov Dec 5, 2024
98f1401
update job schedule
sdinkov Dec 5, 2024
06b2d6f
fix conflict
sdinkov Dec 5, 2024
3b1a028
update vidis sync strategy; init test
sdinkov Dec 6, 2024
36fe5a2
apply suggestions from code review
sdinkov Dec 6, 2024
6589cde
Merge branch 'N21-2103-update-media-activations' of https://github.co…
sdinkov Dec 6, 2024
85054a2
N21-2103 sync logic
mrikallab Dec 9, 2024
af43da2
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
mrikallab Dec 9, 2024
5cefd4f
N21-2103 wip
mrikallab Dec 9, 2024
1a891a6
Merge branch 'main' into N21-2103-update-media-activations
mrikallab Dec 9, 2024
ca9eb0b
N21-2103 WIP
mrikallab Dec 10, 2024
1db0b10
N21-2103 fix request
mrikallab Dec 10, 2024
1e8997a
N21-2103 WIP
mrikallab Dec 11, 2024
2bfa55a
cleanup media school license service
sdinkov Dec 11, 2024
ef1b547
fix for school media licences
sdinkov Dec 11, 2024
da91fc1
cleanup
sdinkov Dec 11, 2024
159148f
update job schedule
sdinkov Dec 11, 2024
1196424
update tests
sdinkov Dec 11, 2024
eeb638b
N21-2103 add vidis to api-init with encryption, placeholder files for…
GordonNicholasCap Dec 12, 2024
847dc2a
Merge remote-tracking branch 'refs/remotes/origin/main' into N21-2103…
GordonNicholasCap Dec 12, 2024
31b4d37
N21-2103 wip tests, mark some code for changes
GordonNicholasCap Dec 12, 2024
c744a29
server-init review changes
GordonNicholasCap Dec 13, 2024
9651e8a
N21-2103 renamed module mediasource -> media-source, fixed imports
GordonNicholasCap Dec 13, 2024
7ce3274
N21-2103 vidis-sync service unit test
GordonNicholasCap Dec 13, 2024
a564ea1
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
GordonNicholasCap Dec 13, 2024
81d7e80
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 13, 2024
71c8853
N21-2103 fix missing imports
GordonNicholasCap Dec 13, 2024
6dae8fc
Merge remote-tracking branch 'origin/N21-2103-update-media-activation…
GordonNicholasCap Dec 13, 2024
8b191f3
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 16, 2024
78113e1
N21-2103 fix media license repo test
GordonNicholasCap Dec 16, 2024
1a3f65f
N21-2103 wip refactor license sync code, fix eslint errors
GordonNicholasCap Dec 16, 2024
33b06d6
N21-2103 wip sync service test
GordonNicholasCap Dec 17, 2024
07bdc4c
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 17, 2024
de94936
N21-2103 finish vidis sync service refactoring, fix tests
GordonNicholasCap Dec 18, 2024
3200b50
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 18, 2024
e423520
Merge branch 'main' into N21-2103-update-media-activations
sdinkov Dec 18, 2024
317ef9a
N21-2103 cleanup, more tests
GordonNicholasCap Dec 18, 2024
260d280
N21-2103 cleanup, add missing unit tests
GordonNicholasCap Dec 19, 2024
47684f6
N21-2103 cleanup, use repo-interface, adjust tests
GordonNicholasCap Dec 19, 2024
e8b10e9
N21-2103 wip vidis response test coverage
GordonNicholasCap Dec 19, 2024
b693121
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 19, 2024
b88a030
test coverage
GordonNicholasCap Dec 20, 2024
30db6c8
add vidis client
sdinkov Dec 20, 2024
06a973a
update client api
sdinkov Dec 20, 2024
2fe346b
add client factory and init test
sdinkov Dec 20, 2024
650cb18
N21-2103 adapt vidis sync code to use vidis client
GordonNicholasCap Dec 20, 2024
7a6b7f8
N21-2103 add config for vidis region
GordonNicholasCap Dec 20, 2024
2a12aa9
Merge branch 'main' into N21-2103-update-media-activations
GordonNicholasCap Dec 20, 2024
ab54ba5
N21-2103 fix tests and eslint
GordonNicholasCap Dec 20, 2024
009a024
N21-2103 fix configs
GordonNicholasCap Dec 20, 2024
72c6ce2
N21-2103 fix configs
GordonNicholasCap Dec 20, 2024
68c6152
N21-2103 fix cronjob tasks condition
GordonNicholasCap Dec 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ansible/roles/media-licenses/defaults/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SERVER_VIDIS_SYNC_CRONJOB_SCHEDULE: "20 4 * * *"
9 changes: 9 additions & 0 deletions ansible/roles/media-licenses/meta/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
galaxy_info:
role_name: media-licenses
author: Schul-Cloud Verbund
description: media-licenses role for the vidis synchronization purposes
company: Schul-Cloud Verbund
license: license (AGPLv3)
min_ansible_version: 2.8
galaxy_tags: []
dependencies: []
19 changes: 19 additions & 0 deletions ansible/roles/media-licenses/tasks/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
- name: media licenses sync CronJob ConfigMap
when: WITH_MEDIA_LICENSES is defined and WITH_MEDIA_LICENSES|bool == true
kubernetes.core.k8s:
kubeconfig: ~/.kube/config
namespace: "{{ NAMESPACE }}"
template: vidis-sync-cronjob-configmap.yml.j2
state: "{{ 'present' if WITH_MEDIA_LICENSES is defined and WITH_MEDIA_LICENSES|bool else 'absent'}}"
tags:
- configmap

- name: media licenses sync CronJob
when: WITH_MEDIA_LICENSES is defined and WITH_MEDIA_LICENSES|bool == true
kubernetes.core.k8s:
kubeconfig: ~/.kube/config
namespace: "{{ NAMESPACE }}"
template: vidis-sync-cronjob.yml.j2
sdinkov marked this conversation as resolved.
Show resolved Hide resolved
state: "{{ 'present' if WITH_MEDIA_LICENSES is defined and WITH_MEDIA_LICENSES|bool else 'absent'}}"
tags:
- cronjob
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: v1
kind: ConfigMap
metadata:
namespace: {{ NAMESPACE }}
name: vidis-sync-cronjob-configmap
labels:
app: vidis-sync-cronjob
data:
NODE_OPTIONS: "--max-old-space-size=3072"
NEST_LOG_LEVEL: "error"
EXIT_ON_ERROR: "true"
VIDIS_API_CLIENT_BASE_URL: "{{ VIDIS_API_CLIENT_BASE_URL }}"
VIDIS_SYNC_REGION: "{{ VIDIS_SYNC_REGION }}"
96 changes: 96 additions & 0 deletions ansible/roles/media-licenses/templates/vidis-sync-cronjob.yml.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
apiVersion: batch/v1
kind: CronJob
metadata:
namespace: {{ NAMESPACE }}
labels:
app: vidis-sync-cronjob
app.kubernetes.io/part-of: schulcloud-verbund
app.kubernetes.io/version: {{ SCHULCLOUD_SERVER_IMAGE_TAG }}
app.kubernetes.io/name: vidis-sync-cronjob
app.kubernetes.io/component: sync
app.kubernetes.io/managed-by: ansible
git.branch: {{ SCHULCLOUD_SERVER_BRANCH_NAME }}
git.repo: {{ SCHULCLOUD_SERVER_REPO_NAME }}
name: vidis-sync-cronjob
spec:
schedule: {{ SERVER_VIDIS_SYNC_CRONJOB_SCHEDULE|default("*/7 * * * *", true) }}
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- name: vidis-sync-cronjob
image: {{ SCHULCLOUD_SERVER_IMAGE }}:{{ SCHULCLOUD_SERVER_IMAGE_TAG }}
envFrom:
- configMapRef:
name: api-configmap
- secretRef:
name: api-secret
- configMapRef:
name: vidis-sync-cronjob-configmap
command: ['/bin/sh','-c']
args: ['npm run nest:start:sync:vidis']
resources:
limits:
cpu: {{ MEDIA_ACTIVATION_CPU_LIMITS|default("2000m", true) }}
memory: {{ MEDIA_ACTIVATION_MEMORY_LIMITS|default("2Gi", true) }}
requests:
cpu: {{ MEDIA_ACTIVATION_CPU_REQUESTS|default("100m", true) }}
memory: {{ MEDIA_ACTIVATION_MEMORY_REQUESTS|default("150Mi", true) }}
restartPolicy: OnFailure
{% if AFFINITY_ENABLE is defined and AFFINITY_ENABLE|bool %}
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 20
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/part-of
operator: In
values:
- schulcloud-verbund
topologyKey: "kubernetes.io/hostname"
namespaceSelector: {}
- weight: 10
podAffinityTerm:
labelSelector:
matchExpressions:
- key: git.repo
operator: In
values:
- {{ SCHULCLOUD_SERVER_REPO_NAME }}
topologyKey: "kubernetes.io/hostname"
namespaceSelector: {}
- weight: 10
podAffinityTerm:
labelSelector:
matchExpressions:
- key: git.branch
operator: In
values:
- {{ SCHULCLOUD_SERVER_BRANCH_NAME }}
topologyKey: "kubernetes.io/hostname"
namespaceSelector: {}
- weight: 10
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/version
operator: In
values:
- {{ SCHULCLOUD_SERVER_IMAGE_TAG }}
topologyKey: "kubernetes.io/hostname"
namespaceSelector: {}
{% endif %}
metadata:
labels:
app: vidis-sync-cronjob
app.kubernetes.io/part-of: schulcloud-verbund
app.kubernetes.io/version: {{ SCHULCLOUD_SERVER_IMAGE_TAG }}
app.kubernetes.io/name: vidis-sync-cronjob
app.kubernetes.io/component: sync
app.kubernetes.io/managed-by: ansible
git.branch: {{ SCHULCLOUD_SERVER_BRANCH_NAME }}
git.repo: {{ SCHULCLOUD_SERVER_REPO_NAME }}
Original file line number Diff line number Diff line change
Expand Up @@ -611,5 +611,31 @@ data:
fi
# ========== End of TSP system creation

# ========== Start of inserting VIDIS media source section.
if [ -n "$MEDIA_SOURCE_VIDIS_USERNAME" ] && [ -n "$MEDIA_SOURCE_VIDIS_PASSWORD" ]; then
echo "Inserting VIDIS media source to the DB..."
# Encrypt the credentials for accessing the VIDIS media source
MEDIA_SOURCE_VIDIS_USERNAME=$(node scripts/secret.js -s $AES_KEY -e $MEDIA_SOURCE_VIDIS_USERNAME)
MEDIA_SOURCE_VIDIS_PASSWORD=$(node scripts/secret.js -s $AES_KEY -e $MEDIA_SOURCE_VIDIS_PASSWORD)
mongosh $DATABASE__URL --quiet --eval 'db.getCollection("media-sources").insertOne(
{
"format" : "VIDIS",
"name" : "vidis",
"sourceId" : "vidis.fwu.de",
"createdAt": new Date(),
"updatedAt": new Date(),
"basicAuthConfig" : {
"username" : "'$MEDIA_SOURCE_VIDIS_USERNAME'",
"password" : "'$MEDIA_SOURCE_VIDIS_PASSWORD'",
"authEndpoint" : "https://service-stage.vidis.schule/o/vidis-rest/v1.0/offers/activated/by-region/test-region",
"_id" : new ObjectId()
}
}
);'
echo "Inserted VIDIS media source to the DB."
fi

# ========== End of inserting VIDIS media source section.

# Database indexes synchronization, it's crucial until we have all the entities in NestJS app.
npm run syncIndexes
4 changes: 3 additions & 1 deletion apps/server/src/infra/sync/console/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ npm run nest:start:console sync run <target>

Where `<target>` is the name of the system you want to start the synchronization for. The currently available systems are:
- `tsp` - Synchronize Thüringer schulportal.
- `vidis` - Synchronize Vidis Activation Data.


If the target is not provided, the synchronization will not start and the available targets will be displayed in an error message.
```bash
{
message: 'Either synchronization is not activated or the target entered is invalid',
data: { enteredTarget: 'tsp', availableTargets: { TSP: 'tsp' }}
data: { enteredTarget: 'tsp', availableTargets: { TSP: 'tsp', VIDIS: 'vidis' }}
}
```

Expand Down
1 change: 1 addition & 0 deletions apps/server/src/infra/sync/index.ts
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
export * from './tsp';
export * from './media-licenses';
3 changes: 3 additions & 0 deletions apps/server/src/infra/sync/media-licenses/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export * from './service';
export * from './strategy';
export * from './vidis-sync-config';
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export { SchoolForSchoolMediaLicenseSyncNotFoundLoggable } from './school-for-school-media-license-sync-not-found.loggable';
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { SchoolForSchoolMediaLicenseSyncNotFoundLoggable } from './school-for-school-media-license-sync-not-found.loggable';

describe(SchoolForSchoolMediaLicenseSyncNotFoundLoggable.name, () => {
describe('getLogMessage', () => {
const setup = () => {
const officialSchoolNumber = '00100';
const exception = new SchoolForSchoolMediaLicenseSyncNotFoundLoggable(officialSchoolNumber);

return { exception, officialSchoolNumber };
};

it('should return the correct log message', () => {
const { exception, officialSchoolNumber } = setup();

const logMessage = exception.getLogMessage();

expect(logMessage).toEqual({
message: 'Unable to sync media school license, because school cannot be found.',
data: {
officialSchoolNumber,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { ErrorLogMessage, Loggable, LogMessage, ValidationErrorLogMessage } from '@src/core/logger';

export class SchoolForSchoolMediaLicenseSyncNotFoundLoggable implements Loggable {
constructor(private readonly officialSchoolNumber: string) {}

getLogMessage(): LogMessage | ErrorLogMessage | ValidationErrorLogMessage {
return {
message: 'Unable to sync media school license, because school cannot be found.',
data: {
officialSchoolNumber: this.officialSchoolNumber,
},
};
}
}
2 changes: 2 additions & 0 deletions apps/server/src/infra/sync/media-licenses/service/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
export { VidisSyncService } from './vidis-sync.service';
export { VidisFetchService } from './vidis-fetch.service';
Loading
Loading