[REPOSITORY] New Repository Strimzi Topic Integration connector

[davidradl]

Name

egeria-connector-integration-topic-strimzi

Owner

davidradl

Deliverable

Provides an integration connector to bring in Kafka Topics by monitoring Strimzi custom resource definitions

Build, test and CI-CD process

answering the questions in order
Same as for the crux
Gradle
Java
I suspect there already is one
Same as for Egeria core
Nothing out of the ordinary
Not needed

Dependencies

The is the gradle.build file we have used for testing
`/*

• SPDX-License-Identifier: Apache-2.0
• Copyright Contributors to the ODPi Egeria project.
  */

repositories {
mavenCentral()
}
group = 'org.odpi.egeria'
version = 1
configurations {
extralibs
}
apply plugin: 'java-library'
ext {
egeriaversion = '3.2'
logbackVersion = '1.2.6'
jacksonVersion = '2.13.0'
junitjupiterVersion = '5.8.1'
}
dependencies {
implementation "org.slf4j:slf4j-api:${logbackVersion}"
implementation "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"
implementation "com.fasterxml.jackson.core:jackson-annotations:${jacksonVersion}"
implementation "com.fasterxml.jackson.core:jackson-core:${jacksonVersion}"
implementation "org.junit.jupiter:junit-jupiter:${junitjupiterVersion}"
implementation "org.odpi.egeria:open-connector-framework:${egeriaversion}"
implementation "org.odpi.egeria:audit-log-framework:${egeriaversion}"
implementation "org.odpi.egeria:repository-services:${egeriaversion}"
implementation "org.odpi.egeria:repository-services-apis:${egeriaversion}"
implementation "org.odpi.egeria:topic-integrator-api:${egeriaversion}"
implementation "org.odpi.egeria:kafka-open-metadata-topic-connector:${egeriaversion}"
extralibs 'org.apache.httpcomponents:httpclient:4.5.13'
configurations.compile.extendsFrom(configurations.extralibs)
}

test {
useJUnitPlatform()
dependencies {
testCompile("org.junit.jupiter:junit-jupiter-api:5.7.0")
testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.7.0"
testImplementation "org.junit.jupiter:junit-jupiter-params:5.7.0"
testImplementation("org.junit.jupiter:junit-jupiter-api:5.7.0")
}
testLogging {
events "passed", "skipped", "failed"
}
reports {
html.enabled = true
}
filter {
includeTestsMatching "*Test"
}
}

// Maven Central (technically sonatype oss) requires we distribute source and javadoc
java {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
// More Java language settings
tasks.withType(JavaCompile) {
options.encoding = 'UTF-8'
sourceCompatibility = "11"
targetCompatibility = "11"
options.incremental = true
options.fork = true
options.failOnError = true
options.compilerArgs << '-Xlint:unchecked' << '-Xlint:deprecation'
}

jar {
from {
configurations.extralibs.filter{it.exists()}
.collect{it.isDirectory() ? it : zipTree(it)}
}
}`

Justification

There is no obvious repo to put this in

Assumptions

Yes all true.

Additional Information

odpi/egeria#6021

Work Plan

Before creating the repo

• review overall request & get clarifications
• get approval on developer/TSC call from maintainers

Creating the repo

• Create the repo under the odpi organization (default gitignore, license, readme)
• Setup branch protection rules
• Set pull request options (allow merge, squash, rebase, suggest updating, allow automerge, do not delete head)
• Update security settings in repo (policy, advisories, alerts)
• set up permissions

First steps

• Initial code-drop (author)

Getting CI/CD started & refining settings

• Add initial build script for PR (including gradle wrapper if required)
• Add initial build script for merge

Further Refinement

• Add link in egeria docs to new repo describing purpose etc
• Add required credentials for publishing to container repos, maven central etc
• Add artifact signing if needed
• Add dependabot config
• Add CodeQL
• Add to LFXSecurity
• Add to LFAnalytics
• Add check for stale defects
• Add standard issue tags

Release

• Add release pipeline

[planetf1]

Thanks for the info.

Note that this work was already pre-approved at a previous developer/maintainer call.

Let's go ahead with that name, we can revise if required when we discuss additional connectors.

[planetf1]

Have started, but best to leave repo as-is until the initial setup is finished (may be tomorrow am)

[planetf1]

@davidradl I think we have enough for you to add the first code drop (or more).
Once there's some code in, I'll look at the buid side - at least initial + code scans.
Do you intend to migrate from an existing repository ? If so it's possible to migrate the history if needed?

[davidradl]

Hi Nigel, we do not want to migrate the history as it contains information that should not be public. I will look at putting the initial code in

[planetf1]

Ok. Once you have something that could potentially build, and it's cheked in, I'll look at the workflows, gradle wrapper, - or you can give them a go if you prefer. Inevitable it will involve a few stages. We'll also need to setup artifact creation/signing (as was done in the postgres repo)

[planetf1]

Quick update - I've ticked off the completed tasks. Builds are running for pr/merge with checks, though I need to fix an issue with publishing the artifacts to maven central. Also still need to verify the built jar (dependencies) and a few doc/external updates to do.

[planetf1]

Since we now have the new repo - let's move this issue over there to track the initial setup tasks..

[planetf1]

Artifacts now being built, signed, & distributed to maven central - or rather the snapshot repo

There's been a fair amount of gradle refactoring, dependencies, additions around source/javadoc/signing/publishing (including more metadata), and the name of the artifact itself.

I've not verified the dependencies in the uber jar yet, and there's a bit more possible cleanup - but it's a good start

There is a release pipeline in place, but untested

Have to finish off some of the docs/lfx security updates, but please feel free to take a look @davidradl

[planetf1]

If using maven/gradle you can retrieve this artifact as long as the snapshot repo in in your repo list.

Via http you can retrieve via wget

wget https://oss.sonatype.org/service/local/artifact/maven/redirect\?r\=snapshots\&g\=org.odpi.egeria\&a\=egeria-connector-integration-topic-strimzi\&v\=1.0-SNAPSHOT\&c\=jar-with-dependencies -O egeria-connector-integration-topic-strimzi.jar-with-dependencies.jar

You can also get the artifact by going to the build under 'Actions' (recommend 'merge') and seeing the attachment

No docker build (egeria+connector) for now - I think it's unnecessary for now, but easy to do if required

[davidradl]

@planetf1 can we close - or is there more you want todo on this ?

[planetf1]

@davidradl there are still a few actions - see tasklist above

[planetf1]

Additional task

• Add to connector catalog

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

still some outstanding tasks on docs + linux foundation

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

Move to docs to ensure all documented there

[planetf1]

• Add to connector catalog
• Add to release process docs

[planetf1]

@juergenhemelt Can you handle this? This is just some missing docs -- documenting the connector in the connector catalog & documenting the release process.