Skip to content

Commit

Permalink
odpi#643 elaborate on gradle build suggestions
Browse files Browse the repository at this point in the history
Signed-off-by: Nigel Jones <nigel.l.jones+git@gmail.com>
  • Loading branch information
planetf1 committed Dec 22, 2022
1 parent dd8aa87 commit 36ab405
Showing 1 changed file with 122 additions and 24 deletions.
146 changes: 122 additions & 24 deletions site/docs/guides/developer/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,41 +225,139 @@ The modules for each registered service that need to run in the OMAG Server Plat

The best guide for building registered services are the existing implementations found in [egeria.git](https://github.com/odpi/egeria/tree/main/open-metadata-implementation){ target=gh }. You can see the way the code is organized and the services that they depend on.

## Build considerations
## Build principles

Egeria's best practise is to use [Gradle](https://gradle.org/) to build the Egeria extensions.
### Which build tool?
Egeria's approach is to use [Gradle](https://gradle.org/){ target = gradle } to build Egeria extensions, and Egeria itself.

Principles around constructing the Gradle build file:
We are migrating our own repositories over to Gradle as we see the following [benefits](https://gradle.org/maven-vs-gradle/){ target = gradle }:

- build against mavenCentral and snapshots if required.
- Much quicker builds due to caching, incremental compile,change detection, parallelism
- better flexibility in defining build tasks
- A more vibrant community with current [discussions](https://discuss.gradle.org){ target = gradle }
-
Our examples will focus on Gradle, but you may also use [Maven](https://maven.apache.org){ target = maven}.


### Use stable versions from official repositories

Egeria core libraries, and other java-oriented artifacts created by the Egeria team are published to [Maven Central](https://search.maven.org){ target = maven } (technically this is in fact just an index)

- Code releases (for example '3.14') go to the *Releases* repository, and will typically be picked up by default by Java build tools
- Snaphots, which are typically produced by Egeria projects as part of code merge, go to the *Snapshots* repository. This typically needs configuration in a build file

Additionally, when compilation is done locally, a local repository may be populated with artifacts. It is not recommended to use a local build for dependencies in code you develop unless working on core Egeria libraries.

It is recommended to:

- Depend on releases if possible (ie unless requiring a new feature/fix in code being developed). This is *essential* if your extension is to be stable and read to use
- Use snapshots if requiring unreleased artifacts, but be aware that the contents can change, or go away
- Only use a local repository if actively making simultaneous changes to multiple repos, for example if developing a new OMAS alongside an integration connector. It's inadvisable to check in code with any dependency on local repositories or jars as behaviour will differ depending on the local environment.

### Example build.gradle fragments

```
repositories {
repositories {
// If only using mavenCentral you likely will not need a repository section at all
mavenCentral()
maven { url("https://oss.sonatype.org/content/repositories/snapshots") }
// uncomment to pick up from local ~/.m2 - but can be unpredictable. recommend to use maven central and snapshots
// mavenLocal()
}
// Uncomment to use snapshots ie egeria 3.15-SNAPSHOT
//maven { url("https://oss.sonatype.org/content/repositories/snapshots") }
// Uncomment to pick up from local ~/.m2 - but can be unpredictable. recommend to use maven central and snapshots
// mavenLocal()
}
```

If using repositories that may change contents without version - such as snapshots or local repo, ensure gradle always picks up current contents. Not needed when using stable versions.
```
configurations.all {
// check for updates every build - critical when using Snapshots
resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}
```
## Keep up to date

Aim to keep new connectors, applications etc up to date with Egeria releases, and ensure the version is current when initially creating and contributing

If your code is hosted on GitHub, [Dependabot](https://github.com/dependabot){ target = gh } is a useful tool for keeping all your dependencies up to date, including on Egeria artifacts.

## Use dependency constraints, especially for multi module projects

[Dependency constraints](https://docs.gradle.org/current/userguide/dependency_constraints.html){ target = gradle } help ensure that anytime you use a dependency in your project, it's the same version, especially in the case of transitive dependencies.

In a [multi-module project](https://docs.gradle.org/current/userguide/multi_project_builds.html){ target = gradle } they should go in the top level `build.gradle` within `AllProjects()` or `SubProjects()`

A [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html){ target = gradle} can then be referenced without a version

### Example build.gradle fragment
```
# Use variables to keep things clear
ext {
egeriaversion = '3.12'
thriftversion = '0.13.0
}
# Setup constraints
dependencies {
constraints {
// this will ensure a consistent version used whenever we refer to this dependency
implementation "org.odpi.egeria:audit-log-framework:${egeriaversion}"
implementation 'org.apache.thrift:libthrift${thriftversion}'
}
// This will inherit from our egeria dependency constraints
implementation platform("org.odpi.egeria:egeria:${egeriaversion}")
}
```

Later when using a dependency
```
dependencies
{
compileOnly "org.odpi.egeria:open-connector-framework"
}
```
## Minimise chance for duplicate contents

This is primarily a concern for building connectors, or other components that run within the Egeria server chassis. In this environment we already have a lot of Egeria code on the classpath so ideally we want to only add what's needed, not duplicate what is already there.

This is to get a balance of:

- ease of use in deploying the connector
- avoid clashing library versions
- minimize size of artifact

Some terms that you may hear of

- 'thin' jar - often taken to mean just your code, ie no dependencies
- 'fat' or 'uber' jar - often taken to mean your code and all it's dependencies.

To achieve a sensible balance what we actually want is to:

- include your code
- omit anything already in the server chassis
- add in other dependencies

To do this, you will often want to build a jar with dependencies, but with care taken over use of 'scope'. Often this will mean Egeria dependencies will be `compileOnly` or `testCompileOnly` (The equivalent in maven is 'provided').

If this is not done, there's a risk of having two different versions of core libraries present in the classpath. This could cause either your code, or other Egeria code to malfunction.

Careful scoping combined with maintaining currency, minimizes this risk.

Other techniques to avoid this issue include:

- Using sharding to rename classes
- Using a dedicated class loader (this requires framework place, and not supported by Egeria at this time)
### Example build.gradle fragment
```
compileOnly "org.odpi.egeria:open-connector-framework"
implementation 'org.apache.thrift:libthrift"
```
## Complex dependency manipulation

It is possible to pick up content from local Maven by uncommenting in the above snippet. This may be required during development if the dependant change is not in a snapshot or release.
For example your new Egeria connector requires changes to an OMAS that you are also developing, but have not yet been merged.

- versions should be defined as variables
- the Egeria version should up to date when the extension is originally contributed. This version and other associated technology versions can be
updated over time if required.
- We are looking to create 'thin' jar, with the minimal content, to avoid bringing in duplicate classes; remember many connectors could be running in the same platform bring in classes to the environment.
- dependencies on artifacts that will already be in the environment for example Egeria jar files that are brought in with the OMAG platform,
should not be brought in again as part of the extension build. Often the approach to take is to use 'compileOnly' on these dependencies.
For dependencies that are not already going to be in the runtime environment then using 'implementation' often makes sense. If you are using junits
in your projects you should follow these principles, but use the appropriate test orientated keywords.
- If you need to have a dependency, but it has vulnerabilities, then may need to exclude libraries with vulnerabilities; for example if they being brought in, but your code is not using them.
For an example on how this has been done see [the HMS connector](https://github.com/odpi/egeria-connector-hivemetastore/blob/main/build.gradle).
For an example on how this has been done see [the HMS connector](https://github.com/odpi/egeria-connector-hivemetastore/blob/main/build.gradle){ target = gh }.
- If you need to pick up a dependency, but there are multiple versions around at runtime, then this build file uses `forceResolution` strategy to force
a particular dependency version; for an example of this see [the HMS connector](https://github.com/odpi/egeria-connector-hivemetastore/blob/main/build.gradle).
- you need to ensure that appropriate versions of jars are available at runtime so that classes can be found.
a particular dependency version; for an example of this see [the HMS connector](https://github.com/odpi/egeria-connector-hivemetastore/blob/main/build.gradle){ target = gh }.
- you need to ensure that appropriate versions of jars are available at runtime so that classes can be found.



## Summary

Expand Down

0 comments on commit 36ab405

Please sign in to comment.