Lint things

.github/workflows/linting.yml

@@ -0,0 +1,31 @@
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Lint Project
+
+on:
+  push:
+    branches: [ "main", "release" ]
+  pull_request:
+    branches: [ "main", "release" ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up JDK
+      uses: actions/setup-java@v4
+      with:
+        java-version: '21'
+        distribution: 'temurin'
+        cache: maven
+    - name: Lint Project
+      run: mvn spotless:check

.github/workflows/maven.yml

@@ -6,7 +6,7 @@
 # separate terms of service, privacy policy, and support
 # documentation.
 
-name: Java CI with Maven
+name: Build and Run Tests
 
 on:
   push:

README.md

@@ -13,6 +13,20 @@ The Dictionary Project's goal is to create a unified process for enabling the fi
 of a query. It needs to work across our current projects, and, in anticipation of our push to productize PIC-SURE, it
 needs to be workable for new PIC-SURE installations.
 
+## Contribution
+
+### Requirements
+
+- Java 21
+- Maven
+
+### Setup
+
+1. Copy the linting commit hook:  
+`cp code-formatting/pre-commit.sh .git/hooks/pre-commit`
+2. Do a clean build of the project:  
+`mvn clean install`
+
 ## Usage
 
 ### Create Env File
@@ -39,47 +53,3 @@ docker exec -ti dictionary-db psql -U picsure dictionary
 ```shell
 docker compose up -d --build
 ```
-
-## Solution
-
-### UI
-
-The Dictionary Project's end result will be an overhaul of the PIC-SURE Query Builder tab. It will provide the user with
-three ways to find variables they wish to query on:
-
-- An expandable tree of variables and their concept paths
-- A series of facets on the page's left gutter
-- A search bar
-
-The tree of variables can be filtered using the facets or the search bar. Users might also need additional information
-to support the filtering options available to them. The UI will have modals that provide meta information about variables, facets, facet categories, and, in relevant environments, datasets.
-
-### API
-
-The frontend will be supported by a new Dictionary API. The API will have four core resources:
-
-- Concepts (`/concepts/`): Variables, the concept paths to them, and associated metadata. The concepts API will support listing (`/concepts/`), viewing details (`/concepts/<dataset>/<concept path>`), and viewing a tree (`/concepts/tree/<dataset>/<concept path>?depth=[1-9]+`). The listing API will return a paginated list of concepts; the response will also be filtered using a universal filter object. That object will be shared among all API endpoints, and will be
-  described later. The tree API returns a hierarchy of concept nodes descending from the requested node, limiting the response to a depth of `depth`. The details API will be used for viewing details such as meta fields, type information, related facets, and related harmonized variables.
-- Facets (`/facets/`): Facets, their categories, and associated metadata. The facets API will support listing (`/facets/`), and viewing details (`/facets/<facet title>/<facet id>`). The listing API will be filtered using the universal filter object. While facets are technically two-dimensional, their hierarchies are quite shallow and broad, so a traditional listing structure will be used, with no pagination. The details API will be used for viewing meta details, related facets, and potentially variable counts for a facet.
-- datasets (`/datasets/`): datasets, and their metadata. There datasets API will support viewing details (`/datasets/<dataset id>`)
-- updates (`/updates/`): update all business objects. This will not be a published, user facing API to start. Instead, it will be for ETL purposes. In the future, we may allow admin users access to this API so that they can update dictionaries in live environments.
-
-The universal filter object can be `POST`ed to filterable listing APIs to filter results. It has the following structure:
-
-```json
-{
-  "search": "search terms",
-  "facets": [
-    {
-      "facet_category": "facet title",
-      "facet_id": "facet id"
-    }
-  ]
-}
-```
-
-### Architecture
-
-The Dictionary project has a UI, API, database, and an ETL. The UI will be part of the unified, next generation monolith currently in progress. The API will be isolated to its own docker container, reachable via the proxy endpoints in the PIC-SURE API. The database will be handled on a per-project basis; in some instances it will make more sense to isolate the database to its own container, but in others, it will remain part of the PIC-SURE relational database monolith.
-
-The queries we will be making to the relational database are in many instances complex, and well outside the standards set out by SQL. This means that this project will likely be locked into whatever database we choose, as the tech debt for moving to a different RDMS would not be justifiable. We would like to use Postgres for this project.

src/main/java/edu/harvard/dbmi/avillach/dictionary/DictionaryApplication.java

@@ -6,8 +6,8 @@
 @SpringBootApplication
 public class DictionaryApplication {
 
-	public static void main(String[] args) {
-		SpringApplication.run(DictionaryApplication.class, args);
-	}
+    public static void main(String[] args) {
+        SpringApplication.run(DictionaryApplication.class, args);
+    }
 
 }

src/main/java/edu/harvard/dbmi/avillach/dictionary/concept/ConceptController.java

@@ -22,25 +22,19 @@ public class ConceptController {
     private Integer MAX_DEPTH;
 
 
-    public ConceptController(
-        @Autowired ConceptService conceptService
-    ) {
+    public ConceptController(@Autowired ConceptService conceptService) {
         this.conceptService = conceptService;
     }
 
 
     @PostMapping(path = "/concepts")
     public ResponseEntity<Page<Concept>> listConcepts(
-        @RequestBody Filter filter,
-        @RequestParam(name = "page_number", defaultValue = "0", required = false) int page,
+        @RequestBody Filter filter, @RequestParam(name = "page_number", defaultValue = "0", required = false) int page,
         @RequestParam(name = "page_size", defaultValue = "10", required = false) int size
     ) {
         PageRequest pagination = PageRequest.of(page, size);
-        PageImpl<Concept> pageResp = new PageImpl<>(
-            conceptService.listConcepts(filter, pagination),
-            pagination,
-            conceptService.countConcepts(filter)
-        );
+        PageImpl<Concept> pageResp =
+            new PageImpl<>(conceptService.listConcepts(filter, pagination), pagination, conceptService.countConcepts(filter));
 
         return ResponseEntity.ok(pageResp);
     }
@@ -52,35 +46,26 @@ public ResponseEntity<Page<Concept>> dumpConcepts(
     ) {
         PageRequest pagination = PageRequest.of(page, size);
         PageImpl<Concept> pageResp = new PageImpl<>(
-            conceptService.listDetailedConcepts(new Filter(List.of(), "", List.of()), pagination),
-            pagination,
+            conceptService.listDetailedConcepts(new Filter(List.of(), "", List.of()), pagination), pagination,
             conceptService.countConcepts(new Filter(List.of(), "", List.of()))
         );
 
         return ResponseEntity.ok(pageResp);
     }
 
     @PostMapping(path = "/concepts/detail/{dataset}")
-    public ResponseEntity<Concept> conceptDetail(
-        @PathVariable(name = "dataset") String dataset,
-        @RequestBody() String conceptPath
-    ) {
-        return conceptService.conceptDetail(dataset, conceptPath)
-            .map(ResponseEntity::ok)
-            .orElse(ResponseEntity.notFound().build());
+    public ResponseEntity<Concept> conceptDetail(@PathVariable(name = "dataset") String dataset, @RequestBody() String conceptPath) {
+        return conceptService.conceptDetail(dataset, conceptPath).map(ResponseEntity::ok).orElse(ResponseEntity.notFound().build());
     }
 
     @PostMapping(path = "/concepts/tree/{dataset}")
     public ResponseEntity<Concept> conceptTree(
-        @PathVariable(name = "dataset") String dataset,
-        @RequestBody() String conceptPath,
+        @PathVariable(name = "dataset") String dataset, @RequestBody() String conceptPath,
         @RequestParam(name = "depth", required = false, defaultValue = "2") Integer depth
     ) {
         if (depth < 0 || depth > MAX_DEPTH) {
             return ResponseEntity.badRequest().build();
         }
-        return conceptService.conceptTree(dataset, conceptPath, depth)
-            .map(ResponseEntity::ok)
-            .orElse(ResponseEntity.notFound().build());
+        return conceptService.conceptTree(dataset, conceptPath, depth).map(ResponseEntity::ok).orElse(ResponseEntity.notFound().build());
     }
 }

src/main/java/edu/harvard/dbmi/avillach/dictionary/concept/ConceptFilterQueryGenerator.java

@@ -34,40 +34,35 @@ AND concept_node_meta.KEY IN (:disallowed_meta_keys)
         """;
 
     private static final String CONSENT_QUERY = """
-                dataset.dataset_id IN (
-                    SELECT
-                        consent.dataset_id
-                    FROM consent
-                        LEFT JOIN dataset ON dataset.dataset_id = consent.dataset_id
-                    WHERE
-                        concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
-                        (dataset.ref IN (:consents) AND consent.consent_code = '')
-                    UNION
-                    SELECT
-                        dataset_harmonization.harmonized_dataset_id
-                    FROM consent
-                        JOIN dataset_harmonization ON dataset_harmonization.source_dataset_id = consent.dataset_id
-                        LEFT JOIN dataset ON dataset.dataset_id = dataset_harmonization.source_dataset_id
-                    WHERE
-                        concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
-                        (dataset.ref IN (:consents) AND consent.consent_code = '')
-                ) AND
-                """;
+        dataset.dataset_id IN (
+            SELECT
+                consent.dataset_id
+            FROM consent
+                LEFT JOIN dataset ON dataset.dataset_id = consent.dataset_id
+            WHERE
+                concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
+                (dataset.ref IN (:consents) AND consent.consent_code = '')
+            UNION
+            SELECT
+                dataset_harmonization.harmonized_dataset_id
+            FROM consent
+                JOIN dataset_harmonization ON dataset_harmonization.source_dataset_id = consent.dataset_id
+                LEFT JOIN dataset ON dataset.dataset_id = dataset_harmonization.source_dataset_id
+            WHERE
+                concat(dataset.ref, '.', consent.consent_code) IN (:consents) OR
+                (dataset.ref IN (:consents) AND consent.consent_code = '')
+        ) AND
+        """;
 
     @Autowired
-    public ConceptFilterQueryGenerator(
-        @Value("${filtering.unfilterable_concepts}") List<String> disallowedMetaFields
-    ) {
+    public ConceptFilterQueryGenerator(@Value("${filtering.unfilterable_concepts}") List<String> disallowedMetaFields) {
         this.disallowedMetaFields = disallowedMetaFields;
     }
 
     /**
-     * This generates a query that will return a list of concept_node IDs for the given filter.
-     * <p>
-     * A filter object has a list of facets, each belonging to a category. Within a category,
-     * facets are ORed together. Between categories, facets are ANDed together.
-     * In SQL, this is represented as N clauses for N facets, all INTERSECTed together. Search
-     * also acts as its own special facet here.
+     * This generates a query that will return a list of concept_node IDs for the given filter. <p> A filter object has a list of facets,
+     * each belonging to a category. Within a category, facets are ORed together. Between categories, facets are ANDed together. In SQL,
+     * this is represented as N clauses for N facets, all INTERSECTed together. Search also acts as its own special facet here.
      *
      * @param filter universal filter object for the page
      * @param pageable pagination, if applicable
@@ -113,8 +108,7 @@ ORDER BY max((1 + rank) * coalesce(rank_adjustment, 1)) DESC, q.concept_node_id
                 LIMIT :limit
                 OFFSET :offset
                 """;
-            params.addValue("limit", pageable.getPageSize())
-                .addValue("offset", pageable.getOffset());
+            params.addValue("limit", pageable.getPageSize()).addValue("offset", pageable.getOffset());
         }
 
         superQuery = " concepts_filtered_sorted AS (\n" + superQuery + "\n)";
@@ -150,26 +144,26 @@ private String createValuelessNodeFilter(String search, List<String> consents) {
                     continuous_max.value <> '' OR
                     categorical_values.value <> ''
                 )
-            """.formatted(rankQuery, rankWhere, consentWhere);
+            """
+            .formatted(rankQuery, rankWhere, consentWhere);
     }
 
     private List<String> createFacetFilter(Filter filter, MapSqlParameterSource params) {
         String consentWhere = CollectionUtils.isEmpty(filter.consents()) ? "" : CONSENT_QUERY;
-        return filter.facets().stream()
-            .collect(Collectors.groupingBy(Facet::category))
-            .entrySet().stream()
-            .map(facetsForCategory ->  {
-                params
-                    // The templating here is to namespace the params for each facet category in the query
-                    .addValue("facets_for_category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getValue().stream().map(Facet::name).toList())
-                    .addValue("category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getKey());
-                String rankQuery = "0";
-                String rankWhere = "";
-                if (StringUtils.hasLength(filter.search())) {
-                    rankQuery = "ts_rank(searchable_fields, (phraseto_tsquery(:search)::text || ':*')::tsquery)";
-                    rankWhere = "concept_node.searchable_fields @@ (phraseto_tsquery(:search)::text || ':*')::tsquery AND";
-                }
-                return """
+        return filter.facets().stream().collect(Collectors.groupingBy(Facet::category)).entrySet().stream().map(facetsForCategory -> {
+            params
+                // The templating here is to namespace the params for each facet category in the query
+                .addValue(
+                    "facets_for_category_%s".formatted(facetsForCategory.getKey()),
+                    facetsForCategory.getValue().stream().map(Facet::name).toList()
+                ).addValue("category_%s".formatted(facetsForCategory.getKey()), facetsForCategory.getKey());
+            String rankQuery = "0";
+            String rankWhere = "";
+            if (StringUtils.hasLength(filter.search())) {
+                rankQuery = "ts_rank(searchable_fields, (phraseto_tsquery(:search)::text || ':*')::tsquery)";
+                rankWhere = "concept_node.searchable_fields @@ (phraseto_tsquery(:search)::text || ':*')::tsquery AND";
+            }
+            return """
                 (
                     SELECT
                         facet__concept_node.concept_node_id AS concept_node_id,
@@ -187,8 +181,7 @@ facet.name IN (:facets_for_category_%s ) AND facet_category.name = :category_%s
                         facet__concept_node.concept_node_id
                 )
                 """.formatted(rankQuery, rankWhere, consentWhere, facetsForCategory.getKey(), facetsForCategory.getKey());
-            })
-            .toList();
+        }).toList();
     }
 
 }