Players Lab Project

Spring lab project with Java Enterprise Stack.

Overview

This project is a Spring Boot application designed to load player data from a CSV file into an H2 database and expose the data through three asynchronous endpoints using Spring WebFlux.
The application reads the CSV file in chunks, processes each chunk asynchronously, and stores the data in the database for efficient and scalable performance.

Features

1. CSV File Processing: The application reads a player.csv file, which is split into small chunks (default size is 1MB). Each chunk is processed to convert the data into a list of Player objects.
2. Asynchronous Data Loading: The chunks are processed and loaded into the H2 database asynchronously to optimize performance and responsiveness.
3. Reactive Endpoints: The application uses Spring WebFlux to expose three endpoints that provide reactive, non-blocking access to the player data.
4. Request Caching: The application employs caching strategies to improve performance and reduce database load.
5. Monitoring with Prometheus: Application metrics are collected and exposed via Prometheus for monitoring.
6. Logging with Log4j2: The application uses Log4j2 for comprehensive and configurable logging.
7. API Documentation with Swagger: The application uses Swagger to provide interactive API documentation.
8. H2 Console: An embedded H2 database console is available for direct database access.

Schema

+--------------------+                     +------------------+
|  Players.csv File  |                     |   Spring Boot    |
|                    |                     |   Application    |
+--------------------+                     +------------------+
          |                                      |
          | (Read in chunks, 1MB default)        |
          V                                      |
+--------------------+                     +------------------+
|  CSV Chunk Reader  |                     |  Asynchronous    |
| (Process each chunk|                     |  Processing      |
|   to List<Player>) |                     |                  |
+--------------------+                     +------------------+
          |                                      |
          V                                      |
+--------------------+                     +------------------+
|  List<Player>      |    Asynchronously   |  H2 Database     |
|  (from each chunk) +-------------------> | (Save Players)   |
+--------------------+                     +------------------+
                                                      |
                                                      V
                                              +------------------+
                                              |  WebFlux API     |
                                              |  Endpoints       |
                                              +------------------+
                                                      |
                                                      V
                                              +------------------+
                                              |  API Consumer    |
                                              |  (e.g., Swagger) |
                                              +------------------+

App Structure Diagram

File Processing Sequence Diagram

Endpoints

Retrieves a paginated and sorted list of players. The response is in JSON format.

GET application/json /api/players?page=0&size=50&sort=birthYear,asc

Streams all player data in JSON streaming format.

GET application/stream+json /api/players/stream

Retrieves a specific player by their ID. The response is in JSON format.

GET application/stream+json /api/players/{playerID}

CURL Commands to Read Data

Retrieve Paginated and Sorted List of Players:

curl -X GET "http://localhost:8080/api/players?page=0&size=50&sort=birthYear,asc" -H "Accept: application/json"

Stream All Player Data:

curl -X GET "http://localhost:8080/api/players/stream" -H "Accept: application/stream+json"

Retrieve Specific Player by ID:

curl -X GET "http://localhost:8080/api/players/{playerID}" -H "Accept: application/stream+json"

Caching Requests

Caching is implemented to enhance performance and reduce the load on the database by storing frequently accessed data in memory.
This can be achieved using Spring Cache with annotations like @Cacheable, @CachePut, and @CacheEvict.

Cache is invalidated for all API requests.

1. Cache Invalidation when Player.csv is Changed Externally:
   • File watcher reacts on changes to the Player.csv file.
   • Whenever a change is detected, application invalidates the cache for all requests.
2. Cache Invalidation after One Hour:
   • Cache is configured to be invalidated after expiration.

Monitoring with Prometheus, documenting with Swagger

Prometheus is used to collect and expose metrics from the application for monitoring purposes.
Spring Boot Actuator provides out-of-the-box integration with Prometheus.

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    export:
      prometheus:
        enabled: true

Prometheus Endpoints:

curl http://localhost:8080/actuator/prometheus

Swagger is used to document and test the API endpoints. Springfox library is often used to integrate Swagger into Spring Boot applications.

Swagger Enpoint:

http://localhost:8080/swagger-ui.html

H2 Console

The H2 Console is enabled to allow direct access to the embedded H2 database.
This is useful for development and debugging purposes.

spring:
  h2:
    console:
      enabled: true
      path: /h2-console
  datasource:
    url: jdbc:h2:mem:testdb
    driverClassName: org.h2.Driver
    username: sa
    password: password

H2 Console URL:

http://localhost:8080/h2-console

Getting Started

For building and running the application you may need:

• JDK 17
• Maven 3
• Docker
• Postman
• IntelliJ IDEA Optional, command line script provided
• Git

Build and run with CMD

To run application with command-line, need to execute following steps.
Build a code:

make build

Run shell script, included in repository:

make run

or use ready script:

sh ./scripts/local-run.sh

or standard java FAT Jar command:

java $JAVA_OPTS -jar players-lab-0.0.1-SNAPSHOT.jar --spring.config.location=${CONFIG_FILE} --logging.config=${LOG_FILE} --players.config.source=${PLAYERS_CONFIG_SOURCE}

NOTE: Make sure, that local-run.sh configured properly with following variables: CONFIG_FILE, LOG_FILE, PLAYERS_CONFIG_SOURCE (need put custom configs)
If these variables are not set, application will use default configuration from resources.

IntelliJ IDEA can be used to run application as well:

Build with Docker

To build Docker Image, there is Makefile provided to simplify process.
To know all function, provided by Makefile, there is command make help need to be invoked.

make help                                                                                             
usage: make [target]

build                          - Compile and Build application
docker-build                   - Build Docker image for  application
docker-delete-containers       - Delete all docker stopped containers
docker-delete-image            - Delete  docker image
docker-delete-images           - Delete all docker unused images
docker-info                    - Docker Info about  docker image
docker-ls                      - List of docker images
docker-run                     - Run in Docker the  docker image
docker-scan                    - Scan for known vulnerabilities the  docker image
env                            - List of Env variables
help                           - Show help message
push-to-aws                    - Push docker image to AWS Elastic Container Registry
scan                           - Scan for known vulnerabilities the  docker image

Before build, following resources could be pre-configured or application will use default configuration from resources:
ATTENTION: You can ignore this step, if you want to use default configuration from resources.

export CONFIG_SOURCE==/Users/some-path/players-lab/players-main/src/main/resources/application.yaml   
export LOG_SOURCE=/Users/some-path/players-lab/players-main/src/main/resources/spring-log4j2.xml
export PLAYERS_CONFIG_SOURCE=/Users/some-path/players-lab/players-main/src/main/resources/player.yaml

To build Docker image, following command must be used with Makefile:

1. Build a code:

make build

2. Start Docker build process:

make docker-build

3. Push Docker image to AWS Elastic Container Registry:

make push-to-aws

Docker image is build with multi-stage build, so it is optimized for production use.
It is based on tiny Linux Alpine image, which is very small and secure (~6MB).
Docker security check results for vulnerabilities:

Running Docker image:

Building with GiFlow is also supported, so it is possible to use GitHub Actions for CI/CD:

Unit Tests and Integration Tests

All relevant tests are included in the project.
Folder and module structure is designed to separate tests from main code.
Tests are ran as part of the build process.

Module for tests is /players-lab/players-main/src/test/java.

Prometheus Monitoring

Prometheus is used to collect and expose metrics from the application for monitoring purposes.
Prometheus is configured to scrape metrics from the application at http://localhost:8080/actuator/prometheus.
This project includes such examples of business metrics:

# HELP updated_players_total Total amount of updated players.
# TYPE updated_players_total counter
updated_players_total 19369.0

# HELP popular_players_total Total amount of requests for popular players.
# TYPE popular_players_total counter
popular_players_total{playerID="acosted01"} 1.0
popular_players_total{playerID="acostjo01"} 1.0

# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players"} 3
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players"} 0.147897249
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/stream"} 2
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/stream"} 0.792809125
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/{playerID}"} 4
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/{playerID}"} 0.03250071

# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players"} 0.141999208
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/stream"} 0.530484334
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/api/players/{playerID}"} 0.022823417

# HELP spring_data_repository_invocations_seconds Duration of repository invocations
# TYPE spring_data_repository_invocations_seconds summary
spring_data_repository_invocations_seconds_count{exception="None",method="findAll",repository="PlayersRepository",state="SUCCESS"} 2
spring_data_repository_invocations_seconds_sum{exception="None",method="findAll",repository="PlayersRepository",state="SUCCESS"} 0.21089425
spring_data_repository_invocations_seconds_count{exception="None",method="findById",repository="PlayersRepository",state="SUCCESS"} 2
spring_data_repository_invocations_seconds_sum{exception="None",method="findById",repository="PlayersRepository",state="SUCCESS"} 0.001884626
spring_data_repository_invocations_seconds_count{exception="None",method="saveAll",repository="PlayersRepository",state="SUCCESS"} 3
spring_data_repository_invocations_seconds_sum{exception="None",method="saveAll",repository="PlayersRepository",state="SUCCESS"} 0.779419876

# HELP spring_data_repository_invocations_seconds_max Duration of repository invocations
# TYPE spring_data_repository_invocations_seconds_max gauge
spring_data_repository_invocations_seconds_max{exception="None",method="findAll",repository="PlayersRepository",state="SUCCESS"} 0.188516792
spring_data_repository_invocations_seconds_max{exception="None",method="findById",repository="PlayersRepository",state="SUCCESS"} 0.001578834
spring_data_repository_invocations_seconds_max{exception="None",method="saveAll",repository="PlayersRepository",state="SUCCESS"} 0.305815084

Reference Documentation

For further reference, please consider the following sections:

• Official Apache Maven documentation
• Spring Boot Maven Plugin Reference Guide
• Create an OCI image
• Spring Web
• Spring Reactive Web
• Spring Security
• Spring Data JPA
• Spring Batch
• Spring Boot Actuator
• Prometheus

Guides

The following guides illustrate how to use some features concretely:

• Building a RESTful Web Service
• Serving Web Content with Spring MVC
• Building REST services with Spring
• Building a Reactive RESTful Web Service
• Securing a Web Application
• Spring Boot and OAuth2
• Authenticating a User with LDAP
• Accessing Data with JPA
• Creating a Batch Service
• Building a RESTful Web Service with Spring Boot Actuator