GOQkview

The project aims to develop a system that handles qkview files, which are uploaded to a MinIO object storage service. The system utilizes a consumer application built with Go to retrieve new messages from a Kafka topic. When a new message is received, the consumer application downloads the corresponding qkview file from MinIO. It then decompresses the file and saves its contents locally, using the file name as the destination directory.

Dependencies

Env variables

go get github.com/joho/godotenv

Kafka connection

go get github.com/Shopify/sarama

MinIO connection

go get github.com/minio/minio-go/v7

Elastic Search

go get github.com/elastic/go-elasticsearch/v8

UUID

go get github.com/google/uuid

Configuration

Kafka ( .docker/kafka ):

• Create a .env file containing:
  Max amount of memory used by JVM: KAFKA_BROKER_HEAP_OPTS="-XX:MaxRAMPercentage=70.0"
  Used to communicate externally using MinIO: DOCKER_HOST_IP="local or server ip"

MinIO ( .docker/minio ):

• Create a .env file containing:
  MINIO_USER="miniouser"
  MINIO_PASSWORD="miniopassword"
• Create a new Event using Kafka Destination in the MinIO web instance ( http://localhost:9000 )
• After creating the bucket subscribe to the event destination

ElasticSearch ( .docker/elastic ):

• Create a .env file containing:
  ELASTIC_PASSWORD="password"
  MEM_LIMIT="memorylimit"