Proposal of meta management and integrated query technology using AGE for DW

[zestspace]

[Proposal of meta management and integrated query technology using AGE for DW]

[Problems with traditional DW]

1. The more data you have, the more expensive the cost of storing and processing your data
2. Lack of flexibility in collected data
3. High redundancy data and ETL or ELT redundancy costs

[DW problems in big data environment]

1. In Data Lake, it is not easy to apply data security for personal information protection, and there is a problem of cost due to the up-to-date data

• Since the data in the data lake is data before it is refined, it is difficult to select personal information and apply access control, masking, and encryption.
• Difficult to ensure up-to-date due to different data collection cycles and levels

2. Data Lake is huge and expensive to find and refine the data you want or store unnecessary data

3. Data Lake has difficulty managing data.

• Because you don't know who and how to use it from the perspective of managers who need to manage integrated management

The DW that data consumers want requires an environment that is flexible in responding to the environment, does not load existing systems, and can use high-quality data quickly and inexpensively. It is difficult to find an ideal technology that meets all of the requirements. However, there are platforms that provide technologies accordingly, and they want to share related architectures and directions.

I'll attached detail document. checkup my PDF document. please!
Proposal of meta management and integrated query technology using AGE for DW_v1.0.pdf

[aked21]

Hi @zestspace, thanks for proposing an interesting idea!
Can you refine your idea/proposal in terms of how AGE, a postgres extension, can be enhanced with modifying existing or adding new features?