Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance Improvement for Documents, Interval Adjustements, Adjustements for UI Logic and Various Fixes #483

Merged
merged 68 commits into from
Apr 8, 2024
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
d546c29
adjusting documentValues comp string to fix caching
datomo Mar 17, 2024
ceddf42
mini adjustement to document webui component, some refactoring
datomo Mar 17, 2024
4017792
various optimizations for map and doc serializations and deserializat…
datomo Mar 17, 2024
7d77055
Fix issue with resetting docker containers on startup
vogti Mar 18, 2024
cdba0ad
fix for rex filter in sources
datomo Mar 18, 2024
192214c
some performance testing for serialization, while preserving function…
datomo Mar 18, 2024
c1bda7c
enabled namespace.collection.command( syntax for mql, switched to bin…
datomo Mar 19, 2024
693e7ff
added omitting of db. keyword for mql queries
datomo Mar 19, 2024
01ac879
adjustment to typed json deserializer to improve caching
datomo Mar 19, 2024
1cb33cb
fix mongo docbench edge case with missing quotes
datomo Mar 19, 2024
cf6acd7
changed editor config to adjusted one
datomo Mar 20, 2024
b1fdb04
removed all interval types except millis and month
datomo Mar 20, 2024
3b0987d
adjustment to fixed mapping of intervals
datomo Mar 20, 2024
644d0a3
partial fix for restoring
datomo Mar 20, 2024
495625e
added missing adapter specific restores
datomo Mar 21, 2024
b5389c8
removed unused fields from joins
datomo Mar 21, 2024
5f13135
fixed special query monitoringType errors and limit clauses
datomo Mar 21, 2024
2f058e4
fixed joint trim logic
datomo Mar 22, 2024
a210df8
bug fixes for joins and additional parameter for queryType of result …
datomo Mar 22, 2024
a6927d6
adjustement to IntervalTypes
datomo Mar 23, 2024
88fba63
streamlined interval logic and adjusted test with well-defined syntax…
datomo Mar 24, 2024
05ccdb5
fixed parsing logic, handling and various tests regarding intervals
datomo Mar 24, 2024
e4e1950
fix for config update
datomo Mar 25, 2024
c625131
fixed graph logic and parsing, added some error handling
datomo Mar 25, 2024
1e7a73e
Fix SQL query splitting
gartens Mar 26, 2024
dfd3e29
minor adjustments for stability, fixed images and videos, fixed norma…
datomo Mar 26, 2024
3aadd78
Merge branch 'document-fix' of https://github.com/polypheny/Polypheny…
datomo Mar 26, 2024
7dfe55c
fixed edge-cases for limit parsing
datomo Mar 26, 2024
4f4709b
increased test timeout
datomo Mar 26, 2024
7483353
fixes for source deleting, plan builder
datomo Mar 26, 2024
841158d
fix for monitoring test to improve reliablity
datomo Mar 26, 2024
d3130d2
fixed typo
datomo Mar 26, 2024
414757d
Remove attribute
vogti Mar 27, 2024
0fb952a
various adjustments, to fix inconsistencies and potential edge-cases
datomo Apr 2, 2024
fb5bb0a
removed correct method
datomo Apr 2, 2024
79d84cf
clean up for PolyNumbers
datomo Apr 2, 2024
55153a3
added default namespace to adapters and removed hardcoded namespace f…
datomo Apr 2, 2024
48b5066
separated create source and store, adjusted some PolyValues
datomo Apr 2, 2024
8964aee
fixes for array casting and parsing
datomo Apr 2, 2024
94d72cb
fixes for multiple queries results
datomo Apr 2, 2024
4408f36
temporary adjustments for statement calls from avatica
datomo Apr 2, 2024
726a6a9
fixes for constraint management from the webui
datomo Apr 3, 2024
5f2b618
fixes for tuplecount race condition and neo4j errors during graph bench
datomo Apr 4, 2024
9cb220b
fixed constraints for primary and foreign in webui
datomo Apr 4, 2024
f5eb2e2
fixed wrong constraint name
datomo Apr 4, 2024
3b27c04
minor QoL adjustment
datomo Apr 4, 2024
8887779
added missing existing transactions and statements for webui updates …
datomo Apr 5, 2024
b813e75
moved PolyLong into correct folder added own hashcode for values back
datomo Apr 5, 2024
e0ac963
adjustment of placement webui model to fix placement modification
datomo Apr 5, 2024
74d44ee
fix informationmanager for non-relationa queries
datomo Apr 6, 2024
1f4ba2f
fix for rollback with data sources and fix for data source column rem…
datomo Apr 7, 2024
509ff33
fix for config errors and added missing rollback for constraint testing
datomo Apr 7, 2024
80648a3
fixed some edge-cases on casting with parameterization and potential …
datomo Apr 7, 2024
94b7dc4
fix for null values in rexliteral
datomo Apr 7, 2024
021e5d6
minor adjustment to mongo functions, reformatting and cleanup for som…
datomo Apr 8, 2024
a5df4f3
Merge branch 'document-fix' of github.com:polypheny/Polypheny-DB into…
datomo Apr 8, 2024
3e3aeea
Add additional SQL function tests
vogti Apr 8, 2024
9cf0eed
Rename monitoring plugin
vogti Apr 8, 2024
ace2547
Remove unnecessary license
vogti Apr 8, 2024
0232611
Fixed PI and USER queries, disabled and marked subquery and BETWEEN b…
datomo Apr 8, 2024
55290ba
Merge branch 'document-fix' of github.com:polypheny/Polypheny-DB into…
datomo Apr 8, 2024
9b1ab57
Fixed unnecessary getter for record
datomo Apr 8, 2024
e1c4e18
added missing escape
datomo Apr 8, 2024
ecb3b39
added constraint validity test on create
datomo Apr 8, 2024
e153166
Minor formatting adjustments and adjust year in copyright
vogti Apr 8, 2024
9b31286
disabled some of the new tests for failing stores
datomo Apr 8, 2024
5a52cc2
Merge branch 'document-fix' of https://github.com/polypheny/Polypheny…
datomo Apr 8, 2024
e34e1ff
added correct "exclude" tag
datomo Apr 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ ij_csv_keep_indents_on_empty_lines = true
ij_csv_wrap_long_lines = false

[*.feature]
indent_size = 2
ij_continuation_indent_size = 8
ij_visual_guides = none
ij_gherkin_keep_indents_on_empty_lines = false
Expand Down Expand Up @@ -315,7 +314,6 @@ ij_java_wrap_first_method_in_call_chain = false
ij_java_wrap_long_lines = false

[*.less]
indent_size = 2
ij_continuation_indent_size = 8
ij_visual_guides = none
ij_less_align_closing_brace_with_properties = false
Expand All @@ -340,8 +338,6 @@ ij_less_use_double_quotes = true
ij_less_value_alignment = 0

[*.proto]
indent_size = 2
tab_width = 2
ij_visual_guides = none
ij_protobuf_keep_blank_lines_in_code = 2
ij_protobuf_keep_indents_on_empty_lines = false
Expand All @@ -353,8 +349,6 @@ ij_protobuf_spaces_within_braces = false
ij_protobuf_spaces_within_brackets = false

[*.sass]
indent_size = 2
tab_width = 2
ij_visual_guides = none
ij_sass_align_closing_brace_with_properties = false
ij_sass_blank_lines_around_nested_selector = 1
Expand All @@ -377,8 +371,6 @@ ij_sass_use_double_quotes = true
ij_sass_value_alignment = 0

[*.scss]
indent_size = 2
tab_width = 2
ij_visual_guides = none
ij_scss_align_closing_brace_with_properties = false
ij_scss_blank_lines_around_nested_selector = 1
Expand Down Expand Up @@ -433,8 +425,6 @@ ij_xml_text_wrap = normal
ij_xml_use_custom_settings = false

[{*.bash,*.sh,*.zsh}]
indent_size = 2
tab_width = 2
ij_visual_guides = none
ij_shell_binary_ops_start_line = false
ij_shell_keep_column_alignment_padding = false
Expand Down Expand Up @@ -812,8 +802,6 @@ ij_markdown_wrap_text_if_long = true
ij_markdown_wrap_text_inside_blockquotes = true

[{*.pb,*.textproto}]
indent_size = 2
tab_width = 2
ij_visual_guides = none
ij_prototext_keep_blank_lines_in_code = 2
ij_prototext_keep_indents_on_empty_lines = false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import io.javalin.Javalin;
import java.io.IOException;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
Expand Down Expand Up @@ -78,9 +79,8 @@ private void configRoutes( final Javalin http ) {
// Save changes from WebUi
http.post( PREFIX + "/updateConfigs", ctx -> {
log.trace( ctx.body() );
TypeReference<Map<String, Object>> typeRef = new TypeReference<>() {
};
Map<String, Object> changes = mapper.convertValue( ctx.body(), typeRef );
HashMap<String, Object> changes = mapper.readValue( ctx.body(), new TypeReference<>() { // we need explicit typing to force the correct map type
} );
StringBuilder feedback = new StringBuilder();
boolean allValid = true;
for ( Map.Entry<String, Object> entry : changes.entrySet() ) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ public abstract class AbstractAdapterSetting {
@Getter
public List<DeploySetting> appliesTo;

public List<String> filenames = new ArrayList<>();


public AbstractAdapterSetting( final AdapterSettingType type, final String name, final boolean canBeNull, final String subOf, final boolean required, final boolean modifiable, List<DeploySetting> appliesTo, String defaultValue, int position ) {
this.type = type;
Expand All @@ -79,7 +81,7 @@ public AbstractAdapterSetting( final AdapterSettingType type, final String name,
*
* @param annotations collection of annotations
* @param properties which are defined by the corresponding Adapter
* @return a map containing the available modes and the corresponding collections of AdapterSettings
* @return a collection containing the available modes and the corresponding collections of AdapterSettings
*/
public static List<AbstractAdapterSetting> fromAnnotations( Annotation[] annotations, AdapterProperties properties ) {
List<AbstractAdapterSetting> settings = new ArrayList<>();
Expand Down
3 changes: 1 addition & 2 deletions core/src/main/java/org/polypheny/db/adapter/Adapter.java
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,7 @@ public abstract class Adapter<ACatalog extends AdapterCatalog> implements Scanna
private final AdapterProperties properties;
protected final DeployMode deployMode;
protected String deploymentId;
@Getter
private final String adapterName;
public final String adapterName;
public final ACatalog adapterCatalog;


Expand Down
19 changes: 13 additions & 6 deletions core/src/main/java/org/polypheny/db/adapter/AdapterManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -170,10 +170,7 @@ public ImmutableMap<String, DataSource<?>> getSources() {
}





public Adapter<?> addAdapter( String adapterName, String uniqueName, AdapterType adapterType, DeployMode mode, Map<String, String> settings ) {
public Adapter<?> addAdapter( String adapterName, String uniqueName, long defaultNamespace, AdapterType adapterType, DeployMode mode, Map<String, String> settings ) {
uniqueName = uniqueName.toLowerCase();
if ( getAdapters().containsKey( uniqueName ) ) {
throw new GenericRuntimeException( "There is already an adapter with this unique name" );
Expand All @@ -184,7 +181,7 @@ public Adapter<?> addAdapter( String adapterName, String uniqueName, AdapterType

AdapterTemplate adapterTemplate = AdapterTemplate.fromString( adapterName, adapterType );

long adapterId = Catalog.getInstance().createAdapter( uniqueName, adapterName, adapterType, settings, mode );
long adapterId = Catalog.getInstance().createAdapter( uniqueName, adapterName, defaultNamespace, adapterType, settings, mode );
try {
Adapter<?> adapter = adapterTemplate.getDeployer().get( adapterId, uniqueName, settings );
adapterByName.put( adapter.getUniqueName(), adapter );
Expand All @@ -209,7 +206,9 @@ public void removeAdapter( long adapterId ) {
// Check if the store has any placements
List<AllocationEntity> placements = Catalog.getInstance().getSnapshot().alloc().getEntitiesOnAdapter( logicalAdapter.id ).orElseThrow( () -> new GenericRuntimeException( "There is still data placed on this data store" ) );
if ( !placements.isEmpty() ) {
throw new GenericRuntimeException( "There is still data placed on this data store" );
if ( adapterInstance instanceof DataStore<?> ) {
throw new GenericRuntimeException( "There is still data placed on this data store" );
}
}

// Shutdown store
Expand Down Expand Up @@ -264,4 +263,12 @@ public interface Function4<P1, P2, P3, R> {

}


@FunctionalInterface
public interface Function5<P1, P2, P3, P4, R> {
datomo marked this conversation as resolved.
Show resolved Hide resolved

R get( P1 p1, P2 p2, P3 p3, P4 p4 );

}

}
Original file line number Diff line number Diff line change
Expand Up @@ -52,21 +52,20 @@ public List<PhysicalEntity> createTable( Context context, LogicalTableWrapper lo


@Override
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities ) {
scannable.restoreTable( alloc, entities );
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities, Context context ) {
scannable.restoreTable( alloc, entities, context );
}


@Override
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities ) {
Scannable.restoreGraphSubstitute( scannable, alloc, entities );
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities, Context context ) {
Scannable.restoreGraphSubstitute( scannable, alloc, entities, context );
}



@Override
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities ) {
scannable.restoreCollection( alloc, entities );
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities, Context context ) {
scannable.restoreCollection( alloc, entities, context );
}


Expand All @@ -93,6 +92,7 @@ public AlgNode getGraphScan( long allocId, AlgBuilder builder ) {
return Scannable.getGraphScanSubstitute( scannable, allocId, builder );
}


@Override
public List<PhysicalEntity> createCollection( Context context, LogicalCollection logical, AllocationCollection allocation ) {
return scannable.createCollection( context, logical, allocation );
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,20 +56,20 @@ public List<PhysicalEntity> createTable( Context context, LogicalTableWrapper lo


@Override
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities ) {
scannable.restoreTable( alloc, entities );
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities, Context context ) {
scannable.restoreTable( alloc, entities, context );
}


@Override
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities ) {
scannable.restoreGraph( alloc, entities );
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities, Context context ) {
scannable.restoreGraph( alloc, entities, context );
}


@Override
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities ) {
Scannable.restoreCollectionSubstitute( scannable, alloc, entities );
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities, Context context ) {
Scannable.restoreCollectionSubstitute( scannable, alloc, entities, context );
}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,20 +60,20 @@ public List<PhysicalEntity> createTable( Context context, LogicalTableWrapper lo


@Override
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities ) {
scannable.restoreTable( alloc, entities );
public void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities, Context context ) {
scannable.restoreTable( alloc, entities, context );
}


@Override
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities ) {
Scannable.restoreGraphSubstitute( scannable, alloc, entities );
public void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities, Context context ) {
Scannable.restoreGraphSubstitute( scannable, alloc, entities, context );
}


@Override
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities ) {
Scannable.restoreCollectionSubstitute( scannable, alloc, entities );
public void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities, Context context ) {
Scannable.restoreCollectionSubstitute( scannable, alloc, entities, context );
}


Expand Down
10 changes: 5 additions & 5 deletions core/src/main/java/org/polypheny/db/adapter/Scannable.java
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,12 @@ static PhysicalEntity createSubstitutionEntity( Scannable scannable, Context con
AdapterCatalog getCatalog();


static void restoreGraphSubstitute( Scannable scannable, AllocationGraph alloc, List<PhysicalEntity> entities ) {
static void restoreGraphSubstitute( Scannable scannable, AllocationGraph alloc, List<PhysicalEntity> entities, Context context ) {
throw new GenericRuntimeException( "todo restore" );
}


static void restoreCollectionSubstitute( Scannable scannable, AllocationCollection alloc, List<PhysicalEntity> entities ) {
static void restoreCollectionSubstitute( Scannable scannable, AllocationCollection alloc, List<PhysicalEntity> entities, Context context ) {
throw new GenericRuntimeException( "todo restore" );
}

Expand Down Expand Up @@ -148,13 +148,13 @@ static AlgNode getDocumentScanSubstitute( Scannable scannable, long allocId, Alg
List<PhysicalEntity> createTable( Context context, LogicalTableWrapper logical, AllocationTableWrapper allocation );


void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities );
void restoreTable( AllocationTable alloc, List<PhysicalEntity> entities, Context context );


void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities );
void restoreGraph( AllocationGraph alloc, List<PhysicalEntity> entities, Context context );


void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities );
void restoreCollection( AllocationCollection alloc, List<PhysicalEntity> entities, Context context );


void dropTable( Context context, long allocId );
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ public class AdapterTemplate {
public long id;
public String description;


public AdapterTemplate( long id, Class<?> clazz, String adapterName, List<AbstractAdapterSetting> settings, List<DeployMode> modes, String description, Function4<Long, String, Map<String, String>, Adapter<?>> deployer ) {
this.id = id;
this.adapterName = adapterName;
Expand Down
Loading
Loading