Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always request all study vocabs #342

Merged
merged 12 commits into from
Jan 5, 2024
Merged

Always request all study vocabs #342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
08b1897
add some helpers to abstract plugin
d-callan Dec 20, 2023
af3af4f
update method to request study vocabs
d-callan Dec 20, 2023
0d19759
update sam floater plugins to request all study vocabs all the time
d-callan Dec 20, 2023
8be5b0f
finish moving from dynamicdataspecimpl to dynamicdataspec
d-callan Dec 20, 2023
3a264ab
use most recent edacommon
d-callan Dec 20, 2023
c856401
typo
d-callan Dec 21, 2023
cb75952
add methods to compare variable and collection specs
d-callan Dec 21, 2023
ead71d6
remove deprecated methods
d-callan Dec 21, 2023
ebfda2e
update map plugins to not use deprecated methods
d-callan Dec 21, 2023
761a1dd
update some helpers i missed
d-callan Dec 21, 2023
b3f4482
update variables list to include unplotted study vocabs
d-callan Dec 21, 2023
8da81d0
update map floaters to use new helper to find variables list w study …
d-callan Dec 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ repositories {
// versions
val coreLib = "6.19.1" // Container core lib version
val edaCompute = "2.4.4" // EDA Compute version (used to pull in compute plugin RAML)
val edaCommon = "11.6.6" // EDA Common version
val edaCommon = "11.6.7" // EDA Common version
val fgputil = "2.12.9-jakarta" // FgpUtil version

// use local EDA compute compiled schema if project exists, else use released version;
Expand Down
287 changes: 221 additions & 66 deletions src/main/java/org/veupathdb/service/eda/ds/core/AbstractPlugin.java
View file
Open in desktop

Large diffs are not rendered by default.

23 changes: 14 additions & 9 deletions ...va/org/veupathdb/service/eda/ds/plugin/standalonemap/CollectionFloatingBarplotPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,32 +67,37 @@ protected void validateVisualizationSpec(CollectionFloatingBarplotSpec pluginSpe
}

@Override
protected List<StreamSpec> getRequestedStreams(CollectionFloatingBarplotSpec pluginSpec) {
protected List<StreamSpec> getRequestedStreams(CollectionFloatingBarplotSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.addAll(pluginSpec.getOverlayConfig().getSelectedMembers());

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVars(pluginSpec.getOverlayConfig().getSelectedMembers())
);
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear to me what you want to be automagical? We probably can with some inheritance if you want to generalize some things. Just hit me up on slack if you want to discuss. Looks like there might be a lot of repetition/cut-and-paste in some of these plugins which we would like to avoid.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a lot of repetition, and i hate it, and every time i think about improving the situation it seems to turn into a can of worms.. i start thinking things like 'well in an ideal world id actually be doing some entirely other thing' and things spiral out of control. so if you have ideas, id be open to hearing them

// this will probably need revisited when/ if we introduce study-dependent vocabs for collections
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
CollectionFloatingBarplotSpec spec = getPluginSpec();
PluginUtil util = getUtil();
String outputEntityId = spec.getOutputEntityId();
String barMode = spec.getBarMode().getValue();
String overlayValues = getRBinListAsString(spec.getOverlayConfig().getSelectedValues());
List<VariableSpec> inputVarSpecs = new ArrayList<>(spec.getOverlayConfig().getSelectedMembers());

Map<String, CollectionSpec> varMap = new HashMap<>();
varMap.put("overlay", spec.getOverlayConfig().getCollection());

List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findCollectionSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithRemoteFiles(Resources.RSERVE_URL, dataStreams, connection -> {
connection.voidEval(DEFAULT_SINGLE_STREAM_NAME + " <- data.table::fread('" + DEFAULT_SINGLE_STREAM_NAME + "', na.strings=c(''))");
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalCollectionMetadataList(varMap));
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalCollectionMetadataListWithStudyDependentVocabs(varMap, outputEntityId));
String cmd =
"plot.data::bar(data=" + inputData + ", " +
"variables=variables, " +
Expand Down
22 changes: 14 additions & 8 deletions ...va/org/veupathdb/service/eda/ds/plugin/standalonemap/CollectionFloatingBoxplotPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,21 +73,27 @@ protected void validateVisualizationSpec(CollectionFloatingBoxplotSpec pluginSpe

@Override
protected List<StreamSpec> getRequestedStreams(CollectionFloatingBoxplotSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.add(pluginSpec.getXAxisVariable());
plotVariableSpecs.addAll(pluginSpec.getOverlayConfig().getSelectedMembers());

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVars(pluginSpec.getOverlayConfig().getSelectedMembers())
.addVar(pluginSpec.getXAxisVariable())
);
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
PluginUtil util = getUtil();
CollectionFloatingBoxplotSpec spec = getPluginSpec();
String outputEntityId = spec.getOutputEntityId();
List<VariableSpec> inputVarSpecs = new ArrayList<>(spec.getOverlayConfig().getSelectedMembers());
inputVarSpecs.add(spec.getXAxisVariable());
CollectionSpec overlayVariable = spec.getOverlayConfig().getCollection();
Map<String, DynamicDataSpecImpl> varMap = new HashMap<>();
Map<String, DynamicDataSpec> varMap = new HashMap<>();
varMap.put("xAxis", new DynamicDataSpecImpl(spec.getXAxisVariable()));
varMap.put("overlay", new DynamicDataSpecImpl(overlayVariable));

Expand All @@ -106,13 +112,13 @@ protected void writeResults(OutputStream out, Map<String, InputStream> dataStrea
conn.voidEval(name + " <- data.table::fread('" + name + "', na.strings=c(''))")
);

List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findDataSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithProcessedRemoteFiles(Resources.RSERVE_URL, filesProcessor, connection -> {
String inputData = getRInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalDynamicDataMetadataList(varMap));
String inputData = getRInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalDynamicDataMetadataListWithStudyDependentVocabs(varMap, outputEntityId));
String cmd =
"plot.data::box(data=" + inputData + ", " +
"variables=variables, " +
Expand Down
22 changes: 13 additions & 9 deletions .../org/veupathdb/service/eda/ds/plugin/standalonemap/CollectionFloatingContTablePlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,30 +58,34 @@ protected void validateVisualizationSpec(CollectionFloatingContTableSpec pluginS
}

@Override
protected List<StreamSpec> getRequestedStreams(CollectionFloatingContTableSpec pluginSpec) {
protected List<StreamSpec> getRequestedStreams(CollectionFloatingContTableSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.addAll(pluginSpec.getXAxisVariable().getSelectedMembers());

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVars(pluginSpec.getXAxisVariable().getSelectedMembers())
);
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
PluginUtil util = getUtil();
CollectionFloatingContTableSpec spec = getPluginSpec();
String outputEntityId = spec.getOutputEntityId();
String overlayValues = getRBinListAsString(spec.getXAxisVariable().getSelectedValues());
List<VariableSpec> inputVarSpecs = new ArrayList<>(spec.getXAxisVariable().getSelectedMembers());
Map<String, CollectionSpec> varMap = new HashMap<>();
varMap.put("xAxis", spec.getXAxisVariable().getCollection());

List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findCollectionSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithRemoteFiles(Resources.RSERVE_URL, dataStreams, connection -> {
connection.voidEval(DEFAULT_SINGLE_STREAM_NAME + " <- data.table::fread('" + DEFAULT_SINGLE_STREAM_NAME + "', na.strings=c(''))");
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalCollectionMetadataList(varMap));
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalCollectionMetadataListWithStudyDependentVocabs(varMap, outputEntityId));
String cmd = "plot.data::mosaic(data=" + inputData + ", " +
"variables=variables, " +
"statistic='chiSq', " +
Expand Down
19 changes: 12 additions & 7 deletions .../org/veupathdb/service/eda/ds/plugin/standalonemap/CollectionFloatingHistogramPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,31 +68,36 @@ protected void validateVisualizationSpec(CollectionFloatingHistogramSpec pluginS

@Override
protected List<StreamSpec> getRequestedStreams(CollectionFloatingHistogramSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.addAll(pluginSpec.getOverlayConfig().getSelectedMembers());

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVars(pluginSpec.getOverlayConfig().getSelectedMembers())
);
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
PluginUtil util = getUtil();
CollectionFloatingHistogramSpec spec = getPluginSpec();
List<VariableSpec> inputVarSpecs = new ArrayList<>(spec.getOverlayConfig().getSelectedMembers());
String outputEntityId = spec.getOutputEntityId();
CollectionSpec overlayVariable = spec.getOverlayConfig().getCollection();
Map<String, CollectionSpec> varMap = new HashMap<>();
varMap.put("overlay", overlayVariable);
String barMode = spec.getBarMode().getValue();
String collectionType = util.getCollectionType(overlayVariable);

List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findCollectionSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithRemoteFiles(Resources.RSERVE_URL, dataStreams, connection -> {
connection.voidEval(DEFAULT_SINGLE_STREAM_NAME + " <- data.table::fread('" + DEFAULT_SINGLE_STREAM_NAME + "', na.strings=c(''))");
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalCollectionMetadataList(varMap));
String inputData = getRCollectionInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalCollectionMetadataListWithStudyDependentVocabs(varMap, outputEntityId));

String viewportRString = getViewportAsRString(spec.getViewport(), collectionType);
connection.voidEval(viewportRString);
Expand Down
22 changes: 14 additions & 8 deletions ...a/org/veupathdb/service/eda/ds/plugin/standalonemap/CollectionFloatingLineplotPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,21 +69,27 @@ protected void validateVisualizationSpec(CollectionFloatingLineplotSpec pluginSp

@Override
protected List<StreamSpec> getRequestedStreams(CollectionFloatingLineplotSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.add(pluginSpec.getXAxisVariable());
plotVariableSpecs.addAll(pluginSpec.getOverlayConfig().getSelectedMembers());

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVars(pluginSpec.getOverlayConfig().getSelectedMembers())
.addVar(pluginSpec.getXAxisVariable())
);
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
PluginUtil util = getUtil();
CollectionFloatingLineplotSpec spec = getPluginSpec();
String outputEntityId = spec.getOutputEntityId();
List<VariableSpec> inputVarSpecs = new ArrayList<>(spec.getOverlayConfig().getSelectedMembers());
inputVarSpecs.add(spec.getXAxisVariable());
CollectionSpec overlayVariable = spec.getOverlayConfig().getCollection();
Map<String, DynamicDataSpecImpl> varMap = new HashMap<>();
Map<String, DynamicDataSpec> varMap = new HashMap<>();
varMap.put("xAxis", new DynamicDataSpecImpl(spec.getXAxisVariable()));
varMap.put("overlay", new DynamicDataSpecImpl(overlayVariable));
String errorBars = spec.getErrorBars() != null ? spec.getErrorBars().getValue() : "FALSE";
Expand All @@ -93,14 +99,14 @@ protected void writeResults(OutputStream out, Map<String, InputStream> dataStrea
String denominatorValues = spec.getYAxisDenominatorValues() != null ? PluginUtil.listToRVector(spec.getYAxisDenominatorValues()) : "NULL";
String overlayValues = getRBinListAsString(spec.getOverlayConfig().getSelectedValues());

List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findDataSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithRemoteFiles(Resources.RSERVE_URL, dataStreams, connection -> {
connection.voidEval(DEFAULT_SINGLE_STREAM_NAME + " <- data.table::fread('" + DEFAULT_SINGLE_STREAM_NAME + "', na.strings=c(''))");
String inputData = getRInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalDynamicDataMetadataList(varMap));
String inputData = getRInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalDynamicDataMetadataListWithStudyDependentVocabs(varMap, outputEntityId));
String viewportRString = getViewportAsRString(spec.getViewport(), collectionType);
connection.voidEval(viewportRString);
BinWidthSpec binSpec = spec.getBinSpec();
Expand Down
22 changes: 14 additions & 8 deletions src/main/java/org/veupathdb/service/eda/ds/plugin/standalonemap/FloatingBarplotPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -82,18 +83,23 @@ protected void validateVisualizationSpec(FloatingBarplotSpec pluginSpec) throws

@Override
protected List<StreamSpec> getRequestedStreams(FloatingBarplotSpec pluginSpec) {
String outputEntityId = pluginSpec.getOutputEntityId();
List<VariableSpec> plotVariableSpecs = new ArrayList<VariableSpec>();
plotVariableSpecs.add(pluginSpec.getXAxisVariable());
plotVariableSpecs.add(Optional.ofNullable(pluginSpec.getOverlayConfig()).map(OverlayConfig::getOverlayVariable).orElse(null));

return ListBuilder.asList(
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, pluginSpec.getOutputEntityId())
.addVar(pluginSpec.getXAxisVariable())
.addVar(Optional.ofNullable(pluginSpec.getOverlayConfig())
.map(OverlayConfig::getOverlayVariable)
.orElse(null)));
new StreamSpec(DEFAULT_SINGLE_STREAM_NAME, outputEntityId)
.addVars(plotVariableSpecs)
// TODO can we make this automagical?
.addVars(getVariableSpecsWithStudyDependentVocabs(pluginSpec.getOutputEntityId(), plotVariableSpecs)));
}

@Override
protected void writeResults(OutputStream out, Map<String, InputStream> dataStreams) throws IOException {
FloatingBarplotSpec spec = getPluginSpec();
PluginUtil util = getUtil();
String outputEntityId = spec.getOutputEntityId();
String barMode = spec.getBarMode().getValue();
VariableSpec overlayVariable = _overlaySpecification != null ? _overlaySpecification.getOverlayVariable() : null;
String overlayValues = _overlaySpecification == null ? "NULL" : _overlaySpecification.getRBinListAsString();
Expand All @@ -103,15 +109,15 @@ protected void writeResults(OutputStream out, Map<String, InputStream> dataStrea
varMap.put("overlay", overlayVariable);

// TODO can we make this automagical? override useRConnectionWithRemoteFiles ? i wasnt clear how..
List<DynamicDataSpecImpl> dataSpecsWithStudyDependentVocabs = findVariableSpecsWithStudyDependentVocabs(varMap);
List<DynamicDataSpec> dataSpecsWithStudyDependentVocabs = getDynamicDataSpecsWithStudyDependentVocabs(outputEntityId);
Map<String, InputStream> studyVocabs = getVocabByRootEntity(dataSpecsWithStudyDependentVocabs);
dataStreams.putAll(studyVocabs);

useRConnectionWithRemoteFiles(Resources.RSERVE_URL, dataStreams, connection -> {
connection.voidEval(DEFAULT_SINGLE_STREAM_NAME + " <- data.table::fread('" + DEFAULT_SINGLE_STREAM_NAME + "', na.strings=c(''))");

String inputData = getRVariableInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, "variables");
connection.voidEval(getVoidEvalVariableMetadataList(varMap));
String inputData = getRVariableInputDataWithImputedZeroesAsString(DEFAULT_SINGLE_STREAM_NAME, varMap, outputEntityId, "variables");
connection.voidEval(getVoidEvalVariableMetadataListWithStudyDependentVocabs(varMap, outputEntityId));
String cmd =
"plot.data::bar(data=" + inputData + ", " +
"variables=variables, " +
Expand Down
Loading