Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mathml -> sympy conversion #188

Merged
merged 56 commits into from
Aug 14, 2023
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
7e9a776
Add mathml -> sympy parser using sbmlmath package
kkaris Jul 5, 2023
080506c
Ensure sbmlmath is installed for tests
kkaris Jul 5, 2023
3c39830
Add sbmlmath in setup.cfg
kkaris Jul 5, 2023
6fb0b02
Add function to __all__
kkaris Jul 5, 2023
0ca8505
Add closing </math> in template
kkaris Jul 5, 2023
730d3b5
Write test for some expressions in scenario1a json
kkaris Jul 5, 2023
8b23a33
Handle mathml -> sympy in state_to_concept
kkaris Jul 5, 2023
f3335f9
Handle sympy -> mathml for initials
kkaris Jul 5, 2023
4eec718
More clear logic for state_to_concept mathml handling
kkaris Jul 5, 2023
9d4e599
Clearer logic for mathml handling for initals
kkaris Jul 5, 2023
b576070
Add mathml support for observables
kkaris Jul 5, 2023
fb0c565
Handle mathml for time units
kkaris Jul 5, 2023
0ceedc2
Handle mathml for rates
kkaris Jul 5, 2023
4702fe0
DRY implementation of sympy and mathml parsing
kkaris Jul 5, 2023
aa13ae7
Clean up mathml->sympy tests
kkaris Jul 5, 2023
1b7c549
Add test for sympy -> mathml
kkaris Jul 5, 2023
03c2fa7
Fix bug
kkaris Jul 6, 2023
bf28d48
Test askenet petrinet sir example
kkaris Jul 6, 2023
1370e08
Fix(?) sbmlmath install
kkaris Jul 6, 2023
9b078c0
Check for None in _get_sympy
kkaris Jul 6, 2023
d5a60ca
Simplify logic
kkaris Jul 6, 2023
373f35c
Move sorted_json_str to tests.__init__.py
kkaris Jul 6, 2023
5483076
Add test for recovering expressions from mathml
kkaris Jul 6, 2023
f93a5be
Move helpers to tests/__init__.py
kkaris Jul 7, 2023
f3b1d64
Move tests
kkaris Jul 7, 2023
41d367a
Delete unused file
kkaris Jul 7, 2023
c728777
Add method and inplace args in _remove_all_sympy
kkaris Jul 7, 2023
cfdde56
Add docstring to helpers
kkaris Jul 7, 2023
6824ae0
Rename helpers
kkaris Jul 7, 2023
ddf29fb
Remove inplace option
kkaris Jul 10, 2023
d6088b1
Add option to 'null' values in sympy deleter
kkaris Jul 10, 2023
a5ebbc4
Add test for no sympy in model api tests
kkaris Jul 10, 2023
5b45061
Use safe_parse_expr in sympy helper
kkaris Jul 12, 2023
536ddad
Clean up after rebase conflicts
kkaris Jul 12, 2023
212a88a
Rename _get_sympy -> get_sympy, add docstring
kkaris Jul 12, 2023
36f9e82
Use askenet.petrinet's get_sympy
kkaris Jul 12, 2023
32935d7
Clean up import
kkaris Jul 12, 2023
08657b3
Initial try to install sbmlmath for tests
kkaris Jul 12, 2023
caca254
Force install despite wrong python version
kkaris Jul 12, 2023
85b4718
Revert --ignore-requires-python, test bump lower python version
kkaris Jul 12, 2023
fc5c070
Import sbmlmath locally to avoid missing package
kkaris Jul 12, 2023
2f98870
Skip tests that use sbmlmath
kkaris Jul 12, 2023
fdae5b9
Put sbmlmath under its own extra
kkaris Jul 12, 2023
59c4b61
Revert to python 3.8, skip sbmlmath install
kkaris Jul 12, 2023
0b5f08c
Update skip message
kkaris Jul 13, 2023
b797b3b
Use sir model with expressions filled out
kkaris Jul 13, 2023
f1b78e1
Add functionality to test a docker running locally
kkaris Jul 13, 2023
d57aeb5
Add comment on how to install sbmlmath locally
kkaris Jul 13, 2023
fcfece4
Update dkg docker file
kkaris Jul 13, 2023
7fdc3ce
Fix grammar
kkaris Jul 13, 2023
ddbbd06
Try disabling codecov
kkaris Jul 13, 2023
6bfd262
Skip another sbmlmath dependent test
kkaris Jul 13, 2023
e30399c
Mark sbmlmath tests and skip them on github
kkaris Jul 13, 2023
6c28d39
Import SympyExprStr
kkaris Jul 13, 2023
ca20063
Restore docker build branch
kkaris Jul 21, 2023
37ee300
pip install sbmlmath
kkaris Aug 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ jobs:
run: |
export MIRA_REST_URL=http://34.230.33.149:8771
tox -e py
- name: Upload coverage report to codecov
uses: codecov/codecov-action@v1
if: success()
with:
file: coverage.xml
# - name: Upload coverage report to codecov
# uses: codecov/codecov-action@v1
# if: success()
# with:
# file: coverage.xml
12 changes: 9 additions & 3 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:20.04
FROM ubuntu:22.04

WORKDIR /sw

Expand All @@ -10,6 +10,8 @@ RUN apt-get update && \
apt-get install -y neo4j && \
apt-get install -y git zip unzip bzip2 gcc graphviz graphviz-dev \
pkg-config python3 python3-pip && \
# purge blinker here to avoid pip uninstall error below
apt-get purge -y python3-blinker && \
ln -s /usr/bin/python3 /usr/bin/python

ARG version=2023-07-07
Expand All @@ -25,10 +27,14 @@ RUN wget -O /sw/nodes.tsv.gz https://askem-mira.s3.amazonaws.com/dkg/$domain/bui
neo4j-admin import --delimiter='TAB' --skip-duplicate-nodes=true --skip-bad-relationships=true --nodes /sw/nodes.tsv.gz --relationships /sw/edges.tsv.gz

# Python packages
RUN python -m pip install git+https://github.com/indralab/mira.git@main#egg=mira[web,uvicorn,dkg-client] && \
RUN python -m pip install --upgrade pip && \
python -m pip install git+https://github.com/indralab/mira.git@main#egg=mira[web,uvicorn,dkg-client] && \
python -m pip uninstall -y flask_bootstrap && \
python -m pip uninstall -y bootstrap_flask && \
python -m pip install bootstrap_flask
python -m pip install bootstrap_flask && \
python -m pip install --no-dependencies pint && \
python -m pip install --no-dependencies "lxml>=4.6.4" && \
python -m pip install --no-dependencies git+https://github.com/dweindl/sbmlmath.git

# Copy the example json for reconstructing the ode semantics
RUN wget -O /sw/sir_flux_span.json https://raw.githubusercontent.com/indralab/mira/main/tests/sir_flux_span.json
Expand Down
14 changes: 13 additions & 1 deletion mira/metamodel/io.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
__all__ = ["model_from_json_file", "model_to_json_file", "expression_to_mathml"]
__all__ = ["model_from_json_file", "model_to_json_file",
"expression_to_mathml", "mathml_to_expression"]

import json
import sympy
Expand Down Expand Up @@ -53,3 +54,14 @@ def expression_to_mathml(expression: sympy.Expr, *args, **kwargs) -> str:
for old_symbol, new_symbol in mappings.items():
mml = mml.replace(new_symbol, old_symbol)
return mml


def mathml_to_expression(xml_str: str) -> sympy.Expr:
"""Convert a MathML string to a sympy expression."""
from sbmlmath import SBMLMathMLParser
template = """<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML">
{xml_str}
</math>"""
xml_str = template.format(xml_str=xml_str)
return SBMLMathMLParser().parse_str(xml_str)
9 changes: 3 additions & 6 deletions mira/metamodel/units.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,9 @@ class Config:

@classmethod
def from_json(cls, data: Dict[str, Any]) -> "Unit":
expr_str = data.get('expression')
if expr_str:
data['expression'] = sympy.parse_expr(
expr_str, local_dict=UNIT_SYMBOLS
)

# Use get_sympy from askenet.petrinet, but avoid circular import
from mira.sources.askenet.petrinet import get_sympy
data["expression"] = get_sympy(data, local_dict=UNIT_SYMBOLS)
assert data.get('expression') is None or not isinstance(
data['expression'], str
)
Expand Down
98 changes: 63 additions & 35 deletions mira/sources/askenet/petrinet.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,15 @@
- Initials only have a value, cannot be expressions so information on
initial condition parameter relationship is lost
"""
__all__ = ["model_from_url", "model_from_json_file", "template_model_from_askenet_json"]
__all__ = [
"model_from_url",
"model_from_json_file",
"template_model_from_askenet_json",
"get_sympy"
]

import json
from typing import Optional
from copy import deepcopy

import sympy
Expand Down Expand Up @@ -113,43 +119,40 @@ def template_model_from_askenet_json(model_json) -> TemplateModel:
# Next we process initial conditions
initials = {}
for initial_state in ode_semantics.get("initials", []):
initial_expression = initial_state.get("expression")
if initial_expression:
initial_sympy = safe_parse_expr(initial_expression,
local_dict=symbols)
initial_sympy = initial_sympy.subs(param_values)
try:
initial_val = float(initial_sympy)
except TypeError:
continue

initial_expr = get_sympy(initial_state, symbols)
if initial_expr is None:
continue

# If we have an expression, try to evaluate it
initial_expr = initial_expr.subs(param_values)
try:
initial_val = float(initial_expr)
initial = Initial(
concept=concepts[initial_state['target']].copy(deep=True),
value=initial_val
)
initials[initial.concept.name] = initial
except TypeError:
continue

# We get observables from the semantics
observables = {}
for observable in ode_semantics.get("observables", []):
observable_expression = observable.get("expression")
if observable_expression:
observable_sympy = safe_parse_expr(observable_expression,
local_dict=symbols)
observable = Observable(name=observable['id'],
expression=observable_sympy)
observables[observable.name] = observable
observable_expr = get_sympy(observable, symbols)
if observable_expr is None:
continue

observable = Observable(name=observable['id'],
expression=observable_expr)
observables[observable.name] = observable

# We get the time variable from the semantics
time = ode_semantics.get("time")
if time:
time_units = time.get('units')
time_units_obj = None
if time_units:
time_expr = time_units.get('expression')
time_units_expr = safe_parse_expr(time_expr,
local_dict=UNIT_SYMBOLS)
time_units_obj = Unit(expression=time_units_expr)
time_units_expr = get_sympy(time_units, UNIT_SYMBOLS)
time_units_obj = Unit(expression=time_units_expr) \
if time_units_expr else None
model_time = Time(name=time['id'], units=time_units_obj)
else:
model_time = None
Expand Down Expand Up @@ -241,13 +244,8 @@ def state_to_concept(state):
identifiers = grounding.get('identifiers', {})
context = grounding.get('modifiers', {})
units = state.get('units')
units_obj = None
if units:
# TODO: if sympy expression isn't given, parse MathML
expr = units.get('expression')
if expr:
units_expr = safe_parse_expr(expr, local_dict=UNIT_SYMBOLS)
units_obj = Unit(expression=units_expr)
units_expr = get_sympy(units, UNIT_SYMBOLS)
units_obj = Unit(expression=units_expr) if units_expr else None
return Concept(name=name,
display_name=display_name,
identifiers=identifiers,
Expand All @@ -271,10 +269,8 @@ def parameter_to_mira(parameter):
def transition_to_templates(transition_rate, input_concepts, output_concepts,
controller_concepts, symbols, transition_id):
"""Return a list of templates from a transition"""
rate_law_expression = transition_rate.get('expression')
rate_law = safe_parse_expr(rate_law_expression,
local_dict=symbols) \
if rate_law_expression else None
rate_law = get_sympy(transition_rate, local_dict=symbols)

if not controller_concepts:
if not input_concepts:
for output_concept in output_concepts:
Expand Down Expand Up @@ -331,3 +327,35 @@ def transition_to_templates(transition_rate, input_concepts, output_concepts,
subject=input_concepts[0],
outcome=output_concepts[0],
rate_law=rate_law)


def get_sympy(expr_data, local_dict=None) -> Optional[sympy.Expr]:
"""Return a sympy expression from a dict with an expression or MathML

Sympy string expressions are prioritized over MathML.

Parameters
----------
expr_data :
A dict with an expression and/or MathML
local_dict :
A dict of local variables to use when parsing the expression

Returns
-------
:
A sympy expression or None if no expression was found
"""
if expr_data is None:
return None

# Sympy
if expr_data.get("expression"):
expr = safe_parse_expr(expr_data["expression"], local_dict=local_dict)
# MathML
elif expr_data.get("expression_mathml"):
expr = mathml_to_expression(expr_data["expression_mathml"])
# No expression found
else:
expr = None
return expr
7 changes: 7 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,13 @@ docs =
sbml =
python-libsbml
lxml
sbmlmath =
# sbmlmath is py3.9+ only. For version below 3.9, use the following install command locally:
# pip install sbmlmath@git+https://github.com/dweindl/sbmlmath.git --python-version py39 --ignore-requires-python
# It might be necessary to install sbmlmath, pint and lxml using --no-deps as well.
pint
lxml>=4.6.4
sbmlmath @ git+https://github.com/dweindl/sbmlmath.git
bgyori marked this conversation as resolved.
Show resolved Hide resolved

[mypy]
plugins = pydantic.mypy
Expand Down
135 changes: 135 additions & 0 deletions tests/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,136 @@
"""Tests for MIRA."""
import json

from mira.metamodel import SympyExprStr


def sorted_json_str(json_dict, ignore_key=None, skip_empty: bool = False) -> str:
"""Create a sorted json string from a json compliant object

Parameters
----------
json_dict :
A json compliant object
ignore_key :
Key to ignore in dictionaries
skip_empty :
Skip values that evaluates to False, except for 0, 0.0, and False

Returns
-------
:
A sorted string representation of the json_dict object
"""
if isinstance(json_dict, str):
if skip_empty and not json_dict:
return ""
return json_dict
elif isinstance(json_dict, (int, float, SympyExprStr)):
if skip_empty and not json_dict and json_dict != 0 and json_dict != 0.0:
return ""
return str(json_dict)
elif isinstance(json_dict, (tuple, list, set)):
if skip_empty and not json_dict:
return ""
out_str = "[%s]" % (
",".join(sorted(sorted_json_str(s, ignore_key, skip_empty) for s in
json_dict))
)
if skip_empty and out_str == "[]":
return ""
return out_str
elif isinstance(json_dict, dict):
if skip_empty and not json_dict:
return ""

# Here skip the key value pair if skip_empty is True and the value
# is empty
def _k_v_gen(d):
for k, v in d.items():
if ignore_key is not None and k == ignore_key:
continue
if skip_empty and not v and v != 0 and v != 0.0 and v is not False:
continue
yield k, v

dict_gen = (
str(k) + sorted_json_str(v, ignore_key, skip_empty)
for k, v in _k_v_gen(json_dict)
)
out_str = "{%s}" % (",".join(sorted(dict_gen)))
if skip_empty and out_str == "{}":
return ""
return out_str
elif json_dict is None:
return json.dumps(json_dict)
else:
raise TypeError("Invalid type: %s" % type(json_dict))


def expression_yielder(model_json, is_unit=False):
"""Recursively yield all (sympy, mathml) string pairs in the model json

Parameters
----------
model_json :
The model json to yield from
is_unit :
Whether the current expression is a unit

Yields
------
:
A (sympy, mathml) string pair
"""
if isinstance(model_json, list):
for item in model_json:
yield from expression_yielder(item)
elif isinstance(model_json, dict):
if "expression" in model_json and "expression_mathml" in model_json:
yield (model_json["expression"],
model_json["expression_mathml"],
is_unit)

# Otherwise, check if 'units' key is in the dict, indicating that
# the expression is a unit
is_units = "units" in model_json
for value in model_json.values():
# Otherwise, recursively yield from the value
yield from expression_yielder(value, is_units)
# Otherwise, do nothing since we only care about the expression and
# expression_mathml fields in a dict


def remove_all_sympy(json_data, method="pop"):
"""Remove all sympy expressions from the model json by either popping or
clearing the expression field.

Parameters
----------
json_data :
The data to check completion for
method :
The method to use to remove the sympy expression. Either "pop" or
"clear" (default: "pop"). If "pop", the expression is removed from
the dict. If "clear", the expression is set to an empty string.
"""
if method not in ("pop", "clear"):
raise ValueError(f"Invalid method: {method}, must be 'pop', 'clear' "
f"or 'null'")
# Recursively remove all sympy expressions
if isinstance(json_data, list):
for item in json_data:
remove_all_sympy(item)
elif isinstance(json_data, dict):
if "expression" in json_data:
# Remove value
if method == "pop":
json_data.pop("expression")
elif method == "clear":
json_data["expression"] = ""
elif method == "null":
json_data["expression"] = None
else:
# Recursive call
for val in json_data.values():
remove_all_sympy(val)
Loading
Loading