directory structure for the arkouda-contrib repo

[mhmerrill]

1. Proposed directory structure

arkouda-contrib/ExtensionPackage1/client/package/*.py
                                        /setup.py
                                 /server/package/PackageModule.chpl
                                        /server-config.py
                                 /test/client/...
                                      /server/...
               /ExtensionPackage2/client
                                 /server

2. How or can we use git submodule? here is a tutorial
3. recursive build structure using python and/or cmake or some other tools? best practices?
4. other things?

[bmcdonald3]

For adding code to the client, someone from the Chapel team recommended using importlib https://docs.python.org/3/library/importlib.html#importlib.import_module to get the code into the client in a more standard way, rather than just setting ak.__dict__.

I haven't deeply looked into this, but it seems like it would work to me.

[Ethan-DeBandi99]

I agree with @bmcdonald3 on using importlib. I do think it will depend on how we are intending this to work though. importlib.import_module will allow us to run dynamic imports based on some input. For example, if you have a file, required_modules.json, we could read in a list of module names (stored as strings) and import each one.

import json
import importlib
with open('required_modules.json') as json_file:
    mods = json.load(json_file)
    for m in mods:
        importlib.import_module(m)

I am not sure this will have the desired outcome though because you won't be able to access the imported module directly. It would still need to be imported by the user.

If we wanted to allow direct access, we would need a top level package that subpackages can be placed under. Example code of how this would like using akutil below:

import arkouda as ak
empty_df = ak.akutil.DataFrame()

I may be missing something obvious here. I discussed with @pierce314159 as well and we are both thinking it is worth further discussion to ensure we are all at least on the same page with the desired outcome in functionality.

As for the directory structure, I do not have an issue with the proposed configuration. I am interested to see if @bmcdonald3 has an issues when converting it.

[Ethan-DeBandi99]

Adding a note from conversation with @mhmerrill last week.

@glitch wrote code in the arkouda-njit repo to give the commands that need to be run to pull in an external module. It looks like we should be able to adapt this to work here.

Additionally, we should be able to configure something similar if we want to handle pulling the python in directly. I think we have some other options than this as well, but thought I would add this as a discussion point.

[Ethan-DeBandi99]

I wanted to add a few notes here after running down an issue with the Python imports this morning.

In PR #6 and #7, I set it up to run export PYTHONPATH=pathToModule/client. The issue is that unless the user manually runs this, it will not stick. I initially went this route because it is minimal effort on the end users. However, I have listed out some alternatives. This is by no means an exhaustive list, but provides some options to think about.

• Write the export command out to a .sh file. We should then be able to run source ./file.sh. This command should allow for execution manually, via a pipe, or via another script.
• We could install the package locally as an editable copy via conda or pip. This would require validation that the preferred pkg manager is installed. Commands listed for reference pip install -e pkg or conda develop pkg.
• We could require a setup.py file be included in modules and run python3 setup.py install. Noting that this needs to be done from the directory containing setup.py.

All of these would function in the same manner:

import arkouda as ak
import yourModule

# Write some code

If anyone has a preferred method or has other ideas, please let me know. Again, this list is not exhaustive just the ideas I have looked at and seen as viable solutions so far. I plan to test each today and see which will be the easiest for managing on the Arkouda side, but also easiest for a user.

UPDATE
After testing out the options I listed above. I set the code in PR #6 and PR #7 up so that they include a setup.py for the client module. This allows for the module_configuration.py script to configure the installation command pip install <path to pkg>. This has the desired functionality of being able to print the commands only, or to pipe them into another shell for installation. This functionality matches the installation method @glitch configured for the arkouda-njit method for the server modules.

[mhmerrill]

I go confused about where the README.md goes... i didn't see it in the top-level for the package so I put in some issues then it was pointed out that it's in the arkouda-contrib/package/client subdir.