Feedback and improvements for iBridges 2.0

[qubixes]

We are currently thinking about what is next for iBridges. Currently, our main priority for the next release is implementing asynchronous transfers #202. This has the potential to improve the performance of iBridges significantly, but it will most likely be a very big change internally, and also the API will have to change with it. Therefore, we are planning this to be part of the next major release of iBridges, which will be 2.0. Since it will be a major release, we have the option for other breaking changes as well, for example:

• Change arguments, names of functions, module names, etc.
• Change how methods/functions behave
• Change the class structure

Currently, I have some soft list of things we could potentially work on:

• Asynchronous transfers
• Upgrading the CLI to become closer to parity to the API (metadata, checksums, etc)
• Improving our documentation
• A more interactive CLI, for example changing collections

This is where we would like some feedback from the community (if anyone even sees this!). If you have an idea for something you would like to see in iBridges 2.0, let us know here and we can think along.

[JTvD]

A few wishes:

• its happens from time to time to that transfers get interupted, this leads to corrupt files. It would be nice to have a function to detect these for admins and to be able to apply a modrepl:
  """
  with iRODSSession(irods_env_file=env_file,**ssl_settings) as session:
  listofmodrepl = []
  query = session.query(Collection.name,DataObject.name,DataObject.replica_number).filter(Criterion('<>',DataObject.replica_status,'1'))
  for result in query:
  command = 'iadmin modrepl logical_path {}/{} replica_number {} DATA_REPL_STATUS 1'.format(result[Collection.name],result[DataObject.name],result[DataObject.replica_number])
  listofmodrepl.append(command)

for item in listofmodrepl:
print(item)
"""

• Extend the search with an option to check what is on resource X and/not on Y, mainly intended to track replication.
  A common workflow at WUR is that data is ingested in iRODS, stays in hot storage for a while and gets archived.
  If someone make a replicate on hot again to download/process it again, it is quite tricky to find these files.
  We can write our own querries like we did for the modrepl, but this might occur for more people?

[qubixes]

Hi @wurDevTim. Thanks for the suggestions! It sounds like both of those are more for admins than for your regular researcher. Perhaps it would be interesting to have an ibridges-admin-tools companion package that contains some of these workflows that are more geared towards admins? Of course, there might also need to be some changes to the iBridges API to facilitate the integration between the two packages. What do you think?

[chStaiger]

I think we can create a query for the status of replicas and their resource. This would also be useful for "normal" users. We can think of extending the replica information function.

However, for real diagnostics what went wrong in data transfers one would also need to have access to the iRODS server logs. So I think what you request @wurDevTim is more a feature/rule/integration with something like the ELK stack on the server side?!

[chStaiger]

Here the reference to the iadmin modrepl:
https://docs.irods.org/4.3.3/icommands/administrator/#modrepl

[JTvD]

Within WUR I have similar discussions to the one we have here. On the one hand you need admins and icommands to debug and run commands like modrepl. As these commands are to my knowledge not available in the PRC we cannot offer them in iBridges.

At the same time users have no clue what is going on when a resource is full... In our case the cause if often that data is not yet send to tape or someone retrieved a file but forgot to remove the replica on hot again. Or something might have gone wrong somewhere either during the upload or replication. Then even through the user cannot fix the problem (yet, for some things rules are requested), it would be nice to have a way to check for it. Instead of assuming everything is fine unless an admin calls... Especially since we are all busy and therefore the admins are not watching the dashboard 24/7.

There is a fine line here, these are two practical examples I encountered a few times already and I think it's at least worth thinking through if we can make working with iRODS even easier. This might be a step in that direction.