Fairmat 2023 #1271

sanbrock · 2023-06-13T13:21:35Z

Base Classes and Application Definitions to support:

MPES(ARPES,XPS,PEEM)
OptSpec(Ellipsometry,Luminescence,Raman)
EM(EBSD,EDX,EELS)
APM

…efinitions

initial commit docs-gen workflow

typo

initial commit docs-gen workflow

experimenting with using original nexus_definitions docs publication workflow file with our fork of the NIAC repo

restricted workflow to treat only pushes on the fairmat branch and support for manual triggering of the action

remove default steps executed when building nexus pdf manual

…he NeXus-FAIRmat-Proposal and modified descriptions, taking into account technique-specific rst documents from Carola, Tamas, and MarkusK, and templates for rerouting the manual into versioned sub-directories

… extentions to existing base classes

…ced NXellipsometry block selector for plain rich text, file ending yaml should be changed in the future, FAIRmat-related NXDLs were updated having now a single-line docstring

…Xsample, NXmirror

Updated arpes definitions for fairmat-pages

Update our forked main branch with remote developments

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXapm.nxdl.xml # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcoordinate_system_set.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_computer.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXimage_set.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXapm.nxdl.xml # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcoordinate_system_set.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_computer.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXimage_set.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXregistration.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # base_classes/NXaberration.nxdl.xml # base_classes/NXactivity.nxdl.xml # base_classes/NXapm_sim.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXroi.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_obj.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcomponent.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcomponent.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcomponent_em.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcomponent.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_recon.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # base_classes/NXaberration.nxdl.xml # base_classes/NXactivity.nxdl.xml # base_classes/NXapm_sim.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_recon.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXion.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_sys.nxdl.xml # contributed_definitions/NXcs_gpu_obj.nxdl.xml # contributed_definitions/NXcs_gpu_sys.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # applications/NXem.nxdl.xml # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polygon_set.nxdl.xml # base_classes/NXcg_polyhedron_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_roi_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_tetrahedron_set.nxdl.xml # base_classes/NXcg_triangle_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXevent_data_em.nxdl.xml # base_classes/NXevent_data_em_set.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_cpu_obj.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_marching_cubes.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_triangulated_surface_mesh.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_mtex_config.nxdl.xml # contributed_definitions/NXms_odf.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_face_list_data_structure.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_hexahedron_set.nxdl.xml # base_classes/NXcg_parallelogram_set.nxdl.xml # base_classes/NXcg_point_set.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcorrector_cs.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXem_ebsd.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXoptical_system_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXregistration.nxdl.xml # base_classes/NXroi.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # base_classes/NXscanbox_em.nxdl.xml # base_classes/NXspectrum_set.nxdl.xml # base_classes/NXstage_lab.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_mtex_config.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # base_classes/NXaberration.nxdl.xml # base_classes/NXcg_alpha_complex.nxdl.xml # base_classes/NXcg_cylinder_set.nxdl.xml # base_classes/NXcg_ellipsoid_set.nxdl.xml # base_classes/NXcg_geodesic_mesh.nxdl.xml # base_classes/NXcg_grid.nxdl.xml # base_classes/NXcg_half_edge_data_structure.nxdl.xml # base_classes/NXcg_polyline_set.nxdl.xml # base_classes/NXcg_sphere_set.nxdl.xml # base_classes/NXcg_unit_normal_set.nxdl.xml # base_classes/NXchamber.nxdl.xml # base_classes/NXcs_filter_boolean_mask.nxdl.xml # base_classes/NXcs_prng.nxdl.xml # base_classes/NXcs_profiling.nxdl.xml # base_classes/NXcs_profiling_event.nxdl.xml # base_classes/NXdeflector.nxdl.xml # base_classes/NXebeam_column.nxdl.xml # base_classes/NXfabrication.nxdl.xml # base_classes/NXfit_parameter.nxdl.xml # base_classes/NXibeam_column.nxdl.xml # base_classes/NXlens_em.nxdl.xml # base_classes/NXpump.nxdl.xml # base_classes/NXrotation_set.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcomponent_em.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_adf.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_conventions_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXimage_r_set_diff.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXinteraction_vol_em.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_ipf.nxdl.xml # contributed_definitions/NXms_ipf_set.nxdl.xml # contributed_definitions/NXms_odf_set.nxdl.xml # contributed_definitions/NXms_pf.nxdl.xml # contributed_definitions/NXms_pf_set.nxdl.xml # contributed_definitions/NXms_recon.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_recon.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXem_img.nxdl.xml # contributed_definitions/NXem_sim.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXem_img.yaml # contributed_definitions/nyaml/NXem_sim.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXem_img.nxdl.xml # contributed_definitions/NXem_sim.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXem_img.yaml # contributed_definitions/nyaml/NXem_sim.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst

@phyy-nx

* Refactored cg_primitive base classes for two key aspects: i) make all primitives inherit from new base class NXcg_primitive_set which makes a substantial number of repeated properties of specialized primitive base classes unnecessary, ii) introducing a rigorous mathematical set theoretical approach to describe the number type N, N0, Z, R, R+, R, iii) introduce short hand notation to reduce writing costs for shape/dimensions of arrays: now numpy syntax is used, for scalars (), (1, can be omitted, experience in FAIRmat has shown that in almost all cases the optional doc string to a dimension was almost never used and thus yaml based descriptions can be written more succinct, iv) proof-read docstrings and shortened them, introduced the suggestion to use info: as a keyword which should for now be just appended to the docstring as usual but in the future should be treated as a comment, i.e. only meant for human contextualization but ignored for semantic interpretation of concepts, a key difference that I hope this will bring is to reduce also the length of base classes and appdef as they show up in html pages, specifically for appdefs like NXem many of the conceptual ideas which these docstrings currently contain would be much better placed in a proper documentation instead of the actual class definition in the hope that the overall appearance of classes becomes easier and faster for people to power read; maybe this also works against some criticism that NeXus is considered complex, I have to say with the outsourcing of NXcg_primitive_set I feel it is damn simple now * Refactoring of NXcs base classes * First lot of refactoring EM base classes, the rest tomorrow, NXapm will not be refactored before the APT&M2023 conference in Sept2023, also because feedback from Leoben was positive enough that no immediate changes wrt to appdefs and base classes were required * Summarize the current state of the discussion about coordinate systems with the proposed NXcoordinate_system and NXcoordinate_system_set base classes and best practice recommendations how they should be used including feedback from discussions with Florian Dobner and Tamás Haraszti * Summary of the discussion how to handle conventions and method-specific conventions for EM and methods used in EM * Base classes for implemented examples for pole figures, orientation distribution function, event_data, stage_lab, and ebsd_crystal_structure_model * NXms_ipf base class and dos2unix to correct newlines * Introducing NXem_method base class as a template how to write method-specific deep base classes to describe the terms and taxononmy of method-specific branches to be hooked into appdefs like NXem * Base class inheritance proposal for a now even stronger modularized schema set for electron microscopy research, two tasks remain: i) refactor what is now a method-specific instance rather than an appdef (NXem_ebsd) (mainly to be able to fuse all examples of em-specific data converters including new ebsd examples into one em parser for pynxtools), ii) refactor NXem which is now clearly a specific appdef specializing the NXem_base deep base class, specialization work needs to define which fields and groups from all the possible ones now composable via base classes (inheritance) are required in an appdef NXem for NOMAD OASIS * NXem_ebsd refactored into a base class to use it as a method-specific group inside the NXem application definition, next step: i) refactor NXem_sim, ii) finalize refactoring of NXem appdef (for Nomad oasis) * finished draft of NXem_ebsd, NXem_correlation, and NXem appdef, cleaning the branch * Add proposal for storing mtx cfg, fixed nxtime datatypes * 2d microstructure projection * Inspection how proposed, info, N0, N, R, Z value type abbreviations, and dimensions could be added to nyamlforward * A likely too simplistic but at least working nyaml2nxdl forward mapping to explore further usage of refactored EM base classes. Info keyword has to be a child of doc or the respective text be removed from the standard and put into proposal-specific documentation, how to store what and where so that the schema docstring remain succinct and slim but all these conceptual ideas get not forgotten, typically the would be part of a tech report, i.e. in my opinion all what is under info: sections of a docstring should move to some documentation to tell the story to humans, next test these NXDLs with the NeXus documentation system * Minor fix to handle info keyword spotted while compiling the documentation * Fixes to compile with NeXus documentation test suite and sphinx * Deactivated the annoying clean yaml via make clean for dev purposes * Minor fix in em_base, this completes the appdef/base class work for now on the refactored EM, there are still some spurious info fields now, which should be removed when a decision has been made wrt to how to deal with info: keyword fields in general, next steps: i) make decision on info, ii) test refactored EM proposal with pynxtools em-parser v2, iii) implement backwards * Styling via black * Added yaml2nxdl-forward-converted NXDL files to all refactored base classes and the refactored NXem * Added NXroot header for the em appdef and its base template appdef * Continuing the refactoring of EM and APM plus related base classes for CG and MS based on suggestions from user meetings, discussions with Sandor, represents work with the MPES sprint #83 * Continuing on #83 * Continuing #83, NXcs_* * Continuing #83, ipf, pf, odf * Continuing on #83, support classes for EM * Continued on #83, coordinate system, further base classes supporting EM * Continuing #83, event_data_set and event_data description substantially condensed amongst other points * Added cross-references to base classes for rst, continuing #83 * Aligned old NXem_ebsd_conventions with NXcoordinate_system for #83 * Reviewed method-specific base classes, ebsd, eds, eels, #83 * #83, NXms_recon * #83, composed constraints on the NXem appdef * Consolidated with changes that happened in between on the fairmat branch based on 1016aa0, NXms_recon has still an issue and is therefore deactivated currently, method-specific landing pages need to be updated * Consolidated further with fairmat 15624c0 * Fixing some missing references * Fixed syntax error to compile NXms_recon, docs building also now, reviewing intro pages remains * Consistencies of dimensionality to use NX_POSINT and an enumeration * Recompiled NXDL files using new nyaml module 3d500ced7e4ca57683957c1d61a8d0cb62eccf53, removed, modified by taking the one from fairmat, and synced all files which were binarily different between this feature branch and the fairmat branch specifically commit a15798b of the fairmat branch * Deactivated em-based tests which because of a refactoring of em are not expected to work anymore * Fix improper Latex notation in math environment for polyline, face_list, nanochem * Added recompiled NXidentifier, NXserialized NXDLs which triggered pipeline errors in CatChen gh action * Some round of proof-reading * Fixed test_nxdl_utils to reflect and use refactored locations of refactored NXem * Added feedback from @phyy-nx, @PeterC-DLS, and @prjemian from discussed here https://github.com/nexusformat/definitions PR nexusformat#1271 * Black formatting * Reactivated data type check for e.g. NXem NX_NUMBER * Implementing NX_DATE_TIME suggestion of @sanbrock --------- Co-authored-by: markus.kuehbach <markus.kuehbach@hu-berlin.de> # Conflicts: # contributed_definitions/NXaberration.nxdl.xml # contributed_definitions/NXaberration_model.nxdl.xml # contributed_definitions/NXaberration_model_ceos.nxdl.xml # contributed_definitions/NXaberration_model_nion.nxdl.xml # contributed_definitions/NXaperture_em.nxdl.xml # contributed_definitions/NXapm.nxdl.xml # contributed_definitions/NXapm_paraprobe_results_nanochem.nxdl.xml # contributed_definitions/NXcg_alpha_complex.nxdl.xml # contributed_definitions/NXcg_cylinder_set.nxdl.xml # contributed_definitions/NXcg_ellipsoid_set.nxdl.xml # contributed_definitions/NXcg_face_list_data_structure.nxdl.xml # contributed_definitions/NXcg_geodesic_mesh.nxdl.xml # contributed_definitions/NXcg_grid.nxdl.xml # contributed_definitions/NXcg_half_edge_data_structure.nxdl.xml # contributed_definitions/NXcg_hexahedron_set.nxdl.xml # contributed_definitions/NXcg_marching_cubes.nxdl.xml # contributed_definitions/NXcg_parallelogram_set.nxdl.xml # contributed_definitions/NXcg_point_set.nxdl.xml # contributed_definitions/NXcg_polygon_set.nxdl.xml # contributed_definitions/NXcg_polyhedron_set.nxdl.xml # contributed_definitions/NXcg_polyline_set.nxdl.xml # contributed_definitions/NXcg_roi_set.nxdl.xml # contributed_definitions/NXcg_sphere_set.nxdl.xml # contributed_definitions/NXcg_tetrahedron_set.nxdl.xml # contributed_definitions/NXcg_triangle_set.nxdl.xml # contributed_definitions/NXcg_triangulated_surface_mesh.nxdl.xml # contributed_definitions/NXcg_unit_normal_set.nxdl.xml # contributed_definitions/NXchamber.nxdl.xml # contributed_definitions/NXcoordinate_system_set.nxdl.xml # contributed_definitions/NXcorrector_cs.nxdl.xml # contributed_definitions/NXcs_computer.nxdl.xml # contributed_definitions/NXcs_filter_boolean_mask.nxdl.xml # contributed_definitions/NXcs_io_obj.nxdl.xml # contributed_definitions/NXcs_io_sys.nxdl.xml # contributed_definitions/NXcs_mm_sys.nxdl.xml # contributed_definitions/NXcs_prng.nxdl.xml # contributed_definitions/NXcs_profiling.nxdl.xml # contributed_definitions/NXcs_profiling_event.nxdl.xml # contributed_definitions/NXdeflector.nxdl.xml # contributed_definitions/NXebeam_column.nxdl.xml # contributed_definitions/NXem.nxdl.xml # contributed_definitions/NXem_conventions.nxdl.xml # contributed_definitions/NXem_ebsd.nxdl.xml # contributed_definitions/NXem_ebsd_crystal_structure_model.nxdl.xml # contributed_definitions/NXem_img.nxdl.xml # contributed_definitions/NXem_sim.nxdl.xml # contributed_definitions/NXevent_data_em.nxdl.xml # contributed_definitions/NXevent_data_em_set.nxdl.xml # contributed_definitions/NXfabrication.nxdl.xml # contributed_definitions/NXgraph_edge_set.nxdl.xml # contributed_definitions/NXgraph_node_set.nxdl.xml # contributed_definitions/NXgraph_root.nxdl.xml # contributed_definitions/NXibeam_column.nxdl.xml # contributed_definitions/NXimage_set.nxdl.xml # contributed_definitions/NXimage_set_em_adf.nxdl.xml # contributed_definitions/NXimage_set_em_kikuchi.nxdl.xml # contributed_definitions/NXion.nxdl.xml # contributed_definitions/NXisocontour.nxdl.xml # contributed_definitions/NXlens_em.nxdl.xml # contributed_definitions/NXms.nxdl.xml # contributed_definitions/NXms_feature_set.nxdl.xml # contributed_definitions/NXms_score_results.nxdl.xml # contributed_definitions/NXoptical_system_em.nxdl.xml # contributed_definitions/NXorientation_set.nxdl.xml # contributed_definitions/NXpump.nxdl.xml # contributed_definitions/NXrotation_set.nxdl.xml # contributed_definitions/NXscanbox_em.nxdl.xml # contributed_definitions/NXspectrum_set.nxdl.xml # contributed_definitions/NXspectrum_set_em_eels.nxdl.xml # contributed_definitions/NXspectrum_set_em_xray.nxdl.xml # contributed_definitions/NXstage_lab.nxdl.xml # contributed_definitions/nyaml/NXaberration.yaml # contributed_definitions/nyaml/NXaberration_model.yaml # contributed_definitions/nyaml/NXaberration_model_ceos.yaml # contributed_definitions/nyaml/NXaberration_model_nion.yaml # contributed_definitions/nyaml/NXaperture_em.yaml # contributed_definitions/nyaml/NXapm.yaml # contributed_definitions/nyaml/NXapm_paraprobe_results_nanochem.yaml # contributed_definitions/nyaml/NXcg_alpha_complex.yaml # contributed_definitions/nyaml/NXcg_cylinder_set.yaml # contributed_definitions/nyaml/NXcg_ellipsoid_set.yaml # contributed_definitions/nyaml/NXcg_face_list_data_structure.yaml # contributed_definitions/nyaml/NXcg_geodesic_mesh.yaml # contributed_definitions/nyaml/NXcg_grid.yaml # contributed_definitions/nyaml/NXcg_half_edge_data_structure.yaml # contributed_definitions/nyaml/NXcg_hexahedron_set.yaml # contributed_definitions/nyaml/NXcg_marching_cubes.yaml # contributed_definitions/nyaml/NXcg_parallelogram_set.yaml # contributed_definitions/nyaml/NXcg_point_set.yaml # contributed_definitions/nyaml/NXcg_polygon_set.yaml # contributed_definitions/nyaml/NXcg_polyhedron_set.yaml # contributed_definitions/nyaml/NXcg_polyline_set.yaml # contributed_definitions/nyaml/NXcg_roi_set.yaml # contributed_definitions/nyaml/NXcg_sphere_set.yaml # contributed_definitions/nyaml/NXcg_tetrahedron_set.yaml # contributed_definitions/nyaml/NXcg_triangle_set.yaml # contributed_definitions/nyaml/NXcg_triangulated_surface_mesh.yaml # contributed_definitions/nyaml/NXcg_unit_normal_set.yaml # contributed_definitions/nyaml/NXchamber.yaml # contributed_definitions/nyaml/NXcoordinate_system_set.yaml # contributed_definitions/nyaml/NXcorrector_cs.yaml # contributed_definitions/nyaml/NXcs_computer.yaml # contributed_definitions/nyaml/NXcs_filter_boolean_mask.yaml # contributed_definitions/nyaml/NXcs_io_obj.yaml # contributed_definitions/nyaml/NXcs_io_sys.yaml # contributed_definitions/nyaml/NXcs_mm_sys.yaml # contributed_definitions/nyaml/NXcs_prng.yaml # contributed_definitions/nyaml/NXcs_profiling.yaml # contributed_definitions/nyaml/NXcs_profiling_event.yaml # contributed_definitions/nyaml/NXdeflector.yaml # contributed_definitions/nyaml/NXebeam_column.yaml # contributed_definitions/nyaml/NXem.yaml # contributed_definitions/nyaml/NXem_ebsd.yaml # contributed_definitions/nyaml/NXem_img.yaml # contributed_definitions/nyaml/NXem_sim.yaml # contributed_definitions/nyaml/NXevent_data_em.yaml # contributed_definitions/nyaml/NXevent_data_em_set.yaml # contributed_definitions/nyaml/NXfabrication.yaml # contributed_definitions/nyaml/NXgraph_edge_set.yaml # contributed_definitions/nyaml/NXgraph_node_set.yaml # contributed_definitions/nyaml/NXgraph_root.yaml # contributed_definitions/nyaml/NXibeam_column.yaml # contributed_definitions/nyaml/NXimage_set.yaml # contributed_definitions/nyaml/NXinteraction_vol_em.yaml # contributed_definitions/nyaml/NXion.yaml # contributed_definitions/nyaml/NXisocontour.yaml # contributed_definitions/nyaml/NXlens_em.yaml # contributed_definitions/nyaml/NXms.yaml # contributed_definitions/nyaml/NXms_feature_set.yaml # contributed_definitions/nyaml/NXms_score_results.yaml # contributed_definitions/nyaml/NXoptical_system_em.yaml # contributed_definitions/nyaml/NXpump.yaml # contributed_definitions/nyaml/NXrotation_set.yaml # contributed_definitions/nyaml/NXscanbox_em.yaml # contributed_definitions/nyaml/NXspectrum_set.yaml # contributed_definitions/nyaml/NXstage_lab.yaml # dev_tools/tests/test_nxdl_utils.py # manual/source/classes/contributed_definitions/cgms-structure.rst # manual/source/classes/contributed_definitions/em-structure.rst # manual/source/classes/contributed_definitions/icme-structure.rst

sherjeelshabih and others added 30 commits January 27, 2022 11:38

Added EM and APM base_classes and MPES, EM, APM, Ellips application d…

6f4d371

…efinitions

Removed not needed Application definitions for now

0fc141a

Added required base_class by MPES reader

2798f71

Updated version of NXellipsometry.nxdl file

39fd5e4

NXellipsometry.nxdl NX_COMPLEX changed to NX_NUMBER

87746b4

Added definitions for default plottables NXapm

719a5d9

Updated NXellipsometry with attribute in the plot group

c2dc05b

Updated doc string of axes attribute in plot in entry.

b9c6524

Create nexus-fairmat-gen-docs

bb1b16c

initial commit docs-gen workflow

Delete nexus-fairmat-gen-docs

95167ff

typo

Create nexus-fairmat-gen-docs.yml

6c155aa

initial commit docs-gen workflow

Update nexus-fairmat-gen-docs.yml

22ef08d

Create nexus-fairmat-gen-docs.yml

7a84a9e

experimenting with using original nexus_definitions docs publication workflow file with our fork of the NIAC repo

Update nexus-fairmat-gen-docs.yml

c482ccd

restricted workflow to treat only pushes on the fairmat branch and support for manual triggering of the action

Update nexus-fairmat-gen-docs.yml

0b9f8c9

remove default steps executed when building nexus pdf manual

Integrated Tommaso Pincelli's customization of the NeXus manual for t…

bbdfc36

…he NeXus-FAIRmat-Proposal and modified descriptions, taking into account technique-specific rst documents from Carola, Tamas, and MarkusK, and templates for rerouting the manual into versioned sub-directories

Fixed ellipsometry, appdefs and makefile

d122a69

updates to NXmpes and related base classes, including manually merged…

1d46b94

… extentions to existing base classes

__init__.py files added

f8bde29

Moved all FAIRmat-related base and appdefs to contributed_defs, repla…

aa4914d

…ced NXellipsometry block selector for plain rich text, file ending yaml should be changed in the future, FAIRmat-related NXDLs were updated having now a single-line docstring

Fixed single line docstring scanbox and stage_lab

bf7fcf0

Recovered NXdeflector, edited structure.rst from FAIRmat hunted for N…

09ae1f0

…Xsample, NXmirror

Updated arpes definitions for fairmat-pages

17ca677

Fixes pipe error

dd9fd8c

Renams spindispersion fields + small docstring corrections

87017df

Fixes some rst formattings for proper html output

621ea62

Additional naming symplification in NXspindisperision

415cc5d

Merge pull request #2 from FAIRmat-Experimental/fairmat-pages-mpes

c121f30

Updated arpes definitions for fairmat-pages

Updates xsi:schemaLocation

9224acb

Merge pull request #4 from nexusformat/main

8b75670

Update our forked main branch with remote developments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fairmat 2023 #1271

Fairmat 2023 #1271

sanbrock commented Jun 13, 2023

Fairmat 2023 #1271

Fairmat 2023 #1271

Conversation

sanbrock commented Jun 13, 2023