Skip to content

graphcore/ad-to-ldap-sync

Repository files navigation

black mypy flake8 unit-test integration-test code-analysis markdown-lint yaml-lint docstr-coverage

MS AD to OpenLDAP sync tooling

This tool is used to synchronize groups and users between MS AD and OpenLDAP. In general it is run via a cron job, but can also be used on an ad-hoc basis.

Release process

  • Make a new branch and merge your changes to it
    • Make sure your changes in your feature branch are checked in
    • make test Make sure all tests pass (you might need -i if you have outstanding test failures)
    • In gitlab, create a merge request from your feature branch to main
      • Make sure you squash commits to reduce the amount of clutter in the comments on main
    • Create your new release branch from main
      • git branch to make sure you are on "main"
      • git checkout main to swap to main if you are not there - git pull to ensure you pull your earlier merge from the feature branch
      • git checkout -b v1.0 to create and switch to the new feature branch
      • git push --set-upstream origin v1.0 to push your new release branch to the git server
  • Deploy your release to the production server
    • ssh server.name to log in to the LDAP master
    • sudo -u service_user -i to become the service account user
    • Disable the existing sync scripts
      • crontab -e and comment out (with i and Esc) with a # at the start of each line, both cron jobs (one for user_sync, and one for group_sync)
      • :wq to save and quit the cron editor
    • Update the git repo to your new release version
      • cd ad-to-ldap-sync/ to change to the git repo
      • git pull to get the new branch names
      • git checkout v1.0 to change to the new release branch
      • You are now using the new feature branch
    • Update config file if necessary to include and new config options
      • diff -i --side-by-side --suppress-common-lines config/config-template.yaml config/config.yaml | grep -Ev 'bind_user|bind_pass|ca_certs_file'
    • Make sure that python has all the requirements installed
      • pip install -r requirements.txt
    • Update the commented out cron lines to take account of any new command line options
    • Run a manual test of user_sync and group_sync in noop mode to verify they run as expected. Look for unexpected errors
      • /home/service_user/.pyenv/versions/3.10.1/bin/python /home/service_user/ad-to-ldap-sync/ad_to_ldap_sync.py group_sync --config_file /home/service_user/ad-to-ldap-sync/config/config.yaml --console error for group_sync
      • /home/service_user/.pyenv/versions/3.10.1/bin/python /home/service_user/ad-to-ldap-sync/ad_to_ldap_sync.py user_sync --config_file /home/service_user/ad-to-ldap-sync/config/config.yaml --console error for user_sync
      • Check the monitoring log files both have "True" in them
        • for i in *monitoring.log; do echo $(cat $i); done
      • Look for errors or warnings in either of the script log files
        • less group_sync_ad-to-ldap-sync.log for group sync
        • less user_sync_ad-to-ldap-sync.log for user sync
    • Reinstate the cron job so that the scripts will run automatically
      • crontab -e and remove the two commented out (with i and Esc) lines by removing # from the start (one for user_sync, and one for group_sync)
      • :wq to save and quit the cron editor
    • Release is completed. Bask in the glorious admiration of your colleagues.

Configuration details

Configuration details

Auto-generated API documentation

API documentation

Metrics

Metrics

Development

Breakpoints

At any point in the code you can add breakpoint() to add a breakpoint. In addition to that, set the environment variable $PYTHONBREAKPOINT = 'web_pdb.set_trace'. This will then launch Pdb on port 5555 (web-pdb required). Use your browser of preference to open it and step through.

Sphinx setup

  • Create the basic configuration:
mkdir api_docs && pushd api_docs
sphinx-quickstart \
--no-sep \
--project "MS AD <-> OpenLDAP sync" \
--author "Peter & Martinus" \
--ext-autodoc \
--ext-coverage \
--extensions sphinx.ext.napoleon \
--makefile \
--no-batchfile \
--language en \
--release "v1"

sphinx-apidoc -o . ../src
popd
  • Install the autodocsumm module:
pip install autodocsumm
  • Set the path correctly:
sed -i '1 i\
import os\
import sys\
\
sys.path.insert(0, os.path.abspath(".."))' api_docs/conf.py
  • Set the autodoc options
sed -i '/Options for HTML output/i\
autodoc_default_options = {\
    "autosummary": True,\
    "members": True,\
    "undoc-members": True,\
    "private-members": True,\
}' api_docs/conf.py
  • Set the theme as the default is ....
sed -i 's/alabaster/classic/' api_docs/conf.py
  • Set the side bar and body width:
echo 'html_theme_options = {"sidebarwidth": 400, "body_max_width": "none"}' >> api_docs/conf.py
  • Ignore the modules.rst file
sed -i '1 i\:orphan:\' api_docs/modules.rst

Docstring coverage

pip install docstr-coverage

Profiling

If you want to see where things are slow:

python -m cProfile -o output.pstats ad_to_ldap_sync.py --conf config/config.yaml --exc config/exceptions.yaml --cons debug --count config/country_control.yaml

And to then graph it use gprof2dot:

gprof2dot -f pstats output.pstats | dot -Tpng -o output.png

Creating a compiled executeable

In order to easily share the program over a wide set of systems and not worry about either containers or Python versions and module, you can compile the entire program into a single binary.

For further information see PyInstaller

NOTE: As of this writing, UPX is not used on Linux.

Ensure you have shared libraries

If you use pyenv:

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.10.1

Create the binary

pyinstaller --clean -F ad-to-ldap-sync.py

The binary will be located with the same name in the dist/ directory.

TODO

New feature(s)

  • Email user with link to Password reset page (in check_change) (Future)

  • Cope with large LDAP queries, and pagination (Future)

  • If the AD object in a group is itself a group, recursively descend and build full member list. (soonish)

  • Set recursion limit based on len(called_groups) and config file in _flatten_nested_group

  • Investigate below 'md4' as it is a broken algorythm: https://cwe.mitre.org/data/definitions/327.html

  • Need integration testing for adding multiple users at once.

  • Duplicate duplicate duplicate code between user and group sync

  • Clean up unit tests so they don't leave log files behind.

  • Ensure 100% code coverage for unit tests.

  • Create integration tests for user_sync.

Testing

  • We would benefit from AD "test users" instead of named ones. This would make testing cleaner and safer.

  • Configure OpenLDAP Docker container for integration tests.

  • Run through all methods and compare with unit-tests, verify we do all the things