End-to-end differentiable blind tip reconstruction

This repository contains Jupyter notebooks for the end-to-end differentiable blind tip reconstruction (BTR) proposed in Matsunaga et al (2023).

Originally, this method was implemented in the Julia language, but currently, we are implementing the method in PyTorch that can be executed on Google Colab.

For explanations on the method in Japanese, please refer to the following presentation video.

All notebooks are written in Julia programming language. You need to install julia before using the notebooks. Also, the notebooks depend on several packages. The packages can be installed as follows:

$ julia
julia> 
# enter the package mode by pressing ]
pkg> add IJulia Flux Plots cuDNN Statistics BSON Revise HTTP ArgParse MLBase
pkg> add https://github.com/matsunagalab/MDToolbox.jl.git
# return to the REPL mode by pressing BACKSPACE or DELETE
julia> using IJulia
julia> exit()

Descriptions on files

The files are organized as follows. Note that all scripts assume Angstrom as the unit.

single_tip/ contains the twin experiment notebooks for single tip shape
- prepare_test_data.ipynb notebook for preparing pseudo AFM data used in the twin experiment. Recommended to start from this notebook.
- blindtip_original.ipynb notebook performs the original BTR under noise-free condition
- blindtip_opening.ipynb notebook performs the differentiable BTR under noise-free condition
- blindtip_original_randn.ipynb notebook performs the original BTR under noisy condition
- blindtip_opening_randn.ipynb notebook performs the differentiable BTR under noisy condition
double_tip/ contains the twin experiment notebooks for double tip shape
- prepare_test_data.ipynb notebook for preparing pseudo AFM data used in the twin experiment. Recommended to start from this notebook.
- blindtip_original_randn.ipynb notebook performs the original BTR under noisy condition
- blindtip_opening_randn.ipynb notebook performs the differentiable BTR under noisy condition
myosin/ contains the notebooks for the BTR of high-speed AFM data of Myosin V walking
- blindtip_myosin_original_863-892.ipynb notebook performs the original BTR
- blindtip_myosin_opening_863-892.ipynb notebook performs the differentiable BTR

Standalone scripts

Standalone scripts written in Julia are available in script/ directory for the end-to-end differentiable blind tip reconstruction dblindtip.jl, cross validation dblindtip_cv.jl, erosion erosion.jl, dilation dilation.jl, visualization of AFM data (afm2png.jl and afm2gif.jl), visualization of tip (tip2png.jl), and RANSAC ransac.jl (for correcting tilt in AFM images). All the scripts read ASD file or CSV files and write CSV files. For each usage, please see the ouptus of --help option. Note that all scripts assume nm as the unit.

A typical work flow using the scripts would be follows. You can try them using test data in script/data/ (a double-tip case).

0. check the usages and options of scripts

$ cd script/
$ julia dblindtip.jl --help # check usage and options

1. visualize AFM data

# make PNG of CSV files
$ julia afm2png.jl data/
# make GIF of CSV files
$ julia afm2gif.jl --output original.gif data/

2. perform cross validaton (LOOCV) and select an appropriate lambda value

# Perform LOOCV and choose an lambda value according to the one standard error rule
$ julia dblindtip_cv.jl --output cv.png --lambda_start 1.0e-5 --lambda_stop 0.01 --lambda_length 4 data/

Following the one standard error rule, it is recommended to select the largest lambda whose mean (indicate by the solid line in the figure below) falls within or around the one standard deviation of the minimum (indicated by the shaded area). In this case, lambda = 1.0e-4 would be recommended.

3. perform the end-to-end differentiable blind tip reconstruction

# perform the end-to-end differentiable blind tip reconstruction
$ julia dblindtip.jl --output tip.csv --lambda 1.0e-4 data/
# visualize the reconstructed tip
$ julia tip2png.jl tip.csv

4. perform erosion (deconvolution) with the reconstructed tip shape

# perform erosion (deconvolution)
$ julia erosion.jl --tip tip.csv data/
# visualize eroded (deconvoluted) molecular surfaces
$ julia afm2png.jl --ext csv_erosion data/
$ julia afm2gif.jl --output erosion.gif --ext csv_erosion data/

Citation and Acknowledgement

Original blind tip reconstruction

The original BTR in the notebooks is based on the algorithm and code provided by Villarrubia, J. Res. Natl. Inst. Stand. Technol. 102, 425 (1997). If you use the end-to-end differentiable BTR or the original BTR, please cite this paper.

J. S. Villarrubia, 
Algorithms for Scanned Probe Microscope Image Simulation, Surface Reconstruction, and Tip Estimation, 
J. Res. Natl. Inst. Stand. Technol. 102, 425 (1997).

Regularized blind tip reconstruction

The original BTR with an improved regularization scheme is based on F. Tian, X. Qian, and J. S. Villarrubia, Ultramicroscopy 109, 44 (2008), and G. Jóźwiak, A. Henrykowski, A. Masalska, and T. Gotszalk, Ultramicroscopy 118, 1 (2012).

F. Tian, X. Qian, and J. S. Villarrubia, 
Blind Estimation of General Tip Shape in AFM Imaging, 
Ultramicroscopy 109, 44 (2008).

G. Jóźwiak, A. Henrykowski, A. Masalska, and T. Gotszalk, 
Regularization Mechanism in Blind Tip Reconstruction Procedure, 
Ultramicroscopy 118, 1 (2012).