+
+
+ 404
+ +Page not found :(
+The requested page could not be found.
+Page not found :(
+The requested page could not be found.
+This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at jekyllrb.com
+ +You can find the source code for Minima at GitHub: +jekyll / +minima
+ +You can find the source code for Jekyll at GitHub: +jekyll / +jekyll
+ diff --git a/algorithms.html b/algorithms.html new file mode 100644 index 0000000..f1c8877 --- /dev/null +++ b/algorithms.html @@ -0,0 +1,376 @@ + + + + + + + + + + + + + + + + + + + + +The framework can be easily extended to handle most codes that use gridded data. Currently, any compiled code or Python code can be handled.
+ +TODO
+ +TODO
+ +**
+ + +| Flag | +Description | +Mandatory | +
|---|---|---|
-ti |
+ Training images (one or more images). If multivariate, the last dimension should be the same size as the number of variables, and should also match the size of the array given for the parameter dt. NaN values in the training image will be ignored. Unlike other MPS-algorithms, if there are multiple variables they will not be automatically normalized to be in the same range. | +✔ | +
-di |
+ Destination image (one image, aka simulation grid). The size of di will be the size of the simulation grid. di can be identical as ti for gap-filing. NaN values will be simulated. Non-NaN values will be considered as conditioning data. | +✔ | +
-dt |
+ Data type. 0 -> continuous and 1 -> categorical. This is where the number of variables is defined. If multiple variables, use a single N value if identical for all variables or use an array of N values if each variable has a different N. | +✔ | +
-k |
+ Number of best candidates to consider to compute the narrowness, range [5, inf]. | +✔ | +
-n |
+ N closest neighbors to consider. | +✔ | +
-ki |
+ Image of the weighting kernel. Can be used to normalize the variables. If multiple variables, use a single kernel value if identical for all variables. If each variable has a different kernel, stack all kernels along the last dimension. The number of kernels should then match the size of the array given for the parameter dt. | ++ |
-nw |
+ Narrowness range: 0 -> max-min, 1 -> median, default IQR -> 0.5. | ++ |
-nwv |
+ Number of variables to consider in the narrowness (start from the end), default: 1. | ++ |
-cs |
+ Chunk size, the number of pixels to simulate at the same time, at each iteration, default: 1. | ++ |
-uds |
+ Area to update around each simulated pixel, the M closest pixel default: 10. | ++ |
-mp |
+ Partial simulation, 0 -> empty, 1 -> 100%. | ++ |
-s |
+ Seed value. | ++ |
-j |
+ To run in parallel (default is single core). To use as follows: -j, N1, N2, N3 (all three are optional but N3 needs N2, which in turn needs N1). Use integer values to specify a number of threads (or logical cores). Use decimal values ∈ ]0,1[ to indicate fraction of the maximum number of available logical cores (e.g., 0.5=50% of all available logical cores). N1 threads used to parallelize the path (path-level) Default: the maximum number of threads available. N2 threads used to parallelize over training images (if many TIs are available, each is scanned on a different core). Default: 1. N3 threads used to parallelize FFTs (node-level). Default: 1. N1 and N2 are recommended over N3. N1 is usually more efficient than N2, but requires more memory. |
+ + |
-W_GPU |
+ Use integrated GPU if available. | ++ |
-nV |
+ No Verbatim (experimental). | ++ |
Gravey, M., Rasera, L. G., & Mariethoz, G. (2019). Analogue-based colorization of remote sensing images using textural information. ISPRS Journal of Photogrammetry and Remote Sensing, 147, 242–254. https://doi.org/10.1016/j.isprsjprs.2018.11.003
+ + +Usage: [sim,index,time,finalprogress,jobid] = g2s(flag1,value1, flag2,value2, ...)
+Outputs: sim = simulation, index = index of the simulated values in the flattened TI, time = simulation time, finalprogress = final progression of the simulation (normally 100), jobid = job ID.
| Flag | +Description | +Mandatory | +
|---|---|---|
-ti |
+ Training images (one or more images). If multivariate, the last dimension should be the same size as the number of variables, and should also match the size of the array given for the parameter dt. NaN values in the training image will be ignored. Unlike other MPS-algorithms, if there are multiple variables they will not be automatically normalized to be in the same range. | +✔ | +
-di |
+ Destination image (one image, aka simulation grid). The size of di will be the size of the simulation grid. di can be identical as ti for gap-filing. NaN values will be simulated. Non-NaN values will be considered as conditioning data. | +✔ | +
-dt |
+ Data type. 0 → continuous and 1 → categorical. |
+ ✔ | +
-k |
+ Number of best candidates to consider ∈ [1 ∞]. | +✔ | +
-n |
+ N closest neighbors to consider. If multiple variables: use a single N value if identical for all variables or use an array of N values if each variable has a different N. | +✔ | +
-ki |
+ Image of the weighting kernel. Can be used to normalize the variables. If multiple variables: use a single kernel value if identical for all variables or if each variable has a different kernel, stack all kernels along the last dimension. The number of kernels should then match the size of the array given for the parameter dt. | ++ |
-f |
+ f=1/k equivalent to f of DS with a threshold to 0. | ++ |
-j |
+ To run in parallel (default is single core). To use as follows ‘-j’, N1, N2, N3 (all three are optional but N3 needs N2, which in turn needs N1). Use integer values to specify a number of threads (or logical cores). Use decimal values ∈]0,1[ to indicate fraction of the maximum number of available logical cores (e.g., 0.5=50% of all available logical cores). N1 threads used to parallelize the path (path-level) Default: the maximum number of threads available. N2 threads used to parallelize over training images (if many TIs are available, each is scanned on a different core). Default: 1. N3 threads used to parallelize FFTs (node-level). Default: 1. N1 and N2 are recommended over N3. N1 is usually more efficient than N2, but requires more memory. | ++ |
-sp |
+ Simulation path, array of the size of the simulation grid containing values that specify the simulation path (from low to high). Default is a random path. Equal values are accepted but will always be ordered in the same way (i.e. not random). -∞ values are not simulated. In case of multiple variables, a vector simulation is default (same path for all variables) and the simulation path should be one dimension less than the number of variables. If you prefer a full simulation, provide an array containing the path for each variable and use the “-fs” flag below. | ++ |
-s |
+ Random seed value. | ++ |
-W_GPU |
+ Use integrated GPU if available. | ++ |
-W_CUDA |
+ Use Nvidia Cuda compatible GPU: specify the device id. (e.g., '-W_CUDA',0,1 to use the first two GPUs). |
+ + |
| Flag | +Description | +Mandatory | +
|---|---|---|
-ii |
+ Array that specifies for each pixel which training image to sample from. Default: all training images are searched for the best match. | ++ |
-far |
+ Fast and risky 😄, like -ii but with a random input (experimental). | ++ |
-cti |
+ With this flag QS will treat the training image(s) as periodic (aka circular or cyclic) over each dimension. | ++ |
-csim |
+ With this flag QS will make sure to create a periodic (aka circular or cyclic) simulation over each dimension. | ++ |
-adsim |
+ Augmented dimentionality simulation: allows for 3D simulation using 2D training image, only for categories (Coming maybe some day!). | ++ |
-fs |
+ Full simulation: follows a different simulation path for each variable (as opposed to vector simulation, where the same simulation path is used for all variables). | ++ |
-nV |
+ No Verbatim, i.e. prohibits neighbors in the training image to be neighbors in the simulation. (experimental). | ++ |
Below are several examples showcasing different applications of QS. For these examples the G2S server should be installed and running, either on your own machine or remotely. A Google Colab notebook with more examples and an automatic installation of G2S can be found here.
+ ++ +
+#This code requires the G2S server to be running
+
+import numpy
+from PIL import Image
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+
+# load example training image ('stone')
+ti = numpy.array(Image.open(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff').content)));
+
+# QS call using G2S
+simulation,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',numpy.zeros((200,200))*numpy.nan,
+ '-dt',[0], #Zero for continuous variables
+ '-k',1.2,
+ '-n',50,
+ '-j',0.5);
+
+#Display results
+fig, (ax1, ax2) = plt.subplots(1, 2,figsize=(7,4))
+fig.suptitle('QS Unconditional simulation',size='xx-large')
+ax1.imshow(ti)
+ax1.set_title('Training image');
+ax1.axis('off');
+ax2.imshow(simulation)
+ax2.set_title('Simulation');
+ax2.axis('off');
+plt.show()
+
+%This code requires the G2S server to be running
+
+%load data
+ti=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff');
+
+% QS call using G2S
+simulation=g2s('-a','qs','-ti',ti,'-di',nan(200,200),'-dt',[0],'-k',1.2,'-n',50,'-j',0.5);
+
+%Display results
+sgtitle('Unconditional simulation');
+subplot(1,2,1);
+imshow(ti);
+title('Training image');
+subplot(1,2,2);
+imshow(simulation);
+title('Simulation');
+
++ +
+#This code requires the G2S server to be running
+
+import numpy
+from PIL import Image
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+
+# load example training image ('stone')
+ti = numpy.array(Image.open(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff').content)));
+
+# create empty grid and add conditioning data
+conditioning = numpy.zeros((200,200))*numpy.nan;
+# fill the grid with 50 random points
+conditioning.flat[numpy.random.permutation(conditioning.size)[:50]]=ti.flat[numpy.random.permutation(ti.size)[:50]];
+
+# QS call using G2S
+simulation,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',conditioning,
+ '-dt',[0],
+ '-k',1.2,
+ '-n',30,
+ '-j',0.5);
+
+# Display results
+fig, (ax1, ax2, ax3) = plt.subplots(1, 3,figsize=(7,4),subplot_kw={'aspect':'equal'})
+fig.suptitle('QS Conditional simulation',size='xx-large',y=0.9)
+ax1.imshow(ti)
+ax1.set_title('Training image');
+ax1.axis('off');
+ax2.scatter(*numpy.meshgrid(numpy.arange(200),numpy.arange(200,0,-1)),s=5,c=conditioning,marker='.')
+ax2.set_title('Conditioning');
+ax2.axis('off');
+ax3.imshow(simulation)
+ax3.set_title('Simulation');
+ax3.axis('off');
+plt.show()
+
+%This code requires the G2S server to be running
+
+%load data
+ti=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff');
+% empty grid
+conditioning=nan(200,200);
+% fill the grid with 50 random points
+conditioning(randperm(numel(conditioning),50))=ti(randperm(numel(ti),50));
+
+% QS call using G2S
+simulation=g2s('-a','qs','-ti',ti,'-di',conditioning,'-dt',[0],'-k',1.2,'-n',50,'-j',0.5);
+
+%Display results
+sgtitle('Conditional simulation');
+subplot(1,3,1);
+imshow(ti);
+title('Training image');
+subplot(1,3,2);
+imshow(conditioning);
+title('Conditioning');
+subplot(1,3,3);
+imshow(simulation);
+title('Simulation');
+
++ +
+#This code requires the G2S server to be running
+
+import numpy
+from PIL import Image
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+
+#ti1 contains horizontal lines and ti2 vertical lines
+ti1 = numpy.tile(numpy.sin(range(150)),(150,1))
+ti2 = ti1.transpose()
+
+# QS call using only the horizontal lines as TI
+simulation1,index1,*_ = g2s('-a','qs',
+ '-ti',ti1,
+ '-di',numpy.zeros((150,150))*numpy.nan,
+ '-dt',[1], #1 for categorical variable
+ '-k',1.2,
+ '-n',25,
+ '-j',0.5);
+
+# QS call using both horizontal and vertical lines as TI's
+simulation2,index2,*_ = g2s('-a','qs',
+ '-ti',[ti1,ti2],
+ '-di',numpy.zeros((150,150))*numpy.nan,
+ '-dt',[1],
+ '-k',1.2,
+ '-n',25,
+ '-j',0.5);
+
+#Display results
+fig, ([[ax1,ax2],[ax3,ax4]]) = plt.subplots(2,2,figsize=(10,10),sharex = True,sharey = True)
+fig.suptitle('QS Multiple TI simulation',size='xx-large')
+ax1.imshow(ti1)
+ax1.set_title('Training image 1');
+ax1.axis('off');
+ax2.imshow(ti2)
+ax2.set_title('Training image 2');
+ax2.axis('off');
+ax3.imshow(simulation1)
+ax3.set_title('Simulation 1');
+ax3.axis('off');
+ax4.imshow(simulation2)
+ax4.set_title('Simulation 2');
+ax4.axis('off');
+
+%This code requires the G2S server to be running
+
+%ti1 contains horizontal lines and ti2 vertical lines
+ti=repmat((sin(1:150)>0)',1,150);
+ti1=ti;
+ti2=rot90(ti,1);
+
+%QS call using only the horizontal lines as TI
+simulation1=g2s('-a','qs', ...
+ '-di',nan(150,150,1), ...
+ '-ti',{ti1}, ...
+ '-dt',[1], ...
+ '-k',1.2, ...
+ '-n',25, ...
+ '-j',0.5);
+
+% QS call using both horizontal and vertical lines as TI's
+simulation2=g2s('-a','qs', ...
+ '-di',nan(150,150,1), ...
+ '-ti',{ti1 ti2}, ...
+ '-dt',[1], ...
+ '-k',1.2, ...
+ '-n',25, ...
+ '-j',0.5);
+
+%Display results
+figure(4);clf
+sgtitle('multi-TI');
+subplot(2,2,1);
+imshow(ti1);
+title('Training image 1');
+subplot(2,2,2);
+imshow(ti2);
+title('Training image 2');
+subplot(2,2,3);
+imshow(simulation1);
+title('Simulation with training image 1');
+subplot(2,2,4);
+imshow(simulation2);
+title('Simulation with training images 1 and 2');
+
++ +
+#This code requires the G2S server to be running
+
+import numpy
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+import tifffile as tf
+
+ti = tf.imread(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/ti_3_variables.tiff').content))
+
+###Example 1 - Complete multivariate simulation
+#Here all three variables are entirely unkwown and will be simulated simultaneously
+
+# QS call using G2S (with dt set to two continuous and one categorical variable)
+simulation1,index1,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',numpy.zeros([400,400,3])*numpy.nan,
+ '-dt',[0,0,1],
+ '-k',1.2,
+ '-n',50,
+ '-j',0.5);
+
+#Display results
+fig, ([[ax1,ax2,ax3],[ax4,ax5,ax6],[ax7,ax8,ax9]]) = plt.subplots(3,3,figsize=(15,10),sharex = True,sharey = True)
+fig.suptitle('QS Multivariate simulation - example 2',size='xx-large')
+ax1.imshow(ti[:,:,0])
+ax1.set_title('Training image dim 1');
+ax1.axis('off');
+ax2.imshow(ti[:,:,1])
+ax2.set_title('Training image dim 2');
+ax2.axis('off');
+ax3.imshow(ti[:,:,2])
+ax3.set_title('Training image dim 3');
+ax3.axis('off');
+ax4.imshow(numpy.zeros([400,400])*numpy.nan)
+ax4.set_title('Destination image dim 1')
+ax4.axis('off')
+ax5.imshow(numpy.zeros([400,400])*numpy.nan)
+ax5.set_title('Destination image dim 2')
+ax5.set_aspect('equal')
+ax5.axis('off')
+ax6.imshow(numpy.zeros([400,400])*numpy.nan)
+ax6.set_title('Destination image dim 3')
+ax6.axis('off')
+ax7.imshow(simulation1[:,:,0])
+ax7.set_title('Simulation dim 1');
+ax7.axis('off');
+ax8.imshow(simulation1[:,:,1])
+ax8.set_title('Simulation dim 2');
+ax8.axis('off');
+ax9.imshow(simulation1[:,:,2])
+ax9.set_title('Simulation dim 3');
+ax9.axis('off');
+
+
+
+### Example 2 - Multivariate simulation with partially informed covariables
+#In many situations we will have fully and/or partially informed covariables (e.g. topography + point measurements)
+#To simulate a different environment than the training images, we will use the results from example 1 above
+#Variable 2 is partially informed and variable 3 is fully informed
+
+#Take random values from the previous simulation of variable 2 and add them to the di
+di_var2 = numpy.zeros([400,400])*numpy.nan
+di_var2.flat[numpy.random.permutation(di_var2.size)[:500]] = ti[:,:,1].flat[numpy.random.permutation(ti[:,:,1].size)[:500]]
+di_example2 = numpy.stack([numpy.zeros([400,400])*numpy.nan,
+ di_var2,
+ simulation1[:,:,2]],axis = 2)
+
+# QS call using G2S (with dt set to two continuous and one categorical variable)
+simulation2,index,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',di_example2,
+ '-dt',[0,0,1],
+ '-k',1.2,
+ '-n',50,
+ '-j',0.5);
+
+#Display results
+fig, ([[ax1,ax2,ax3],[ax4,ax5,ax6],[ax7,ax8,ax9]]) = plt.subplots(3,3,figsize=(15,10),sharex = True,sharey = True)
+fig.suptitle('QS Multivariate simulation - example 2',size='xx-large')
+ax1.imshow(ti[:,:,0])
+ax1.set_title('Training image dim 1');
+ax1.axis('off');
+ax2.imshow(ti[:,:,1])
+ax2.set_title('Training image dim 2');
+ax2.axis('off');
+ax3.imshow(ti[:,:,2])
+ax3.set_title('Training image dim 3');
+ax3.axis('off');
+ax4.imshow(di_example2[:,:,0])
+ax4.set_title('Destination image dim 1')
+ax4.axis('off')
+ax5.scatter(*numpy.meshgrid(numpy.arange(400),numpy.arange(400,0,-1)),s=5,c=di_example2[:,:,1],marker='.')
+ax5.set_title('Destination image dim 2')
+ax5.set_aspect('equal')
+ax5.axis('off')
+ax6.imshow(di_example2[:,:,2])
+ax6.set_title('Destination image dim 3')
+ax6.axis('off')
+ax7.imshow(simulation2[:,:,0])
+ax7.set_title('Simulation dim 1');
+ax7.axis('off');
+ax8.imshow(simulation2[:,:,1])
+ax8.set_title('Simulation dim 2');
+ax8.axis('off');
+ax9.imshow(simulation2[:,:,2])
+ax9.set_title('Simulation dim 3');
+ax9.axis('off');
+
+
+%This code requires the G2S server to be running
+
+%load data
+ti=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/ti_3_variables.tiff');
+
+% QS call using G2S (with dt set to two continuous and one categorical
+% variable)
+simulation=g2s('-a','qs','-ti',ti,'-di',nan(400,400,3),'-dt',[0,0,1],'-k',1.2,'-n',30,'-j',0.5);
+
+%Display results
+sgtitle('Unconditional simulation');
+subplot(2,3,1);
+imshow(ti(:,:,1));
+title('Training image dim 1');
+subplot(2,3,2);
+imshow(ti(:,:,2));
+title('Training image dim 2');
+subplot(2,3,3);
+imshow(ti(:,:,3));
+title('Training image dim 3');
+subplot(2,3,4);
+imshow(simulation(:,:,1));
+title('Simulation dim 1');
+subplot(2,3,5);
+imshow(simulation(:,:,2));
+title('Simulation dim 2');
+subplot(2,3,6);
+imshow(simulation(:,:,3));
+title('Simulation dim 3');
+colormap winter;
+
++ +
+#This code requires the G2S server to be running
+
+import numpy
+from PIL import Image
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+
+# load example training image ('stone') and cut out a part of it
+tiWithGap = numpy.array(Image.open(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff').content)));
+tiWithGap[60:140,60:140]=numpy.nan;
+
+# QS call using G2S
+simulation,*_=g2s('-a','qs',
+ '-ti',tiWithGap,
+ '-di',tiWithGap,
+ '-dt',[0],
+ '-k',1.2,
+ '-n',25,
+ '-j',0.5);
+
+#Display results
+fig, (ax1, ax2) = plt.subplots(1, 2,figsize=(7,4))
+fig.suptitle('QS Gap filling',size='xx-large')
+ax1.imshow(tiWithGap)
+ax1.set_title('Training image');
+ax1.axis('off');
+ax2.imshow(simulation)
+ax2.set_title('Simulation');
+ax2.axis('off');
+plt.show()
+
+%This code requires the G2S server to be running
+
+%load data
+tiWithGap=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff');
+tiWithGap(60:140,60:140)=nan;
+
+% QS call using G2S
+simulation=g2s('-a','qs','-ti',tiWithGap,'-di',tiWithGap,'-dt',[0],'-k',1.2,'-n',25,'-j',0.5);
+
+%Display results
+sgtitle('Gap filling');
+subplot(1,2,1);
+imshow(tiWithGap);
+title('Training image');
+subplot(1,2,2);
+imshow(simulation);
+title('Simulation');
+
++ +
+#This code requires the G2S server to be running
+
+import numpy as np
+from PIL import Image
+from g2s import g2s
+import matplotlib.pyplot as plt
+from matplotlib import colors
+from io import BytesIO
+import requests
+
+#Load example training image "Dunes gobi"
+ti_full = Image.open(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/dunes_gobi.tiff').content))
+#Keep one version at fine resolution
+ti_fine = np.array(ti_full)
+#Make a synthetic coarse resolution image by upscaling and reprojecting to the original grid
+ti_coarse = np.array(ti_full.resize(
+ (int(ti_full.width/5),int(ti_full.height/5))).resize(
+ (ti_full.width,ti_full.height),resample=Image.NEAREST))
+
+#Display the full training image at both resolutions
+norm =colors.Normalize(vmin=ti_fine.min(),vmax=ti_fine.max())
+f,(ax1,ax2) = plt.subplots(2,1,figsize=(12,12))
+ax1.imshow(ti_fine,norm=norm)
+ax1.set_title('Fine resolution training image')
+ax2.imshow(ti_coarse, norm =norm)
+ax2.set_title('Coarse resolution training image')
+
+#Crop half of the image to be used as ti
+ti = np.stack((ti_fine[:500,:500],ti_coarse[:500,:500]),axis=2)
+#Crop upper right corner to be used as di
+di_coarse = ti_coarse[:200,-200:]
+di_fine = np.zeros((200,200))*np.nan
+di=np.stack((di_fine,di_coarse),axis=2)
+#dt consists of two zeros representing two continuous variables
+dt = [0]*ti.shape[-1]
+
+# QS call using G2S
+simulation,index,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',di,
+ '-dt',dt,
+ '-k',1.0,
+ '-n',30,
+ '-j',0.5)
+
+#Display results
+norm =colors.Normalize(vmin=ti_fine.min(),vmax=ti_fine.max())
+f,((ax1,ax2),(ax3,ax4)) = plt.subplots(2,2,figsize=(12,12),sharex=True,sharey=True)
+plt.subplots_adjust(wspace=0.1,hspace=0.1)
+plt.suptitle('QS Downscaling',size='xx-large')
+ax1.imshow(di_coarse,norm=norm)
+ax1.set_title('Coarse res di')
+ax2.imshow(simulation[:,:,0], norm =norm)
+ax2.set_title('Simulation')
+ax3.imshow(index)
+ax3.set_title('Index')
+ax4.imshow(ti_fine[:200,-200:],norm=norm)
+ax4.set_title('True image')
+ax4.set_xticks(np.arange(0,200,25),np.arange(ti_full.width-200,ti_full.width,25));
+
+%This code requires the G2S server to be running
+
+%load example training image 'Dunes gobi'
+ti_fine=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/dunes_gobi.tiff');
+
+%create an artificial coarse resolution image on the same grid
+ti_coarse = imresize(imresize(ti_fine,0.2,'nearest'),5,'nearest');
+
+%display the full training image at both resolutions
+figure(1);
+tiledlayout(2,1);
+nexttile;
+image(ti_fine);
+title('Fine TI');
+nexttile;
+image(ti_coarse);
+title('Coarse TI');
+
+ti_size = 500;
+di_size = 200;
+
+%crop half of the image, to be used as ti
+ti = cat(3,ti_fine(1:ti_size,1:ti_size),ti_coarse(1:ti_size,1:ti_size));
+%crop upper right corner to be used as di
+di_coarse = ti_coarse(1:di_size,(end-di_size+1):end);
+di_fine = nan(di_size,di_size);
+di = cat(3,di_fine,di_coarse);
+
+
+% QS call using G2S
+[simulation,index,time]=g2s('-a','qs',...
+ '-ti',ti,...
+ '-di',di,...
+ '-dt',[0 0],... #Zero for continuous variables
+ '-k',1.0,...
+ '-n',30,...
+ '-j',0.5);
+
+figure(2);
+tiledlayout(2,2)
+nexttile;
+image(di_coarse);
+title('Coarse DI');
+nexttile;
+image(simulation(:,:,1));
+title('Simulation');
+nexttile;
+image(index(:,:,1),'CDataMapping','scaled');
+title('Index');
+nexttile;
+image(ti_fine(1:di_size,(end-di_size+1):end));
+title('True image');
+
+
+
++ +
+#This code requires the G2S server to be running
+
+import numpy as np
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+import tifffile as tf
+from matplotlib import cm
+
+#Load example boolean training image "Damier 3D" and crop it to reduce computational requirements
+dim = 30
+ti = tf.imread(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/Damier3D.tiff').content))[:dim,:dim,:dim]
+
+# QS call using G2S
+simulation,index,time,*_=g2s('-a','qs',
+ '-ti',ti,
+ '-di',np.zeros_like(ti)*np.nan,
+ '-dt',[1], #1 for categorical variables
+ '-k',1.2,
+ '-n',30)
+
+#Display results
+fig = plt.figure(figsize=(20,15))
+ax1=fig.add_subplot(121,projection='3d')
+ax1.voxels(ti,alpha=0.9,facecolor='tab:blue',edgecolor='black')
+ax1.view_init(azim=45)
+ax1.set_title('3D Training image')
+ax2=fig.add_subplot(122,projection='3d')
+ax2.voxels(simulation,alpha=0.9,facecolor='tab:blue',edgecolor='black')
+ax2.view_init(azim=45)
+ax2.set_title('QS 3D simulation')
+
+#Display the indices of the ti-values found in the di-image
+viridis = cm.get_cmap('viridis',index.max())
+colormap = viridis(index)
+fig = plt.figure(figsize=(10,10))
+ax1=fig.add_subplot(111,projection='3d')
+ax1.voxels(index,facecolors=colormap)
+ax1.view_init(azim=45);
+
+%This code requires the G2S server to be running
+
+%% 3D multivariate
+
+clear;home;clf
+%download TI (first variable, continuous)
+ti1=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/fold_continuous.tiff');
+%create second variable (categorical) by thresholding the first
+ti2=ti1>0.5;
+ti=cat(4,ti1,ti2);
+ti=double(ti(1:80,1:50,101:120,:));
+
+%display TI
+subplot(2,2,1)
+[X,Y,Z] = meshgrid(1:size(ti,2),1:size(ti,1),1:size(ti,3));
+slice(X,Y,Z,ti(:,:,:,1),[1 size(ti,2)],[1 size(ti,1)],[1 size(ti,3)])
+shading flat
+camproj('perspective')
+axis equal
+title('TI variable 1');
+
+subplot(2,2,2)
+[X,Y,Z] = meshgrid(1:size(ti,2),1:size(ti,1),1:size(ti,3));
+slice(X,Y,Z,ti(:,:,:,2),[1 size(ti,2)],[1 size(ti,1)],[1 size(ti,3)])
+shading flat
+camproj('perspective')
+axis equal
+title('TI variable 2');
+
+% QS call using G2S
+tic
+simulation=g2s('-a','qs','-ti',ti,'-di',nan(80,50,10,2),'-dt',[0,1],'-k',1.2,'-n',{20 10},'-j',0.5);
+toc
+
+%display simulation
+subplot(2,2,3)
+[X,Y,Z] = meshgrid(1:size(simulation,2),1:size(simulation,1),1:size(simulation,3));
+slice(X,Y,Z,simulation(:,:,:,1),[1 size(simulation,2)],[1 size(simulation,1)],[1 size(simulation,3)])
+shading flat
+camproj('perspective')
+axis equal
+title('Simulation variable 1');
+
+subplot(2,2,4)
+[X,Y,Z] = meshgrid(1:size(simulation,2),1:size(simulation,1),1:size(simulation,3));
+slice(X,Y,Z,simulation(:,:,:,2),[1 size(simulation,2)],[1 size(simulation,1)],[1 size(simulation,3)])
+shading flat
+camproj('perspective')
+axis equal
+title('Simulation variable 2');
+sgtitle('3D Simulation with two variables');
+
++ +
+#This code requires the G2S server to be running
+
+import numpy
+from PIL import Image
+import requests
+from io import BytesIO
+from g2s import g2s
+import matplotlib.pyplot as plt
+import time
+
+# load example training image ('stone')
+ti = numpy.array(Image.open(BytesIO(requests.get(
+ 'https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff').content)));
+
+
+# asynchronous QS call using G2S with the "-submitOnly" flag
+jobid_1=g2s('-a','qs',
+ '-submitOnly',
+ '-ti',ti,
+ '-di',numpy.zeros((200,200))*numpy.nan,
+ '-dt',[0],
+ '-k',1.2,
+ '-n',50,
+ '-j',0.5);
+
+# second asynchronous call that waits for job 1 to finish using the "-after" flag
+jobid_2 = g2s('-a','qs',
+ '-after',jobid_1,
+ '-submitOnly',
+ '-ti',ti,
+ '-di',numpy.zeros((200,200))*numpy.nan,
+ '-dt',[0],
+ '-k',1.2,
+ '-n',50,
+ '-j',0.5);
+
+# check the status of both jobs in 2-second intervals using the "-statusOnly" flag
+status_1 = 0
+status_2 = 0
+while status_2 < 95:
+ time.sleep(2)
+ status_1,*_ = g2s('-statusOnly',jobid_1)
+ status_2,*_ = g2s('-statusOnly',jobid_2)
+ print('Status jobs 1 & 2: ', status_1,status_2)
+
+# retrieve the simulation results from the server using the "-waitAndDownload" flag
+# if the simulation would not be ready yet this call would wait for it to be ready
+sim1,*_ = g2s('-waitAndDownload',jobid_1)
+sim2,*_ = g2s('-waitAndDownload',jobid_2)
+
+# display results
+fig, (ax1, ax2,ax3) = plt.subplots(1, 3,figsize=(7,4))
+fig.suptitle('QS Unconditional simulation',size='xx-large')
+ax1.imshow(ti)
+ax1.set_title('Training image');
+ax1.axis('off');
+ax2.imshow(sim1)
+ax2.set_title('Simulation 1');
+ax2.axis('off');
+ax3.imshow(sim2)
+ax3.set_title('Simulation 2');
+ax3.axis('off');
+plt.show()
+
+%This code requires the G2S server to be running
+
+% load example traning image ('stone')
+ti=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/stone.tiff');
+
+% asynchronous QS call using G2S with the "-submitOnly" flag
+jobid_1=g2s('-a','qs','-submitOnly','-ti',ti,'-di',nan(200,200),'-dt',[0],'-k',1.2,'-n',50,'-j',0.5);
+
+% 2nd async call that waits for job 1 to finish using the "-after" flag
+jobid_2=g2s('-a','qs','-after',jobid_1,'-submitOnly','-ti',ti,'-di',nan(200,200),'-dt',[0],'-k',1.2,'-n',50,'-j',0.5);
+
+% check the status of both jobs in 2-second intervals using the
+% "-statusOnly" flag
+status_1 = 0;
+status_2 = 0;
+while status_2 < 95
+ pause(2);
+ status_1=g2s('-statusOnly',jobid_1);
+ status_2=g2s('-statusOnly',jobid_2);
+ fprintf('Status jobs 1 & 2: %s %s\n', status_1, status_2);
+end
+
+% retrieve the simulation results from the server using the
+%"-waitAndDownload" flag. If the simulation would not be ready yet this
+%call would wait for it to be ready
+sim1=g2s('-waitAndDownload',jobid_1);
+sim2=g2s('-waitAndDownload',jobid_2);
+
+% display results
+sgtitle('Unconditional asynchronous simulation');
+subplot(1,3,1);
+imshow(ti);
+title('Training image');
+subplot(1,3,2);
+imshow(sim1);
+title('Simulation 1');
+subplot(1,3,3);
+imshow(sim2);
+title('Simulation 2');
+
+
+
+Gravey, M., & Mariethoz, G. (2020). QuickSampling v1.0: a robust and simplified pixel-based multiple-point simulation approach. Geoscientific Model Development, 13(6), 2611–2630. https://doi.org/10.5194/gmd-13-2611-2020
+ +This code CAN be used for benchmarking (and I invite you to do so 😉), the code needs to run natively on macOS or on Linux using the Intel Compiler with MKL library. The version needs to be reported, and the time needs to be the time reported by the algorithm (that is the true execution time without taking into account interfaces overhead).
+ +When benchmarking, the code should NOT be used inside a Virtual Machine or through WSL on Windows 10+.
+ +This page provides some additional information on how to do common tasks. It also serves as personal note to myself 😁
+ +A question that is frequently asked is how to do multiple realizations. Currently there are three different ways to do it, each one of them has pros and cons.
+ +The simplest solution is to do a for-loop over the realizations. However, in each step the algorithm needs to wait for the data to load, and this creates an overhead
+ +sims1=numpy.empty((250,250,numberOfSimulation));
+for i in range(numberOfSimulation):
+ sims1[:,:,i],*_=g2s('-sa',computationServer,'-a','qs','-ti',ti,'-di',numpy.zeros((250,250))*numpy.nan,'-dt',numpy.ones((1,)),'-k',1.2,'-n',50,'-j',0.5,'-s',100+i);
+By using the G2S submission queue, it’s possible to remove most of the overhead. This is the most versatile solution and it is recommended over the first solution, especially in case the computations are run on another machine. Furthermore, in this solution the parameters can be changed for each realization.
+ +ds=numpy.empty((numberOfSimulation,), dtype='long');
+for i in range(numberOfSimulation):
+ ids[i]=g2s('-sa',computationServer,'-a','qs','-ti',ti,'-di',numpy.zeros((250,250))*numpy.nan,'-dt',numpy.ones((1,)),'-k',1.2,'-n',50,'-j',0.5,'-s',100+i,'-submitOnly');
+
+sims2=numpy.empty((250,250,numberOfSimulation));
+for i in range(numberOfSimulation):
+ sims2[:,:,i],*_=g2s('-sa',computationServer,'-waitAndDownload',ids[i]);
+This is the third solution to get multiple simulations at once. Although this solution looks easier, it has more limitations (e.g., being limited same size, same parameters, etc.), and therefore it is ⚠️ not guaranteed to stay functional in the future. This last solution should only be considered in case of extremely parallelized simulations (number of threads >50) and/or extremely small simulations (less than 1e4 pixels per simulation)
+ +sims3,*_=g2s('-sa',computationServer,'-a','qs','-ti',ti,'-di',numpy.zeros((numberOfSimulation,250,250))*numpy.nan,'-dt',numpy.ones((1,)),'-k',1.2,'-n',50,'-j',0.5);
+Although QS tends to reduce issues related to non-stationarity, it won’t remove them completely. Therefore in cases with high non-stationarity, the challenge is to respect each zone separately. Here we assume that we have different training images for each specific zone.
+ +The first solution for doing a simulation by segment is to add a secondary variable with the information of non-stationarity. In this case the setting of the algorithm has to be identical for both training images.
+ +ti1=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/bangladesh.tiff');
+ti2=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/strebelle.tiff');
+
+dt=[0];
+N=300;
+
+ti1_bis=cat(3,ti1,zeros(size(ti1)));
+ti2_bis=cat(3,ti2,ones(size(ti2)));
+
+localKernel=cat(3,kernel,padarray(15,25*[1,1]));
+% localKernel=cat(3,kernel,kernel);
+
+sim=g2s('-a','qs','-ti',ti1_bis,ti2_bis,'-di',cat(3,nan(size(tiSelection)),tiSelection),'-dt',[0,1],'-n',50,'-k',1.5,'-j',0.5,'-ki',localKernel);
+imshow(medfilt2(sim(:,:,1)))
+drawnow;
+pause(5)
+The second solution for doing a simulation by segment is to use a training image index map. This requires that each training image represents a stationary subdomain. Also in this case the setting of the algorithm needs to be identical between training images.
+ +
+ti1=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/bangladesh.tiff');
+ti2=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/strebelle.tiff');
+
+dt=[0];
+N=300;
+
+sim=g2s('-a','qs','-ti',ti1,ti2,'-di',nan(size(tiSelection)),'-dt',dt,'-n',50,'-k',1.5,'-j',0.5,'-ki',kernel,'-ii',tiSelection);
+imshow(medfilt2(sim))
+pause(5)
+Using the -ni and -kii …
+TODO!!
The last solution for doing a simulation by segment is to do simulations per subdomain. This allows for changing the settings between each subdomain. However each subdomain is simulated sequentially and therefore can be at the origin of some artifacts, regarding the order of simulations.
+ +ti1=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/bangladesh.tiff');
+ti2=imread('https://raw.githubusercontent.com/GAIA-UNIL/TrainingImagesTIFF/master/strebelle.tiff');
+
+dt=[0];
+N=300;
+
+path=reshape(randperm(numel(tiSelection)),size(tiSelection));
+p1=path;
+p1(tiSelection==0)=-inf;
+p2=path;
+p2(tiSelection==1)=-inf;
+sim=g2s('-a','qs','-ti',ti1,'-di',nan(size(tiSelection)),'-dt',dt,'-n',50,'-k',1.5,'-j',0.5,'-ki',kernel,'-sp',p1);
+sim=g2s('-a','qs','-ti',ti2,'-di',sim,'-dt',dt,'-n',50,'-k',1.5,'-j',0.5,'-ki',kernel,'-sp',p2);
+imshow(medfilt2(sim))
+pause(5)
+The GeoStatistical Server (G2S) is a framework that allows you to use state-of-the-art Multiple Point Statistics (MPS) algorithms to run stochastic simulations.
+ +G2S is designed to run simulations in a generic way, independently of the code used or language it is written in. For example, it enables you to run a C/C++ simulation code using Python, or Python using MATLAB (or any other combination).
+ +Currently, the framework is provided with:
+ +The framework can be easily extended to handle most codes that use gridded data. Currently, any compiled code or Python code can be handled.
+ +For a hands-on introduction to MPS (PPT slides, Colab Notebook & recorded tutorial), follow this link.
+ +To use the GeoStatistical Server (G2S), you need to install both the G2S server and an interface that matches your preferred programming language. The server handles the core functionality and computations, while the interface enables communication between your code and the server. Follow the steps below to install both components:
+ +Install the G2S server: The server is the backbone of the G2S framework, responsible for running the simulations. To install the G2S server, follow the instructions provided in the server installation guide.
+Install the G2S interface: Choose the interface that corresponds to your preferred programming language (Python, MATLAB, or R). These interfaces are written in C++, ensuring seamless integration and performance. To install the appropriate G2S interface, refer to the instructions in the interface installation guide.
+After completing the installation of both the G2S server and the corresponding interface, you can start running stochastic simulations using the G2S framework in your preferred programming language.
+ +The GeoStatistical Server (G2S) utilizes a server-interface architecture to provide a versatile, efficient, and user-friendly platform for running geostatistical simulations. This design choice offers several key advantages:
+ +Language and code-independence: By decoupling the server and algorithm implementation (written in highly optimized C++) from the interface, G2S interface can work seamlessly with various programming languages, including Python, MATLAB, and R. Users can run simulations using their preferred language without worrying about compatibility issues.
+Flexibility and extensibility: The server-interface architecture allows for easy integration of new algorithms and techniques as they become available. This modular design ensures that the framework can be updated and improved without disrupting existing workflows.
+Scalability and performance: The algorithms are written in C++, a language known for its high performance and efficiency, ensuring that the simulations are executed quickly and reliably. Additionally, the server can be set up to handle multiple simultaneous requests and remotely, making it suitable for large-scale applications.
+The G2S framework supports Python, MATLAB, and R interfaces, ensuring seamless integration with popular scientific programming languages while maintaining the performance benefits of the underlying C++ implementation. A Python server interface (to implement algorithms in Python) is also available. While originally designed for prototyping, it allows for a quick conversion of existing workflows in G2S.
+ +When running the server on a workstation, sometimes an automatic reboot may be needed (e.g., update, power cut, etc). To make matters worse, we may need to login to start again the server. In addition to machine reboots, sometimes servers can crash.
+ +On Linux machines, it is possible to run the software in the background and launch it automatically at start-up. Here is the installation for Ubuntu, but it’s very simple to adapt it for other systems.
+ +Many solutions exist with cron, etc., but my favorite one is with systemd
+ +/etc/systemd/system/G2S.service [Unit]
+ Description=G2S service
+ After=network-online.target
+ StartLimitIntervalSec=0
+
+ [Service]
+ Type=simple
+ Restart=always
+ RestartSec=1
+ User=tesla-k20c
+ # update {pathToTheServer} with the path where "./server" is located,
+ # intel-build or c++-build, regarding your choice
+ WorkingDirectory={pathToTheServer}
+ # if you use the Intel version, you probably want to load the libraries
+ # like : /bin/bash -c "source /opt/Intel/parallel_studio_xe_2019/bin/psxevars.sh intel64; ./server -kod -age 864000"
+ ExecStart=/bin/bash -c "./server -kod -age 864000"
+ # -age 864000 ==> keep the file for 10 days
+ # -kod ==> don't remove old files in case of reboot or crash
+
+ [Install]
+ WantedBy=multi-user.target
+systemctl start G2Ssystemctl enable G2SIn this section, I will give you a brief overview of how to set up G2S on a cluster. In this mode, the computation is not distributed through the cluster, but runs on a single cluster node.
+ +The installation on a cluster can be a quite difficult task. Please don’t hesitate to ask for help from the team that manages the cluster.
+ +The installation really depends on what the cluster team allows you to do or not.
+ +In the best-case scenario, fftw3 is installed in a non-standard path (you may not find two clusters that have the stuff at the same place. Why? To make your life harder, of course! 🤪). Potentially you can have the Intel compiler installed with all the libraries. Please refer to your cluster documentation for this.
+ +Probably ZMQ is not already installed, and don’t expect that they will install it for you. Also, jsoncpp is most probably missing too. In order to make life a bit easier for you, I prepared a small script that downloads and compiles all of these libraries and even modifies the Makefile.
+ +build/forClusters/compileExternalLibs.sh (I assume that the front and the computational nodes are from the same generation, in other cases you can run into some unsupported instruction issues).The previous script updates the Makefile automatically, so refer to the standard Linux installation. ⚠️ Be careful to compile in a computational node and not on the front node; otherwise, you can have lots of painful issues later.
+ +Take a deep breath and don’t give up; it’s almost finished!
+ +Now the challenging part starts. It is highly dependent on how lucky you are and how much energy you want to spend to make it work. But keep this in mind, spending some time now can make you win a lot later.
+ +You can use this solution if you don’t have a queuing system. You really don’t? Are you sure you are on a cluster?
+ +ssh -L 8128:theNodeName:8128 clusterAddress, this will redirect your localhost to the computation node.The goal here is to create a task for each simulation and to add it to the queue. Again each cluster is different, some use PBS (qsub), some use SLURM(sbatch, srun),… (ok, it’s not the place to debate about this). We need to convert each call to an algorithm that doesn’t starts the computation, but instead submits it in the queue.
+ +⚠️ currently computation interruption does not work on some clusters (Ctrl+C as well as ‘-kill’). Therefore, you need to manage job interruptions with the submission queue system. I will try to fix this issue as soon as I can, however it’s not on my priority list now.
+ +If you are lucky you may be able to keep the server running all the time on the front node. Then ….
+ +You can try to run server in demons -d, if it’s authorized. So you can always cut the connection even if you submit one million tasks at the same time.
+ +Then… if you’re even luckier you can open a port on the front node (here 8128). Don’t hesitate to ask the support if they can do this favor for you. If you can, 1) invest in a good bottle of wine for the support department, 2) If you are that lucky you should play lottery!
+ +So, once the port is open and the server are executed as a deamon, you can close all connections, and simply put the address of the front node in each call to g2s ‘-sa’,clusterAddress
+ +Once the server runs on the front node use ‘‑submitOnly’,’‑statusOnly’ and ‘‑waitAndDownload’ to manage your computations. The G2S server is coming with its own submission queue and dependency manager. Because the dependencies are unknown, each job runs sequentially by default. To over pass this limitation we can simply specify dependency as :’‑after’,0, so each job can start in parallel.
+ +A standard job submission should look like id=g2s(… usualParameter,’‑statusOnly’,’‑after’,0). +and to get the final result sim=g2s(‘‑waitAndDownload’,id)
+ +NodeParallel (NP) allow to forward command from one node to another, in our context from the login node to some worker nodes (it require a shared file system). Therefore we can use NP to forward computation command to computation nodes without making a new task in the submission queue. This is particularly interesting when we have lots of short jobs (it allows removing the initialization for each task), or when we have significantly more jobs than the number of tasks that we are allowed to submit to the queue.
+ +To use it simple download NP from its github page, follow the instruction to install and add it to the PATH (at least each time you want to use it :) ). Then adapt the compilation for use on cluster using ./configureCluster.sh NP available in the forClusters folder. This configuration is not compatible with a queue submission one, so if you plan to use both strategies in parallel you need to duplicate the G2S installation. Run the NP and G2S servers, both on the login node.
+ ++ +
+Simply use pip install G2S
sudo apt install python3-distutils python3-dev python3-numpy -ybuild/python-buildpython3 setup.py install --userSimply run from g2s import g2s; g2s('--version')
+ +
+build/matlab-buildCompileG2Sbuild/build-matlabCompileG2Sbuild/build-matlabCompileG2SSimply run g2s('--version')
TODO !
+ ++ + + + +
+/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"brew install open-geocomputing/brew/g2sg2s command/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"brew install open-geocomputing/brew/g2sg2s command/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"brew install open-geocomputing/brew/g2sg2s commandgit clone https://github.com/GAIA-UNIL/G2S.gitbuild/c++-build/install_needs_W_VM.shgit clone https://github.com/GAIA-UNIL/G2S.gitsudo apt install build-essential libzmq3-dev libjsoncpp-dev zlib1g-dev libfftw3-dev libcurl4-openssl-dev -y (libcurl4-openssl-dev is optional).wget "https://raw.githubusercontent.com/zeromq/cppzmq/master/zmq.hpp" -O ./include/zmq.hppmake -j, if the Intel C++ compiler is installed, the adapted version will be compiled too. The Intel compiler can be downloaded freely in many cases: here.
+To manually select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.make installgit clone https://github.com/GAIA-UNIL/G2S.gitbrew install fftw zmq jsoncpp cppzmq curl Note: curl is optional.build subfolder.make -j, if the Intel C++ compiler is installed the adapted version will be compiled too. The Intel compiler can be downloaded freely in many cases: here. Obviously, Intel compiler is only for Intel CPUs 😁. To manually select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.make installgit clone https://github.com/GAIA-UNIL/G2S.gitsudo port install zmq-devel jsoncpp-devel zlib cppzmq-devel fftw-3 fftw-3-single curl Note: curl is optional.build subfolder.make -j, if the Intel C++ compiler is installed the adapted version will be compiled too. The Intel compiler can be downloaded freely in many cases: here.
+To manually select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.make installbuild/c++-build and run/double-click install.bat.Simply run g2s server to launch the server, optionally accompanied by any of the below settings. Alternatively, run g2s enable to launch the server and make sure it relaunches at start-up when you reboot your machine (and run g2s disable to disable this service again).
+If you did not install g2s through linuxbrew but chose the manual installation from source instead, launch the server by running ./server from within the G2S/build/c++-build (or G2S/build/intel-build for the intel version).
ⓘ The server generates logs- and data-files in subfolders /tmp/G2S/logs and /tmp/G2S/data that are saved for debug purposes. They are currently automatically removed at the reboot of the computer, at the launch of the server or after one day. This parameterization can be changed with -kod and -age.
+ +| Flag | +Description | +
|---|---|
-d |
+ Run as daemon | +
-To n |
+ Shutdown the server if there is no activity after n seconds, 0 for ∞ (default: 0) | +
-p |
+ Select a specific port for the server (default: 8128) | +
-kod |
+ Keep old data, if the flag is not present all files from previous runs are erased at launch | +
-age |
+ Set the time in second after files need to be removed (default: 86400 s = 1d) | +
| Flag | +Description | +
|---|---|
-mT |
+ Single job at a time (excludes the use of -after, only use if you’re sure to be the only one accessing the server) | +
-fM |
+ Run as a function, without fork | +
-maxCJ n |
+ Limit the maximum number of jobs running in parallel. | +
A call to g2s is needed to launch any computation. Each call to g2s is composed of parameters of G2S and of the name of the algorithm used. Flags do NOT have a specific order.
+ + ++ +
+from g2s import g2s
+data = g2s(...) # it returns a tuple that contains all the output maps and the computing duration
+You can either write '-flag',value or flag=value or a mix of both.
data = g2s(...) % principal output, the simulation
+[data, t] = g2s(...) % the simulation and the computation time
+[data, ...,t] = g2s(...) % the simulation, other outputs map and the computation time
+A call to g2s is needed to launch any computation. Each call to g2s is composed of parameters of G2S and of the name of the algorithm used.
+Flags do NOT have a specific order.
+You can either write '-flag',value or flag=value or a mix of both.
The R interface is not available at this moment. Please contact g2s@mgravey.com if you need it.
+ +| Flag | +Description | +
|---|---|
--version |
+ Return the version and compilation date of the interface | +
-a |
+ The simulation algorithm to be used, it can be ‘qs’, ‘nds’, ‘ds-l’ (DS-like, not maintained) | +
-sa |
+ Server address (default: localhost (the server is local), otherwise provide IP address). Nice when we have a powerful machine dedicated for computation | +
-p |
+ Port where to look for the server (default: 8128). Should be passed as an integer. | +
-silent |
+ Don’t display the progression, useful for scripts | +
-serverStatus |
+ Inform if the server is working properly <1 → error, such as comunication, server shutdown,…) =0 → undefined >1 → server is operational 1: standard server working normaly |
+
-noTO |
+ Deactivate TimeOut on ZMQ communication, useful for slow network (e.g., through internet) | +
-TO |
+ Specify a custom TimeOut on ZMQ communication( uint32 in millisecond ) | +
-shutdown |
+ Shutdown the server, useful at the end of scripts | +
The following options represent the Asynchronous mode, which allows you to submit multiple jobs simultaneously and retrieve the results of each of them later on (as opposed to synchronous communication with the server, where you need to wait until a job is finished before you can submit a new one). You launch the async mode by simply adding the -submitOnly flag to your g2s call. This will give only the job ID as an output, so the g2s call becomes jobid = g2s(flag1,value1, flag2,value2, ...). Don’t forget to always include the server address if it’s not local! See the example section for a demonstration in MATLAB and Python.
| Flag | +Description | +
|---|---|
-submitOnly |
+ Submit a job | +
-statusOnly |
+ Check progression Usage: status = g2s('-statusOnly',jobid) |
+
-waitAndDownload |
+ Download the result Usage: sim,_ = g2s('-waitAndDownload',jobid) |
+
-kill |
+ Kill a given task Usage: g2s('-kill',jobid) |
+
-after |
+ Execute the job after another one is finished (e.g. -after,previousJobId ) |
+