Workflow
OptiNiSt makes it easy to create analysis pipelines using the GUI.
In the workflow field, you can:
Select the data and the algorithms or analysis methods (node).
Connect these nodes to define the order of processing (workflow).
The analysis pipeline can be parallel or bifurcating.
You can create a new workflow by clicking the + button.
Caution By creating workflow, current workflow will be deleted. The records are kept if you have already run the workflow. You can reproduce the workflow from RECORD tab. See details here. By default, an Image node is displayed. This node defines the path to the data to use.
To select the data, follow these steps: Click on the check list icon on the Image node. Select a file or a folder. Choosing a folder makes all the TIFF files in the shown sequence an input set of continuous frames. You can select files in
The image’s shapes are displayed in the file select dialog.
Algorithms for image analysis assumes (time, y, x) or (time, z, y, x).
Please check the data if you looks getting wrong results. If you replace the image file with the same file name, shape cannot be updated automatically.
Please click reload icon besides the checkbox. Note Currently, only image files with {.tif, .TIF, .tiff, .TIFF} extensions are supported. OptiNiSt uses Choosing a folder makes all the TIFF files in the shown sequence an input set of continuous frames. You may not want to modify your original data folder, or you may want to make your data folder visible and accessible to OptiNiSt because imaging data can be large and take time to copy. You can take either strategy in assigning your data path: Upload from GUI Click on the image icon on the node. The LOAD button copies the selected file to your
By this method, you cannot upload multiple files or folder at once. If you want to upload multiple files or folder at once, use the next method. Copy files to Copy your raw data to Warning Be sure to copy under You can copy folder into the input directory. If you put folder, you can see the folder from GUI, SELECT IMAGE dialog like this.
Get file via URL Click on the link icon on the node.
Then input dialog will be shown. You can input the URL of the file you want to use.
Note The URL should be direct link to the one file.
It should be started with ended with the file name with extension. And more, download links require authentication are not supported. Change the setting of This requires modifying source codes. See Each Platforms for Developer installation guide section.
CSV, FLUO, BEHAVIOR These nodes are for csv data.
Once the file selected, you can preview and change settings for the data.
transpose You can use transposed the data as the input by checking this box.
eta, cca, correlation, cross_correlation, granger, glm, lda, and svm assume the input neural data shape is frames x cells matrix.
Because the output of CaImAn and Suite2P on the pipeline is cell x frames, the default setting for neural data for these analyses is set to transpose. Pca and tsne can be done in either direction depending on your purpose.
The function assumes their input to be samples x features. set_header If your csv data has header, check this box.
Set the header index to your data’s header row. (first row is 0)
The data below the header row will be used as the data.
set_index You can add index column to the data by checking this box.
HDF5 This node id for
Once the file selected, you can dig into the data structure and select the data to use.
Connecting the hdf5 node to the pipeline, you can use the selected data in the hdf5 file.
There are some utility algorithms for hdf5 data.
fluo_from_hdf5 is to extract fluorescence data from hdf5 data.
Matlab This node is for Note CSV, HDF5 and Matlab nodes have black output connectors.
Black output connector can be connected to any input connector.
Be careful; this means that it does not check the format correspondence between input and output. MICROSCOPE Currently, we’ve implemented to read following microscope data format. Inscopix(.isxd) NIKON(.nd2) Olympus(.oir) Note We use py_isx package with version 1.0.* for Inscopix data.
Currently, the library supports only CellSet and Movie file types, so you can use only them. Note We implement loading C libraries for reading NIKON and Olympus file.
The libraries are available only on Linux and Windows (Not on MacOS). And due to the license issue, we cannot distribute their library in this project.
You need to get the libraries by yourself and add them to the path. To do that, get the C library files from the manufacturer’s support. use OptiNiSt by developer mode. set environment value put the libraries into If you use Linux, open
You can convert these data into ImageData so that you can use them in the pipeline.
To convert, connect microscope_to_img Algorithm node to the microscope data node. Select Data or Algorithm node from the treeview on the left.
Clicking the + button adds the analysis nodes to the Workflow field.
The left side of the window displays all available analysis methods. ROI detection tools (currently Suite2P, CaImAn and LCCD) are in the “Algorithm” category, and all other pre-installed analyses are in the “optinist” category. Each algorithm node has PARAM button and OUTPUT button.
PARAM You can see or edit parameters for the algorithm.
Clicking param button, the parameters are displayed in the right side of the window.
The names, types, and default values of the parameters are the same as the original algorithms. Refer to the original documentation to confirm the meaning of the parameters. The link list is available at Implemented Analysis. OUTPUT You can check the results of algorithm quickly after the successful execution of the pipeline. For details about the charts, see Visualize.
Connect colored connectors of the nodes by dragging your cursor from the output connector(right side of the nodes) to the next input connector(left side of the nodes) to create connecting edges.
The color of the connector indicates the data type of the input and the output.
You can only connect the input and output connectors of the same color. DataType List ImageData Suite2pData Fluorescence Behavior Iscell Clicking on the x mark on a node or on an edge removes it from the workflow field. You can create same workflow by importing workflow config yaml format file. This feature is useful to share workflow template. Workflow config file can be downloaded from RECORD tab’s executed workflow.
Clicking the import icon button and select shared workflow config yaml.
Then you can create same workflow as the file’s record.
Input file selection will not reproduced because it may not be in your device.
Click the RUN button at the top right. Note that while the workflow running, you cannot There are 2 types of execution. You can select the type by clicking the dropdown button.
Runs the entire process. Assigns a new folder for saving the results. This folder name is only for the user’s convenience. The actual folder name is long digit random letter+number. Available only when the pipeline has been executed. Useful to re-run the workflow with different parameters. By checking parameter changes and addition of new nodes, it would skip already executed processes. e.g. Let’s say you have executed following workflow.
Then you have changed the parameter “block_size” of suite2p_registration from 128,128 to 256,256.
With “RUN”, only following nodes would be executed again. suite2p_registration: because you have changed parameter for this node suite2p_roi: because its upstream data (from suite2p_registration) would be changed. Nodes which would be executed by “RUN” are highlighted in yellow.
Following changes would affect whole workflow. So you cannot use “RUN” button after changing them. Input data NWB settings Caution With RUN, results will be overwritten. To avoid this, use RUN ALL. OptiNiSt’s pipeline construction is based on Snakemake, a pipeline controlling tool for Python scripts.
The Snakemake parameter setting is following. use_conda: If this is on, snakemake uses conda environment. cores: Specifies the number of cores to use. If not specified, snakemake uses number of available cores in the machine. forceall: Flag to indicate the execution of the target regardless of already created output. forcetargets: Users may not want to change this. lock: Users may not want to change this. For details about snakemake parameters please refer to snakemake official page. Defines the metadata associated with the input data.
By configuring this, the output NWB file includes the information set here.
You can leave this setting as the default.
The details of NWB setting in OptiNiSt are following. session_description: a description of the session where this data was generated identifier: a unique text identifier for the file experiment_description: general description of the experiment device: device used to acquire the data (information such as manufacturer, firmware version, model etc.) optical_channel: information about the optical channel used to acquire the data imaging_plane: information about imaging such as sampling rate, excitation wave length, calcium indicator. image_serises: information about imaing time ophys: general information about imaging For general information about NWB, refer to NWB official page. Note Basically, these parameters are description for your experiment.
So, they are not used for analysis except for following parameters. This will be used as parameter for frame rate.
See details in Switch time course plot units. If True, raw image data will be saved to NWB file’s acquisition.
If not, only the path to the image data will be saved.
Creating workflow
Setting Input data
input
directory under OPTINIST_DIR
.
To put files there, see next Directory Setting section.
Directory Setting
OPTINIST_DIR
for retrieving data and saving results. OptiNiSt searches for input data in the ‘input’ directory within OPTINIST_DIR
. The default OPTINIST_DIR
is /tmp/studio
on your computer.
OPTINIST_DIR/input
.
OPTINIST_DIR
OPTINIST_DIR/input/1/
by your OS’s file manager or command lines.input/1/
. 1
is the default workspace id for standalone mode.
If you copy under input/
directly, the file cannot be found from GUI.
http://
or https://
.OPTINIST_DIR
OPTINIST_DIR
is defined in optinist/studio/app/dir_path.py
. Change line for OPTINIST_DIR
, INPUT_DIR
, and OUTPUT_DIR
according to your demand. Changing dir_path.py
may also be necessary when running the pipeline on your cluster computers. Also, you can quickly change OPTINIST_DIR
by changing the environment variable before launching. The change is effective after relaunching.Other Data Formats As The Input
.hdf5
and .nwb
. NWB is compatible with the HDF5 data format. So you can use your NWB result’s data as the input..mat
file.
Like hdf5 node, you can dig into the data structure and select the data to use.
MICROSCOPES_LIBRARY_DIR
in studio/config/.env
to your library path.MICROSCOPES_LIBRARY_DIR/dll/{nikon | olympus}/{linux | windows}/
studio/app/optinist/wrappers/optinist/conda/microscope.yaml
and uncomment the - gcc=12
line.Adding Nodes
Algorithm Nodes
Connecting Nodes
Removing Nodes or Connects
Import existing workflow by yaml file
Running pipelines
RUN ALL
RUN
SNAKEMANE settings
NWB setting
imaging_plane.imaging_rate
image_series.save_raw_image_to_nwb