CHANGES.txt

V3.10.6:
  - Improving subset creation
  - PLUGIN_MODULE variable can be used in host.conf to customize mpirun command: mpirun.PLUGIN_MODULE --> mpirun.relion)
       This approach implies having this pattern for all mpi plugins that use mpi.
  - Alternatively, in the scipion.conf file a variable called PARALLEL_COMMAND_XXX could be defined for a more
       versatile way to have a mpi command only for some plugins. Example:
       PARALLEL_COMMAND_RELION=mpirun.relion -np %%(JOB_NODES)d --map-by node %%(COMMAND)s
       NOTE: in this config file, %_ (valid and needed in host.conf) should be replaced by %%
V3.10.5: cancel usage of SCIPION_PROJECT in the host template
V3.10.4: hotfix: Parallelization with GPUs is fixed and step queue executor fixed with the new implementation.
V3.10.3: Failed hotfix
V3.10.2: hotfix: Executor frees the GPU/s used via step id instead of node.
V3.10.1: hotfix: Avoid double Gpu assignment in scipion parallelize protocols with concurrent GPU steps
V3.10.0
developers:
  - stepsExecutionMode = STEPS_SERIAL at class level. Should be a class attribute for future versions. Now both approaches are valid.

V3.9.2 - hotfix: rescue commented methods: Protocol._insertRunJobStep and Protocol._insertCopyFileStep
V3.9.1 - hotfix: loading related objects (CTF - Mics in particle extraction)
V3.9.0
  - Project can have a comment (tooltip over the project name and click will edit it). Will be populated with templates description.
  - Project Color modes:
       size color mode removed: Problematic in slow setups or huge projects.
       labels color mode merged with status color mode.
  - Manual subsets works when SCIPION_USE_QUEUE is active. For now this execution is not sent to the QUEUE: Might need to ask for the queue parameters.
  - Step details show it step needs a GPU
  - Folder size not showing in protocol's tooltip: poor performance in some setups
  - Threads field and help improved: Distinguish between "Scipion threads" and "Threads" (program threads)
  - create script accepts a comment (location optional as -)
  - GPU "queue":
      Basic GPU queue implemented for a more robust GPU specification in combination with threads. See threadmpigpu help page in the doc. Linked in the help.
      Tests added
  - Tolerate non exisiting SCIPION_SPRITES_FILE falling back to default value.
  - Scroll and zoom gestures swap to more standard usage: Control + scroll = zoom, Shift + Scroll = Horizontal scroll.

developers:
  - New constants for "id" and "_objId": ID_COLUMN='id', ID_ATTRIBUTE='_objId'
V3.8.0
users:
  - Inform when a plugin does not import properly
developers:
  - Handling the name and the suffix to be used for a new output
  - New method: Config.isCondaInstallation

V3.7.0
users:
  - Add protocol is an icon.
  - Right clik on canvas offers to add a protocol on the clicked location.
  - Scipion log is a project.log (only for project events)
  - Variables window saves non default variables changed (bug)
  - Queue interoperability review:
    - jobs cancelled in the queue are detected in scipion
    - job and pids handling improved, pids = 0 when use queue for protocol rely on jobid
    - Change Protocols Lock to Rlocks objects to avoid concurrency when updating the protocols jobid
  - add information in the protocol log regarding execution type (pid or jobid)f
  - Streaming protocols based on new streaming (generation steps) are resumable
  - Avoiding a double process when launching a workflow using the protocol form
  - pyworkflowtest tests fixed
  - GUI will detect when protocols processes have been killed(cancelled) in all cases?
  - fix outdated pyworkflow scripts
  - scipion3 template (template manager) lists workflows at workflowhub.eu from "Scipion CNB" organization
  - Join heterogeneous sets are possible for tomography objects TS and tomograms.
developers:
  - Canvas has a testing method: _drawGrid to draw a grid on the canvas (unused now)
  - String.toSeconds:converts to seconds any "duration" string like "1h 4m 34s" -> 3600 + 240 + 34.
  - ListDialog has a refresh method to refresh the content taking into account the filter
  - Improve the DDBB lock in the method "_store". If Protocol._lock is locked, a nullcontext is used instead of another
    Protocol._lock context, which causes a permanent lock.
  - XmlMapper removed.
  - FlatMapper instanatiation centralized:
     New variable: SCIPION_MAPPER_USE_TEMPLATE. By default uses the template as it was now. If deacivated, instantiation happens on each iteration (less performant) but dev friendly
  - Adding a parameter to launch jobs (e.g: bash)
  - Handling item-click event in the treeview dialog
  - Method copyAttributes expanded to manage the complex sets properly

V3.6.0
users:
  - Window to see/edit config variables. Launched from Project menu in project window.
  - Fix installation error: matplotlib imported at installation time
  - Fixing an error drawing the graph
developers:
  - FileBrowser window tweaks: Can be called from a dialog.
  - Windows can be called as dialogs with show(modal=True)
  - askPath: Method to ask for a path (PATH, FOLDER or BASENAME)
  - backup: Method in pyworkflow.utils.path to backup files
  - All variables are centralized and Plugin._defineVar or _defineEmVar accept variable description, and type (VarTypes enum) to better describe the variable.
  - ProgressBar handles the step in combination with increase method call
  - Set.aggregate more flexible and versatile.
        Accepts strings or lists in operations, labels or groupby args.
        Accepts id or _objId ar fields
  - String.getListFromValues: works now with strings with an x inside (e.g.: _x _y)
  - New utils method: valueToList (converts the argument into a list if is not a list): Useful for "dual type arguments) that can be a list or not

V3.5.0
------
users:
  - New Icons.
  - Icons based on sprites, easier to resize or customized
  - New variable: SCIPION_SPRITES_FILE to point to a custom sprite. It should be compatible with sprites.png at pyworkflow/resources.
  - Scipion GUI log moved to the log tabs and removed from menu
  - New variable: SCIPION_SHOW_TEXT_IN_TOOLBAR, set it to anything to show the text of the icons below
  - New variable: SCIPION_ICON_ZOOM, defaults to 50 (percent) to show icons at 16x16. 100 will set them to 32 or 75 to 24.
  - Icons in toolbar grouped. Better row height adjustment in trees.
  - Improved handling of the case where protocols steps are send to the queue, fix stop protocol functionality
developers:
  - ProjectConfig class removed
  - New methods in path: hasChangedSince and isFileFinished
  - New method in utils: strToDuration (useful to translate "1h 35m" to seconds). For duration parameters.
  - New class Variable to be tested in Config class. A step towards a Config editing GUI.
  - SCIPION_DEBUG_SQLITE rescued to activate SQL clauses in debug.
  - Active jobs are stored in the 'jobIds' protocol variable. Iteration through these job IDs allows for canceling each active job sent to the queue.
  - Implemented cleaning of old protocol job IDs and PIDs.
  - Extending the ToolbarDialog buttons. Now is possible to define a tooltip and a shortcut
  - Viewers has getName class method.
  - Object.getClassName is now a class method
  - Validating the Close method in the Dialog class. Now we can to reimplement de validateClose method.
  - Better case-specific implementation of pID and jobID. Three scenarios: normal execution, sending the entire protocol to the queue, or sending individual steps.
  - Object.clone a copy optionally clones the enable flag
  - Set.getItem: returns the first item with a value in the fields passed. Useful for querying by TsId in tomo.
  - Correctly update the protocols in streaming workflows when the input pointer points to a protocol, the protocol points to a set (i.e., classes), and this set points to another set (i.e., particles).

V3.4.0
------
users:
  - Prevent cyclic loops in protocols. Cannot chose as input, as output that is derived or will be derived from the current protocol.
  - Tolerate cyclic loops saved projects projects, disconnecting the "problematic" protocol.
developers:
  - Dialog class: add lockGui input to allow the viewers to prevent getting the screen focus if desired.

V3.3.1
------
users:
  - Steps window can force finish steps too. Useful for some CS cases.
  - "Continue workflow" from improved. Works with more cases.
  - Fix output deletion when right-clicking on an output compatible with this option.
developers:
  - Set class has isItemEnabled to be used as callback based on item.isEnabled()
  - config.py.OS class (not for GUI): to return distro info (isCentos or isUbuntu) or isWSL (--> new dependency "distro")"...

V3.3.0
------
users:
  - Boxes from Plugins in devel mode are prefixed with * now (not delta anymore)
  - Partial exporting/copying will carry also the input of those protocols not being exported
  - Partial importing/pasting will be linked to existing protocols in the project with the same id.
  - Fix: Scheduling failed with direct pointers (impossible to get form GUI but programming or tests)
  - tests: possible to use " tomo." (e.g.) to test just a plugin. Unfortunately needs adaptation in the launcher
  - "Return" keypress "clicks" the active button in a window
  - Avoid adding the same viewer twice

developers:
  - Control and Shift keypress are captured when clicking in analyze result button and passed to the viewer.
  - Project.getProtocol() has a fromRuns mode to prevent going to the db. Used in a couple of calls.(performance)
  - Plotter.createSubPlot: allow plots without axis passing xlabel and ylabel as None
  - Tolerate type change in attributes in protocol execution processes.
  - Fixing an error when loading a project where it is not possible to retrieve the current version of an installed plugin

V3.2.0
------
users:
  - Templates validate booleans properly (false, true, 0 or 1)
  - Labels and box positions are saved after Labels being edited.
  - scipion3 project list (or without list) should list existing projects.
  - Windows are grouped by project under the project name.
  - Do not allow empty Config variables unless is the default value
  - SCIPION_MAIN_COLOR: If not valid, fallsback to Firebrick
  - Project names: Only accepts digits, "letters", - or _. Rest are turned into -
  - Fix wrong PARALLEL_COMMAND in hosts.conf created during tests.
  - SCIPION_USE_QUEUE: New variable (default=False). If defined to anything else than False, new protocols will be sent to queue by default.

developers:
  - Tolerate loading of extra set properties but WARN the developer.
  - ListDialog: allows for a preview panel on the right if previewCallback is passed.
  - SQL DEBUG deprecated: active when regular debug is on.
  - Boolean Config variables refactored to be "reversible".
  - Show info id debug mode (print) when an infinite loop happens in the protocols graph
V3.1.1
------
  - Hotfix: Fix resume single. Protocol was reset.

V3.1.0
------
users:
  - New optional config variable: SCIPION_MAIN_COLOR. Any medium tone color name or hexadecimal should work(like in CSS)
  - New optional config variable: SCIPION_BG_COLOR. Any light tone color name or hexadecimal should work(like in CSS)
  - Control + Plus (numpad) will increase the font of the logs. Control + subtract will restore it.
  - Toolbar and actions refactored: Most of them have shortcuts and appear in the toolbar without text, but have tooltips showing the shortcut
  - Plugins in devel mode are shown with a delta suffix after the package name.
  - Green lines in log are lighter, colored lines are back in the log.
  - Commands in the log are "announced" by "** Running command: **"
  - Project list window: F5 on the filter box will refresh the list.
  - Project list window: Auto refresh on window activate when data is older than 60 seconds
  - New actions: Copy (Control+c) and Paste (Control +v) will allow to copy the selection to and from the clipboard.
  - File browser thumbnail: Writes if the file is a link
  - Protocol in devel mode is shown prefixing the protocol in the box and shown in log
  - Unused graphical gui nodes are cleaned up upon protocols deletion and graph painting (to clean up old projects)
  - Error in host.conf for clusters better described.
  - Fixed a bug related to organizing the workflow when importing a template(Import workflow template option)
  - Terminal where Scipion is being launched only shows ERROR level info. Rest goes to scipion.log shown by the Scipion log menu.
  - Scipion windows of the same process are now grouped under the same icon. Window Tooltip says Scipion framework instead of Toplevel
  - Refactoring Restart & Continue workflow option. Allowing to continue and restart a workflow from a protocol

  
 developers:
  - Plugin.inDevelMode(): returns true is plugin installed in devel mode.
  - Shortcut for devel mode is now Control+D since Control+d is for duplicate (old copy)
  - Protocol.getClassPlugin deprecated. Use instead getPlugin()
  - Protocol._cleanExtraFiles(): optional method to clean files other than tmp folder.
  - Config.isScipionRunning(): To distinguish from just importing or running.
  - Set implements copy() leaving mapper_path out.
  - TaseCase: compareSetProperties() added to compare 2 set's properties.
  - Add debugging messages to check the job status and add renameGpuIds method in the StepQueueExecutor

V3.0.32
-------
  - Hotfix: fix links more versatile.
V3.0.31
-------
  - Hotfix: Fixed updated protocol icon

V3.0.30
-------
users:
  - File browser: error while getting the preview thumbnail are reported as the preview string and logged.
  - Wrong status when updating: When scheduling protocols, they were wrongly showed as failure temporarily.


developers:
  - Using logger in executor.py
  - Fix: --overwrite problem. CONDA_ACTIVATION_CMD was disappearing form the config
  - Fix: plugin manager error. wait.png renamed to wait.gif
  - Improvement: Tolerate protocol loading when attribute type is changed by developers.
  - CONDA_ACTIVATION_CMD is now a Config variable.
  - Flag to indicate if steps list has new steps. Useful for new streaming approach.
  - Base streaming protocol: ProtStreamingBase.

V3.0.29
-------
users:
  - Fix: with some host.conf, where current working dir is not "transferred" to the node, execution failed when
    creating the logs cause they where relative. Now they (log paths) are passed to pw_protocol_run as absolute paths

V3.0.28
-------
users:
  - Fix: rare error making subsets of self referencing output.
  - Schedule process: Logging starts from the very beginning. Better feedback when there is an error.
  - Message is shown in project's info log when discovering viewers for the firs time.
  - Fix: Expert level pointers not clearer anymore after copying or deacivating expert level.
  - Protocol list output improved --> scipion3 protocols --with-doc
  - Scheduling: more informative log lines.
  - Exporting a workflow does not show the final confirmation dialog (just a log line)
  - Font size responsive in file browser text box.
developers:
  - Fix: rare error when loading pointers with extended from the database
  - More modules using logging: gui/project.py, mapper/sqlite.py
  - Pointer.set: optional parameter to cancel extended reset. Used from mapper.
  - Pyworkflow Domain now is operational. Still ED plugins needs adaptation.
  - String class offers getListFromRange and getListFromValues for convenience when param any ot the Ranges
  - getListFromValues, returns str list as before, but optionally you can pass a caster (int, float,..)
  - GE validator: tolerates None values.
  - Form.addBoolean() helper added to facilitate Boolean param definitions. TODO: Do it with the rest?
  - dependencies updated
  - icons changed to png 32, so transparency is understood by pillow latest version
  - unused icons removed
  - icon names all centralized in Icon class
  - getImage: caches the images, rename gif to png.

V3.0.27
-------
Users:
 - Fixing an error refreshing the workflow:
     * the protocol status changed intermittently
 - "Use queue?" label and help reworded and linked to documentation site
 - "Wait for" label and help reworded and linked to documentation site
 - Fix: New protocols, without sibilings were wrongly located at the top.


V3.0.26 (hotfix)
-------
 - Reporting url is now pointing to https.

V3.0.25 (hotfix)
-------
 - Fix logger error when downloading a new dataset.

V3.0.24 (hotfix)
-------
 - Do not fail when getting the size of an empty protocol.

V3.0.23
-------
Users:
 - Fixing a bug for static protocols: when a protocol had another protocol as input, it failed to wait until it had finished.
 - Improvements in responsiveness to font size: Tables and automatic layouts. Object browser, Search windows.
 - Filebrowser has 2 new shortcuts: "recent" and "Protocol folder"
 - Text widget: "File does not exist message" removed.
 - Rescue colors in terminal
 - Rescue output in tests.
 - New color mode: Size (GREEN < 1MB, ORANGE < 1GB, RED < 1TB, DARK > 1TB
 - Color mode info is shown in the "info line" when switching
 - Color mode age: fixed.
 - New config variables:
      SCIPION_LOG_FORMAT: Define log lines format base on python convention,
      SCIPION_TEST_NOSYNC: Disable dataset syncing,
      SCIPION_GUI_REFRESH_INITIAL_WAIT: Initial time to wait after a manual refresh. Defaults to 5.
      SCIPION_GUI_CANCEL_AUTO_REFRESH: True will cancel automatic refreshes.
 - scipion printenv returns also int variables.
 - Tolerate project update errors (e.g.: read only filesystem due to quota + update)
 - Check if scheduled process is alive when updating.

Developers:
 - Some modules switched to use logger instead of print: viewprotocols.py, browser.py and launch.py,
   pw_schedule_run.py, viewprojects.py, canvas.py, project.py, protocol.py.
 - LoggerConfigurator adapted to received logfile, flag for console handler and line format.
 - pw_run_tests.py uses logging now.
 - tests mode with --log now works. All goes to the same file and is not html.
 - Protocol.getPath added = Protocol._getPath
 - form.py, process.py, pw_sync_data.py, utils.py use logging now.
 - All logging messages are affected by SCIPION_LOG_FORMAT (defaults to %(message)).
 - stdout handler does exclude ERROR and CRITICAL messages.
 - Test output can be sent to a log file (--log) and uses SCIPION_LOG_FORMAT.
 - testdata --check-all renamed to --check and optionally can check a single dataset.

V3.0.22
-------
 - Fixing a persistent error related with the logic of the EXECUTE/SCHEDULE button


V3.0.21
-------
 - Generating an interface to create the scipion log file.
 - Creating an option to display the scipion log
 - New "Search protocol in graph" (Control + l) window to locate a box in the graph
 - At loading time, selected protocol is "focused" (window scrolls to it)
 - Control + MouseWheel scrolls horizontally
 - Links are also detected when they are just "http"
 - Bugfixed: Input list is shown when there are possible outputs and need to "expose details" (like selecting a specific volume in a set)
 - Bugfixed: Outputs that a numeric do not break the "Browse object window"
 - Fixing an error related with the logic of the EXECUTE/SCHEDULE button

Developers:
 - prerequisites could be just the id or the previous list of ids when adding a step.

V3.0.20
-------
 - Empty sections/subsection (without protocols) in the left tree will not be shown

Developers:
 - Allow pointers to be defined with a list of classes instead of a string: "SetOfParticles, SetOfAverages" or [ SetOfParticles, SetOfAverages ]
 - Two "utils" functions: hasAnyFileChanged, hasFileChangedSince to check if a file/s have changed since a "time" passed.
 - BaseTest now has: setupTestOutput exposed --> self.setupTestOutput()
 - Allow getting an item in a set not only by id but by a query: myset["_tsId=tomo1"].


V3.0.19
-------
 - Scalar parameters pointing to outputs now works also when scheduling
 - Fix: outputs after a Scalar output (boxSize from pickings) weren't offered to the chosen
 - Defining a new format to handler the log
 - Creating in the config new constants to prevent sqlite locks when updating any set
    - SCIPION_UPDATE_SET_ATTEMPTS: Number of attempts when trying a set update. The default value is 3
    - SCIPION_UPDATE_SET_ATTEMPT_WAIT: Time in seconds to wait between updates. The default value is 2 seconds

Developers:
 - In _showWizard, passing the paramName as a form variable so the wizard can check from which parameter it was called


V3.0.18
-------
 - Fixed GUI errors:
   - fixing the workflow collapse/expand nodes
   - resetting nodes with manual subset looses the parent
   - reporting the protocol wall time
   - finished protocol can't be modified
 - Fixed: execute a protocol directly that is not streaming and that has an open input must be scheduled
 - Close all sets when stopping a protocol
 - fixing an error closing some viewers(the action of closing some viewers closes Scipion completely)
 - fixing an error drawing the protocols step tree
 - fixing an error deleting a protocol by the keyboard
 - logging DEBUG mode will be activated when toggling debug mode
 - A dramatic plugin import error will be tolerated. Plugin will not be loaded.
 - Possible outputs definition is now possible. Protocols can define possible outputs to allow designing
   a workflow in advance without executing them.
 - New variable added: SCIPION_STEPS_CHECK_SEC. Defaults to 3 (seconds). It represents how often new input is checked in streamified protocols. Increasing this   
   value will relax the stress in the system, but will feel less "alive".
 - Workflow templates can now be imported into an existing project

Developers:
 - Possible Outputs allowed: dict or enum definition possible, see Protocol._possibleOutput
 - Allow readonly parameters in protocols. Useful in combination with wizards or just informative parameters.
 - Added the variable SCIPION_PRIORITY_PACKAGE_LIST that is used in the view
   protocols in order to load first the plugins that contains the main
   protocols.conf sections. This variable is defined as string with the packages names separated by a space.
   e.g: SCIPION_PRIORITY_PACKAGE_LIST = "pwem tomo cistem"


v3.0.17
-------
 - Second tab of logs have just error messages (stderr output)
 - Tests show stderr output too (last lines)
 - Report host.conf submit command problems when parsing/formatting fails

 Developers:
 - Logging mechanism closer to a standard way. import logging + logging.getLogger() whould work in any module


v3.0.16
-------
 - Do not suggest deleteing hosts-conf when parsing fails
 - "Select from" option in context menu to select all protocols hanging from the current selection.
 - Schedule optimized
 - dataset download more verbose
 - pyworkflow version shown in main project window
 - new color chooser. More user friendly
 - Viewers are not asked to open outputs generated by themselves. In case a viewer creates an output.
 - File browser shows links
 - File browser allow multiple file handlers for the same extension

 Developers:
 - weakImport tool: context manager to tolerate import errors

-------
v3.0.15
-------
 - Cancel fastcopy of shutil if SCIPION_CANCEL_FASTCOPY is defined (fix GPFS copying errors)
 Developers:
 - Allow renaming parameters using pyworkflow.protocol.params.DeprecatedParam

-------
v3.0.13
-------

April 21, 2021:
 - GUI performance and speed up
 - Improvement of the scheduling process
 - Fixed the options: Restart, Continue, Reset and Stop a workflow