Skip to main content

Folder Scan

The Folder Scan Task scans any folder and returns :

  • on the yellow port of the task any filtered files.
  • any growing or locked files on the orange port of the task

Main tab​

We need to set some parameters in this tab of the task.

main

Input folder
Let you define a file connector and a sub path to the folder to scan. This could be a simple directory, a S3 bucket, a SDTP server, or a AVID Media Central folder

Iterate for each link
Instead of triggering all linked tasks for one iteration, this parameter will trigger one iteration for any linked task. Each task will receive a unique loop_index, and a new iteration will be triggered as soon as a previous one is terminated.

Iterate SmartPath as
When scanning folder, a smartPath for the current file will be created inside context. The value of the parameter 'Iterate SmartPath as' becomes the name of the smartPath created.

Sort by
This parameter allows to sort out iterated files by file name, full path, last updated and size using ascendant or descendant method.

Max running

Max running define the maximum iterations running simultaneously by folder. When it is greater than one, this task will trigger a batch of n iterations and wait for termination of the first one before triggering another one.

Batch size
If Batch size greater than one, each loop_value will be a subset of n elements of the founded files list. Each element will contain file_path, file_name and rel_path. For more information and example, see for each > Batch size.

File filters tab​

To limit dealing with non-valuable files, we can filter the folder by file name or file size.

scan

by File name
The single or list of file name set in this parameter includes or excludes them on the iterated files.

by File size
Files can also be filtered base on its size (Min and Max).

  • Min : this is the minimum file size that will be scanned.
  • Max : this is the maximum file size that will be scanned.

Scan tab​

scan

Keep scanning
It Keeps scanning the folder for new incoming files as long as the condition value is True. Otherwise, if the value of keep scanning is a number(seconds), the folder will to be scanning while the lifetime is not reached.
In the case of the condition is set to False, a single scan of the folder will be executed. While the task is alive, a cache of scanned files will be kept in memory to avoid repetitive iteration of the same files. When a new instance of this task is created, files cache is cleared.

Sleep
Number of seconds before the task will be suspended after each complete folder scan.

Wait for Growing File
Files size can be different from one another and then the time for them to be scanned will be also be different. In fact, the more the file size is big the more the time for them to scanned is big. checking this box means the folder wait until all files are scanned.

growing_file

Deep Scan
If checked, deep scan will scan any files in input folder and all its sub folders. If not checked, only the first level of the input folder is scanned.

Persistent scan
Persistent scan allows in each new scan of the folder to return only previously un-scanned files. The workflow executing the task should be run by a scheduler

The Folder Scan task will add 2 variables into the context The list of scanned files

   iterated_files

The list of removed path which allow you to now when a file has been deletde between 2 folder_scan

    remove_persistant_paths

Persistent file can be view, searched and deleted from the “Persistent paths" tab of the scheduler.

persistent_file