Addressable data store (aka CID store)

[pditommaso]

Tentative implementation for addressable data store (very basic POC so far).

Update on 1 Mar 2025 from #5787 by @jorgee

M1 Implementation of CID store for provenance

Changes:

• CID store is specified by workflow.data.store.location
• Workflow Hash is created based on the workflow and parameters description
• workflow, tasks and outputs metadata are stored in <cid.store.location>/.meta
• references to other cid metadata are cid://<workflow_hash|task_hash/output_target_path
• CID NIO Filesystem to access data based on CIS URLs
• nextflow cid command to log, show and get lineage from CID store metadata

Known Limitations:

• Outputs which are not published in absolutePaths or URLs which are not subfolders both the outputDir, we can not infer the relative output target path. They are not currently tracked in the CID store. We could create a hash for the parent directory of the URL or absolute path and use it as relative folder.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	10f2b87
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67f9773f9f61560008d2c2da

[pditommaso]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

[jorgee]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

I have reverted the changes in this branch and created a new one in PR #5787

[pditommaso]

Some quick wins discussed this morning with @jorgee

• Use Instant object instead of strings in model objects
• Unify nomenclature using taskRun and workflowRun instead of publishBy, runBy, etc
• Use #outputs fragment using run CID instead of a separate result CID
• Use DataOutput in place of WorkflowOutput and TaskOutput

[bentsherman]

TODO: decide whether to use tags or phase it out in favor of annotations

[pditommaso]

Made a few changes to cleanup and simplify some commons patterns.

As general comment try reducing average cyclomatic complexity using smaller method and simpler if-block using if-guard pattern (an example here)

Also it should be increased the test coverage, there are large part not tested.

[pditommaso]

A few more comments:

1. the use of run in DataOutput is ambiguous. It should be used taskRun or workflowRun. I guess the problem is used for both outputs. In the case it should be considered to add both taskRun and workflowRun.
2. we should think how to unify the model for inputs and outputs. Now are used both List<DataPath>, List<Parameter> and Map<String, Object>
3. along the same manner it would be nice unify mainScriptFile and otherScriptFile into a single collection scriptFiles
4. it would be useful to have a symmetry on the usage /outputs and #outputs for tasks along the same manner of workflow

[pditommaso]

This should be review to make it work without the need for locks.

[jorgee]

I followed the same as the executions log file. Two executions could write the log. The other option is converting the .history to a folder and have a file per run, the history log will be the list of the content of the files.

[pditommaso]

The other option is converting the .history to a folder and have a file per run, the history log will be the list of the content of the files

I was thinking the same

[pditommaso]

Likely this can be a class attribute to avoid creating a new instance for each task, see below

[pditommaso]

Let's add a description for these fields

[pditommaso]

This could introduce some unexpected side effect with symlinks. We may need to introduce a specific method for it.

nextflow/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy

Lines 1945 to 1947 in f1a8770

	if (path instanceof RealPathAware){
	path = path.toRealPath()
	}