Rewrite links to fix all remaining GTN:003 errors

bin/lint.rb

@@ -151,16 +151,17 @@ def self.youtube_bad(contents)
   def self.link_gtn_tutorial_external(contents)
     find_matching_texts(
       contents,
-      %r{\((https?://(training.galaxyproject.org|galaxyproject.github.io)/training-material/[^)]*)\)}
+      %r{\(https?://(training.galaxyproject.org|galaxyproject.github.io)/training-material/([^)]*)\)}
     )
       .map do |idx, _text, selected|
+        #puts "#{idx} 0 #{selected[0]} 1 #{selected[1]} 2 #{selected[2]} 3 #{selected[3]}"
       ReviewDogEmitter.error(
         path: @path,
         idx: idx,
         # We wrap the entire URL (inside the explicit () in a matching group to make it easy to select/replace)
-        match_start: selected.begin(1),
-        match_end: selected.end(1) + 1,
-        replacement: "{% link #{selected[3]}.md %}",
+        match_start: selected.begin(0) + 1,
+        match_end: selected.end(0),
+        replacement: "{% link #{selected[2].gsub('.html', '.md')} %}",
         message: 'Please use the link function to link to other pages within the GTN. ' \
                  'It helps us ensure that all links are correct',
         code: 'GTN:003'
@@ -908,7 +909,7 @@ def self.format_reviewdog_output(message)
 
       # puts "#{original[0..start_coln - 2]} + #{repl} + #{original[end_coln-1..-1]}"
       fixed = original[0..start_coln - 2] + repl + original[end_coln - 1..]
-      warn "Fixing #{original} to #{fixed}"
+      warn "DIFF\n-#{original}\n+#{fixed}"
       lines[start_line - 1] = fixed
 
       # Save our changes

faqs/galaxy/account_reduce_quota_usage.md

@@ -6,10 +6,10 @@ layout: faq
 box_type: tip
 contributors: [jennaj, bernandez]
 ---
-* [**Download**](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_download_datasets.html) Datasets as individual files or entire Histories as an archive. Then *purge* them from the public server.
-* **Transfer/Move** [Datasets](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_moving_datasets_between_galaxy_servers.html) or [Histories](https://training.galaxyproject.org/training-material/faqs/galaxy/histories_transfer_entire_histories_from_one_galaxy_server_to_another.html) to another Galaxy server, including your own Galaxy. Then purge.
-* [**Copy**](https://training.galaxyproject.org/training-material/faqs/galaxy/histories_copy_dataset.html) your most important Datasets into a new/other History (inputs, results), then purge the original full History.
-* [**Extract**](https://training.galaxyproject.org/training-material/faqs/galaxy/workflows_extract_from_history.html) a **Workflow** from the History, then purge it.
+* [**Download**]({% link faqs/galaxy/datasets_download_datasets.md %}) Datasets as individual files or entire Histories as an archive. Then *purge* them from the public server.
+* **Transfer/Move** [Datasets]({% link faqs/galaxy/datasets_moving_datasets_between_galaxy_servers.md %}) or [Histories]({% link faqs/galaxy/histories_transfer_entire_histories_from_one_galaxy_server_to_another.md %}) to another Galaxy server, including your own Galaxy. Then purge.
+* [**Copy**]({% link faqs/galaxy/histories_copy_dataset.md %}) your most important Datasets into a new/other History (inputs, results), then purge the original full History.
+* [**Extract**]({% link faqs/galaxy/workflows_extract_from_history.md %}) a **Workflow** from the History, then purge it.
 * **Back-up your work**. It is a best practice to download an archive of your FULL original Histories periodically, even those still in use, as a backup.
 
 **Resources** Much discussion about all of the above options can be found at the [Galaxy Help forum](https://help.galaxyproject.org/).

faqs/galaxy/analysis_reporting_issues.md

@@ -8,17 +8,17 @@ contributors: [jennaj, garimavs]
 
 - For reporting **Usage Problems**, related to tools and functions, head to the [Galaxy Help](https://help.galaxyproject.org/) site.
     - **Red Error Datasets**:
-        - Refer to the [Troubleshooting errors](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_troubleshooting.html) FAQ for red error in datasets. 
+        - Refer to the [Troubleshooting errors]({% link faqs/galaxy/analysis_troubleshooting.md %}) FAQ for red error in datasets.
     - **Unexpected results in Green Success Dataset**:
-        - To resolve it you may be asked to send in a shared history link and possibly a shared workflow link. For sharing your history, refer to this [link](https://training.galaxyproject.org/training-material/faqs/galaxy/histories_sharing.html).
-        - To reach our support team, visit [Support FAQs](https://training.galaxyproject.org/training-material/faqs/galaxy/#support).
+        - To resolve it you may be asked to send in a shared history link and possibly a shared workflow link. For sharing your history, refer to this [these instructions]({% link faqs/galaxy/histories_sharing.md %}).
+        - To reach our support team, visit [Support FAQs]({% link faqs/galaxy/ %}#support).
     - **Functionality problems**:
         - Using [Galaxy Help](https://help.galaxyproject.org/) is the best way to get help in most cases.
-        - If the problem is more complex, email a description of the problem and how to reproduce it. 
+        - If the problem is more complex, email a description of the problem and how to reproduce it.
     - **Administrative problems**:
-        - If the problem is present in your own Galaxy, the administrative configuration may be a factor.  
-        - For the fastest help directly from the development community, admin issues can be alternatively reported to the mailing list or the GalaxyProject [Gitter channel](https://gitter.im/galaxyproject/Lobby). 
+        - If the problem is present in your own Galaxy, the administrative configuration may be a factor.
+        - For the fastest help directly from the development community, admin issues can be alternatively reported to the mailing list or the GalaxyProject [Gitter channel](https://gitter.im/galaxyproject/Lobby).
 
 - For **Security Issues**, do not report them via GitHub. Kindly disclose these as explained in this [document](https://github.com/galaxyproject/galaxy/blob/dev/SECURITY_POLICY.md).
-- For **Bug Reporting**, create a Github issue. Include the steps mentioned [here](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_troubleshooting_reporting.html).
+- For **Bug Reporting**, create a Github issue. Include the steps mentioned [in these instructions]({% link faqs/galaxy/analysis_troubleshooting_reporting.md %}).
 - Search the [GTN Search](https://training.galaxyproject.org/search) to find prior Q & A, FAQs, tutorials, and other documentation across all Galaxy resources, to verify in case your issue was already faced by someone.

faqs/galaxy/datasets_chromosome_identifiers.md

@@ -11,9 +11,9 @@ contributors: [jennaj, Melkeb]
 
 - **If using a Custom Reference genome**, the methods below also apply, but the first step is to make certain that the Custom Genome is formatted correctly. Improper formatting is the most common root cause of CG related errors.
 
-Method 1: [Finding BAM dataset identifiers](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_BAM_dataset_identifiers.html)
+Method 1: [Finding BAM dataset identifiers]({% link faqs/galaxy/datasets_BAM_dataset_identifiers.md %})
 
-Method 2: [Directly obtaining UCSC sourced *genome* identifiers](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_UCSC_sourced_genome.html)
+Method 2: [Directly obtaining UCSC sourced *genome* identifiers]({% link faqs/galaxy/datasets_UCSC_sourced_genome.md %})
 
 Method 3: [Adjusting identifiers for UCSC sourced data used with other sourced data](https://galaxyproject.org/support/chrom-identifiers/#adjusting-identifiers-or-input-source)
 

faqs/galaxy/datasets_download_using_command_line.md

@@ -20,7 +20,7 @@ From the terminal window on your computer, you can use **wget** or **curl**.
    > **``curl -o outfile '<link>' ``  
    > ``curl -o outfile --insecure '<link>'     # ignore SSL certificate warnings``  
    > ``curl -C - -o outfile '<link>'           # continue an interrupted download``**
-4. For dataset collections and datasets within collections you have to supply your [API key](https://training.galaxyproject.org/training-material/faqs/galaxy/preferences_admin_api_key.html) with the request
+4. For dataset collections and datasets within collections you have to supply your [API key]({% link faqs/galaxy/preferences_admin_api_key.md %}) with the request
    - Sample commands for **wget** and **curl** respectively are:
    > 
    > **``wget https://usegalaxy.org/api/dataset_collections/d20ad3e1ccd4595de/download?key=MYSECRETAPIKEY``**

faqs/galaxy/datasets_hidden.md

@@ -10,6 +10,6 @@ If you have run a workflow with hidden datasets, in your History:
 - Click the **gear icon** {% icon galaxy-gear %} → Click **Unhide Hidden Datasets**
 - Or use the toggle ``hidden`` to view them
 
-When using the [Copy Datasets](https://training.galaxyproject.org/training-material/faqs/galaxy/histories_copy_dataset.html) feature, hidden datasets will not be available to transfer from the **Source History** list of datasets. To include them:
+When using the [Copy Datasets]({% link faqs/galaxy/histories_copy_dataset.md %}) feature, hidden datasets will not be available to transfer from the **Source History** list of datasets. To include them:
 1. Click the **gear icon** {% icon galaxy-gear %} → Click **Unhide Hidden Datasets**
 2. Click the **gear icon** {% icon galaxy-gear %} → Click **Copy Datasets** 

faqs/galaxy/datasets_quotas_for_datasets_and_histories.md

@@ -10,4 +10,4 @@ contributors: [jennaj, garimavs]
 - _Permanently deleted datasets_ and _permanently deleted histories_ containing datasets **are not considered**.
 - _Histories/datasets that are shared with you_ are only **partially considered** unless you import them.
 
-**Note:** To reduce quota usage, refer to [How can I reduce quota usage while still retaining prior work (data, tools, methods)?](https://training.galaxyproject.org/training-material/faqs/galaxy/reduce_quota_usage.html) FAQ.
+**Note:** To reduce quota usage, refer to [How can I reduce quota usage while still retaining prior work (data, tools, methods)?]({% link faqs/galaxy/account_reduce_quota_usage.md %}) FAQ.

faqs/galaxy/datasets_working_with_fasta.md

@@ -14,11 +14,11 @@ contributors: [jennaj, garimavs]
     - Formatting criteria:
         - All sequence identifiers must be unique.
         - Some tools will require that there is no description line content, only identifiers, in the fasta title line (“>” line). Use **NormalizeFasta** to remove the description (all content after the first whitespace) and wrap the sequences to 80 bases.
-- [Custom genome](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_add_custom_build.html), transcriptome exome?
+- [Custom genome]({% link faqs/galaxy/analysis_add_custom_build.md %}), transcriptome exome?
     - Only appropriate for smaller genomes (bacterial, viral, most insects).
     - Not appropriate for any mammalian genomes, or some plants/fungi.
     - Sequence identifiers must be an exact match with all other inputs or expect problems. See **GFF GFT GFF3**.  
     - Formatting criteria:
         - All sequence identifiers must be unique.
         - ALL tools will require that there is no description content, only identifiers, in the fasta title line (“>” line). Use **NormalizeFasta** to remove the description (all content after the first whitespace) and wrap the sequences to 80 bases. 
-        - The only exception is when executing the **MakeBLASTdb** tool and when the input fasta is in NCBI BLAST format (see the tool form).
+        - The only exception is when executing the **MakeBLASTdb** tool and when the input fasta is in NCBI BLAST format (see the tool form).

faqs/galaxy/datasets_working_with_reference_annotation.md

@@ -17,7 +17,7 @@ contributors: [jennaj, garimavs]
         - If only GFF3 is available, you can attempt to transform it with the tool **gffread**.
     - Was GTF data detected as GFF during Upload? It probably has headers.
         -Remove the headers (lines that start with a "#") with the **Select** tool using the option "NOT Matching" with the regular expression: ^#
-        - [Redetect the datatype](https://training.galaxyproject.org/training-material/faqs/galaxy/#detecting-the-datatype-file-format). It should be GTF once corrected.
+        - [Redetect the datatype]({% link faqs/galaxy/datasets_detect_datatype.md %}). It should be GTF once corrected.
     - UCSC annotation
         - Find annotation under their Downloads area. The path will be similar to: `https://hgdownload.soe.ucsc.edu/goldenPath/<database>/bigZips/genes/`
         - Copy the URL from UCSC and paste it into the Upload tool, allowing Galaxy to detect the datatype.

faqs/galaxy/datatypes_fastq_and_fastqsanger.md

@@ -11,4 +11,4 @@ Before assigning `fastqsanger` or `fastqsanger.gz`, be sure to confirm the forma
 **TIP:** 
 - Using *non-fastqsanger* scaled quality values will cause scientific problems with tools that expected `fastqsanger` formatted input.
 - Even if the tool does not fail, get the format right from the start to avoid problems. Incorrect format is still one of the most common reasons for tool errors or unexpected results (within Galaxy or not).
-- For more information on [How to format fastq data for tools that require .fastqsanger format?](https://training.galaxyproject.org/training-material/faqs/galaxy/datatypes_fastq_format.html)
+- For more information on [How to format fastq data for tools that require .fastqsanger format?]({% link faqs/galaxy/datatypes_fastq_format.md %})

faqs/galaxy/datatypes_understanding_datatypes.md

@@ -8,8 +8,8 @@ contributors: [jennaj, garimavs]
 
 - Allow Galaxy to detect the datatype during Upload, and adjust from there if needed.
 - Tool forms will filter for the appropriate datatypes it can use for each input. 
-- [Directly changing](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_change_datatype.html) a datatype can lead to errors. Be intentional and consider [converting](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_convert_datatype.html) instead when possible. 
-- Dataset content can also be adjusted (tools: Data manipulation) and the expected [datatype detected](https://training.galaxyproject.org/training-material/faqs/galaxy/#detecting-the-datatype-file-format). Detected datatypes are the most reliable in most cases.
+- [Directly changing]({% link faqs/galaxy/datasets_change_datatype.md %}) a datatype can lead to errors. Be intentional and consider [converting]({% link faqs/galaxy/datasets_convert_datatype.md %}) instead when possible. 
+- Dataset content can also be adjusted (tools: Data manipulation) and the expected [datatype detected]({% link faqs/galaxy/datasets_detect_datatype.md %}). Detected datatypes are the most reliable in most cases.
 - If a tool does not accept a dataset as valid input, it is not in the correct format with the correct datatype.
 - Once a dataset’s content matches the datatype, and that dataset is repeatedly used (example: Reference annotation) use that same dataset for all steps in an analysis or expect problems. This may mean rerunning prior tools if you need to make a correction. 
 - Tip: Not sure what datatypes a tool is expecting for an input?
@@ -21,5 +21,5 @@ contributors: [jennaj, garimavs]
     - Why? This can also unexpectedly create hidden datasets that are near duplicates of your original data, only in a different format. 
     - For large data, that can quickly consume working space (quota). 
     - Deleting/purging any hidden datasets can lead to errors if you are still using the original datasets as an input. 
-    - Consider [converting](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_convert_datatype.html) to the expected datatype yourself when data is large.
-    - Then test the tool directly on converted data. If it works, purge the original to recover space.
+    - Consider [converting]({% link faqs/galaxy/datasets_convert_datatype.md %}) to the expected datatype yourself when data is large.
+    - Then test the tool directly on converted data. If it works, purge the original to recover space.

faqs/galaxy/histories_shared.md

@@ -15,4 +15,4 @@ contributors: [jennaj, garimavs]
 - Import the History into your account via copying it to work with it.
 - Unshare Histories that you no longer want shared with you or that you have already made a copy of.
 
-**Note:** Shared Histories (when copied into your account or not) do count in portion toward your total account data quota usage. More details on histories shared concerning account quota usage can be found in [this link](https://training.galaxyproject.org/training-material/faqs/galaxy/#unsharing-unwanted-histories).
+**Note:** Shared Histories (when copied into your account or not) do count in portion toward your total account data quota usage. More details on histories shared concerning account quota usage can be found in [this link]({% link faqs/galaxy/histories_unsharing_histories.md %}).

faqs/galaxy/reference_genomes_troubleshooting_custom_genome.md

@@ -7,7 +7,7 @@ contributors: [jennaj, Nurzhamalyrys]
 ---
 
 
-If a custom genome/transcriptome/exome dataset is producing errors, double check the format and that the chromosome identifiers between **ALL** inputs. Clicking on the bug icon {% icon galaxy-bug %} will often provide a description of the problem. This does not automatically submit a bug report, and it is not always necessary to do so, but it is a good way to get some information about [why a job is failing](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_troubleshooting.html). 
+If a custom genome/transcriptome/exome dataset is producing errors, double check the format and that the chromosome identifiers between **ALL** inputs. Clicking on the bug icon {% icon galaxy-bug %} will often provide a description of the problem. This does not automatically submit a bug report, and it is not always necessary to do so, but it is a good way to get some information about [why a job is failing]({% link faqs/galaxy/analysis_troubleshooting.md %}). 
 
 - Custom genome not assigned as FASTA format
 

faqs/galaxy/troubleshooting_cluster_failure.md

@@ -25,5 +25,5 @@ Remote job server indicated a problem running or monitoring this job.
 
 - Solutions:
     - Try at least one rerun. Server/cluster errors like this are usually transient.
-    - Review the Solutions section of the [Understanding input error messages](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_job_failure_input_problem.html) FAQ.
-    - If after any corrections, the job still fails, please report the technical issue [following the extended issue guidelines](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_reporting_issues.html).
+    - Review the Solutions section of the [Understanding input error messages]({% link faqs/galaxy/troubleshooting_input_problem.md %}) FAQ.
+    - If after any corrections, the job still fails, please report the technical issue [following the extended issue guidelines]({% link faqs/galaxy/analysis_reporting_issues.md %}).

faqs/galaxy/troubleshooting_excess_memory.md

@@ -43,5 +43,5 @@ Memory Error
     - The most common reasons for this error are input and tool parameters problems that must be adjusted/corrected.
 - Solutions:
     - Try at least one rerun to execute the job on a different cluster node.
-    - Review the Solutions section of the [Understanding input error messages](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_job_failure_input_problem.html) FAQ.
-    - Your data may actually be too large to process at a public Galaxy server. Alternatives include setting up a [private Galaxy server](https://training.galaxyproject.org/training-material/faqs/gtn/galaxy_usage.html).
+    - Review the Solutions section of the [Understanding input error messages]({% link faqs/galaxy/troubleshooting_input_problem.md %}) FAQ.
+    - Your data may actually be too large to process at a public Galaxy server. Alternatives include setting up a [private Galaxy server]({% link faqs/gtn/galaxy_usage.md %}).