add in updates from master

README.md

@@ -1,12 +1,35 @@
 --------------
-About Juicebox
+Breaking News!
 --------------
-Juicebox is visualization software for Hi-C data.  This distribution includes the source code for Juicebox, <a href="https://github.com/theaidenlab/juicer/wiki/Download">Juicer Tools</a>, and <a href="https://aidenlab.org/assembly/">Assembly Tools</a>.  <a href="https://github.com/theaidenlab/juicebox/wiki/Download">Download Juicebox here</a>, or use <a href="https://aidenlab.org/juicebox">Juicebox on the web</a>.  Detailed documentation is available <a href="https://github.com/theaidenlab/juicebox/wiki">on the wiki</a>. Instructions below pertain primarily to usage of command line tools and the Juicebox jar files.
+This codebase is in the process of being split up to better support long-term maintenance and modular aspects of different tools
+available. The new repositories will be:
+
+- [JuiceboxGUI](https://github.com/aidenlab/juiceboxgui) for visualization of Hi-C maps with Juicebox Desktop and genome assembly correction with Assembly Tools.
+- [HiCTools](https://github.com/aidenlab/hictools) for building and writing .hic files (Pre, Addnorm, and Statistics)
+- [JuicerTools](https://github.com/aidenlab/juicertools) for downstream analysis of .hic files (HiCCUPS, Arrowhead, APA, etc.)
+- [Java Straw](https://github.com/aidenlab/java-straw) to quickly read and stream data from .hic files into Java, and is used by the above repositories.
+
+We also have new tools:
+
+- [Straw](https://github.com/aidenlab/straw) to quickly read and stream data from .hic files into C++, python, R, and MATLAB.
+- [POSSUMM](https://github.com/sa501428/EigenVector) for new C++ code to rapidly calculate A-B compartments (i.e.
+  eigenvectors) for higher resolutions
+- [EMT](https://github.com/sa501428/hic-emt) for upgrading older .hic files or making smaller .hic files for regions of
+  interest.
 
-Juicebox can now be used to visualize and interactively (re)assemble genomes. 
-Check out the Juicebox Assembly Tools Module website https://aidenlab.org/assembly for 
-more details on how to use Juicebox for assembly.
+--------------
+About Juicebox
+--------------
+Juicebox is visualization software for Hi-C data. This distribution includes the source code for
+Juicebox, <a href="https://github.com/theaidenlab/juicer/wiki/Download">Juicer Tools</a>,
+and <a href="https://aidenlab.org/assembly/">Assembly Tools</a>
+.  <a href="https://github.com/theaidenlab/juicebox/wiki/Download">Download Juicebox here</a>, or
+use <a href="https://aidenlab.org/juicebox">Juicebox on the web</a>. Detailed documentation is
+available <a href="https://github.com/theaidenlab/juicebox/wiki">on the wiki</a>. Instructions below pertain primarily
+to usage of command line tools and the Juicebox jar files.
 
+Juicebox can now be used to visualize and interactively (re)assemble genomes. Check out the Juicebox Assembly Tools
+Module website https://aidenlab.org/assembly for more details on how to use Juicebox for assembly.
 
 Juicebox was created by <a href="https://github.com/jrobinso">Jim Robinson</a>,
 <a href="https://github.com/nchernia">Neva C. Durand</a>, and <a href="http://www.erez.com/">Erez Lieberman Aiden</a>. Past contributors include <a href="https://github.com/imachol">Ido Machol</a>, <a href="https://github.com/zgire">Zulkifl Gire</a>, <a href="https://github.com/mhoeger">Marie Hoeger</a>, <a href="https://github.com/asddf123789">Fanny Huang</a>, <a href="https://github.com/mikeehman">Nam Hee Kim</a>, <a href="https://github.com/nguyenkvi">Vi Nguyen</a>, <a href="https://github.com/bluejay9676">Jay Ryu</a>, <a href="https://github.com/musianat">Nathaniel T. Musial</a>, and <a href="https://github.com/ranganmostofa11">Ragib Mostofa</a>.
@@ -30,7 +53,7 @@ then you'll point to it (IntelliJ has lots of documentation on this sort of thin
 
 * Then go to `VCS` -> `checkout from version control`.
 * You'll need to do is be sure `*.sizes` is included as a file to be copied over to the class files.
-Set this up via IntelliJ `Preferences` -> `Compiler`. Add `?*.sizes` to the list of `Resource Patterns`.
+  Set this up via IntelliJ `Preferences` -> `Compiler`. Add `?*.sizes` to the list of `Resource Patterns`.
 * While there, also go to `Java Compiler` and put this into additional command line options: `-Xlint:all -target 1.7`
 The former turns on all warnings, the latter gives some flexibility since some people haven't updated Java to 1.8 yet.
 * Then go to `Run` -> `Edit Configurations`.
@@ -51,11 +74,11 @@ The former turns on all warnings, the latter gives some flexibility since some p
         -Xmx2000m
 
 * Note that the `Xmx2000m` flag sets the maximum memory heap size to 2GB.
-Depending on your computer you might want more or less.
-Some tools will break if there's not enough memory and the file is too large,
-but don't worry about that for development; 2GB should be fine.
+  Depending on your computer you might want more or less.
+  Some tools will break if there's not enough memory and the file is too large,
+  but don't worry about that for development; 2GB should be fine.
 * One last note: be sure to `Commit and Push` when you commit files, it's hidden in the dropdown menu button in the
-commit window.
+  commit window.
 
 ----------------------------------
 Hardware and Software Requirements
@@ -70,8 +93,7 @@ Java Runtime Environment (JRE), please go to https://www.java.com/download.
 We recommend having at least 2GB free RAM for the best user experience with
 Juicebox.
 
-To launch the Juicebox application from command line, type
-  java -Xms512m -Xmx2048m -jar Juicebox.jar
+To launch the Juicebox application from command line, type java -Xms512m -Xmx2048m -jar Juicebox.jar
 
 Note: the -Xms512m flag sets the minimum memory heap size at 512 megabytes, and
 the -Xmx2048m flag sets the maximum size at 2048 megabytes (2 gigabytes). These
@@ -96,16 +118,10 @@ on how to use the Juicer tools.
 --------------------------------
 Compiling Jars from Source Files
 --------------------------------
-1. You should have Java 1.8 JDK and Apache Ant installed on your system. See
-   below for more information.
-2. Go to the folder containing the Juicebox source files and edit the
-   juicebox.properties file with the proper Java JDK Address.
-3. Open the command line, navigate to the folder containing the build.xml file
-   and type
-     ant
-   The process should take no more than a minute to build on most machines.
-4. The jars are written to the directory out/.  You can change this by editing
-   the build.xml file.
+1. You should have Java 1.8 JDK and Apache Ant installed on your system. See below for more information.
+2. Go to the folder containing the Juicebox source files and edit the juicebox.properties file with the proper Java JDK Address.
+3. Open the command line, navigate to the folder containing the build.xml file and type ant The process should take no more than a minute to build on most machines.
+4. The jars are written to the directory out/. You can change this by editing the build.xml file.
 
 * Installing Java 1.8 JDK
 
@@ -115,25 +131,11 @@ https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.
 https://tecadmin.net/install-oracle-java-8-jdk-8-ubuntu-via-ppa/
 
 * Installing Apache Ant
-Mac
-  Ant should be installed on most Macs. To verify installation via the command
-  prompt, type
-    ant -version
-  If Ant is not on your Mac, install it via homebrew. At the command prompt, type
-    brew update
-    brew install ant
-  You may need to install Homebrew (https://brew.sh/) on your machine
-  See the following Stackoverflow post for more details:
+  Mac Ant should be installed on most Macs. To verify installation via the command prompt, type ant -version If Ant is not on your Mac, install it via homebrew. At the command prompt, type
+  brew update brew install ant You may need to install Homebrew (https://brew.sh/) on your machine See the following Stackoverflow post for more details:
   https://stackoverflow.com/questions/3222804/how-can-i-install-apache-ant-on-mac-os-x
 
-Windows
-  Installing Ant requires some minor changes to your system environment. Follow
-  the instructions in this article:
-  https://www.nczonline.net/blog/2012/04/12/how-to-install-apache-ant-on-windows/
-
-Linux
-  In the command prompt, type
-    sudo apt-get install ant
-  or
-    sudo yum install ant
-  depending on your package installer
+Windows Installing Ant requires some minor changes to your system environment. Follow the instructions in this article:
+https://www.nczonline.net/blog/2012/04/12/how-to-install-apache-ant-on-windows/
+
+Linux In the command prompt, type sudo apt-get install ant or sudo yum install ant depending on your package installer

src/juicebox/HiCGlobals.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2024 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -36,7 +36,7 @@
  */
 public class HiCGlobals {
 
-    public static final String versionNum = "2.13.07";
+    public static final String versionNum = "2.24.00";
     public static final String juiceboxTitle = "[Juicebox " + versionNum + "] Hi-C Map ";
 
     // MainWindow variables

src/juicebox/data/AbstractDatasetReader.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2022 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -110,10 +110,4 @@ public double[] readEigenvector(String chrName, HiCZoom zoom, int number, String
     public NormalizationVector readNormalizationVector(NormalizationType type, int chrIdx, HiC.Unit unit, int binSize) throws IOException {
         return null;  // Override as necessary
     }
-
-    @Override
-    public String readStats() throws IOException {
-        return null; // Override for Combined Dataset Reader
-    }
-
 }

src/juicebox/data/CombinedDatasetReader.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2023 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -62,7 +62,14 @@ public Dataset read() throws IOException {
         List<Dataset> tmpDatasets = new ArrayList<>();
         version = 100000;
         for (DatasetReader r : readers) {
-            tmpDatasets.add(r.read());
+            Dataset ds = r.read();
+
+            if (ds == null) {
+                System.err.println("Dataset is null!! : " + r.getPath());
+            } else {
+                tmpDatasets.add(ds);
+            }
+
             version = Math.min(version, r.getVersion());
         }
 
@@ -97,13 +104,7 @@ public int getVersion() {
     @Override
     public String getPath() {
         // we use this for peaks and blocks list, maybe the best thing to do is to somehow combine them
-        return null;
-    }
-
-    @Override
-    public String readStats() {
-        // again we need to somehow combine from constituent datasets
-        return null;
+        return "combined_dataset";
     }
 
     @Override
@@ -138,12 +139,23 @@ public Matrix readMatrix(String key) throws IOException {
         List<Matrix> tmpDatasets = new ArrayList<>();
         for (DatasetReader r : readers) {
             if (r.isActive()) {
-                tmpDatasets.add(r.readMatrix(key));
+                Matrix m = r.readMatrix(key);
+                if (m == null) {
+                    System.err.println("Matrix Region is null!! : " + r.getPath());
+                    System.err.println("Key " + m.getKey());
+                } else {
+                    tmpDatasets.add(m);
+                }
             }
         }
 
-        return mergeMatrices(tmpDatasets);
-
+        if (tmpDatasets.size() == 0) {
+            return null;
+        } else if (tmpDatasets.size() == 1) {
+            return tmpDatasets.get(0);
+        } else {
+            return mergeMatrices(tmpDatasets);
+        }
     }
 
     @Override

src/juicebox/data/Dataset.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2023 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -105,7 +105,8 @@ public Matrix getMatrix(Chromosome chr1, Chromosome chr2) {
                 matrices.put(key, m);
 
             } catch (Exception e) {
-                System.err.println("Error fetching matrix for: " + chr1.getName() + "-" + chr2.getName());
+                System.err.println("Error fetching matrix for: " + chr1.getName() + "-" + chr2.getName() +
+                        " in " + reader.getPath());
                 e.printStackTrace();
             }
         }
@@ -353,16 +354,10 @@ public String getHiCFileScalingFactor() {
     public String getStatistics() {
         String stats = null;
         if (attributes != null) stats = attributes.get(Preprocessor.STATISTICS);
-        if ((stats == null) || !stats.contains("<table>")) {
-            try {
-                attributes.put(Preprocessor.STATISTICS, reader.readStats());
-            } catch (IOException error) {
-                if (stats != null) {
-                    attributes.put(Preprocessor.STATISTICS, convertStats(stats));
-                } else return null;
-            }
+        if (stats != null && (!stats.contains("<table>")) && HiCGlobals.guiIsCurrentlyActive) {
+            return convertStats(stats);
         }
-        return attributes.get(Preprocessor.STATISTICS);
+        return stats;
     }
 
     private String convertStats(String oldStats) {
@@ -389,10 +384,7 @@ private String convertStats(String oldStats) {
         newStats += "<table><tr><th colspan=2>Experiment Information</th></tr>\n" +
                 "        <tr> <td> Experiment #:</td> <td>";
         String filename = reader.getPath();
-        boolean mapq30 = false;
-        if (filename.lastIndexOf("_30") > 0) {
-            mapq30 = true;
-        }
+        boolean mapq30 = filename.lastIndexOf("_30") > 0;
         String[] parts = filename.split("/");
         newStats += parts[parts.length - 2];
         newStats += "</td></tr>";

src/juicebox/data/DatasetReader.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2022 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -73,8 +73,6 @@ public interface DatasetReader {
 
     String getPath();
 
-    String readStats() throws IOException;
-
     List<JCheckBox> getCheckBoxes(List<ActionListener> actionListeners);
 
     NormalizationVector getNormalizationVector(int chr1Idx, HiCZoom zoom, NormalizationType normalizationType);

src/juicebox/data/DatasetReaderV2.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2022 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -244,31 +244,6 @@ private SeekableStream getValidStream() throws IOException {
         return stream;
     }
 
-
-    public String readStats() throws IOException {
-        String statsFileName = path.substring(0, path.lastIndexOf('.')) + "_stats.html";
-        String stats;
-        BufferedReader reader = null;
-        try {
-            StringBuilder builder = new StringBuilder();
-            reader = ParsingUtils.openBufferedReader(statsFileName);
-            String nextLine;
-            int count = 0; // if there is an big text file that happens to be named the same, don't read it forever
-            while ((nextLine = reader.readLine()) != null && count < 1000) {
-                builder.append(nextLine);
-                builder.append("\n");
-                count++;
-            }
-            stats = builder.toString();
-        } finally {
-            if (reader != null) {
-                reader.close();
-            }
-        }
-
-        return stats;
-    }
-
     @Override
     public List<JCheckBox> getCheckBoxes(List<ActionListener> actionListeners) {
         String truncatedName = HiCFileTools.getTruncatedText(getPath(), maxLengthEntryName);

src/juicebox/mapcolorui/HeatmapMouseHandler.java

@@ -1,7 +1,7 @@
 /*
  * The MIT License (MIT)
  *
- * Copyright (c) 2011-2021 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
+ * Copyright (c) 2011-2022 Broad Institute, Aiden Lab, Rice University, Baylor College of Medicine
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal
@@ -1755,14 +1755,16 @@ private String toolTipText(int x, int y) {
                     Collections.sort(selectedFeatures);
                     appendWithSpan(txt, selectedFeatures);
                 } else {
+                    StringBuilder txt0 = new StringBuilder();
                     for (Feature2DGuiContainer loop : allFeaturePairs) {
                         if (loop.getRectangle().contains(x, y)) {
                             // TODO - why is this code duplicated in this file?
-                            txt.append("<br><br><span style='font-family: arial; font-size: 12pt;'>");
-                            txt.append(loop.getFeature2D().tooltipText());
-                            txt.append("</span>");
+                            txt0.append("<br><br><span style='font-family: arial; font-size: 12pt;'>");
+                            txt0.append(loop.getFeature2D().tooltipText());
+                            txt0.append("</span>");
                         }
                     }
+                    txt.insert(0, txt0);
                 }
             } else {
                 int numLayers = superAdapter.getAllLayers().size();