Skip to content

Commit

Permalink
Overhaul pano stitching and viewer interface (#173)
Browse files Browse the repository at this point in the history
  • Loading branch information
trey0 authored Aug 1, 2024
1 parent 3578a0d commit 7e79f15
Show file tree
Hide file tree
Showing 27 changed files with 1,031 additions and 235 deletions.
7 changes: 5 additions & 2 deletions pano/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,11 @@ Note that some of the individual steps within each panorama stitch (e.g., `enble
### View the tour
TODO
In the host OS, point a web browser at `http://127.0.0.1:8080/`. The `run.sh` script forwards port 8080 of the host to port 80 of the container, where an Apache server is running to provide a preview of the tour.
### Install the tour
Recursively copy the `$ISAAC_PANO_OUTPUT/html` folder wherever you want on your web server. Make sure permissions are set to enable your users to access files in the folder. No special configuration should be needed as long as the server is configured to serve static files with the appropriate MIME types. None of the application logic runs on the web server; the viewing interfaces run completely in the browser. When users make annotations, they will be stored locally in the user's web browser, not on the web server.

### Generate target pose from image

Expand Down Expand Up @@ -185,4 +188,4 @@ An example output would be:

Things to look for are that the timestamps are not too large, this would mean that results are unreliable.
Pay attention to the point cloud point to vector distance to make sure that the point is not too far away from the target vector, meaning that the point cloud does not include the target which is possible due to different field of views between the cameras.
Lastly the mesh and pcl should not disagree, if they do, some manual analysis is needed. Be aware that the attitude provided is defined by the direction pointing at the target assuming roll zero.
Lastly the mesh and pcl should not disagree, if they do, some manual analysis is needed. Be aware that the attitude provided is defined by the direction pointing at the target assuming roll zero.
37 changes: 29 additions & 8 deletions pano/docker/pano.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,37 @@ ARG REMOTE=ghcr.io/nasa

FROM ${REMOTE}/isaac:latest-ubuntu${UBUNTU_VERSION}

# apache2: Local web server for previewing pano tour interface
# default-jre: Java runtime needed for minifying Pannellum web files
# hugin: pano stitching tools (and hsi Python interface)
# libvips-tools: convert images to multires, zoomable in OpenSeaDragon
# python3-pip: for installing Python packages later in this Dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
apache2 \
default-jre \
gfortran \
hugin \
libfftw3-dev \
libopenblas-dev \
libvips-tools \
python3-pip \
gfortran libopenblas-dev libfftw3-dev \
&& rm -rf /var/lib/apt/lists/*

# pandas: pulled in as pyshtools dependency but install breaks if not mentioned explicitly (?)
# pyshtools: used during Pannellum multires generation
# snakemake: modern build system based on Python, manages stitching workflows

# Install Jupyter explicitly first
RUN pip3 install --no-cache-dir --upgrade pip && \
pip3 install --no-cache-dir jupyter
RUN pip3 install --no-cache-dir --upgrade pip \
&& pip3 install --no-cache-dir jupyter

# Install other Python packages: jupyter package needs to be installed before attempting to build pyshtools
RUN pip3 install --no-cache-dir pandas pyshtools snakemake pulp==2.7 --ignore-installed PyYAML
# - pandas: pulled in as pyshtools dependency but install breaks if not mentioned explicitly (?)
# - pyshtools: used during Pannellum multires generation
# - snakemake: modern build system based on Python, manages stitching workflows
RUN pip3 install --no-cache-dir \
--ignore-installed PyYAML \
pandas \
pulp==2.7 \
pyshtools \
snakemake

# pannellum: library for viewing/navigating panorama tours
RUN mkdir -p /opt \
Expand Down Expand Up @@ -73,3 +81,16 @@ RUN echo 'source "/src/isaac/devel/setup.bash"\nexport ASTROBEE_CONFIG_DIR="/src
# once new pano folder is merged into develop and official docker
# images are updated.
RUN echo 'export ROS_PACKAGE_PATH="${ROS_PACKAGE_PATH}:/src/isaac/src/pano/pano_stitch::/src/isaac/src/pano/pano_view"' >> "${HOME}/.bashrc"

# This is a bit unusual, but starting apache in .bashrc ensures it's
# running whenever we run an interactive session using
# run.sh. (Running it multiple times doesn't cause a problem.)
RUN echo 'apachectl start' >> "${HOME}/.bashrc"

# Make /output/html the DocumentRoot for the apache2 debug web server
RUN perl -i -ple 's:/var/www:/output:g' \
/etc/apache2/apache2.conf \
/etc/apache2/sites-available/000-default.conf

# Expose local web server for pano tour preview
EXPOSE 80
1 change: 1 addition & 0 deletions pano/docker/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,5 @@ docker run \
--mount type=bind,source=${ISAAC_PANO_INPUT},target=/input,readonly \
--mount type=bind,source=${ISAAC_PANO_OUTPUT},target=/output \
--mount type=bind,source=${ISAAC_SRC},target=/src/isaac/src \
-p 127.0.0.1:8080:80 \
isaac/pano
7 changes: 3 additions & 4 deletions pano/pano_stitch/scripts/pano_image_meta.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
import rosbag
from tf.transformations import euler_from_quaternion

IMAGE_TOPIC = ["/hw/cam_sci/compressed", "/hw/cam_sci_info"]
IMAGE_TOPICS = ["/hw/cam_sci/compressed", "/hw/cam_sci_info"]
POSE_TOPIC = "/loc/pose"
FIELD_NAMES = (
"timestamp",
Expand All @@ -50,9 +50,8 @@ def get_image_meta(inbag_path, num_images=None):
images = []
with rosbag.Bag(inbag_path) as bag:
img_meta = None
topics = IMAGE_TOPIC + [POSE_TOPIC]
for topic, msg, t in bag.read_messages(topics):
if topic in IMAGE_TOPIC:
for topic, msg, t in bag.read_messages([*IMAGE_TOPICS, POSE_TOPIC]):
if topic in IMAGE_TOPICS:
if num_images is not None and len(images) == num_images:
break

Expand Down
2 changes: 1 addition & 1 deletion pano/pano_view/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ add_dependencies(point_cloud_intersect ${catkin_EXPORTED_TARGETS})
target_link_libraries(point_cloud_intersect ${catkin_LIBRARIES})


## Declare a C++ executable: find_point_coordinate
## Declare a C++ executable: find_point_coordinate
add_executable(find_point_coordinate tools/find_point_coordinate.cc)
add_dependencies(find_point_coordinate ${catkin_EXPORTED_TARGETS})
target_link_libraries(find_point_coordinate mesh_intersect point_cloud_intersect
Expand Down
183 changes: 183 additions & 0 deletions pano/pano_view/config/phase1x_inspection_results.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
# ======================================================================
# The content below the line is from the file
# isaac/src/pano/pano_view/config/phase1x_inspection_results.config
# and is designed to be copy/pasted at the bottom of pano_meta.yaml.

# How inspection_results was generated: The photos were manually
# selected from all hatch seal targeted inspection photos to be the
# best ones (at most one from each focus stack) for showing
# recognizable damage sites. Pitch/yaw values for each image were
# manually selected using the Pannellum hotSpotDebug feature.
inspection_results:
scene001_isaac11_bumble_usl_bay6:
# ISAAC11 USL aft hatch seal inspection
- pitch: -19.496113033067832
yaw: -178.49501734467697
image: /input/inspection_results/isaac11_usl_aft/1657549131.173.jpg
slug: Hatch Seal ISAAC16 USL AFT

scene004_isaac11_queen_usl_bay1:
# ISAAC16 USL forward hatch seal deck section positions, numbered 0 .. 3 port -> starboard
# Only position 1 has a recognizable damage site, leaving out the others

#- pitch: -23.829139007892287
# yaw: -9.57794857674467
# image: /input/inspection_results/isaac16_usl_fwd/1720198615.236.jpg
# slug: Hatch Seal ISAAC16 USL FWD 0
- pitch: -24.233722398730837
yaw: -2.3869610431019694
image: /input/inspection_results/isaac16_usl_fwd/1720199288.313.jpg
slug: Hatch Seal ISAAC16 USL FWD 1
#- pitch: -24.210064732216285
# yaw: 6.143908859653246
# image: /input/inspection_results/isaac16_usl_fwd/1720199367.399.jpg
# slug: Hatch Seal ISAAC16 USL FWD 2
#- pitch: -24.167778451658435
# yaw: 10.29111636247168
# image: /input/inspection_results/isaac16_usl_fwd/1720199447.824.jpg
# slug: Hatch Seal ISAAC16 USL FWD 3

# ISAAC11 USL forward hatch seal inspection
- pitch: -24.233722398730837
yaw: -1
image: /input/inspection_results/isaac11_usl_fwd/1657551186.506.jpg
slug: Hatch Seal ISAAC11 USL FWD

# How initial_annotations was generated: I used the existing target
# selection feature to annotate the (somewhat) recognizable damage
# sites, saved the results, and simply copy/pasted the file contents
# here. Conveniently, YAML is a superset of the JSON format used for
# the annotations, so it was already valid YAML (even if the style
# clashes with the one used above).
initial_annotations: {
"scene004_isaac11_queen_usl_bay1": {
"1720199288.313": [
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "A: Divot on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:3449.820556640625,1809.1005859375,52.701171875,39.3167724609375"
}
},
"id": "#5e67f4b4-86ed-4eaf-9bf2-caeac4358691"
},
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "B: Cut on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:3723.783203125,1791.1153564453125,124.642333984375,58.97509765625"
}
},
"id": "#f5f60596-a44a-48d2-aafb-49d63621bd20"
}
],
"1657551186.506": [
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "B: Cut on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:3658.030517578125,2492.563720703125,110.6806640625,51.651123046875"
}
},
"id": "#5a8d060a-259e-47f9-9b42-519a09246897"
},
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "A: Divot on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:3269.641845703125,2496.58837890625,52.992431640625,42.930908203125"
}
},
"id": "#49a5daa0-41c1-4856-b21b-09164687415a"
}
]
},
"scene001_isaac11_bumble_usl_bay6": {
"1657549131.173": [
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "F: Divot and Cut on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:2259.4794921875,2235.703125,56.346435546875,41.589111328125"
}
},
"id": "#0b8ab636-3697-425a-b683-089712796e24"
},
{
"@context": "http://www.w3.org/ns/anno.jsonld",
"type": "Annotation",
"body": [
{
"type": "TextualBody",
"value": "G: Divot on Inner Seal",
"purpose": "commenting"
}
],
"target": {
"source": "http://localhost:8080/src/undefined",
"selector": {
"type": "FragmentSelector",
"conformsTo": "http://www.w3.org/TR/media-frags/",
"value": "xywh=pixel:2729.704833984375,2227.653564453125,63.72509765625,38.906005859375"
}
},
"id": "#84eb0fa7-4601-43e1-a900-46077fc93c4a"
}
]
}
}
Binary file added pano/pano_view/media/camera_highlight.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-128x128.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-16x16.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-192x192.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-196x196.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-228x228.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-32x32.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-57x57.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-76x76.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/favicon/favicon-96x96.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pano/pano_view/media/inspection.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 7e79f15

Please sign in to comment.