Nginx default etag and last-modified headers not working as expected

[ajdergute]

Describe the Enhancement

Currently a reproducible build is performed were my image contains my static files of a react front-end application.
Unfortunately, all files modified date at file system level is Jan 01 1980, because all image layers should be reproducible.

Nginx on the other hand uses last modified on file system level to notify clients about changed files. The HTTP headers used are etag and last-modified.

So if I deploy a new image with changed application layer, the client got the old headers and didn't invalidate it's cache.

Possible Solution

I recommend to introduce a flag which enable users of this buildpack to choose whether the original timestamp or 1980 is used for the web server root folder.

Motivation

A useful caching mechanism preserve users of a web application from manually invalidate their cache and therefore increase their experience.

[uqix]

I tried SOURCE_DATE_EPOCH option, but it only changed the creation date of images not the build output files.

[uqix]

I recommend to introduce a flag which enable users of this buildpack to choose whether the original timestamp or 1980 is used for the web server root folder.

This is not gonna happen in short term, as natalieparellano said:

In theory we could in the future allow buildpacks to set metadata on a per-layer basis to have the lifecycle omit zeroing the timestamps for particular layers. But this would require an RFC...

As a workaround, we can implement finer cache control in another way:

1. echo 'include /mnt/config/nginx.conf;' > nginx.conf in project root before lifecycle
2. in helm chart, mount a ConfigMap with nginx.conf key at /mnt/config
3. Add some chart values to template this nginx.config

A nginx.conf example:

{{- $name := include "chart.fullname" . }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ $name }}
  labels:
    {{- include "chart.labels" . | nindent 4 }}
data:
  nginx.conf: |-
    # --- based on BP_WEB_SERVER=nginx ---
    #
    # Number of worker processes running in container
    worker_processes 1;

    # Run NGINX in foreground (necessary for containerized NGINX)
    daemon off;

    # Set the location of the server's PID file
    #
    # comment out, or we got error:
    # [emerg] 1#0: "pid" directive is duplicate in /mnt/config/nginx.conf
    #
    # pid /tmp/nginx.pid;

    # Set the location of the server's error log
    error_log stderr;

    events {
      # Set number of simultaneous connections each worker process can serve
      worker_connections 1024;
    }

    http {
      client_body_temp_path /tmp/client_body_temp;
      proxy_temp_path /tmp/proxy_temp;
      fastcgi_temp_path /tmp/fastcgi_temp;

      charset utf-8;

      # Map media types to file extensions
      types {
        text/html html htm shtml;
        text/css css;
        text/xml xml;
        image/gif gif;
        image/jpeg jpeg jpg;
        application/javascript js;
        application/atom+xml atom;
        application/rss+xml rss;
        font/ttf ttf;
        font/woff woff;
        font/woff2 woff2;
        text/mathml mml;
        text/plain txt;
        text/vnd.sun.j2me.app-descriptor jad;
        text/vnd.wap.wml wml;
        text/x-component htc;
        text/cache-manifest manifest;
        image/png png;
        image/tiff tif tiff;
        image/vnd.wap.wbmp wbmp;
        image/x-icon ico;
        image/x-jng jng;
        image/x-ms-bmp bmp;
        image/svg+xml svg svgz;
        image/webp webp;
        application/java-archive jar war ear;
        application/mac-binhex40 hqx;
        application/msword doc;
        application/pdf pdf;
        application/postscript ps eps ai;
        application/rtf rtf;
        application/vnd.ms-excel xls;
        application/vnd.ms-powerpoint ppt;
        application/vnd.wap.wmlc wmlc;
        application/vnd.google-earth.kml+xml  kml;
        application/vnd.google-earth.kmz kmz;
        application/x-7z-compressed 7z;
        application/x-cocoa cco;
        application/x-java-archive-diff jardiff;
        application/x-java-jnlp-file jnlp;
        application/x-makeself run;
        application/x-perl pl pm;
        application/x-pilot prc pdb;
        application/x-rar-compressed rar;
        application/x-redhat-package-manager  rpm;
        application/x-sea sea;
        application/x-shockwave-flash swf;
        application/x-stuffit sit;
        application/x-tcl tcl tk;
        application/x-x509-ca-cert der pem crt;
        application/x-xpinstall xpi;
        application/xhtml+xml xhtml;
        application/zip zip;
        application/octet-stream bin exe dll;
        application/octet-stream deb;
        application/octet-stream dmg;
        application/octet-stream eot;
        application/octet-stream iso img;
        application/octet-stream msi msp msm;
        application/json json;
        audio/midi mid midi kar;
        audio/mpeg mp3;
        audio/ogg ogg;
        audio/x-m4a m4a;
        audio/x-realaudio ra;
        video/3gpp 3gpp 3gp;
        video/mp4 mp4;
        video/mpeg mpeg mpg;
        video/quicktime mov;
        video/webm webm;
        video/x-flv flv;
        video/x-m4v m4v;
        video/x-mng mng;
        video/x-ms-asf asx asf;
        video/x-ms-wmv wmv;
        video/x-msvideo avi;
      }

      access_log /dev/stdout;

      # Set the default MIME type of responses; 'application/octet-stream'
      # represents an arbitrary byte stream
      default_type application/octet-stream;

      # (Performance) When sending files, skip copying into buffer before sending.
      sendfile on;
      # (Only active with sendfile on) wait for packets to reach max size before
      # sending.
      tcp_nopush on;

      # (Performance) Enable compressing responses
      gzip on;
      # For all clients
      gzip_static always;
      # Including responses to proxied requests
      gzip_proxied any;
      # For responses above a certain length
      gzip_min_length 1100;
      # That are one of the following MIME types
      gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/xml+rss;
      # Compress responses to a medium degree
      gzip_comp_level 6;
      # Using 16 buffers of 8k bytes each
      gzip_buffers 16 8k;

      # Add "Vary: Accept-Encoding” response header to compressed responses
      gzip_vary on;

      # Decompress responses if client doesn't support compressed
      gunzip on;

      # Don't compress responses if client is Internet Explorer 6
      gzip_disable "msie6";

      # Set a timeout during which a keep-alive client connection will stay open on
      # the server side
      keepalive_timeout 30;

      # Ensure that redirects don't include the internal container PORT - <%=
      # ENV["PORT"] %>
      port_in_redirect off;

      # (Security) Disable emitting nginx version on error pages and in the
      # “Server” response header field
      server_tokens off;

      server {
        listen {{ .Values.httpPort }} default_server;
        server_name _;

        # Directory where static files are located
        root dist;

        # https://stackoverflow.com/a/41635866
        location / {
          try_files $uri $uri/ /index.html;
        }
        location = /index.html {
          add_header last-modified '';
          etag off;
        }

        # (Security) Don't serve dotfiles, except .well-known/, which is needed by
        # LetsEncrypt
        location ~ /\.(?!well-known) {
          deny all;
          return 404;
        }
      }
    }

Notes:

1. Comment out the pid directive, or we got error
2. try_files $uri $uri/ /index.html is similar to BP_WEB_SERVER_ENABLE_PUSH_STATE by rewrite
3. For location = /index.html, the last-modified and etag headers have to be removed for correct cache behavior
4. Don't use any templated variables like {{...}}, or we got this issue

[ajdergute]

Thanks @uqix for commenting this issue. I suggest to not mix up things. This issue is about fixed timestamps for static files i.e. css, html and js. It's not about a read only root filesystem (see #463).

Basically you turned cache control off, which is not what I would suggest on a general basis.
Of course anyone is able to provide a custom nginx.conf without cache control if this meets the requirements.

As an intermediate solution we disabled etag-header and configured last-modified-header via helm chart as follows:

  - name: NGINX_LAST_MODIFIED_HEADER
    value: '{{ dateInZone "Mon, 2 Jan 2006 15:04:05" (now) "UTC" }} GMT'

This way some basic caching should be working.

[mecseid]

Any update about this issue?
For SPAs, the Push-state routing is a good start, but without any cache control, these apps do not work as expected.

The SOURCE_DATE_EPOCH env variable only manages the layers' creation time but doesn't change the zero epoch timestamp of application-related source/compiled files (but I think it should change the app-related files mtime too).

@natalieparellano
Do you know anything about it (or buildpack have some settings to control the modification time)?
I read the related RFC and I think the images stay reproducible if the env variable changes the sources' modification time.

[natalieparellano]

@mecseid if we need layer timestamps to be configurable we could certainly add it, we would need an RFC similar to https://github.com/buildpacks/rfcs/blob/main/text/0103-source-date-epoch.md to describe how this would work in pack and the lifecycle.