PDF/A-1b Documents pdf/a Compliant false after signing with CLI

[thomasgundlach]

Hej @MatthiasValvekens ,

i'm creating for test purposes a blank pdf/a-1b file

#create blank pdf-file with convert from package imagemagick on Ubuntu LTS 20.04
gundlac@batch-ost:~/test$ convert xc:none -page A4 pdfa:a.pdf
#convert to valid pdf/a-1b requirements
gundlac@batch-ost:~/test$ gs -dPDFA=1 -sColorConversionStrategy=UseDeviceIndependentColor -sDEVICE=pdfwrite -dPDFACompatibilityPolicy=1 -dNOPAUSE -dBATCH -o test.pdf a.pdf
GPL Ghostscript 9.50 (2019-10-15)
Copyright (C) 2019 Artifex Software, Inc.  All rights reserved.
This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:
see the file COPYING for details.
Processing pages 1 through 1.
Page 1

install verapdf for check compliance

#Download: https://software.verapdf.org/rel/verapdf-installer.zip
#Docs: https://docs.verapdf.org/cli/validation/
gundlac@batch-ost:~$ wget http://downloads.verapdf.org/rel/verapdf-installer.zip
gundlac@batch-ost:~$ unzip verapdf-installer.zip
gundlac@batch-ost:~$ cd verapdf-greenfield-1.20.1/
gundlac@batch-ost:~$ ./verapdf-install
1
<Enter>
O
1
Y
Y
Y
1
Y
<Enter>

check compliance with verapdf: success

#check validity -> compliant:true
gundlac@batch-ost:~/test$ test=$(~/verapdf/verapdf test.pdf 2>&1); echo $?; echo "$test" | less
<?xml version="1.0" encoding="utf-8"?>
<report>
  <buildInformation>
    <releaseDetails id="core" version="1.20.1" buildDate="2022-01-18T11:47:00+01:00"></releaseDetails>
    <releaseDetails id="validation-model" version="1.20.1" buildDate="2022-01-18T13:39:00+01:00"></releaseDetails>
    <releaseDetails id="gui" version="1.20.1" buildDate="2022-01-18T14:50:00+01:00"></releaseDetails>
  </buildInformation>
  <jobs>
    <job>
      <item size="171413">
        <name>/home/gundlach/mytest2/test/test.pdf</name>
      </item>
      <validationReport profileName="PDF/A-1B validation profile" statement="PDF file is compliant with Validation Profile requirements." isCompliant="true">
        <details passedRules="101" failedRules="0" passedChecks="249" failedChecks="0"></details>
      </validationReport>
      <duration start="1645200154279" finish="1645200154571">00:00:00.292</duration>
    </job>
  </jobs>
  <batchSummary totalJobs="1" failedToParse="0" encrypted="0">
    <validationReports compliant="1" nonCompliant="0" failedJobs="0">1</validationReports>
    <featureReports failedJobs="0">0</featureReports>
    <repairReports failedJobs="0">0</repairReports>
    <duration start="1645200154217" finish="1645200154587">00:00:00.370</duration>
  </batchSummary>
</report>
0

sign it with pyhanko

#set test passphrase
gundlac@batch-ost:~/test$ read PASS; export PASS
foobar
#create key/certificate to sign
gundlac@batch-ost:~/test$ openssl req -newkey rsa:4096 -sha256 -passout pass:$PASS -keyform PEM -keyout key.pem -x509 -days 365 -outform PEM -out certificate.pem \
>    -subj '/C=DE/ST=Hamburg/L=Hamburg/O=Company/OU=Group/CN=pyhanko_Signaturtest/emailAddress=mail@domain' \
>    -addext "keyUsage = critical,nonRepudiation,digitalSignature" \
>    -addext "extendedKeyUsage = critical,codeSigning"
Generating a RSA private key
..................................................++++
....................................++++
writing new private key to 'key.pem'
-----
#sign it with pyhanko
gundlac@batch-ost:~/test$ pyhanko sign addsig --use-pades-lta --field Sig1 pemder --key key.pem --passfile <(echo "$PASS") --cert certificate.pem test.pdf test-signed.pdf

check compliance again, now false

#check validity with verapdf -> compliant:false
gundlac@batch-ost:~/test$ test=$(~/verapdf/verapdf test-signed.pdf 2>&1); echo $?; echo "$test" | less
<?xml version="1.0" encoding="utf-8"?>
<report>
  <buildInformation>
    <releaseDetails id="core" version="1.20.1" buildDate="2022-01-18T11:47:00+01:00"></releaseDetails>
    <releaseDetails id="validation-model" version="1.20.1" buildDate="2022-01-18T13:39:00+01:00"></releaseDetails>
    <releaseDetails id="gui" version="1.20.1" buildDate="2022-01-18T14:50:00+01:00"></releaseDetails>
  </buildInformation>
  <jobs>
    <job>
      <item size="180928">
        <name>/home/gundlach/mytest2/test/test-signed.pdf</name>
      </item>
      <validationReport profileName="PDF/A-1B validation profile" statement="PDF file is not compliant with Validation Profile requirements." isCompliant="false">
        <details passedRules="100" failedRules="1" passedChecks="325" failedChecks="1">
          <rule specification="ISO 19005-1:2005" clause="6.9" testNumber="2" status="failed" passedChecks="0" failedChecks="1">
            <description>Every form field shall have an appearance dictionary associated with the field's data.</description>
            <object>PDWidgetAnnot</object>
            <test>AP != null</test>
            <check status="failed">
              <context>root/document[0]/pages[0](4 0 obj PDPage)/annots[0](17 0 obj PDWidgetAnnot)</context>
              <errorMessage>Form field does not have an appearance dictionary associated with the field's data.</errorMessage>
            </check>
          </rule>
        </details>
      </validationReport>
      <duration start="1645200053666" finish="1645200053997">00:00:00.331</duration>
    </job>
  </jobs>
  <batchSummary totalJobs="1" failedToParse="0" encrypted="0">
    <validationReports compliant="0" nonCompliant="1" failedJobs="0">1</validationReports>
    <featureReports failedJobs="0">0</featureReports>
    <repairReports failedJobs="0">0</repairReports>
    <duration start="1645200053611" finish="1645200054013">00:00:00.402</duration>
  </batchSummary>
</report>
1

The problem seems to be described in https://docs.verapdf.org/validation/pdfa-part1/ Rule 6.9-4

Is it possible to fix that to preserve PDF/A-1b-compliance?

Best Regards,

Thomas

[MatthiasValvekens]

Hi Thomas,

Thanks for another detailed report!

First, as a general disclaimer: as stated on the Known Issues page, I'm not currently actively trying to align pyHanko with PDF subsets like the various PDF/A flavours, PDF/UA and PDF/X. I try not to do things that run counter to what the subsets are designed to do, but full compatibility is not really a priority at the moment. FWIW, I currently don't own copies of the subset standards, so take what I'm about to say with a grain of salt (going off memory here).

To your question: this is sort of expected given the way pyHanko generates invisible signatures. Historically, ISO 19005-1 (the standard that defines PDF/A-1) has a "bug", in that it requires appearance streams for invisible annotations as well, which is kind of pointless. That issue was corrected in later parts of ISO 19005. Because of the nature of PDF/A as an archival format, the "fix" wasn't backported to PDF/A-1.

I could add a CLI switch to add a dummy appearance for invisible signatures, but I don't want to make that the default behaviour (I can elaborate on why, if you want). Does that sound workable?

Hope that helps :)

---

PS: If you're really keen on preserving compatibility with PDF/A-1, then arguably signing with PAdES is already out of the question, because PAdES is only relevant for PDF 1.7 and PDF 2.0, while PDF/A-1 is defined in terms of PDF 1.4. :)
PPS: --use-pades-lta currently doesn't imply --with-validation-info. That's arguably a bug (which I'll fix in short order), but in the meantime, you really need both those switches to produce a PAdES-B-LTA signature.

[MatthiasValvekens]

I filed the --use-pades-lta thing as #80.

[thomasgundlach]

Hej @MatthiasValvekens,

thanks for your fast help. I understand, that you don't like to make the fix the default behaviour. i didn't know about the Historical "bug" in ISO 19005-1.
Good thanks to your hint to a newer pdf/a standard. i upgrade my generate-command to pdf/a-3 and now all signed pdfs are fine :-)

#find srgb profile
export SRGB_ICC_PATH=$(locate /srgb.icc)
#generate PDFA_def.ps with proper srgb.icc path
export PDFA_def=$(awk -v iccprof=${SRGB_ICC_PATH} '$2=="(srgb.icc)"{gsub(/srgb.icc/,iccprof,$2)}{print $0}' $(locate /usr*lib/PDFA_def.ps))
#convert pdf document to valid pdf/a-3b
gs --permit-file-read=${SRGB_ICC_PATH} -dPDFA=3 -sColorConversionStrategy=RGB -sDEVICE=pdfwrite -dPDFACompatibilityPolicy=1 -dNOPAUSE -dNOSAFER -dBATCH -o ~/mytest2/test/esult2.pdf -f <(echo "$PDFA_def") ./3222_304246.pdf

And thank you for your suggestion to add a cli switch for the problem. for me, pdf/3-a is fine. but maybe its worthwhile for other people who are depend on pdf/a-1a.

i also add --with-validation-info and works good.

Best Regards and Good Night,

Thomas