Version 1.27.x Tree Updates

[duaneg3]

Are you still taking changes for the 1.27.x tree (I assume this is the version before all the name changes)?
I am still on 1.26, and have a bunch of 1.26 fixes locally coded.
Looking through the 1.27.x release notes, I think some, but not all, might have been addressed.
I am perhaps willing to merge with the1.27.x tree, but it would be too much effort to merge with the most recent release.
(I mess with a lot of the pypdf internals. An all python library is awesome in what it allows)

Are updates to the 1.27.x tree still being supported?

[pubpub-zz]

we are currently at version 4.x
the version are talking about is really too old.
If you think some major changes/fixes could be of interest list propose them as PRs on 1.26 and we see if it worth to try to port them

[duaneg3]

I can check against 1.27.12 .. Here are the changes:

(1) generic.py def __parseContentStream(self, stream):
need to add the following last line

## DEG: need next line with have incomplete command (finished by another object)
if (len(operands)>0): self.operations.append( (operands,b_("")) )

(2) generic.TextStringObject.writeToStream = TextStringObject__writeToStream__
(not exactly sure why I needed this ... happened a while ago ..)

def TextStringObject__writeToStream__(self, stream, encryption_key):
 # Try to write the string out as a PDFDocEncoding encoded string.  It's
 # nicer to look at in the PDF file.  Sadly, we take a performance hit
 # here for trying...
 try:
     bytearr = generic.encode_pdfdocencoding(self)
 except UnicodeEncodeError:
     #bytearr = codecs.BOM_UTF16_BE + self.encode("utf-16be")
     bytearr = self.encode("utf-16be")
 if encryption_key:
     bytearr = RC4_encrypt(encryption_key, bytearr)
     obj = ByteStringObject(bytearr)
     obj.writeToStream(stream, None)
 else:
     stream.write(b_("("))
     if 1==2: print("bytearr=(%s)" %(bytearr))
     for c in bytearr:
         cx = c
         #print("char=(%s) (%s)" %(utils.ord_(cx),utils.chr_(cx)))
         if utils.ord_(cx) == 32: cx = ord(' ')  # alternative nonbreaking space
         #if not utils.chr_(cx).isalnum() and not utils.chr_(cx) in ' ':
         #if not utils.chr_(cx).isalnum():
         #if not utils.chr_(cx).isalnum() and not utils.chr_(cx) in ' .:-#/&|\'\",_':
         if utils.ord_(cx) == 0: continue
         if utils.ord_(cx) < 31 and utils.ord_(cx)>14: continue  ## random stuff  
         if not utils.chr_(cx).isprintable():
             stream.write(b_("\\%03o" % utils.ord_(cx)))
         else:
             if utils.chr_(cx) in '()\\': stream.write(b_("\\"))
             stream.write(b_(utils.chr_(cx)))
     stream.write(b_(")"))

(3) generic.TreeObject.removeChild = generic_TreeObject__removeChild__

change 3 versions of:
   self[NameObject("/Count")] = self[NameObject("/Count")] - 1
to:
   generic.NumberObject(self[NameObject('/Count')] - 1)

(4) pdf.PdfFileWriter.addBookmark = PdfFileWriter__addBookmark__

for a in args:
  #print("a=",a," None=",None,"type=",type(a))
            ## DEG: commented out next line and added line below
            #if a is not None:
            if (isinstance(a,int)):
                zoomArgs.append(generic.NumberObject(a))
            else:
                zoomArgs.append(generic.NullObject())

The other changes are combinations of additions and changes to work-around issues.

Without the fix of (1), the code generates invalid PDF's is some cases. It was a nasty one to debug.

Hope this helps. It is a great library!! Thanks for all your efforts!

[MartinThoma]

Thank you for checking and documenting. Maybe it helps somebody.

Are updates to the 1.27.x tree still being supported?

No. Only changes to 4.x are supported.

[pubpub-zz]

(1) : It's hard to see if a fix has been applied. Can you first prepare a test case with pdf and code to confirm the issue is still present and provide them in a new issue. in any case this will be required to extend tests
(2): the code has been strongly rewritten. as for (1) can you provide testcase
(3) it has been fixed
(4) code has been rewritten but looks like covered

looking forward to your issues

[duaneg3]

I have attached very small example .pdf for example (1).
As you know (in 1.26), the object only gets parsed if it is processed, so you need to get the contents and save the result of the updated contents to cause the parsing to happen and to make the results of the parsing visible:
1.26 example code:

      page = pdf1.getPage(n)
      content_obj = my_content_update_list(page,"pdf1")  # always return list
      for idx,c_obj2 in enumerate(content_obj):
        c_obj = c_obj2.getObject()
        content = pdf.ContentStream(c_obj,pdf1)
        content_obj[idx] = content

test_drop_h2.pdf

[pubpub-zz]

please to ease handling,
a) create an issue
b) provide a code fully stanalone (here my_content_update_list in not clear)
c) confirm the error is still present on latest release

[duaneg3]

def my_content_update_list(mypage,tinfo):
    content_key = generic.NameObject('/Contents')
    content_obj = mypage['/Contents'].getObject()
    if content_obj != None and not isinstance(content_obj, generic.ArrayObject):
        if 1==2: print("my_content_update_list: switching to list (%s)" %tinfo)
        newContentArray = generic.ArrayObject()
        if content_obj != None: newContentArray.append(content_obj)
        mypage[content_key] = newContentArray
    content_obj = mypage[content_key]
    return content_obj

We are sticking with version 1.26.
With all the name changes, the effort to upgrade beyond 1.27 is too high.
Hopefully you can find someone else interested in building a test case with the latest version.

[pubpub-zz]

I've checked the output with .getcontents() :

r = pypdf.PdfReader("test_drop_h2.pdf")
r.pages[0].get_contents().get_data()
# returns b'q\n\nQ\nq\nBT\n0 Tr\n/DeviceRGB cs\n0 0 0 sc\n/FNXH 10 Tf\n1 0 0 1 300 300 Tm\n(1 PAGE DATA)\nTj\nET\nQ\n'
r.pages[0].get_contents().operations
# returns [([], b'q'), ([], b'Q'), ([], b'q'), ([], b'BT'), ([0], b'Tr'), (['/DeviceRGB'], b'cs'), ([0, 0, 0], b'sc'), (['/FNXH', 10], b'Tf'), ([1, 0, 0, 1, 300, 300], b'Tm'), (['1 PAGE DATA'], b'Tj'), ([], b'ET'), ([], b'Q')]

the analysis is correctly performed . No fix is required

[duaneg3]

In 1.26, I get the contents of a per PDF object basis rather than on a per PDF page basis.
I think this is what is causing the issue .. not sure that is allowed in the latest release.

[pubpub-zz]

Array of Contents are still allowed but it is quite difficult to work on. .getcontents() ease the work

[duaneg3]

If you want try this example .. I put the extra characters at the end of the page.
If they get dropped here, I suspect they will be dropped at the end of pdf object if just the contents of that object are gotten ..

test_drop_h3.pdf

[pubpub-zz]

this is the test:

r = pypdf.PdfReader("test_drop_h3.pdf")
r.pages[0].get_contents().get_data()
#returns b'q\n\nQ\nq\nBT\n0 Tr\n/DeviceRGB cs\n0 0 0 sc\n/FNXH 10 Tf\n1 0 0 1 300 300 Tm\n(1 PAGE DATA)\nTj\nET\nQ\n(1 DATA)\n'
r.pages[0].get_contents().operations
#returns [([], b'q'), ([], b'Q'), ([], b'q'), ([], b'BT'), ([0], b'Tr'), (['/DeviceRGB'], b'cs'), ([0, 0, 0], b'sc'), (['/FNXH', 10], b'Tf'), ([1, 0, 0, 1, 300, 300], b'Tm'), (['1 PAGE DATA'], b'Tj'), ([], b'ET'), ([], b'Q')]

It is meaningless to analysis element(partial content) of the ArrayObject directly : you have first to concat them back before doing analysis
(1 PAGE DATA) is properly read/kept and associated with Tj operator, whereas 1 DATA at the end is discarded as can not be properly associated to an operator but it is still visible as a bytestring with .get_data()
I see no issue there

[duaneg3]

If you want to uncompress an object, how do you do it in 4.0?
(it is get_contents() in 1.26) .. just get_data() in 4.0 ??
There are lots of reasons to operate on an obj by obj basis.
If you to uncompress an object .. If you want to remove hidden text or change hidden text to visible .. an object by object basis is fine .. and it leaves the hierarchy as it was ... there are very few (but some) instances where operations span objects. Just want to make sure don't mess these up.

Having get_data() and operations all return the same data seems to make sense to me. Less chance to mess things up. I think dropping things quietly, is never a good thing to do. Adding one line to make sure it does not happen seems like a good trade of to me. Either way, is fine with me. I am stuck at 1.26.

Full 1.26 code with added line:

def ContentStream__parseContentStream__(self, stream):
    # file("f:\\tmp.txt", "w").write(stream.read())
    if 1==2: print("start content stream")
    stream.seek(0, 0)
    operands = []
    while True:
        peek = utils.readNonWhitespace(stream)
        if peek == b_('') or utils.ord_(peek) == 0:
            break
        stream.seek(-1, 1)
        if peek.isalpha() or peek == b_("'") or peek == b_('"'):
            operator = utils.readUntilRegex(stream,
                    NameObject.delimiterPattern, True)
            if operator == b_("BI"):
                # begin inline image - a completely different parsing
                # mechanism is required, of course... thanks buddy...
                assert operands == []
                ii = self._readInlineImage(stream)
                self.operations.append((ii, b_("INLINE IMAGE")))
            else:
                self.operations.append((operands, operator))
                operands = []
        elif peek == b_('%'):
            # If we encounter a comment in the content stream, we have to
            # handle it here.  Typically, readObject will handle
            # encountering a comment -- but readObject assumes that
            # following the comment must be the object we're trying to
            # read.  In this case, it could be an operator instead.
            while peek not in (b_('\r'), b_('\n')):
                peek = stream.read(1)
        else:
            operands.append(generic.readObject(stream, None))
    ## DEG: need next line with have incomplete command (finished by another object)
    if (len(operands)>0): self.operations.append( (operands,b_("")) )

[pubpub-zz]

from the top of my head there is no change since 1.26 in here .get_contents() provides you access to a ContentStream Object , you have then to select how you want to "decode" it : .get_contents().get_data() if you want plain text, .get_contents().operations if you want extracted operations
operations are provided after parsing and typically remove comments(tagged with %) => operations and plain_text may not report the same data.

with your example the content is corrupted, and the operation parsing cleans it.

if a PR would have to be proposed, it would be to raise a warning which may induce lots of warning reports. feel free to submit a PR on 4.x branch if you think it worth it.