Why are so many fields NotRequired?

[maxrabin]

I find that many fields in response TypeDicts are Optional when it seems from the API that they aren't. For example:

import boto3

s3 = boto3.client("s3")
resp = s3.list_buckets()

buckets = resp["Buckets"]
for bucket in buckets:
    name = bucket["Name"]
    print(name)

I get a type checking error from pyright:

tmp.py:8:12 - error: Could not access item in TypedDict
    "Name" is not a required key in "BucketTypeDef", so access may result in runtime exception (reportTypedDictNotRequiredAccess)
1 error, 0 warnings, 0 informations

This is due to the definition for BucketTypeDef:

class ListBucketsOutputTypeDef(TypedDict):
    Buckets: list[BucketTypeDef]
    Owner: OwnerTypeDef
    ContinuationToken: str
    Prefix: str
    ResponseMetadata: ResponseMetadataTypeDef


class BucketTypeDef(TypedDict):
    Name: NotRequired[str]
    CreationDate: NotRequired[datetime]
    BucketRegion: NotRequired[str]

The Name field of BucketTypeDef is NotRequired. Is this really the case that there can be a response to list_buckets which returns ListBucketsOutputTypeDef which has a list of Buckets but some of the items in the list don't have Name?

I have been adding a bunch of asserts to my code to verify that these NotRequired fields exist in order to overcome this typing "error" (assuming that the field exists) but it seems unnecessary.

[vemel]

Good question.

Type annotations are generated based on botocore shapes, and I have no way to check which fields are Required if a shape does not define required key.

    "ListBucketsOutput":{
      "type":"structure",
      # no `required` key at all
      "members":{
        "Buckets":{
          "shape":"Buckets",
          "documentation":"<p>The list of buckets owned by the requester.</p>"
        },
        ...
      },
    },

    "Buckets":{
      "type":"list",
      "member":{
        "shape":"Bucket",
        "locationName":"Bucket"
      }
    },
    "Bucket":{
      "type":"structure",
      # no `required` key as well
      "members":{
        "Name":{
          "shape":"BucketName",
          "documentation":"<p>The name of the bucket.</p>"
        },
        "CreationDate":{
          "shape":"CreationDate",
          "documentation":"<p>Date the bucket was created. This date can change when making changes to your bucket, such as editing its bucket policy.</p>"
        },
        "BucketRegion":{
          "shape":"BucketRegion",
          "documentation":"<p> <code>BucketRegion</code> indicates the Amazon Web Services region where the bucket is located. If the request contains at least one valid parameter, it is included in the response.</p>"
        }
      },
      "documentation":"<p> In terms of implementation, a Bucket is a resource. </p>"
    },

I mark all keys in Output TypedDict if it is a return type annotation for a method. However, marking all keys in nested TypedDicts is dangerous, because some fields could be missing.

Even for direct return type annotations not all keys are marked as required. I have a list of keys that are still optional even though they are in return type annotations. An example for S3 is LastEvaluatedKey that is optional in response. Also, ContinuationToken should probably be marked as NotRequired as well.

So, unless a shape explicitly specifies required keys, there is no actual way to say if a key is required or not. I can use AI to get this info from a docstring for output shape, but again, it won't work for nested shapes :)

[maxrabin]

Thanks for the quick, thorough response!
I was wondering if botocore was the best place source for the fields, so I checked the API docs https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html and found that for Bucket indeed Name is not Required.

I'll keep looking at more APIs and see if I can find case(s) where the API docs offer Required designations where botocore doesn't.

[vemel]

Thank you! Also, let me know if you find any keys that can potentially be missing in *OutputTypeDefs. As I said, ContinuationToken is a good candidate, but it is currently marked as required as well.