Forwarder failing to connect to S3 over VPN

[chadmiracle]

Platform AWS
VPN without a NAT nor Internet gateway
Using Private Links only.

Forwarder is attempting to contact S3 using host:s3.amazonaws.com

This would be fine if going over the open internet, but it is an invalid URL for an VPC Endpoint. Logs are making it Datadog successfully. I have only recently turned on access logs to the bucket to see what is actually accessing the bucket

Oh dang, that's alot of logs, sorry. But this is also according to Datadog support. They won't continue looking into the issue until this timeout is corrected.

`2024-11-22T15:23:50.189Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Event request-created.s3.GetObject: calling handler <function add_retry_headers at 0x7f19a916db20>

2024-11-22T15:23:50.189Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://s3.amazonaws.com/REDACTED/app-ui-bucket-logs/2024-11-22-15-23-49-4F47DF96D978604A, headers={'User-Agent': b'Boto3/1.34.145 md/Botocore#1.34.145 ua/2.0 os/linux#5.10.227-239.884.amzn2.x86_64 md/arch#x86_64 lang/python#3.11.10 md/pyimpl#CPython exec-env/AWS_Lambda_python3.11 cfg/retry-mode#legacy Botocore/1.34.145', 'X-Amzn-Trace-Id': b'Root=1-6740a206-1c4330fa538d265arent=6591e65b3219a5b7;Sampled=0;Lineage=1:e1a66698:0', 'X-Amz-Date': b'20241122T152350Z', 'X-Amz-Security-Token': REDACTED, 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20241122/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=b58b283a3f9bfd117c7247c546355a9f515ac80a01d13d', 'amz-sdk-invocation-id': b'436e19d5-2ce9-4f09-8c88-1f204d9aa192', 'amz-sdk-request': b'attempt=1'}>

2024-11-22T15:23:50.189Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Certificate path: /opt/python/certifi/cacert.pem

2024-11-22T15:23:50.190Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Starting new HTTPS connection (1): s3.amazonaws.com:443

2024-11-22T15:23:50.255Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] https://s3.amazonaws.com:443 ""GET /REDACTED/app-ui-bucket-logs/2024-11-22-15-23-49-4F47DF96D978604A HTTP/1.1"" 200 902

2024-11-22T15:23:50.255Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Response headers: {'x-amz-id-2': 'LKTkUhvSPUdPWD7HAxJC5vnhGPIkRhC1qT2I5THb4gJo=', 'x-amz-request-id': 'YYH5DBNGZ9', 'Date': 'Fri, 22 Nov 2024 15:23:51 GMT', 'Last-Modified': 'Fri, 22 Nov 2024 15:23:50 GMT', 'ETag': '""30e7267f9181a2ee8efb0bcacb7f409f""', 'x-amz-server-side-encryption': 'AES256', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain', 'Content-Length': '902', 'Server': 'AmazonS3'}

2024-11-22T15:23:50.255Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Response body:
<botocore.response.StreamingBody object at 0x7f19a4e72800>

2024-11-22T15:23:50.256Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f19a4118bd0>

2024-11-22T15:23:50.256Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] No retry needed.

2024-11-22T15:23:50.256Z 90f47740-2bd4-4b10-a2a0-2bf895592fe8 [dd.trace_id=6020510453849326526 dd.span_id=7668974733756765316] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f19a41ae390>>

`