In one of our application, while connecting to DynamoDB, our application was getting following exception-
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
The request signature we calculated does not match the signature you provided.
Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
There was no deployment on our side and we were surprised to see this issue. This issue was not happening for all the requests to DynamoDB. So, we were not sure where is the problem coming from.
There are already various posts to troubleshoot this exception-
1. https://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html
2. Problem with the skewed clocks on the server.
3. Access token not valid.
We tried all the steps as suggested by AWS tech support, various posts on google or stack overflow but none of these worked out.
So, we enabled the debug logs and analyzed the POST requests going to DynamoDB. After going through the request payload and headers, we found out that -
1. All failed requests have extra header "lets-say-xyz_now" which was passed to DynamoDB request.
2. All successful requests did NOT have extra header "lets-say-xyz_now" in the request to DynamoDB.
So, AWS was using this extra header in the signing algorithm which caused this exception.
One of our upstream services passed this header which got passed to DynmoDB. So, we removed the extra header and everything started working again.
Another suspect is underscore(_) present in the header. Some blogs suggest that nginx drops headers with underscore. http://nginx.org/en/docs/http/ngx_http_core_module.html#underscores_in_headers
