Can’t save all transaction on WMS due to AWS issue

Creation date: 03/02/2023 14:01    Updated: 03/02/2023 14:01   #aws #unknown environment
Question:

Affected Environment:

< PROD >

Affected warehouse:

< ALL>

Screenshots of error messages:


Time when issue occurred:

< 2023-01-31 17:10 PM >

Affected module:

< ALL MODULES >

Detailed description of issue:

< Can’t save transaction on all modules in WMS including RF and Datahub>

Expected result:

<>

First level investigation/troubleshooting taken:

< Input on investigation and troubleshooting >


Case replication in original warehouse:


< Details of replicated case >


Answer:

Investigation:

The backend server environment turn UNKNOWN.

This is an issue with AWS cloud.

We have recommend to restart the server but it does not help.


From our previous experience with AWS, there seems to be a bug with AWS that we need to deploy the sample application, terminate EC2 for WMS and deploy WMS zip application again.

When FAST request what else can be done, we recommended the steps above but the sample application doesn’t seem to be able to deploy.


The sample application is from AWS. If it cannot then AWS actually have an issue.



In the whatsapp group discussion, we have confirmation that FAST requesting us to get WMS up for operation and we have inform this will be billable SR – on urgent support.

Note that this is AWS issue but we are trying to get WMS to work.

Initially we try not to change any security rule and  group and trying to work within the boundary but after follow through the recommended steps, we still unable to get WMS to work.


Upon escalation at 11pm, we decided to restore all the security rule and group to default and run the deployment again. We have reduce the backend server from 4 to 1 server for easy investigation.

We notice when we did this, we can see the Datahub is not connected to MongoDB. Upon investigation we realise that the IP is out of range and they are unable to connect.

After informing FAST to add in the range, Datahub is able to connect.

We checked on frontend and backend and it has the same issue. Unable to connect to each other. After allowing port 80 and http in backend, it is able to connect.


At this point, we confirm WMS is able to commit transaction.


With this, we spawn the backend instance from 1 to 4. Initially user confirms it is working but after a while, it does not work again.

The setting on AWS seems to be an issue.


We reduce the backend instance to 1 and after checking, we ask that the backend config setting be sync to frontend.

Both backend and frontend now has 1 instance.

After we have sync the setting from the backend server to the frontend server, it seems to be good.





We confirm with operation that it is working.

AWS setting is interfering with WMS. Advise customer to do the setting carefully.


Files