Views:

Applies to Product - FnO Platform



What’s happening?
Customers are experiencing issues with Batch AOS servers crashing intermittently and batch jobs getting stuck in an executing state for extended periods.


Reason:

  • The crashes of Batch AOS servers are likely related to a transient issue impacting the East US region, which has been confirmed to affect Azure SQL services.
  • A T3 environment is not suitable for performance testing due to having less powerful resources than production, which may lead to instability during stress testing.
Resolution:
For Batch AOS crashes:
  1. Confirm that all AOSs (both batch and interactive) crashed at a specific time and check for any further crashes.
  2. Advise the customer that a T3 environment is not suitable for performance testing and recommend considering a more robust environment if issues persist.
  3. Monitor for any further incidents and lower the severity of the case if no additional crashes occur. 

For batch jobs stuck in executing state: 
  1. Request the customer to provide the batch job ID and the timestamp of when the issue started.
  2. Instruct the customer to use the enhanced batch abort option to cancel the stuck batch job. Refer them to the Microsoft documentation for guidance on canceling a running batch job.
  3. If the job remains in a canceling state without progress, consider restarting the batch service as a potential solution.
  4. Even if after restart , batch job still in cancelling state , then reach out to PG team for further mitigation to run the required scripts.