Handle an SFTP failure (incident playbook)

Outcome

EDI transactions stuck in SUBMITTING status are unblocked, the underlying SFTP issue (auth, credential expiry, network) is identified and fixed, and failed transactions are retried.

Prerequisites

• PLATFORM_ADMIN or EDI_OPERATOR (rotation + retry rights).
• SSH client with sftp for direct connectivity probes.
• Access to Key Vault for credential inspection / rotation.

Symptoms

• EDI transactions stuck in SUBMITTING status for > 10 minutes.
• Error logs showing connection timeouts or authentication failures.
• Response poller (271 / 277CA / 835) finding no new files for hours.

Decision flow

Steps

1. Diagnose the affected transactions:

   SELECT edi_transaction_id, direction, tx_type, status,
          error_details, retry_count, created_at
   FROM billing.edi_transaction
   WHERE status IN ('ERROR', 'SUBMITTING')
     AND created_at > NOW() - INTERVAL '24 hours'
   ORDER BY created_at DESC;

   Note the trading_partner and the count — is this one partner or all of them? One partner narrows to credential/connectivity; all partners points at network or app-side.

2. Probe SFTP connectivity from the rcm-core host:

   sftp -o ConnectTimeout=10 rcm_user@sftp.mits.example.gov

   Result	Likely cause	Next step
   Connection refused / ETIMEDOUT	Network or partner down	Wait, then escalate to partner / network team
   Permission denied (publickey,password)	Credential expired or rotated upstream	Rotate via credential vault
   Lands in shell	App-side issue	Continue to step 4

3. Inspect connection config for typos or stale entries:

   SELECT name, connection_config
   FROM rcm_master.trading_partner
   WHERE partner_code = 'OH_MITS';

   Common drift: port changed, remotePath renamed, hostname migrated. Update via the trading partner UI — direct SQL edits skip the audit trail.

4. If credential rotation is needed, follow Rotate trading-partner credentials. The UI confirms the bind with a Run test button after rotation.

5. Retry failed transactions once connectivity is restored:

   # Replay one transaction
   curl -X POST http://localhost:3011/edi/admin/replay/<edi_transaction_id>
   
   # Or replay a batch
   curl -X POST http://localhost:3011/edi/admin/replay/batch \
     -H "Content-Type: application/json" \
     -d '{"trading_partner_id": "<id>", "since": "2026-04-27T00:00:00Z"}'

6. Monitor retry counts — anything > 3 retries on the same transaction needs manual review (the auto-retry logic gives up at 5 by default):

   SELECT edi_transaction_id, retry_count, error_details
   FROM billing.edi_transaction
   WHERE retry_count > 3 AND status = 'ERROR';

Validation

Check	Expected
sftp from rcm-core host	Lands in shell
EDI transactions in SUBMITTING	Drop back to baseline
Response poller pulling files	New files within poll interval
error_details on retried transactions	Cleared on success

Troubleshooting

Symptom	Cause	Fix
Probe works from your laptop, fails from rcm-core host	Outbound firewall / NSG rules differ	Update Azure NSG; whitelist partner IPs.
auth failed despite a fresh password	Partner expects key auth, not password	Switch to private-key auth in connection_config.credentials.
All partners failing simultaneously	Outbound proxy or DNS broken	Check rcm-core logs for DNS errors; restart if proxy is wedged.
Replay returns 200 but transaction stays ERROR	Underlying batch is malformed (not a transport issue)	Inspect error_details; this is no longer an SFTP problem.
Partner says they received our 837 but our submission_status shows ERROR	Partner ack didn't come back; we don't know we succeeded	Check 999/TA1 response; the partner's portal may show the receipt.

Cross-references

• Trading partner credential vault — for any rotation work.
• Submission routing rules — to check which partners + companion guides are in play.
• Distributed tracing — for correlating SFTP errors across services via trace IDs.

Next

2.4 — 278 prior auth playbook