Hi Alert101

From above description, it seems you are facing memory outage issue in first epoch itself.

The stall right after the first epoch and the eventual DATA_CAPABILITY … "ping timeout" strongly suggests your job is getting stuck on writes to the RW mount (checkpointing) while the BlobFuse/AML data runtime becomes unhealthy under load. There are two compounding factors here:

1. Write path + cache mode — frequent appends/writes over a mounted blob can degrade or wedge the mount, especially with block‑based cache enabled; BlobFuse2 has known issues in block‑cache mode for certain write patterns and recommends file‑cache mode for reliability. [github.com]
2. Low‑priority (spot) compute — preemption or transient capacity blips can interrupt the data runtime and make the lifecycler/mount service report timeouts; for long I/O‑heavy training, switching to Dedicated VMs is the recommended way to rule this out. [hi-ml.readthedocs.io], [Architectu...soft Learn | Learn.Microsoft.com]

Please reduce your batch size, cache_size to successfully save the weight periodically through check points.

Below is a pragmatic checklist to stabilize the run and get throughput back.

1) Don’t write checkpoints directly to the RW mount

Write locally (NVMe) and upload/sync in the background. This bypasses the FUSE overhead on every small write/append and avoids re‑upload of whole files that AML mount can trigger. The AML guidance explicitly calls out that file writes over mount can slow down or stall; prefer upload semantics or copy from local. [Troublesho...soft Learn | Learn.Microsoft.com]

YAML (v2) – use outputs and write locally, then sync

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
experiment_name: tfrecord-large
display_name: tfrecord-large-run
# Inputs: RO mount for training data
inputs:
train_data:
type: uri_folder
path: azureml://datastores/<blob-datastore>/paths/<container>/tfrecords/
mode: ro_mount
# Outputs: write locally; AML will upload at job end
outputs:
checkpoints:
type: uri_folder
path: azureml://datastores/<blob-datastore>/paths/checkpoints/${{name}}/
mode: upload # avoids live RW mount writes
environment_variables:
# See section 2 for cache tuning; keep write path local regardless
DATASET_MOUNT_BLOCK_BASED_CACHE_ENABLED: "true" # for RO input
DATASET_MOUNT_BLOCK_FILE_CACHE_ENABLED: "true"
DATASET_MOUNT_CACHE_SIZE: "-8192MB" # keep ~8GB free on disk
Keras callback: write locally, then sync periodically
Python
import os, subprocess, time
from tensorflow.keras.callbacks import Callback, ModelCheckpoint
local_ckpt_dir = "/mnt/outputs/checkpoints" # local NVMe-backed path
os.makedirs(local_ckpt_dir, exist_ok=True)
# Save checkpoint locally each epoch
ckpt_cb = ModelCheckpoint(
filepath=os.path.join(local_ckpt_dir, "ckpt-{epoch:04d}.h5"),
save_weights_only=True,
save_freq="epoch"
)
# Optional: background sync to blob every N epochs to reduce loss risk
class AzCopySync(Callback):
def __init__(self, every_n=5, dest_url=None):
super().__init__()
self.every_n = every_n
self.dest_url = dest_url # SAS or datastore-mounted target
def on_epoch_end(self, epoch, logs=None):
if (epoch + 1) % self.every_n == 0 and self.dest_url:
# azcopy sync is efficient & resumable
subprocess.run([
"azcopy", "sync", local_ckpt_dir, self.dest_url,
"--recursive=true"
], check=False)
# model.fit(..., callbacks=[ckpt_cb, AzCopySync(every_n=5, dest_url="<blob-sas-url>")])
``

Show more lines

The data‑loading best practices doc also emphasizes when to use mount vs download and how to tune them for large workloads. [github.com]

2) Tune mount/cache settings (especially disable block‑cache on write paths)

Given your env vars:

• DATASET_MOUNT_CACHE_SIZE: "-4096 MB" — AML accepts negative values to reserve free disk; prefer no space in the unit and a consistent unit, e.g. "-4096MB" or "-4GB".
• DATASET_MOUNT_MEMORY_CACHE_SIZE: "32768" — MB value; set based on available RAM.
• Critical: Keep block‑based cache ON for the RO training dataset (large sequential reads) but OFF for any RW path (checkpoints). BlobFuse2 cautions that block‑cache has correctness/performance problems for random writes; use file‑cache only for write scenarios. [github.com], [github.com]

If you must keep a live RW mount (e.g., for mid‑run artifact access), set:

For RW mounts (checkpoint path)

DATASET_MOUNT_BLOCK_BASED_CACHE_ENABLED = "false"
DATASET_MOUNT_BLOCK_FILE_CACHE_ENABLED = "true"
DATASET_MOUNT_FILE_CACHE_PRUNE_TARGET = "0.20" # start pruning earlier

Reference: Access data in a job and best practices describe available mount settings and when each cache mode is appropriate. [learn.microsoft.com], [github.com]

3) Use Dedicated compute to rule out spot preemption

Low‑priority clusters can be interrupted any time; under heavy I/O, that looks like random “ping timeouts” from the lifecycler/data runtime. For a large, long‑running training with a single node and big mounts, switch to Dedicated (same VM size) until the pipeline is stable; re‑introduce spot later if needed. [hi-ml.readthedocs.io], [Architectu...soft Learn | Learn.Microsoft.com]

4) Consider Azure Files (Premium) for checkpoint path

If you want a networked RW store that tolerates frequent small writes/appends, Azure Files (SMB) often behaves better than Blob over FUSE for this specific pattern. Register an azure_file datastore and point outputs.checkpoints there (still writing locally and uploading at end is preferable, but SMB can help if you must share intermediate files). Guidance on mount vs download and RW scenarios is covered in the AML docs. [learn.microsoft.com]

5) Validate BlobFuse2 health & versions

• Ensure your workspace/runtime uses a BlobFuse2 version without block‑cache write issues; if you see those, switch to file‑cache for all mounts. Known‑issues page: [github.com]
• Use BlobFuse2 Health Monitor and the AML job logs to check cache pressure and mount errors. Troubleshooting refs: [learn.microsoft.com], [docs.azure.cn]

6) tf.data input pipeline hygiene (RO mount)

Keep the GPU fed but avoid excessive random seeks into TFRecords on a mount: use num_parallel_reads, prefetch, and co‑located storage/compute (same region) as you already do (East US). The AML “Efficient data loading” doc has concrete patterns and throughput notes. [github.com]

7) What about LRS → ZRS?

Changing redundancy (LRS→ZRS) won’t address the mount stalling during checkpoint writes. ZRS improves durability/availability of the storage service, but your failure is happening inside the job’s data runtime/mount path (gRPC ping to the data capability). Focus on write strategy & cache mode first. The behavior you posted matches other recent reports of DATA_CAPABILITY/lifecycler timeouts under heavy I/O. [learn.microsoft.com]

8) Collect the right logs quickly

• Stream job logs: az ml job stream -n <job_id>
• Inspect system_logs → data-capability and user_logs/std_log.txt for mount errors and cache pruning messages; the troubleshooting doc points out these locations and the impact of writes over mount. [Troublesho...soft Learn | Learn.Microsoft.com]

Suggested course of action.

1. Write checkpoints locally (/mnt/outputs) and sync (upload mode or azcopy) rather than a live RW mount. [Troublesho...soft Learn | Learn.Microsoft.com]
2. Disable block‑cache for any RW path; keep file‑cache only there; continue using block‑cache for RO training data. [github.com], [github.com]
3. Run on Dedicated for this debug cycle to eliminate spot preemption. [hi-ml.readthedocs.io], [Architectu...soft Learn | Learn.Microsoft.com]
4. Optionally, move the checkpoint target to Azure Files (Premium) if you need shared intermediate access. [learn.microsoft.com]the stall right after the first epoch and the eventual DATA_CAPABILITY … "ping timeout" strongly suggests your job is getting stuck on writes to the RW mount (checkpointing) while the BlobFuse/AML data runtime becomes unhealthy under load. There are two compounding factors here:
   1. Write path + cache mode — frequent appends/writes over a mounted blob can degrade or wedge the mount, especially with block‑based cache enabled; BlobFuse2 has known issues in block‑cache mode for certain write patterns and recommends file‑cache mode for reliability. [github.com]
   2. Low‑priority (spot) compute — preemption or transient capacity blips can interrupt the data runtime and make the lifecycler/mount service report timeouts; for long I/O‑heavy training, switching to Dedicated VMs is the recommended way to rule this out. [hi-ml.readthedocs.io], [Architectu...soft Learn | Learn.Microsoft.com]

Thank you.