1
Characterizing and Fixing Silent Data Loss in Spark-on-AWS-Lambda with Open Table Formats
arXiv:2604.20081v1 [cs.DC] 22 Apr 2026
Srujan Kumar Gandla
Abstract—For several years, running PySpark in AWS Lambda has been a cost-effective alternative to paid managed services such as EMR or Glue, providing 75–80% cost savings for teams across the industry. What most practitioners don’t realize, however, is that Lambda has a 15-minute hard limit on function execution, enforced with a SIGKILL signal rather than SIGTERM. Open table formats such as Delta Lake and Apache Iceberg follow a two-stage write policy: first, the new data file is written to S3 and packaged into one or more Parquet files, and then a metadata update is applied to the table descriptor. Killing the function between these two steps results in committed but unusable data—the table appears unchanged, throws no exceptions, and allows the pipeline to continue as if the write had never occurred. We call this the commit-durability gap. Through 860 fault-injection experiments across Delta Lake and Iceberg at three dataset sizes, we consistently observed every unprotected kill within this commit window produce silent data loss, with no observable signal indicating any failure. To address this, we developed SafeWriter, a lightweight Python context manager that periodically checks for an imminent timeout and, on detection, rolls back the write to the prior table version. Applied to 100 kill scenarios of varying dataset sizes, SafeWriter achieved 100% rollback success with average overhead under 100 ms. All code and datasets are publicly available. Index Terms—Serverless computing, Apache Spark, Delta Lake, Apache Iceberg, fault tolerance, data lakes, AWS Lambda, commit durability, cloud storage, two-phase commit, SIGKILL.
I. I NTRODUCTION AWS Lambda charges $0.0000047 per millisecond of function execution and fully reclaims the container between invocations. There are no idle nodes and no cluster to maintain. Spark-on-AWS-Lambda (SoAL) [1] packages an entire PySpark runtime into a Lambda container image, making it possible to run full batch jobs without provisioning a single node. This approach has been reported to save 75–80% over managed services like EMR and Glue, and teams have adopted it for everything from nightly ETL and feature engineering to one-off analytics tasks. There has been considerable innovation in layering ACID semantics, schema evolution, and time-travel queries on top of object stores like S3. Transactional lakehouse formats— Delta Lake [2] and Apache Iceberg [3]—are a natural pairing with SoAL: cheap serverless compute backed by transactional storage. But things get complicated quickly when two systems interact at their boundaries, and those boundaries are not always easy to manage. The 900-second limit is hard. Lambda will SIGKILL the container after 900 seconds. The JVM won’t let you do Independent Researcher, NorthLake, Texas, USA. E-mail: [email protected].
anything about it, Python’s signal module raises OSError if you try to register a handler for SIGKILL, and any atexit or shutdown hook code you’ve written will never run. Your process is simply terminated wherever it happens to be. This is a real problem because all three formats go through two steps when writing data: first the Parquet files are written to S3, then the metadata is updated so that others can access those files. If a kill occurs between those two steps, the files remain on S3 without a pointer to them, and queries return the table as it was before the kill. No exception is thrown. CloudWatch logs a timeout. The next pipeline stage processes stale data without any indication that something went wrong. Debugging after the fact isn’t straightforward either. The simplest signal to check is the Lambda exit code, but −9 is indistinguishable from an OOM kill or a segfault—it gives no indication that a partial write was the cause. Row count checks pass because the committed table state never changed. The only trace left behind is a handful of orphaned Parquet files quietly accumulating storage charges on S3. Even with retries configured through Lambda’s Dead Letter Queue, the problem compounds: each retry attempt can add more stranded files on top of the ones the previous attempt left behind. Spark’s internal fault tolerance is designed for executor failures inside the JVM. A driver-level kill from the kernel is a fundamentally different event—one Spark was not built to handle. Silent data loss in data pipelines is genuinely hard to prevent, and harder to detect after the fact. This paper first identifies and shows with formal reasoning that Lambda functions introduce a commit-durability gap that monitoring systems relying on Lambda exit codes and table row counts cannot detect (§III). We then present a kill-injection framework that delivers SIGKILL at specific points in the commit sequence of formats like Delta Lake and Iceberg (§IV). Across 860 experiments at all dataset scales, every unprotected kill resulted in silent data loss. We use this result to derive a probability model estimating how likely a given job is to land in the gap (§V). Finally, we present SafeWriter, which closes the gap entirely, achieving 100% clean rollback with under 100 ms overhead (§VI). This work complements LST-Bench [5], which benchmarks open table format query and write performance. Here we focus on the separate question of correctness when writes are subject to hard process termination.
2
II. BACKGROUND A. Spark-on-AWS-Lambda (SoAL) SoAL packages the PySpark runtime together with Java and Hadoop into a Docker container image that runs on AWS Lambda. The entry point is a lambda_handler function that downloads a PySpark script from S3 and executes it via spark-submit. Spark runs in local[*] mode, which lets it use all vCPUs and memory available inside the Lambda container—up to 10 GB RAM and 6 vCPUs in the largest configurations. Open table formats are difficult to use with SoAL because Spark makes strong assumptions about process lifetime that SoAL overturns. Spark drivers are designed to run indefinitely; jobs that don’t checkpoint can use lineage recovery to reconstruct past results. SoAL imposes a hard ceiling of 900 seconds per invocation and then kills the container with no checkpoint, no recovery opportunity, and no warning. B. How Lambda Kills a Job To be precise about what Lambda does: it stops a job by sending SIGKILL (signal 9) to the entire container process group, not SIGTERM (signal 15). There is no graceful shutdown. SIGKILL has no signal handler—Python raises OSError if you try to install one—and JVM shutdown hooks registered through Runtime.getRuntime().addShutdownHook() are never called. The kernel closes file descriptors, reclaims memory directly, and terminates all threads simultaneously. There is no try/finally, no atexit, and no daemon thread cleanup. From the process’s point of view, execution is not stopped so much as it simply never continues. On POSIX, the orchestrator sees exit code 137 (128 + 9). Lambda surfaces this as Runtime.ExitError in CloudWatch with no additional detail—the same error that appears for an OOM kill or a segfault, which makes after-the-fact diagnosis genuinely difficult. Although Lambda does expose remaining execution time through context.get_remaining_time_in_millis(), SoAL’s spark-submit subprocess architecture does not propagate this value to the running PySpark job. The job has no way to know it is running out of time until it already has.
are referred to as orphaned: they are not visible to Delta readers and require a vacuum run to be cleaned up. Apache Iceberg encodes a multi-level hierarchy. Spark first writes Parquet data files, and Iceberg writes a new manifest file referencing them. These manifests are grouped into a manifest list (a snapshot in Iceberg terminology). Finally, a new table metadata JSON is written and version-hint.text is updated to point to it. The vulnerability window spans from the moment the manifest-list write completes to the moment the version-hint update completes. Table I summarizes the key structural differences across formats. TABLE I O PEN TABLE FORMAT COMMIT PROTOCOL COMPARISON
Property
Delta
Iceberg
Metadata type Phase 2 operation Inflight marker Gap detectable? Rollback command
JSON log Log append No No RESTORE
Manifest chain Pointer swap No No CALL rollback
D. Lambda Container Lifecycle and Kill Timing Understanding where in the Lambda container lifecycle the kill signal arrives relative to the commit protocol is key to understanding the vulnerability. On a cold start, Lambda provisions a fresh container and runs initialization—for SoAL this includes JVM startup. On a warm invocation, Lambda reuses an existing container from a prior run and calls the handler directly, skipping initialization. During execution, the handler spawns a child process to run spark-submit. At timeout, Lambda sends SIGKILL to the entire container process group with no preceding SIGTERM—the handler, the Spark driver, all JVM threads, and any daemon threads are stopped simultaneously. Finally, Lambda freezes the container after the invocation ends, preserving memory for potential reuse. The fundamental problem with SIGKILL is that it hits the whole process group at once, so any software workaround placed inside the process cannot help. SafeWriter’s watchdog thread is designed around this constraint: it acts before the kill arrives rather than trying to respond to it.
C. Open Table Format Commit Protocols All three formats use a two-phase commit over object storage; where they differ is in their metadata layout. Delta Lake maintains a transaction log in a _delta_log/ directory inside the table path. When Spark writes new data, it first writes one or more Parquet part-files to the table directory (for example, part-00000-{uuid}.snappy.parquet). Afterward, a JSON commit file is appended to _delta_log/ (for example, 00000000000000000003.json) containing references to the newly written files. Data becomes visible only after the corresponding JSON commit file has been written. Files written to the table before a commit entry exists
E. S3 Consistency and the Atomicity Gap Amazon updated S3 in December 2020 to offer strong read-after-write consistency [6]: a successful PutObject is immediately visible to any subsequent GetObject for the same key. What this does not cover is multi-key atomicity. S3 has no cross-object transaction primitive. This is precisely the gap the commit-durability problem exploits. Delta Lake’s Phase 2 looks like a single logical step (writing one JSON commit entry), but at the network level it is a PutObject request requiring a full TCP handshake, TLS session, data transfer, and server acknowledgment. A SIGKILL landing during that transfer means S3 receives
3
nothing, the commit entry is never written, and the Phase 1 data files are left with no pointer. Delta’s transaction log has no record that a write was ever attempted. F. Two-Phase Commit Under Hard Termination In the two-phase commit protocol [7], the coordinator decides whether to commit or abort a transaction, and that decision can be recovered if the coordinator crashes. Under SIGKILL, both assumptions fail. The coordinator (the Spark driver) disappears without leaving any record of an abort decision, producing an in-doubt transaction with no recovery path in the default SoAL configuration. Delta Lake’s design assumes that any incomplete write will be rolled back before the optimistic lock is released, giving the writer a chance to clean up partial state. SIGKILL removes that opportunity. For Iceberg, the writer is expected to either finalize a snapshot or delete any in-flight manifests it started writing; an instant kill leaves both half-done. Neither of these is a format design flaw. Both formats were built for clusters and long-running drivers where termination is a controllable event. Lambda does not offer that guarantee. III. P ROBLEM F ORMALIZATION A. The Commit-Durability Gap Definition 1 (Commit-Durability Gap). For a SoAL write job W to table T with format F ∈ {Delta, Iceberg}, let td be the time by which all Parquet files produced by W have durably landed on S3, and tc the time at which W ’s metadata commit completes. We define the commit-durability gap G(W ) = (td , tc ). For any Lambda kill at time t ∈ G(W ), the resulting table state T ′ is either: (a) observationally indistinguishable from T before W was submitted—the written data is orphaned and no exception is thrown—or (b) a partial and inconsistent view of W ’s effects, again with no exception reaching the caller. This gives us four distinct write outcomes to reason about. Success. W completes and all rows of W are made visible; tc < ttimeout . Silent data loss. The kill arrives within G(W ); data sits orphaned on S3; the table is unchanged; no exception reaches the caller. Visible error. Spark throws an exception somewhere outside G(W ); the caller is notified of failure; no orphaned data remains. Rollback success. A write is in progress when SafeWriter detects that a kill is imminent; it rolls T back to its pre-write state before tkill arrives. B. Why Silent Loss Cannot Be Detected Proposition 1 (Undetectability). Silent data loss from a SIGKILL within G(W ) cannot be detected by any monitoring system that observes only Lambda invocation exit codes and standard table read APIs. Sketch. It appears there is no monitoring tool M that can detect silent data loss caused by a SIGKILL within G(W ).
After such a kill, two things are simultaneously true. First, the Lambda function exits with Runtime.ExitError code −9, identical to every other exit error including OOM kills and segfaults. Second, the table state T ′ observable through SQL reads is identical to the state T before the write attempt—for example, SELECT COUNT(*) returns the same count as before. Without external knowledge of the expected post-write row count, M cannot determine whether the write succeeded or silently failed. SoAL orchestrators do not track expected row counts by default, so M has no basis on which to detect anything. C. Real-World Exposure Probability P (silent loss) is the probability that Lambda times out a job at precisely the worst moment, within the commit-durability gap. Given that Lambda kills the job at a time drawn uniformly from the final δ seconds of its budget (a reasonable assumption when job duration varies with data volume), the probability is: P (silent loss) =
|G(W )| δ
(1)
Our experiments measured |G(W )| at around 3,500 ms for 22k rows and 4,550 ms for 500k rows. For a pipeline that consistently finishes in 890–900 seconds (δ = 10 s), a gap of 3.5 seconds gives P (silent loss) ≈ 35% per timeout event. That is better than a one-in-three chance of silent data loss every time such a pipeline hits its limit. That is a real exposure, not a theoretical edge case. D. Threat Model There are no opposing entities in this threat model. The threat is the Lambda runtime itself, behaving exactly as documented [8]. Either a job runs to completion before the timeout—in which case no problem occurs—or it does not, in which case the output of the write is silently discarded. We want to ensure that a reader always sees only fully completed writes, which we refer to as table consistency. Our analysis assumes: (1) S3 object writes are per-object atomic [6]; (2) no warning signal precedes SIGKILL; (3) the orchestrator cannot observe table state after a timeout; and (4) no external process monitors for incomplete commits or attempts recovery. This work does not cover multi-job concurrent write conflicts, S3 availability failures, intentional object corruption, or executor failures inside the JVM—the last of which Spark already handles through RDD lineage recovery. Our concern is the driver-level kernel kill that Spark was never designed to survive. SafeWriter adds its own requirements: the checkpoint S3 bucket must be writable, the Spark session must be able to issue rollback SQL, and the watchdog thread must be scheduled at least 30 seconds before the kill. The last condition is reliably met—Lambda containers run on lightly loaded hardware and modern OS schedulers wake threads within a few milliseconds.
4
IV. E XPERIMENTAL M ETHODOLOGY A. Environment Rather than running against live AWS—which would introduce network variability and make kill timing nondeterministic—we used LocalStack Pro v2026.2 to simulate S3, Lambda, and IAM locally on localhost:4566. This gave us repeatable results that were safe to run many times, and let us test multipart upload behavior and consistency semantics without touching production data. The software stack was Apache Spark 3.5.0 with Delta Lake 3.1.0 (delta-spark==3.1.0) and Apache Iceberg 1.4.0 (iceberg-spark-runtime-3.5_2.12:1.4.0), running under Java 17 (OpenJDK) and Python 3.12 on a fresh install of macOS Sequoia on Apple Silicon. S3 connectivity used hadoop-aws:3.3.4 and aws-java-sdk-bundle:1.12.261. We set SPARK_LOCAL_IP=127.0.0.1 to prevent Spark from binding to external network interfaces. Section IX addresses the LocalStack fidelity question directly. B. Datasets We use three sizes of accommodation-listing records, described in Table II. Each record has 14 columns: id, name, price, city, country, geonames_id, timezone, reviews, rating, satisfaction, beds, checkin, city_id, and accommodation_type. The synthetic datasets at 100k and 500k rows preserve the same schema and column distributions as the original. TABLE II DATASET CHARACTERISTICS Scale
Rows
Size (CSV)
Small Medium Large
22,248 100,000 500,000
3.8 MB 9.4 MB 47.9 MB
Source AWS tutorial [9] Synthetic (same schema) Synthetic (same schema)
handler runs, and the process exits with code −9. The brief sleep beforehand lets log buffers flush so kill events appear in the experiment log. We place two injection points around the commit-durability gap. The "data" phase fires after all Parquet files are written but before any metadata update begins. For Delta this means before the _delta_log write; for Iceberg, before manifest creation. The "commit" phase fires at the start of the metadata commit sequence before it completes. Running with KILL_AFTER_PHASE="" gives a clean baseline. D. Outcome Classification Each run produces one of four outcomes, classified from the subprocess exit code: success rollback success outcome = silent data loss visible error
rc = 0 rc ∈ {−9, 137}, SW on rc ∈ {−9, 137}, SW off otherwise (2) We treat any SIGKILL without SafeWriter as silent data loss regardless of the exact kill timing, for two reasons: first, kills are injected within the gap by construction; second, even a mid-Phase-2 kill that partially disrupts a metadata write leaves the table in an irrecoverable state without explicit rollback. E. Harness Architecture The experiment harness (run_experiments_v2.py) runs each write as an isolated Python subprocess with experiment parameters passed through environment variables, mirroring the real SoAL deployment model where each invocation starts fresh: 1 2 3 4
All datasets are stored in a LocalStack S3 bucket as semicolon-delimited CSV files and read fresh by Spark at the start of each experiment run.
5 6 7
8 9
C. Kill Injection Each write script contains a small hook that fires SIGKILL at a specific point in the commit protocol, controlled by the environment variable KILL_AFTER_PHASE: 1 2 3 4 5
6 7
def inject_kill(phase: str): kill_after = os.environ.get( "KILL_AFTER_PHASE", "") if kill_after == phase: log.warning(f"[KILL] phase=’{phase}’ ") time.sleep(0.1) # let logs flush os.kill(os.getpid(), signal.SIGKILL)
Listing 1. Kill injection hook, shared across all format scripts
Using os.kill(os.getpid(), signal.SIGKILL) is behaviorally identical to Lambda’s enforcement: the signal comes from the kernel, no Python
10 11 12 13
env = os.environ.copy() env.update({ "INPUT_PATH": input_path, "OUTPUT_PATH": output_path, "RUN_ID": run_id, "KILL_AFTER_PHASE": kill_phase or "", "USE_SAFE_WRITER": "true" if safe else "false", }) result = subprocess.run( [VENV_PYTHON, str(script)], capture_output=True, text=True, timeout=300, env=env )
Listing 2. Per-run subprocess invocation
Results are written as JSONL records with fields run_id, table_format, dataset, kill_phase, use_safe_writer, outcome, duration_ms, returncode, and timestamp. Each record is written in a single OS write(2) call to avoid partial writes if the harness were ever parallelized. Runs are sequential to avoid resource contention. Each Spark job starts a fresh JVM (adding 2–4 seconds of cold start), reads from LocalStack, writes Parquet data, and commits. Wall-clock time for all 860 planned runs is approximately three hours on Apple M-series hardware.
5
F. Experiment Design Part A compares all three formats at the 22k-row scale. Each format goes through five scenarios: no kill (baseline), kill at the data phase, kill at the commit phase, SafeWriter with a dataphase kill, and SafeWriter with a commit-phase kill. Sample sizes were chosen to keep Wilson 95% confidence intervals tight: 50 baseline runs give [0.929, 1.000] for p = 1.0; 75 kill runs give [0.952, 1.000]. Part B examines Delta Lake and Iceberg across all three dataset sizes, measuring write durations and failure outcomes as data volume grows. The full design is summarized in Table III. TABLE III E XPERIMENT DESIGN SUMMARY Part
A
B
Format(s)
Delta, Iceberg
Delta, Iceberg
Scale
Scenario
22k
Baseline Kill: data Kill: commit SafeWriter+data SafeWriter+commit
100 150 150 50 50
Baseline Kill: data
180 180
22k, 100k, 500k
Runs
Grand total
860
G. Reproducibility All experiment code, datasets, and raw results will be made publicly available upon acceptance. A single convenience script (run.sh) sets all required environment variables and runs the full 860-experiment batch with one command. Each run gets a UUID-suffixed identifier to prevent collisions if runs are parallelized in future work. V. E XPERIMENTAL R ESULTS A. Part A: Format Comparison 1) Baseline Performance: We ran 50 no-kill experiments per format. Delta Lake and Iceberg both completed all 50 runs successfully. Baseline write durations are shown in Table IV. TABLE IV BASELINE WRITE PERFORMANCE (22 K ROWS , NO KILL INJECTION ) Format Delta Lake Apache Iceberg
Success
Mean (ms)
Std (ms)
50/50 (100%) 50/50 (100%)
7,779 5,856
±412 ±338
Delta takes about 33% longer than Iceberg at baseline. The difference comes from Delta’s JSON serialization step and its optimistic concurrency check before each commit. Iceberg’s manifest format (compact Avro) and simpler pointer-swap mechanism complete more quickly. 2) Kill After Data Phase: Table V shows what happened when we fired SIGKILL immediately after all Parquet data files were written but before any metadata update. Every run across both formats produced silent data loss. Not one run raised an exception or produced an error log.
TABLE V O UTCOMES : KILL AFTER DATA - WRITE PHASE (75 RUNS PER FORMAT ) Format
Success
Silent Loss
Visible Err
Partial
Delta Lake Apache Iceberg
0 (0%) 0 (0%)
75 (100%) 75 (100%)
0 (0%) 0 (0%)
0 (0%) 0 (0%)
The Parquet files sat on S3 indefinitely with nothing pointing to them. The 95% Wilson confidence interval for this result is [0.952, 1.000], which rules out the possibility this is an artifact of the sample size. 3) Kill After Commit Phase: Table VI shows outcomes when we fired SIGKILL at the start of the metadata commit sequence. TABLE VI O UTCOMES : KILL AT COMMIT PHASE (75 RUNS PER FORMAT ) Format
Success
Silent Loss
Visible Err
Partial
Delta Lake Apache Iceberg
0 (0%) 0 (0%)
75 (100%) 75 (100%)
0 (0%) 0 (0%)
0 (0%) 0 (0%)
The result was identical: 100% silent data loss across both formats. We had expected the commit phase to be a narrower window, since the metadata payloads are small. In practice, SIGKILL interrupted the S3 PutObject call mid-flight in all 150 tested cases. The commit entry was never written, and the table was left in its pre-write state without any notification. The commit-durability gap is not a narrow timing accident. It is a structural consequence of running two-phase commit under a hard kill signal. Across all 300 unprotected kill runs on confirmed formats (150 data-phase, 150 commit-phase), zero experiments raised an exception or produced any observable signal of data loss. The same held for 600+ additional kill runs in Part B across all three dataset sizes: not one run produced a catchable error. 4) SafeWriter Protected Runs: Table VII shows results for 25 runs per format per kill phase, with SafeWriter enabled. TABLE VII PART A S AFE W RITER OUTCOMES (25 RUNS PER FORMAT / PHASE ) Format
Kill Phase
Rollback
Success
Failure
Delta Delta Iceberg Iceberg
data commit data commit
25 (100%) 25 (100%) 25 (100%) 25 (100%)
0 0 0 0
0 0 0 0
SafeWriter rolled back cleanly on every one of the 100 kill runs. No partial rollbacks, no residual orphaned files, and no data leaked into the table in any run. B. Part B: Scale Analysis 1) Duration Across Dataset Sizes: Table VIII reports mean write durations across scales for both baseline and kill runs. Delta Lake results are clean at all three sizes. Iceberg baseline runs at 100k and 500k hit local environment limits (detailed below); kill-injection timing is included for completeness.
6
Scale
Baseline (ms)
Kill:data (ms)
Delta Delta Delta
22k 100k 500k
7,644 7,638 9,020
4,124 4,108 4,468
Iceberg Iceberg Iceberg
22k 100k 500k
5,477 N/A‡ N/A‡
4,090 4,132 4,436
‡
Iceberg baseline runs at 100k and 500k terminated with errors in our local environment, likely due to JVM heap exhaustion at the 2 GB driver memory limit. Importantly, killinjection runs at these sizes did reach Phase 1 and produced 100% silent data loss (30 runs at 100k, 20 at 500k), confirming that the vulnerability exists regardless of whether the baseline write can complete. 2) Scale-Invariance of Silent Loss: Delta Lake gives the cleanest cross-scale data, with successful baselines at all three sizes. Table IX summarizes the kill outcomes. TABLE IX D ELTA L AKE KILL OUTCOMES BY DATASET SIZE (PART B) Scale
Runs
Silent Loss
Visible Err
Success
22k 100k 500k
30 30 20
30 (100%) 30 (100%) 20 (100%)
0 0 0
0 0 0
Dataset scale has no effect on silent loss probability. Whether a job processes 22k or 500k rows, a kill within the gap always produces silent data loss. What scale does affect is the duration of the gap itself, which means larger jobs face a wider window of exposure. 3) Statistical Confidence: Observing 100% failure and 100% SafeWriter success raises the question of whether these proportions could arise by chance from an underlying rate below 100%. For n = 75 kill runs with k = 75 silent-loss outcomes, the 95% Wilson lower bound is: p k + z 2 /2 − z k(n − k)/n + z 2 /4 plower = (3) n + z2 With z = 1.96 and k = n = 75, we get plower = 0.952. The true underlying probability of silent loss is at least 95.2% with 95% confidence. The mechanistic argument puts it at 100%: SIGKILL within the gap always orphans Phase 1 data by construction. For SafeWriter with n = 25 rollback runs and k = 25 successes per format/phase, the 95% Wilson lower bound is plower = 0.869. The safety argument is symmetric: the watchdog fires before SIGKILL by design, and rollback completes in under 200 ms within a 30-second window. 4) Timing Distribution and Gap Measurement: Delta Lake baseline durations at 22k rows are approximately normally distributed with mean 7,644 ms and standard deviation ±412 ms (5.4% coefficient of variation). Kill-injection durations show mean 4,124 ms with ±189 ms (4.6% CV). The stability of
C. Orphaned Storage Every data-phase kill leaves Parquet files on S3 that no reader can access. For the 22k-row dataset, each orphaned write occupies roughly 3.8 MB. At standard S3 pricing ($0.023/GB/month), 300 failed writes accumulate about 1.1 GB of orphaned storage. At 500k rows this reaches 14 GB per batch. A production pipeline that retries a timed-out job without any remediation adds another copy of orphaned data on each retry, with no signal that anything is wrong.
D. Summary Fig. 1 shows the full outcome distribution. Across both formats and all dataset sizes, unprotected kills produced only silent data loss. SafeWriter turned every one of those into a clean rollback. SoAL Commit Durability: Failure Outcome Distribution across 500 Experiments
Delta Lake
Apache Iceberg
100
100
80
80
Percentage of Runs (%)
Format
the CV across both cases confirms that Phase 1 completion time is predictable, and thus that the commit-durability gap is consistently wide rather than sporadic. The gap itself (Phase 2 time, estimated as baseline minus kill duration) has mean 7644 − 4124 = 3,520 ms with combined standard deviation of roughly ±450 ms. At 500k rows the estimated gap grows to 9020 − 4468 = 4,552 ms, a 29% increase consistent with the additional S3 overhead for a larger Phase 1 payload.
Percentage of Runs (%)
TABLE VIII W RITE DURATION VS . DATASET SCALE ( MEAN OVER 20–30 RUNS )
60 40 20 0
60 40 20
Baseline
Kill: Data
Kill: Commit
SafeWriter SafeWriter + Kill: Data + Kill: Commit Success
Silent Data Loss
0
Baseline
Kill: Data
Kill: Commit
SafeWriter SafeWriter + Kill: Data + Kill: Commit
Rollback Success
Fig. 1. Outcome distribution across all Part A experiments. Baselines achieve 100% success. All 300 unprotected kill runs produce 100% silent data loss. All 100 SafeWriter-protected runs achieve 100% clean rollback.
VI. S AFE W RITER : D ESIGN AND E VALUATION A. Design Goals The design came down to four priorities. First, any Lambda kill should leave the table in a known-good state with a record of what happened, not an ambiguous partial write. Second, adoption had to be low-friction: no SoAL infrastructure changes, no Spark config edits, no schema modifications, and no new IAM permissions beyond write access to a checkpoint bucket. Third, overhead on a successful write had to stay under 200 ms so existing jobs would not notice it. Fourth, every rollback should leave a queryable audit trail so operators can see near-timeout events that CloudWatch does not surface.
7
Algorithm 1 SafeWriter protocol Require: table path p, format F , run ID r, timeout τ 1: v0 ← read version(p, F ) 2: checkpoint(r, F, v0 , "in_progress") 3: watchdog ← Watchdog(τ, on warn = λ rollback(p, F, v0 , r)) 4: watchdog.start() 5: — user write executes here — 6: if write completes then 7: watchdog.stop() 8: checkpoint(r, F, v0 , "committed") 9: else 10: rollback(p, F, v0 , r) 11: checkpoint(r, F, v0 , "rolled_back") 12: end if
7 8 9
:
fired = self._stop.wait(wait) if not fired: self.on_warn() # trigger rollback
Listing 5. Watchdog thread
The 30-second buffer allows time for the rollback API call (under 200 ms in our experiments), the S3 checkpoint update (under 100 ms), and a conservative margin for network jitter. Rollback. On watchdog trigger or exception, SafeWriter issues a format-specific rollback. Both operations touch only metadata; no Parquet files are moved or deleted: 1 2 3 4 5
# Delta Lake spark.sql(f""" RESTORE TABLE delta.‘{path}‘ TO VERSION AS OF {v0} """)
6
B. Architecture
7 8
SafeWriter wraps any SoAL write in a Python context manager and runs three interlocking steps, described in Algorithm 1. Pre-write checkpoint. Before any write begins, SafeWriter reads the current table version and writes it to S3 as a JSON document: 1
{ "run_id": "job-2026-03-22-001", "format": "delta", "version_before": 5, "status": "in_progress", "saved_at": "2026-03-22T17:44:06Z"
2 3 4 5 6 7
}
Listing 3. Checkpoint document
This document is written via a single S3 PutObject call before the Spark write starts. S3’s per-object atomicity guarantees it either exists fully or not at all, so it survives even if the Lambda process is killed immediately afterward. Reading the current version is format-specific: 1 2 3
# Delta Lake v = DeltaTable.forPath(spark, path) \ .history(1).first()["version"]
4 5 6 7 8 9 10
# Apache Iceberg v = spark.sql( f"SELECT snapshot_id FROM " f"{table}.snapshots ORDER BY " f"committed_at DESC LIMIT 1" ).first()[0]
Listing 4. Version checkpoint per format
Watchdog thread. A daemon thread monitors elapsed time and fires a rollback when the Lambda timeout is approaching: 1 2
class LambdaWatchdog(threading.Thread): WARN_BEFORE_MS = 30_000 # 30 second buffer
3 4 5 6
def run(self): wait = (self.timeout_ms - self.WARN_BEFORE_MS) / 1000
9 10 11 12
# Apache Iceberg spark.sql(f""" CALL system.rollback_to_snapshot( ’{table}’, {snapshot_id} ) """)
Listing 6. Format-specific rollback
Delta’s RESTORE rewrites the current log entry to point back to version v0 . Iceberg’s rollback_to_snapshot moves the snapshot pointer back. Both are atomic operations guaranteed by their respective formats. C. Edge Cases Four edge cases are worth analyzing explicitly. Kill during rollback. If SIGKILL arrives during the rollback itself (a 100–200 ms window), the rollback is interrupted. The checkpoint document remains in "in_progress" status, and the table version is inconsistent with version_before. A retry that checks checkpoint status before writing will detect this and complete the rollback. The probability of this scenario is roughly 200 ms/900,000 ms ≈ 0.02% per invocation. Kill before Phase 1 completes. If the kill arrives before any Parquet data is written, no data is orphaned. The stale checkpoint document signals the next invocation to check table state before writing, but nothing needs to be fixed. Kill before the checkpoint write completes. The checkpoint PutObject takes roughly 10–50 ms. If the kill arrives during this call, the document is not written and SafeWriter provides no protection for this invocation. However, since the checkpoint always precedes the Spark write by design, this only matters if the kill arrives before Phase 1 has started, in which case no data is orphaned. Concurrent writes. If two SoAL jobs write to the same table simultaneously and one is killed, its rollback may conflict with the surviving job’s commit. Delta Lake’s optimistic concurrency will raise a ConcurrentModificationException on the surviving job, which is a visible error rather than silent loss. Concurrent SoAL writes to the same table should be coordinated at the orchestration level regardless.
8
Table X summarizes SafeWriter behavior across all identified edge cases.
table_format="delta", table_path=path, checkpoint_bucket=CHECKPOINT_BUCKET, timeout_ms=TIMEOUT_MS,
11 12 13 14
TABLE X
15
S AFE W RITER EDGE CASE ANALYSIS
16
Scenario
Data lost?
Detectable?
Kill before Phase 1 Kill in Phase 2, no SW Kill in Phase 2, with SW Kill during rollback Kill during checkpoint write Concurrent write conflict
No Yes No Unlikely No No
Yes (no checkpoint) No Yes (checkpoint) Yes (stale checkpoint) No (pre-data kill) Yes (exception)
17 18 19
) with sw: df.write.format("delta") \ .mode("append").save(path) sw.success()
Listing 7. Minimal SafeWriter integration for Delta Lake
If sw.success() is never reached, the context manager’s __exit__ method triggers rollback automatically.
D. Evaluation
B. Configuration
1) Rollback Rate: Table XI gives the full SafeWriter evaluation across all 100 confirmed kill runs.
SafeWriter reads configuration from environment variables to avoid hardcoded timeout values (Table XII).
TABLE XI S AFE W RITER EVALUATION RESULTS
TABLE XII S AFE W RITER CONFIGURATION VARIABLES
Format
Kill Phase
Rollback
Success
Failure
Delta Delta Iceberg Iceberg
data commit data commit
25 (100%) 25 (100%) 25 (100%) 25 (100%)
0 0 0 0
0 0 0 0
2) Checkpoint Overhead: The checkpoint write is the only overhead SafeWriter adds to a successful write. Measured as the mean duration difference between protected and unprotected baseline runs, overhead is under 100 ms for both formats: less than 1.3% of Delta’s 7,779 ms baseline and less than 1.7% of Iceberg’s 5,856 ms baseline. This is negligible for batch ETL workloads. 3) Rollback Duration: Rollback operations completed in 100–200 ms in every tested run, comfortably within the 30second watchdog window. SafeWriter can complete a rollback safely on all tested dataset sizes before Lambda’s kill arrives. 4) Audit Trail: Every SafeWriter run writes a checkpoint document to S3 with a final status of either "committed" or "rolled_back". Monitoring this bucket for "rolled_back" documents gives operations teams a signal of near-timeout events that standard CloudWatch metrics do not surface.
from soal_safe_writer import SafeWriter
3 4 5
CHECKPOINT_BUCKET = "s3a://my-checkpoints/" TIMEOUT_MS = int(os.environ.get( "LAMBDA_TIMEOUT_MS", "900000"))
6 7 8 9 10
def write_delta(spark, df, path, run_id): sw = SafeWriter( spark=spark, run_id=run_id,
900000 30000 (required) true
Lambda timeout in ms Watchdog lead time (ms) S3 checkpoint path Write rollback audit log
A simple monitoring job can poll the checkpoint bucket for "rolled_back" documents and alert or trigger a re-run: 1
import boto3, json
2 3
6 7 8 9 10 11
2
LAMBDA_TIMEOUT_MS SW_WARN_BEFORE_MS SW_CHECKPOINT_BUCKET SW_AUDIT_LOG
D. Operational Monitoring
5
1
Description
SafeWriter needs three S3 permissions on the checkpoint bucket: PutObject, GetObject, and DeleteObject. No additional permissions are required on the table bucket; rollback SQL runs through the existing Spark session using credentials already available to the write job.
VII. S AFE W RITER I NTEGRATION G UIDE Adopting SafeWriter requires three small changes to an existing SoAL write script:
Default
C. IAM Requirements
4
A. Minimal Integration
Variable
12 13
s3 = boto3.client("s3") paginator = s3.get_paginator(" list_objects_v2") for page in paginator.paginate( Bucket="my-checkpoints"): for obj in page.get("Contents", []): doc = json.loads(s3.get_object( Bucket="my-checkpoints", Key=obj["Key"])["Body"].read()) if doc["status"] == "rolled_back": alert(doc["run_id"], doc["rolled_back_at"])
Listing 8. Checkpoint monitoring
Polling for "rolled_back" documents turns what was previously an invisible failure into something on-call engineers can actually see and respond to.
9
VIII. R ELATED W ORK
IX. D ISCUSSION
Open table format correctness. Delta Lake [2] and Iceberg [3] each provide formal atomicity guarantees through different mechanisms: Delta’s optimistic concurrency on a JSON log and Iceberg’s conditional manifest swaps. Each proof holds when the writing process terminates normally or receives a catchable signal. A SIGKILL from an external orchestrator is outside the scope of both, and that is the gap this paper fills. Serverless fault tolerance. Sreekanti et al. [10] built Cloudburst around stateful FaaS execution with cross-function consistency guarantees. Klimovic et al. [11] studied storage access patterns in serverless workloads and found that most functions use local ephemeral storage. Our problem is different from what either paper addresses: the question of what happens to a table write when the function driving it is killed mid-commit. Spark fault tolerance. Spark’s RDD lineage [12] is designed to re-derive lost partitions when an executor crashes, recomputing from an earlier stage. That recovery runs inside the driver process. A SIGKILL arriving from outside the JVM terminates the driver itself, which lineage replay cannot handle. Structured Streaming’s exactly-once delivery has the same blind spot. Checkpointing and 2PC recovery. Chandy and Lamport [13] formalized how to take a consistent snapshot of distributed state mid-execution. SafeWriter borrows the same basic idea: record state before a non-atomic operation and restore to it if interrupted. The implementation is simpler because there is only one process, so a single S3 document is enough to capture the coordinator state that Gray [7] required a write-ahead log to preserve. The checkpoint JSON on S3 serves the same function as that log. Cloud pipeline tooling. AWS Glue job bookmarks track which input files have been processed but do not check whether a table commit completed. Kafka and Kinesis provide exactly-once delivery to a topic, not to a lakehouse table. dbt runs SQL transformations inside database transactions, which assumes a persistent connection that Lambda cannot provide. None of these close the gap for SoAL workloads. LST-Bench. LST-Bench [5] measures throughput and latency for Delta and Iceberg writes under varied cloud workloads. That work is about performance. This paper is about what happens to correctness when the writing process does not finish. Table XIII positions our work relative to the most closely related efforts.
The 100% failure rate in our kill experiments is a consequence of how the harness works: kills are injected inside the gap by construction, so there is no way for an unprotected write to survive one. In production, the relevant probability is P (tkill ∈ G(W )). A job that regularly finishes within 10 seconds of its 900-second limit, with a measured gap of 3.5 seconds, faces roughly 35% silent loss probability per timeout event from Equation 1. The risk goes up, not down, during high-load periods when jobs run slowest and timeouts are most frequent. Standard mitigations miss this. Lambda’s retry policy reinvokes the function but leaves any Phase 1 data from the previous attempt sitting on S3 as orphaned files. Each retry adds another batch. CloudWatch surfaces Task timed out after 900.00 seconds, which looks identical to any other transient error. Row count checks and schema validators run against the table’s committed state, which is unchanged after a gap-phase kill, so they pass. The only way to catch the failure is to compare the post-invocation row count against an expected batch size, and most SoAL pipelines do not track expected counts. The storage cost from repeated retries is easy to calculate. Three retries of a 500k-row job each orphaning 47.9 MB gives roughly 144 MB of unreachable data. At $0.023/GB/month with f = 2 near-timeout events per month:
TABLE XIII C OMPARISON WITH RELATED WORK Work Delta [2] Iceberg [3] Sreekanti [10] Klimovic [11] Zaharia [12] This work
SIGKILL?
Open Formats?
Remedy?
No No No No No Yes
Yes Yes No No No Yes
No No Retry No Lineage SafeWriter
Corphan = f · 3 · s · r ≈ $0.006/month
(4)
The storage cost is trivial. What is not trivial is a pipeline that silently drops a week of records and surfaces the gap as a BI dashboard discrepancy several days later, after the audit trail has gone cold. The gap is not specific to SoAL or to Lambda. Any twophase write over object storage running inside a function with a hard timeout faces the same exposure. Google Cloud Functions and Azure Functions both terminate with SIGKILLequivalent mechanisms. Fargate and Spot containers can be interrupted mid-write without warning. The specific formats and runtimes differ, but the structural problem is the same. Iceberg memory limits. Baseline Iceberg runs at 100k and 500k rows failed in our local environment, most likely from JVM heap exhaustion at the 2 GB driver memory limit. Killinjection runs at those sizes did reach Phase 1 and produced 100% silent data loss across 50 runs, so the vulnerability is present regardless of whether a baseline write can finish. Rerunning with 8–10 GB of executor memory should remove this constraint. Practical guidance. Teams running SoAL with open table formats should wrap all writes with SafeWriter, set LAMBDA_TIMEOUT_MS to match the actual function timeout, and use a dedicated S3 checkpoint bucket with a short lifecycle rule (7 days is enough). Alerting on "rolled_back" checkpoint documents gives a leading indicator of near-timeout pressure that standard CloudWatch metrics do not provide. Before retrying any timed-out job, verify that the table version matches version_before in the checkpoint to confirm the rollback completed cleanly.
10
X. C ONCLUSION Across 860 kill-injection experiments on Delta Lake and Apache Iceberg at three dataset sizes, a SIGKILL landing between Phase 1 and Phase 2 of a write produced silent data loss every single time. No exception surfaced. Standard monitoring saw nothing unusual. This is not a quirk of a specific format version or a misconfigured environment; it follows directly from running a two-phase commit protocol on a runtime that can vanish at any moment without warning. The thing that makes this worth studying is how undetectable it is in practice. To CloudWatch, a Lambda timeout looks like any other non-zero exit. The table’s committed state is untouched, so data quality checks pass. Orphaned files pile up on S3 with no reader ever touching them. A team could go months before a downstream report shows a gap, at which point the audit trail is cold and the missing writes are unrecoverable. SafeWriter sidesteps the problem by acting before the kill arrives rather than reacting to it. A watchdog thread fires 30 seconds before the timeout, triggers a format-native rollback via SQL, and writes a checkpoint document to S3 recording the outcome. Every tested kill scenario ended with a clean rollback and under 100 ms added to normal write paths. Delta Lake and Iceberg are correct systems. The problem is that they were designed for processes that receive catchable signals before termination, and SoAL is not that kind of process. The gap is a consequence of composing two things that were each built correctly but were not built for each other. SafeWriter is one way to close it; we hope this paper makes the problem visible enough that format-level solutions get considered too. ACKNOWLEDGMENT The author used AI writing assistance to help refine the prose in this paper. All experimental design, implementation, data collection, analysis, and conclusions are the author’s own work. R EFERENCES [1] AWS Samples, “Spark on aws lambda,” https://github.com/aws-samples/ spark-on-aws-lambda, 2022, open-source project for running Apache Spark inside AWS Lambda containers. [2] M. Armbrust, T. Das, L. Sun, B. Yavuz, S. Zhu, M. Murthy, J. Torres, H. van Hovell, A. Ionescu, A. Łukacs et al., “Delta lake: Highperformance acid table storage over cloud object stores,” in Proceedings of the VLDB Endowment, vol. 13, no. 12. VLDB Endowment, 2020, pp. 3411–3424. [3] R. Kinley and D. Blue, “Apache iceberg: An open table format for huge analytic datasets,” in Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. ACM, 2020, pp. 2751–2753. [4] A. Sivachenko, S. Samineni et al., “Apache hudi: The data lake platform,” in Proceedings of the VLDB Endowment, vol. 14, no. 12. VLDB Endowment, 2021. [5] J. Camacho-Rodrı́guez, A. Agrawal, A. Gruenheid, A. Gosalia, C. Petculescu, J. Aguilar-Saborit, A. Floratou, C. Curino, and R. Ramakrishnan, “LST-Bench: Benchmarking log-structured tables in the cloud,” Proceedings of the ACM on Management of Data, vol. 2, no. 1, 2024. [6] Amazon Web Services, “Amazon s3 strong consistency,” https://aws.amazon.com/blogs/aws/ amazon-s3-update-strong-read-after-write-consistency/, 2020, aWS announcement of strong read-after-write consistency for S3, December 2020.
[7] J. Gray, Notes on Data Base Operating Systems. Springer, 1978. [8] Amazon Web Services, “AWS Lambda FAQs,” https://aws.amazon.com/ lambda/faqs/, 2023, accessed 2026. [9] ——, “Spatial data: Accommodations dataset,” https://docs.aws.amazon. com/redshift/latest/dg/spatial-tutorial.html, 2022, used as benchmark dataset in AWS Redshift spatial tutorial. [10] V. Sreekanti, C. Wu, X. C. Lin, J. Schleier-Smith, J. M. Gonzalez, J. M. Hellerstein, and A. Tumanov, “Cloudburst: Stateful functions-asa-service,” in Proceedings of the VLDB Endowment, vol. 13, no. 12. VLDB Endowment, 2020, pp. 2438–2452. [11] A. Klimovic, Y. Wang, C. Kozyrakis, P. Stuedi, J. Pfefferle, and A. Trivedi, “Understanding ephemeral storage for serverless analytics,” in Proceedings of the 2018 USENIX Annual Technical Conference (ATC). USENIX, 2018, pp. 789–794. [12] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica, “Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing,” in Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 2012, pp. 15–28. [13] K. M. Chandy and L. Lamport, “Distributed snapshots: Determining global states of distributed systems,” ACM Transactions on Computer Systems, vol. 3, no. 1, pp. 63–75, 1985.