wsteam - Omar Hopman

The problem

Steam downloads content as chunks - individually addressable pieces of a depot, each with its own offset, CRC, and encryption key. Steam's own client downloads them sequentially on one connection, which on a fast connection is not the bottleneck at all. Decomposing a manifest into chunk jobs and running them in parallel is the difference between 2-3 MB/s and roughly 70 MB/s on the same line. This project is mostly that, plus Workshop support.

Why these choices

C# on .NET 10. The point was to learn the language properly rather than use it at work, so everything is hand-written: the Adler32, the chunk scheduler, the manifest source adapters. SteamKit2 handles auth and PICS only.
RandomAccess over a stream. Chunks arrive out of order and land at arbitrary byte offsets. FileStream requires a seek-and-hold pattern; RandomAccess takes the offset per call, which is exactly the shape of the problem.
ArrayPool for every buffer. At parallel degree 32, allocating a fresh array per chunk buries the GC. Renting and returning keeps the collection count flat.
Parallel.ForEachAsync with a clamped degree. More threads than CDN connections just produces throttled retries. Default is cores + 4, clamped between 4 and 32, overridable by env var.
Interface-based manifest sources. IManifestApi with three implementations behind it, so switching provider is a flag, not a code change.

Writing chunks at their offset

This is the piece that makes parallelism safe. Every chunk declares its own offset, so writes never need to be ordered and never need to hold a lock on a shared position. Two workers writing different chunks of the same file is not a race.

csharpCore/Downloads/ChunkedFileWriter.cs

file = File.OpenHandle(
    fileDirectory,
    FileMode.OpenOrCreate,
    FileAccess.ReadWrite,
    FileShare.Read,
    FileOptions.Asynchronous | FileOptions.RandomAccess
);

public async Task WriteChunkAsync(DepotManifest.ChunkData chunk, Memory<byte> data)
{
    try
    {
        var offset = (long)checked(chunk.Offset);
        await RandomAccess.WriteAsync(file, data, offset);
    }
    catch (Exception ex)
    {
        throw new Exception($"Failed to write chunk {chunk.Offset} of {fileName} ({ex.Message})");
    }
}

checked on the cast because a malformed manifest offset would otherwise wrap silently and write to the wrong part of the file. The exception message includes the offset, which is the only thing you need to identify a bad chunk.

The downloader is 30 lines of scheduling

Flatten every depot's manifest into a list of jobs, then run them with a bounded degree of parallelism. All the complexity is in DownloadChunkAsync underneath: verify, decrypt, retry with a different CDN server.

csharpCore/Downloads/DownloadManager.cs

var dop = GetChunkParallelism();
AnsiConsole.MarkupLine($"[grey]downloader[/] chunk parallelism={dop} (override via {ChunkParallelismEnv})");

await Parallel.ForEachAsync(
    chunkJobs,
    new ParallelOptions { MaxDegreeOfParallelism = dop },
    async (job, ct) =>
    {
        await DownloadChunkAsync(
            job.chunk, manifestInfo.DepotId, manifestInfo.DepotKey,
            job.file, cdnClient, cdnServers,
            depotProgress, overallProgress);
    });

The finally block around this disposes every open file handle. SafeFileHandle would do it on GC, but with hundreds of open depots that's a real handle leak for the length of a large download.

Resumability without a database

A chunk that's already on disk gets verified by Adler32 instead of re-downloaded. That means an interrupted download resumes by re-running the same command - no checkpoint file, no partial-state tracking, because the disk is the checkpoint.

Thread count is the other half of it. The default scales with the machine, but an env var override exists because the right number depends on the CDN, not the CPU.

csharpCore/Downloads/DownloadManager.cs

private static int GetChunkParallelism()
{
    var env = Environment.GetEnvironmentVariable(ChunkParallelismEnv);
    if (int.TryParse(env, out var parsed) && parsed > 0)
        return Math.Clamp(parsed, 1, 256);

    int dynamicDefault = Environment.ProcessorCount + 4;
    int minThreads = Math.Max(4, Environment.ProcessorCount);
    return Math.Clamp(dynamicDefault, minThreads, 32);
}

Two different clamps deliberately: the env var range is wide because the user asked for it, the automatic range is conservative because it was guessed.

Rotating away from bad CDN servers

Some CDN servers throttle or fail repeatedly. A counter per server takes it out of rotation after two failures, so one slow node can't eat the whole download's parallelism.

csharpCore/Downloads/DownloadManager.cs

private const int MaxServerFailures = 2;
private readonly ConcurrentDictionary<Server, int> serverFailures = new();

private bool IsServerFailed(Server server)
    => serverFailures.TryGetValue(server, out var failures) && failures >= MaxServerFailures;

A ConcurrentDictionary rather than a locked list because this is read on the hot path by every one of the 32 workers, on every chunk.

Current limitations

It depends on third-party manifest APIs. Steam's own CDN doesn't serve manifests without a depot key. This only works with a source you're authorised to use, which makes it an educational project rather than something to hand a friend.
Manifests aren't cached. Re-running a fetch re-queries the source. It's on the roadmap and it shows.
Plugin sources aren't done. The interface is there and two implementations ship; "bring your own source" is still a roadmap item.
Retry is recursion. DownloadChunkAsync calls itself with an incremented counter. It works and it's the wrong shape - it puts the backoff policy in the stack depth.