Skip to content

Instantly share code, notes, and snippets.

@VirtualPirate
Last active January 7, 2025 14:25
Show Gist options
  • Save VirtualPirate/21a776cd9366c82fdab333dfe3d32486 to your computer and use it in GitHub Desktop.
Save VirtualPirate/21a776cd9366c82fdab333dfe3d32486 to your computer and use it in GitHub Desktop.
This is a script that downloads a file using multi-segmentation. Making multiple connections to the url and downloading the file in segments and then merges all the downloaded data after download is completed
import axios from "axios";
import * as fs from "node:fs";
import * as path from "node:path";
interface DownloadSegment {
start: number;
end: number;
index: number;
}
async function getFileSize(url: string): Promise<number> {
const response = await axios.head(url);
return parseInt(response.headers["content-length"]);
}
async function downloadSegment(
url: string,
segment: DownloadSegment,
outputDir: string
): Promise<string> {
const { start, end, index } = segment;
console.log(`Segment Download started start: ${start} end: ${end}`);
const outputPath = path.join(outputDir, `part_${index}`);
const writer = fs.createWriteStream(outputPath);
return new Promise((resolve, reject) => {
axios({
method: "get",
url: url,
headers: { Range: `bytes=${start}-${end}` },
responseType: "stream",
})
.then((response) => {
response.data.pipe(writer);
let error: any = null;
writer.on("error", (err) => {
error = err;
writer.close();
reject(err);
});
writer.on("close", () => {
if (!error) {
console.log(`Downloaded segment ${index} (bytes ${start}-${end})`);
resolve(outputPath);
}
});
})
.catch((err) => {
reject(err);
});
});
}
async function downloadFile(
url: string,
segments: number,
outputFilePath: string
) {
const fileSize = await getFileSize(url);
const segmentSize = Math.ceil(fileSize / segments);
const outputDir = path.dirname(outputFilePath);
const segmentPaths: string[] = [];
const segmentPromises = Array.from({ length: segments }, (_, index) => {
const start = index * segmentSize;
const end = index === segments - 1 ? fileSize - 1 : start + segmentSize - 1;
const segment: DownloadSegment = { start, end, index };
return downloadSegment(url, segment, outputDir).then((path) =>
segmentPaths.push(path)
);
});
await Promise.all(segmentPromises);
// Merge all parts
const writeStream = fs.createWriteStream(outputFilePath);
for (const segmentPath of segmentPaths.sort()) {
const data = fs.readFileSync(segmentPath);
writeStream.write(data);
fs.unlinkSync(segmentPath); // Delete segment after merging
}
writeStream.close();
console.log(`Download complete: ${outputFilePath}`);
}
// Example usage
const url =
"https://rr4---sn-qxaeenl6.googlevideo.com/videoplayback?expire=1731000837&ei=pKUsZ_OXNpKJ9fwP3NO2kAc&ip=34.96.33.87&id=o-AKkWUuNwaTJIgO01fTpJnQs7F2_458w2KqLEX4xhDQjF&itag=137&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&met=1730979237%2C&mh=DQ&mm=31%2C29&mn=sn-qxaeenl6%2Csn-qxaelned&ms=au%2Crdu&mv=u&mvi=4&pl=27&rms=au%2Cau&vprv=1&svpuc=1&mime=video%2Fmp4&rqh=1&gir=yes&clen=101803581&dur=635.720&lmt=1726430152436097&mt=1730979090&fvip=2&keepalive=yes&fexp=51312688%2C51326932&c=IOS&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRAIgWLZzJ20EWzch7xywvZuuRRk5sxubZli9CNkRIRMx6QICIENKy_k_PjSWKYVF0tV7eSyzbEIlebNcA4qGkMqepbKN&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Crms&lsig=ACJ0pHgwRAIgC3OHLLVbbxFtUgic47v2ENYwkJYbSt0lrNWpIveaHOQCIF4wLMfSZmaCmYPrytRux6dBgRpCo_Cqdq3riZNNl-am";
const outputFilePath = path.resolve("./largefile.mp4");
downloadFile(url, 10, outputFilePath)
.then(() => console.log("File downloaded successfully"))
.catch((error) => console.error("Download failed", error));
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment