Luca Soldaini soldni

🤗Huggingface Model Downloader

Note

(2025-01-08) Add feature for 🏷️Tag(Revision) Selection, contributed by @Bamboo-D.
(2024-12-17) Add feature for ⚡Quick Startup and ⏭️Fast Resume, enabling skipping of downloaded files, while removing the git clone dependency to accelerate file list retrieval.

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, This command-line tool leverages curl and aria2c for fast and robust downloading of models and datasets.

Features

⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.

	#
	# Run like this:
	# beaker experiment logs <experiment id> \| python logs.py
	#

	import sys
	import re
	from collections import defaultdict, Counter
	from typing import List, Dict

	import asyncio
	import os
	import random
	from collections import deque
	import ssl
	from urllib.parse import urlparse

	import aiohttp
	import polars as pl
	import aiofiles

	from typing import *
	import time
	import logging

	def logging_tqdm(
	i,
	*,
	logger: Optional[logging.Logger] = None,
	desc: str = "Working",
	total: Optional[int] = None,

	import json
	from twitter import *

	# Twitter app credentials
	token = "..."
	token_secret = "..."
	consumer_key = "..."
	consumer_secret = "..."

	# Twitter client

	#!/bin/sh


	# this script does absolutely ZERO error checking. however, it worked
	# for me on a RHEL 6.3 machine on 2012-08-08. clearly, the version numbers
	# and/or URLs should be made variables. cheers, [email protected]


	mkdir mosh
	cd mosh

	default['sshd']['sshd_config']['AuthenticationMethods'] = 'publickey,keyboard-interactive:pam'
	default['sshd']['sshd_config']['ChallengeResponseAuthentication'] = 'yes'
	default['sshd']['sshd_config']['PasswordAuthentication'] = 'no'