got

package module
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 25, 2020 License: MIT Imports: 25 Imported by: 0

README

Got.

Simple and fast concurrent downloader.

InstallationCLI UsageModule UsageLicense

Tests

Comparison

Comparison in cloud server:


[root@centos-nyc-12 ~]# time got -o /tmp/test -c 20 http://www.ovh.net/files/1Gio.dat
URL: http://www.ovh.net/files/1Gio.dat done!

real    0m8.832s
user    0m0.203s
sys 0m3.176s


[root@centos-nyc-12 ~]# time curl http://www.ovh.net/files/1Gio.dat --output /tmp/test1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
								 Dload  Upload   Total   Spent    Left  Speed
100 1024M  100 1024M    0     0  35.6M      0  0:00:28  0:00:28 --:--:-- 34.4M

real    0m28.781s
user    0m0.379s
sys 0m1.970s

Installation

Download and install the latest release:
# go to tmp dir.
cd /tmp

# Download latest version.
curl -sfL https://git.io/getgot | sh

# Make the binary executable.
chmod +x /tmp/bin/got

# Move the binary to your PATH
sudo mv /tmp/bin/got /usr/bin/got
Or Go ahead compile it yourself:
go get github.com/melbahja/got/cmd/got
Or from the AUR

Install got for the latest release version or got-git for the latest development version.

Note: these packages are not maintained by melbahja

Command Line Tool Usage

Simple usage:
got https://example.com/file.mp4
You can specify destination path:
got -o /path/to/save https://example.com/file.mp4
You can download multiple URLs and save them to directory:
got --dir /path/to/dir https://example.com/file.mp4 https://example.com/file2.mp4
You can download multiple URLs from a file:
got --dir /path/to/dir -f urls.txt

You can pipe multiple URLs:

cat urls.txt | got --dir /path/to/dir
Docs for available flags:
got help

Module Usage

You can use Got to download large files in your go code, the usage is simple as the CLI tool:

package main

import "github.com/melbahja/got"

func main() {

	g := got.New()

	err := g.Download("http://localhost/file.ext", "/path/to/save")

	if err != nil {
		// ..
	}
}

For more see PkgDocs.

How It Works?

Got takes advantage of the HTTP range requests support in servers RFC 7233, if the server supports partial content Got split the file into chunks, then starts downloading and merging the chunks into the destinaton file concurrently.

License

Got is provided under the MIT License © Mohammed El Bahja.

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source
var ChunkPool = &sync.Pool{
	New: func() interface{} {
		return new(Chunk)
	},
}

ChunkPool helps in multi *Download files.

View Source
var DefaultClient = &http.Client{
	Transport: &http.Transport{
		MaxIdleConns:        10,
		IdleConnTimeout:     30 * time.Second,
		TLSHandshakeTimeout: 5 * time.Second,
		Proxy:               http.ProxyFromEnvironment,
	},
}

DefaultClient is the default http client for got requests.

View Source
var DefaultFileName = "got.output"

DefaultFileName is the fallback name for GetFilename.

View Source
var ErrDownloadAborted = errors.New("Operation aborted")

ErrDownloadAborted - When download is aborted by the OS before it is completed, ErrDownloadAborted will be triggered

View Source
var HeaderSlice []GotHeader
View Source
var UserAgent = "Got/1.0"

UserAgent is the default Got user agent to send http requests.

Functions

func CliDownload

func CliDownload(args []string)

func GetFilename

func GetFilename(URL string) string

GetFilename it returns default file name from a URL.

func NewRequest

func NewRequest(ctx context.Context, method, URL string, header []GotHeader) (req *http.Request, err error)

NewRequest returns a new http.Request and error if any.

Types

type Chunk

type Chunk struct {

	// Chunk start pos.
	Start uint64

	// Chunk end pos.
	End uint64

	// Path name where this chunk downloaded.
	Path string

	// Done to check is this chunk downloaded.
	Done chan struct{}
}

Chunk is a partial content range.

type Download

type Download struct {
	Client *http.Client

	Concurrency uint

	URL, Dir, Dest string

	Interval, ChunkSize, MinChunkSize, MaxChunkSize uint64

	Header []GotHeader

	StopProgress bool
	// contains filtered or unexported fields
}

Download holds downloadable file config and infos.

Example
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

dl := got.NewDownload(ctx, testUrl, destPath)

// Init
if err := dl.Init(); err != nil {
	fmt.Println(err)
}

// Start download
if err := dl.Start(); err != nil {
	fmt.Println(err)
}

fmt.Println("Done")
Output:

Done

func NewDownload

func NewDownload(ctx context.Context, URL, dest string) *Download

NewDownload returns new *Download with context.

func (Download) AvgSpeed

func (d Download) AvgSpeed() uint64

AvgSpeed returns average download speed.

func (Download) Context

func (d Download) Context() context.Context

Context returns download context.

func (*Download) DownloadChunk

func (d *Download) DownloadChunk(c Chunk, dest *os.File) error

DownloadChunk downloads a file chunk.

func (Download) GetInfo

func (d Download) GetInfo() (*Info, error)

GetInfo returns URL info, and error if any.

func (*Download) Init

func (d *Download) Init() (err error)

Init set defaults and split file into chunks and gets Info, you should call Init before Start

func (Download) IsRangeable

func (d Download) IsRangeable() bool

IsRangeable returns file server partial content support state.

func (*Download) Name

func (d *Download) Name() string

Name returns the downloaded file path.

func (*Download) RunProgress

func (d *Download) RunProgress(fn ProgressFunc)

RunProgress runs ProgressFunc based on Interval and updates lastSize.

func (Download) Size

func (d Download) Size() uint64

Size returns downloaded size.

func (Download) Speed

func (d Download) Speed() uint64

Speed returns download speed.

func (*Download) Start

func (d *Download) Start() (err error)

Start downloads the file chunks, and merges them.

func (Download) TotalCost

func (d Download) TotalCost() time.Duration

TotalCost returns download duration.

func (Download) TotalSize

func (d Download) TotalSize() uint64

TotalSize returns file total size (0 if unknown).

func (*Download) Write

func (d *Download) Write(b []byte) (int, error)

Write updates progress size.

type Got

type Got struct {
	ProgressFunc

	Client *http.Client
	// contains filtered or unexported fields
}

Got holds got download config.

Example
// Just for testing
destPath := createTemp()
defer clean(destPath)

g := got.New()

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")
Output:

done
Example (WithContext)
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

g := got.NewWithContext(ctx)

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")
Output:

done

func New

func New() *Got

New returns new *Got with default context and client.

func NewWithContext

func NewWithContext(ctx context.Context) *Got

NewWithContext wants Context and returns *Got with default http client.

func (Got) Do

func (g Got) Do(dl *Download) error

Do inits and runs ProgressFunc if set and starts the Download.

func (Got) Download

func (g Got) Download(URL, dest string) error

Download creates *Download item and runs it.

type GotHeader

type GotHeader struct {
	Key   string
	Value string
}

type Info

type Info struct {
	Size      uint64
	Name      string
	Rangeable bool
}

Info holds downloadable file info.

type ProgressFunc

type ProgressFunc func(d *Download)

ProgressFunc to show progress state, called by RunProgress based on interval.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL