g2cache

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 22, 2023 License: Apache-2.0 Imports: 15 Imported by: 2

README

g2cache

g2cache

介绍

分布式多级缓存方案g2cache

软件架构

软件架构说明
主要内容：

模块 功能 特点 注意
local interface 本机内存高速缓存 纳秒，毫秒级别响应速度；有效期较短 实现LocalCache接口，注意控制内存
out interface 外部高速缓存，如Redis 较快的访问速度，容量很足 实现OutCache接口
LoadDataSourceFunc 外部数据源 外部加载函数，灵活，并发保护 需要自行处理panic
PubSub 外部缓存发布订阅 外部缓存可以自选实现 未实现就不支持分布式

local 本地内存高速缓存接口

// Local memory cache，Local memory cache with high access speed
type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // local storage should set Entry.Obsolete
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

out 外部缓存接口，值得一提的是外部缓存需要实现发布订阅接口，这是因为g2cache支持分布式多级缓存

// External cache has faster access speed, such as Redis
type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // out storage should set Entry.Expiration
	Subscribe(data chan *ChannelMeta) error
	Publish(gid ,key string, action int8, data *Entry) error
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

外部缓存发布订阅

// only out storage pub sub
type PubSub interface {
	Subscribe(data chan *ChannelMeta) error
	Publish(gid, key string, action int8, data *Entry) error
}

LoadDataSourceFunc 原始数据加载函数，请自行处理panic并以error形式返回
加载函数支持返回string，map，slice，struct，ptr类型

// Shouldn't throw a panic, please return an error
type LoadDataSourceFunc func() (interface{}, error)

缓存效果-初始

[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:222               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:183                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:83                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:18                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:103               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:90                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:170               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:101               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:165               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [local] hit percentage [[5.8824]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [out] hit percentage [[1.9608]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [data source] hit percentage [[90.1961]]

缓存效果-N分钟后

[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:106               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:136               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:100               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:219               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:13                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:200               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:172               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [local] hit percentage [[45.3865]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [out] hit percentage [[6.7332]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [data source] hit percentage [[48.1297]]

缓存效果-N+M分钟后

[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [local] hit percentage [[82.3877]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [out] hit percentage [[16.2689]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [data source] hit percentage [[1.3641]]
Usage

go get -u gitee.com/kelvins-io/[email protected]

模拟测试

修改example/main.go中Redis的配置
sh example-run.sh

分支说明
  1. copyobj分支支持返回特定object，为不稳定版本
  2. dev分支Get返回object序列化值，过渡版
  3. master分支只提供sync.Map的local实现，早期版
  4. release分支提供发布版，与copyobj有较大变化
参与贡献
  1. Fork 本仓库
  2. 新建 Feat_xxx 分支
  3. 提交代码
  4. 新建 Pull Request
合作交流

QQ群：
g2cache

邮箱： [email protected]

Documentation

Index

Constants

View Source 
const (
	SetPublishType int8 = iota
	DelPublishType
)

Entry expire is UnixNano

Variables

View Source 
var (
	CacheDebug                  bool
	CacheMonitor                bool
	OutCachePubSub              bool
	CacheMonitorSecond          = 5
	DefaultGPoolWorkerNum       = 200
	DefaultGPoolJobQueueChanLen = 1000
)
View Source 
var (
	CacheKeyEmpty           = errors.New("cache key is empty")
	CacheObjNil             = errors.New("cache object is nil")
	LoadDataSourceFuncNil   = errors.New("cache load func is nil")
	LocalStorageClose       = errors.New("local storage close !!! ")
	OutStorageClose         = errors.New("out storage close !!! ")
	CacheClose              = errors.New("g2cache close !!! ")
	DataSourceLoadNil       = errors.New("data source load nil")
	OutStorageLoadNil       = errors.New("out storage load nil")
	CacheNotImplementPubSub = errors.New("cache not implement pubsub interface")
)
View Source 
var (
	DefaultFreeCacheSize = 50 * 1024 * 1024 // 50MB
)
View Source 
var DefaultPubSubRedisChannel = "g2cache-pubsub-channel"
View Source 
var (
	EntryLazyFactor = 32
)

Functions

func GenKey 

func GenKey(args ...interface{}) string

func GetRedisPool 

func GetRedisPool(conf *RedisConf) (*redis.Pool, error)

func LogDebug 

func LogDebug(s ...interface{})

func LogDebugF 

func LogDebugF(f string, s ...interface{})

func LogErr 

func LogErr(s ...interface{})

func LogErrF 

func LogErrF(f string, s ...interface{})

func LogInfo 

func LogInfo(s ...interface{})

func LogInfoF 

func LogInfoF(f string, s ...interface{})

func NewUUID 

func NewUUID() (string, error)

func RedisDelKey 

func RedisDelKey(key string, pool *redis.Pool) error

func RedisGetString 

func RedisGetString(key string, pool *redis.Pool) (string, error)

func RedisPublish 

func RedisPublish(channel, message string, pool *redis.Pool) error

func RedisSetString 

func RedisSetString(key, value string, ttl int, pool *redis.Pool) error

Types

type ChannelMeta 

type ChannelMeta struct {
	Key    string `json:"key"`    // cache key
	Gid    string `json:"gid"`    // Used to identify working groups
	Action int8   `json:"action"` // SetPublishType,DelPublishType
	Data   *Entry `json:"data"`
}

type Entry 

type Entry struct {
	Value      interface{} `json:"value"`
	TtlSecond  int         `json:"ttl"`
	Obsolete   int64       `json:"obsolete"`
	Expiration int64       `json:"expiration"`
}

func NewEntry 

func NewEntry(v interface{}, second int) *Entry

func (*Entry) Expired 

func (e *Entry) Expired() bool

Expired means that the data is unavailable and data needs to be synchronized

func (*Entry) GetExpireTTL 

func (e *Entry) GetExpireTTL() (second int64)

func (*Entry) GetObsoleteTTL 

func (e *Entry) GetObsoleteTTL() (second int64)

func (*Entry) Obsoleted 

func (e *Entry) Obsoleted() bool

Outdated data means that the data is still available, but not up-to-date

func (*Entry) String 

func (e *Entry) String() string

type FreeCache 

type FreeCache struct {
	// contains filtered or unexported fields
}

func NewFreeCache 

func NewFreeCache() *FreeCache

func (*FreeCache) Close 

func (c *FreeCache) Close()

func (*FreeCache) Del 

func (c *FreeCache) Del(key string) error

func (*FreeCache) Get 

func (c *FreeCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*FreeCache) Set 

func (c *FreeCache) Set(key string, e *Entry) error

func (*FreeCache) ThreadSafe 

func (c *FreeCache) ThreadSafe()

type G2Cache 

type G2Cache struct {
	GID string // Identifies the number of an instance
	// contains filtered or unexported fields
}

func New 

func New(out OutCache, local LocalCache) (g *G2Cache, err error)

func (*G2Cache) Close 

func (g *G2Cache) Close()

func (*G2Cache) Del 

func (g *G2Cache) Del(key string, wait bool) (err error)

func (*G2Cache) Get 

func (g *G2Cache) Get(key string, ttlSecond int, obj interface{}, fn LoadDataSourceFunc) error

Get return err DataSourceLoadNil if fn exec return nil

func (*G2Cache) Set 

func (g *G2Cache) Set(key string, obj interface{}, ttlSecond int, wait bool) (err error)

type Harsher 

type Harsher interface {
	Sum64(string) uint64
}

type HitStatistics 

type HitStatistics struct {
	HitOutStorageTotalRate   float64 `json:"hit_out_storage_total_rate"`
	HitDataSourceTotalRate   float64 `json:"hit_data_source_total_rate"`
	HitLocalStorageTotalRate float64 `json:"hit_local_storage_total_rate"`
	HitDataSourceTotal       int64   `json:"hit_data_source_total"`
	HitLocalStorageTotal     int64   `json:"hit_local_storage_total"`
	HitOutStorageTotal       int64   `json:"hit_out_storage_total"`
	AccessGetTotal           int64   `json:"access_get_total"`
}
var HitStatisticsOut HitStatistics

func (*HitStatistics) Calculation 

func (h *HitStatistics) Calculation()

func (*HitStatistics) StatisticsDataSource 

func (h *HitStatistics) StatisticsDataSource()

func (*HitStatistics) StatisticsLocalStorage 

func (h *HitStatistics) StatisticsLocalStorage()

func (*HitStatistics) StatisticsOutStorage 

func (h *HitStatistics) StatisticsOutStorage()

func (*HitStatistics) String 

func (h *HitStatistics) String() string

type Job 

type Job func()

Represents user request, function which should be executed in some worker.

type LoadDataSourceFunc 

type LoadDataSourceFunc func() (interface{}, error)

Shouldn't throw a panic, please return an error

type LocalCache 

type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // local storage should set Entry.Obsolete
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

Local memory cache，Local memory cache with high access speed

type LoggerInterface 

type LoggerInterface interface {
	LogInfoF(f string, s ...interface{})
	LogInfo(s ...interface{})
	LogDebug(s ...interface{})
	LogDebugF(f string, s ...interface{})
	LogErr(s ...interface{})
	LogErrF(f string, s ...interface{})
}

外部调用者可实现此日志接口用于将日志导出 

var (
	Logger LoggerInterface = &sysLogger{}
)

type OutCache 

type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // out storage should set Entry.Expiration
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

External cache has faster access speed, such as Redis

type Pool 

type Pool struct {
	// contains filtered or unexported fields
}

func NewPool 

func NewPool(numWorkers int, jobQueueLen int) *Pool

Will make pool of gorouting workers. numWorkers - how many workers will be created for this pool queueLen - how many jobs can we accept until we block

Returned object contains JobQueue reference, which you can use to send job to pool.

func (*Pool) Release 

func (p *Pool) Release()

Will release resources used by pool

func (*Pool) SendJob 

func (p *Pool) SendJob(job func())

func (*Pool) SendJobWithDeadline 

func (p *Pool) SendJobWithDeadline(job func(), t time.Time) bool

func (*Pool) SendJobWithTimeout 

func (p *Pool) SendJobWithTimeout(job func(), t time.Duration) bool

type PubSub 

type PubSub interface {
	Subscribe(data chan<- *ChannelMeta) error
	Publish(gid, key string, action int8, data *Entry) error
}

only out storage pub sub

type RedisCache 

type RedisCache struct {
	// contains filtered or unexported fields
}

func NewRedisCache 

func NewRedisCache() (*RedisCache, error)

func (*RedisCache) Close 

func (r *RedisCache) Close()

func (*RedisCache) Del 

func (r *RedisCache) Del(key string) error

func (*RedisCache) DistributedEnable 

func (r *RedisCache) DistributedEnable() bool

func (*RedisCache) Get 

func (r *RedisCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*RedisCache) Publish 

func (r *RedisCache) Publish(gid, key string, action int8, value *Entry) error

func (*RedisCache) Set 

func (r *RedisCache) Set(key string, obj *Entry) error

func (*RedisCache) Subscribe 

func (r *RedisCache) Subscribe(ch chan<- *ChannelMeta) error

func (*RedisCache) ThreadSafe 

func (r *RedisCache) ThreadSafe()

type RedisConf 

type RedisConf struct {
	DSN     string
	Pwd     string
	DB      int
	MaxConn int
}
var DefaultPubSubRedisConf RedisConf
var DefaultRedisConf RedisConf

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.