Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache for the Swift AppFS #4178

Merged
merged 1 commit into from
Oct 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ require (
github.com/gorilla/websocket v1.5.0
github.com/h2non/filetype v1.1.3
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/howeyc/gopass v0.0.0-20210920133722-c8aef6fb66ef
github.com/justincampbell/bigduration v0.0.0-20160531141349-e45bf03c0666
github.com/labstack/echo/v4 v4.11.2
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,8 @@ github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+l
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/hcl v0.0.0-20170914154624-68e816d1c783/go.mod h1:oZtUIOe8dh44I2q6ScRibXws4Ajl+d+nod3AaR9vL5w=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
Expand Down
39 changes: 37 additions & 2 deletions pkg/appfs/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,13 @@ import (
"path"
"strconv"
"strings"
"sync"
"time"

"github.com/andybalholm/brotli"
"github.com/cozy/cozy-stack/pkg/consts"
web_utils "github.com/cozy/cozy-stack/pkg/utils"
lru "github.com/hashicorp/golang-lru/v2"
"github.com/labstack/echo/v4"
"github.com/ncw/swift/v2"
"github.com/spf13/afero"
Expand Down Expand Up @@ -99,19 +101,52 @@ func (g gzipReadCloser) Close() error {
return nil
}

type cacheEntry struct {
content []byte
headers swift.Headers
}

var cache *lru.Cache[string, cacheEntry]
var initCacheOnce sync.Once

// NewSwiftFileServer returns provides the apps.FileServer implementation
// using the swift backend as file server.
func NewSwiftFileServer(conn *swift.Connection, appsType consts.AppType) FileServer {
initCacheOnce.Do(func() {
c, err := lru.New[string, cacheEntry](128)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why such a small cache? Don't we have more konnectors and applications already?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it can take a lot of RAM. I prefer to start small and see later to increase.

if err != nil {
panic(err)
}
cache = c
})
return &swiftServer{
c: conn,
container: containerName(appsType),
ctx: context.Background(),
}
}

func (s *swiftServer) openWithCache(objName string) (io.ReadCloser, swift.Headers, error) {
entry, ok := cache.Get(objName)
if !ok {
f, h, err := s.c.ObjectOpen(s.ctx, s.container, objName, false, nil)
if err != nil {
return f, h, err
}
entry.headers = h
entry.content, err = io.ReadAll(f)
if err != nil {
return nil, h, err
}
cache.Add(objName, entry)
}
f := io.NopCloser(bytes.NewReader(entry.content))
return f, entry.headers, nil
}

func (s *swiftServer) Open(slug, version, shasum, file string) (io.ReadCloser, error) {
objName := s.makeObjectName(slug, version, shasum, file)
f, h, err := s.c.ObjectOpen(s.ctx, s.container, objName, false, nil)
f, h, err := s.openWithCache(objName)
if err != nil {
return nil, wrapSwiftErr(err)
}
Expand All @@ -127,7 +162,7 @@ func (s *swiftServer) Open(slug, version, shasum, file string) (io.ReadCloser, e

func (s *swiftServer) ServeFileContent(w http.ResponseWriter, req *http.Request, slug, version, shasum, file string) error {
objName := s.makeObjectName(slug, version, shasum, file)
f, h, err := s.c.ObjectOpen(s.ctx, s.container, objName, false, nil)
f, h, err := s.openWithCache(objName)
if err != nil {
return wrapSwiftErr(err)
}
Expand Down
Loading