-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: cmd/go: support embedding static assets (files) in binaries #35950
Comments
It's worth considering whether |
Some people would need the ability to serve the embedded assets with HTTP using the I personally use either mjibson/esc (which does that) or in some cases my own file embedding implementation which renames files to create unique paths and adds a map from the original paths to the new ones, e.g. |
I think a consequence of this would be that it would be nontrivial to figure out what files are necessary to build a program. The (Just musing out loud here.) |
@opennota,
Yes, the first link above is a package I wrote (in 2011, before Go 1) and still use, and it supports using http.FileServer: https://godoc.org/perkeep.org/pkg/fileembed#Files.Open |
Yes, good point. That's a very strong argument for using a package. It also makes it more readable & documentable, since we can document it all with regular godoc, rather than deep in cmd/go's docs. |
@agnivade, thanks for finding that! I thought I remembered that but couldn't find it. Let's leave it open for now and see what others think. |
If we go with the magic package, we could use the unexported type trick to ensure that callers pass compile-time constants as arguments: https://play.golang.org/p/RtHlKjhXcda. (This is the strategy referenced here: https://groups.google.com/forum/#!topic/golang-nuts/RDA9Hag8RZw/discussion) |
One concern I have is how would it hanle invividual or all assets being too big to fit into memory and whether there would be maybe a build tag or per file access option to choose between pritorizing access time vs memory footprint or some middle ground implementation. |
the way i've solved that problem (because of course i also have my own implementation :) ) is to provide an http.FileSystem implementation that serves all embedded assets. That way, you don't to rely on magic comments in order to appease the typechecker, the assets can easily be served by http, a fallback implementation can be provided for development purposes (http.Dir) without changing the code, and the final implementation is quite versatile, as http.FileSystem covers quite a bit, not only in reading files, but listing directories as well. One can still use magic comments or whatever to specify what needs to be embedded, though its probably easier to specify all the globs via a plain text file. |
@AlexRouSg This proposal would only be for files which are appropriate to include directly in the final executable. It would not be appropriate to use this for files that are too big to fit in memory. There's no reason to complicate this tool to handle that case; for that case, just don't use this tool. |
@ianlancetaylor, I think the distinction @AlexRouSg was making was between having the files provided as global |
@bradfitz both http.File itself is an interface with no technical dependencies to the |
@urandom, it couldn't implement http.FileSystem, though, without referring to the "http.File" name (https://play.golang.org/p/-r3KjG1Gp-8). |
@robpike and I talked through a proposal for doing this years ago (before there was a proposal process) and never got back to doing anything. It's been bugging me for years that we never finished doing that. The idea as I remember it was to just have a special directory name like "static" containing the static data and automatically make them available through an API, with no annotations needed. I'm not convinced about the complexity of a "compressed vs not" knob. If we do that, then people will want us to add control over which compression, compression level, and so on. All we should need to add is the ability to embed a file of plain bytes. If users want to store compressed data in that file, great, the details are up to them and there's no API needed on Go's side at all. |
A couple thoughts:
Either way, this blocks embedding In general, I'm worried about expanding the scope of the go command too much. However, the fact that there are so many solutions to this problem means there probably ought to be one official solution. I'm familiar with |
Blocking certain files shouldn't be too hard, especially if you use a I'm not particularly a fan of a comment that compiles to code, but I also find the pseudo-package that affects compilation to be a bit strange as well. If the directory approach isn't used, maybe it might make a bit more sense to have some kind embed ui "./ui/build"
func main() {
file, err := ui.Open("version.txt")
if err != nil {
panic(err)
}
version, err = ioutil.ReadAll(file)
if err != nil {
panic(err)
}
file.Close()
log.Printf("UI Version: %s\n", bytes.TrimSpace(version))
http.ListenAndServe(":8080", http.EmbeddedDir(ui))
} Edit: You beat me to it, @jayconrod. |
To expand on #35950 (comment), there is a puzzle about the exposed API. The obvious ways to expose the data are The typical case is that you want the embedded data to be immutable. However, all interfaces exposing I'd suggest route (3): be immutable despite being a (A third party codegen package could do this by generating a generic assembly file containing |
I'm unsure what you're talking about in terms of immutability. |
I agree with @DeedleFake. I don't want to play games with magic |
Just another wrinkle here -- I have a different project which uses DTrace source code (embedded). This is sensitive to differences between \n and \r\n. (We can argue whether this is a dumb thing in DTrace or not -- that's beside the point and it is the situation today.) It's super useful that backticked strings treat both as \n regardless of how they appear in source, and I rely on this with a go-generate to embed the DTrace. So if there is an embed file added to the go command, I would gently suggest that options to change the handling of CR/CRLF might come in very handy, particularly for folks who might be developing on different systems where the default line endings can be a gotcha. |
Like with compression, I'd really like to stop at "copy the file bytes into the binary". CR/CRLF normalization, Unicode normalization, gofmt'ing, all that belongs elsewhere. Check in the files containing the exact bytes you want. (If your version control can't leave them alone, maybe check in gzipped content and gunzip them at runtime.) There are many file munging knobs we could imagine adding. Let's stop at 0. |
It may be too late to introduce a new reserved directory name, as much as I'd like to. Suppose we define a type runtime.Files. Then you could imagine writing:
And then at runtime you just call files.Open to get back an Names TBD but as far as the mechanism it seems like that should be enough, and I don't see how to make it simpler. (Simplifications welcome of course!) |
Whatever we do, it needs to be possible to support with Bazel and Gazelle too. That would mean having Gazelle recognize the comment and write out a Bazel rule saying the globs, and then we'd need to expose a tool (go tool embedgen or whatever) to generate the extra file to include in the build (the go command would do this automatically and never actually show the extra file). That seems straightforward enough. |
If various munging won't do the trick, then that's an argument against using this new facility. It's not a stopper for me -- I can use go generate like I've been doing, but it means I cannot benefit from the new feature. With respect to munging in general -- I can imagine a solution where someone provides an implementation of an interface (something like a Reader() on one side, and something to receive the file on the other -- maybe instantianted with an io.Reader from the file itself) -- which the go cmd would build and run to prefilter the file before embedding. Then folks can provide whatever filter they want. I imagine some folks would provide quasi-standard filters like a dos2unix implementation, compression, etc. (Maybe they should be chainable even.) I guess there'd have to be an assumption that whatever the embedded processor is, it must be compilable on ~every build system, as go would be building a temporary native tool for this purpose. |
If the files are only accessible through a special package, say |
Has anybody looked at markbates/pkger? It's a pretty simple solution of using It's also worth mentioning that Go doesn't have any significant structure requirements for a project as far as I could see. |
As we already have a way of injecting (albeit limited) data into a build via the existing I realise this doesn't cover all of the stated goals of the original issue, but as an extension of the existing (And if that works out then extending the |
That's a very interesting idea, but I'm worried about the security implications. It's pretty common to do Then there's the other issue of making sure |
Using the linker precludes support for WASM, and possibly other targets that don't use a linker. |
Based on the discussion here, @bradfitz and I worked out a design that sits somewhere in the middle of the two approaches considered above, taking what seems to be the best of each. I've posted a draft design doc, video, and code (links below). Instead of comments on this issue, please use the Reddit Q&A for comments on this specific draft design - Reddit threads and scales discussions better than GitHub does. Thanks! Video: https://golang.org/s/draft-embed-video |
@rsc In my opinion, the go:embed proposal is inferior to providing universal sandboxed Go code execution at compile-time which would include reading files and transforming read data into an optimal format best suitable for consumption at runtime. |
@atomsymbol That sounds like something waaay outside the scope of this issue. |
I am aware of that. |
I read through the proposal and scanned the code, but couldn't find an answer to this: Will this embedding scheme contain information about the file on disk (~os.Stat)? Or will these timestamps get reset to build time? Either way, these are useful pieces information that gets used in various places, e.g. we can send a 304 for unchanged assets based on this. Thanks! Edit: found it in the reddit thread.
|
An ETag header based on the file data hash would solve that problem without having to know anything about dates. But that would have to be known by http.HandlerFS or something to be able to work and to not waste resources it would have to be done only once per file. |
How would http.HandlerFS know that the fs.FS was immutable? Should there be an |
I don't want to get into implementation details because I'm not the designer of these things but http.HandlerFS could check if it's an embed.FS type and act upon that as a special case, I don't think anyone wants to expand the FS API right now. There could also be an option argument to HandlerFS specifically to tell it to treat a filesystem as immutable. Also if this is done on application start up and all ctime/mtime have zero value handlerFS could use that info to "know" that the file hasn't changed but there are also file systems which might not have mtime or have it disabled so there might be problems there as well. |
I wasn't watching the comments on this issue. @atomsymbol welcome back! It's great to see you commenting here again. @kokes I am not sure about the details, |
I have filed #41191 for accepting the design draft posted back in July. |
There are many tools to embed static asset files into binaries:
Actually, https://tech.townsourced.com/post/embedding-static-files-in-go/ lists more:
Proposal
I think it's time to do this well once & reduce duplication, adding official support for embedding file resources into the cmd/go tool.
Problems with the current situation:
go install
-able or making people write their own Makefiles, etc.Goals:
go install
/go build
do the embedding automaticallyfunc() io.Reader
,io.ReaderAt
, etc)io.Reader
)? (edit: but probably not; see comments below)go build
orgo install
can not run arbitrary code, just likego:generate
doesn't run automatically at install time.The two main implementation approaches are
//go:embed Logo logo.jpg
or a well-known package (var Logo = embed.File("logo.jpg")
).go:embed approach
For a
go:embed
approach, one might say that anygo/build
-selected*.go
file can contain something like:Which, say, compiles to:
(adding a dependency to the
io
package)Or:
compiling to, say:
Obviously this isn't fully fleshed out. There'd need to be something for compressed files too that yield only an
io.Reader
.embed package approach
The other high-level approach is to not have a magic
//go:embed
syntax and instead just let users write Go code in some new"embed"
or"golang.org/x/foo/embed"
package:Then have cmd/go recognize the calls to embed.Foo("foo/*.js") etc and glob do the work in cmd/go, rather than at runtime. Or maybe certain build tags or flags could make it fall back to doing things at runtime instead. Perkeep (linked above) has such a mode, which is nice to speed up incremental development where you don't care about linking one big binary.
Concerns
../../../../../../../../../../etc/shadow
.git
tooThe text was updated successfully, but these errors were encountered: