A tool that generates FS API calls to generate a filesystem, and packages the files
to work with that.
This is called by emcc. You can also call it yourself.
You can split your files into "asset bundles", and create each bundle separately
with this tool. Then just include the generated js for each and they will load
the data and prepare it accordingly. This allows you to share assets and reduce
data downloads.
* If you run this yourself, separately/standalone from emcc, then the main program
compiled by emcc must be built with filesystem support. You can do that with
-s FORCE_FILESYSTEM=1 (if you forget that, an unoptimized build or one with
ASSERTIONS enabled will show an error suggesting you use that flag).
Usage:
file_packager.py TARGET [--preload A [B..]] [--embed C [D..]] [--exclude E [F..]]] [--js-output=OUTPUT.js] [--no-force] [--use-preload-cache] [--indexedDB-name=EM_PRELOAD_CACHE] [--no-heap-copy] [--separate-metadata] [--lz4] [--use-preload-plugins]
--preload ,
--embed See emcc --help for more details on those options.
--exclude E [F..] Specifies filename pattern matches to use for excluding given files from being added to the package.
See https://docs.python.org/2/library/fnmatch.html for syntax.
--from-emcc Indicate that `file_packager.py` was called from `emcc.py` and will be further processed by it, so some code generation can be skipped here
--js-output=FILE Writes output in FILE, if not specified, standard output is used.
--export-name=EXPORT_NAME Use custom export name (default is `Module`)
--no-force Don't create output if no valid input file is specified.
--use-preload-cache Stores package in IndexedDB so that subsequent loads don't need to do XHR. Checks package version.
--indexedDB-name Use specified IndexedDB database name (Default: 'EM_PRELOAD_CACHE')
--no-heap-copy If specified, the preloaded filesystem is not copied inside the Emscripten HEAP, but kept in a separate typed array outside it.
The default, if this is not specified, is to embed the VFS inside the HEAP, so that mmap()ing files in it is a no-op.
Passing this flag optimizes for fread() usage, omitting it optimizes for mmap() usage.
--separate-metadata Stores package metadata separately. Only applicable when preloading and js-output file is specified.
--lz4 Uses LZ4. This compresses the data using LZ4 when this utility is run, then the client decompresses chunks on the fly, avoiding storing
the entire decompressed data in memory at once. See LZ4 in src/settings.js, you must build the main program with that flag.
--use-preload-plugins Tells the file packager to run preload plugins on the files as they are loaded. This performs tasks like decoding images
and audio using the browser's codecs.
Notes:
* The file packager generates unix-style file paths. So if you are on windows and a file is accessed at
subdir\file, in JS it will be subdir/file. For simplicity we treat the web platform as a *NIX.
AV_WORKAROUND=0# Set to 1 to randomize file order and add some padding, to work around silly av false positives
data_files=[]
excluded_patterns=[]
export_name='Module'
leading=''
has_preloaded=False
compress_cnt=0
plugins=[]
jsoutput=None
from_emcc=False
force=True
# If set to True, IndexedDB (IDBFS in library_idbfs.js) is used to locally cache VFS XHR so that subsequent
# page loads can read the data from the offline cache instead.
use_preload_cache=False
indexeddb_name='EM_PRELOAD_CACHE'
# If set to True, the blob received from XHR is moved to the Emscripten HEAP, optimizing for mmap() performance.
# If set to False, the XHR blob is kept intact, and fread()s etc. are performed directly to that data. This optimizes for minimal memory usage and fread() performance.
no_heap_copy=True
# If set to True, the package metadata is stored separately from js-output file which makes js-output file immutable to the package content changes.
# If set to False, the package metadata is stored inside the js-output file which makes js-output file to mutate on each invocation of this packager tool.
at_position=arg.replace('@@','__').find('@')# position of @ if we're doing 'src@dst'. '__' is used to keep the index same with the original if they escaped with '@@'.
uses_at_notation=(at_position!=-1)# '@@' in input string means there is an actual @ character, a single '@' means the 'src@dst' notation.
ifuses_at_notation:
srcpath=arg[0:at_position].replace('@@','@')# split around the @
dstpath=arg[at_position+1:].replace('@@','@')
else:
srcpath=dstpath=arg.replace('@@','@')# Use source path as destination path.
# Absolutize paths, and check that they make sense
curr_abspath=os.path.abspath(os.getcwd())# os.getcwd() always returns the hard path with any symbolic links resolved, even if we cd'd into a symbolic link.
forfile_indata_files:
ifnotfile_['explicit_dst_path']:
# This file was not defined with src@dst, so we inferred the destination from the source. In that case,
# we require that the destination not be under the current location
path=file_['dstpath']
abspath=os.path.realpath(os.path.abspath(path))# Use os.path.realpath to resolve any symbolic links to hard paths, to match the structure in curr_abspath.
print('Error: Embedding "%s" which is below the current directory "%s". This is invalid since the current directory becomes the root that the generated code will see'%(path,curr_abspath),file=sys.stderr)
sys.exit(1)
file_['dstpath']=abspath[len(curr_abspath)+1:]
ifos.path.isabs(path):
print('Warning: Embedding an absolute file/directory name "'+path+'" to the virtual filesystem. The file will be made available in the relative path "'+file_['dstpath']+'". You can use the explicit syntax --preload-file srcpath@dstpath to explicitly specify the target location the absolute source path should be directed to.',file=sys.stderr)
forfile_indata_files:
file_['dstpath']=file_['dstpath'].replace(os.path.sep,'/')# name in the filesystem, native and emulated
iffile_['dstpath'].endswith('/'):# If user has submitted a directory name as the destination but omitted the destination filename, use the filename from source file
print('warning: file packager is creating an asset bundle of %d MB. this is very large, and browsers might have trouble loading it. see https://hacks.mozilla.org/2015/02/synchronous-execution-and-filesystem-access-in-emscripten/'%(start/(1024*1024)),file=sys.stderr)
Module['removeRunDependency']('fp ' + that.name); // workaround for chromium bug 124926 (still no audio with this, but at least we don't hang)
} else {
err('Preloading file ' + that.name + ' failed');
}
}, false, true); // canOwn this data in the filesystem, it is a slide into the heap that will never change
'''
create_data='''
Module['FS_createDataFile'](this.name, null, byteArray, true, true, true); // canOwn this data in the filesystem, it is a slide into the heap that will never change
Module['removeRunDependency']('fp ' + that.name);
'''
# Data requests - for getting a block of data out of the big archive - have a similar API to XHRs
code+='''
function DataRequest(start, end, audio) {
this.start = start;
this.end = end;
this.audio = audio;
}
DataRequest.prototype = {
requests: {},
open: function(mode, name) {
this.name = name;
this.requests[name] = this;
Module['addRunDependency']('fp ' + this.name);
},
send: function() {},
onload: function() {
var byteArray = this.byteArray.subarray(this.start, this.end);
// copy the entire loaded file into a spot in the heap. Files will refer to slices in that. They cannot be freed though
// (we may be allocating before malloc is ready, during startup).
if (Module['SPLIT_MEMORY']) err('warning: you should run the file packager with --no-heap-copy when SPLIT_MEMORY is used, otherwise copying into the heap may fail due to the splitting');
throw new Error(xhr.statusText + " : " + xhr.responseURL);
}
};
xhr.send(null);
};
function handleError(error) {
console.error('package error:', error);
};
'''
code+=r'''
function processPackageData(arrayBuffer) {
Module.finishedDataFileDownloads++;
assert(arrayBuffer, 'Loading data file failed.');
assert(arrayBuffer instanceof ArrayBuffer, 'bad input to processPackageData');
var byteArray = new Uint8Array(arrayBuffer);
var curr;
%s
};
Module['addRunDependency']('datafile_%s');
'''%(use_data,shared.JS.escape_for_js_string(data_target))# use basename because from the browser's point of view, we need to find the datafile in the same dir as the html file
code+=r'''
if (!Module.preloadResults) Module.preloadResults = {};
'''
ifuse_preload_cache:
code+=r'''
function preloadFallback(error) {
console.error(error);
console.error('falling back to default preload behavior');
# Overwrite the old jsoutput file (if exists) only when its content differs from the current generated one, otherwise leave the file untouched preserving its old timestamp