mirror of
https://github.com/tensorflow/tensorflow.git
synced 2026-09-28 05:13:36 +08:00
Write ExecutableAndOptionsProto records uncompressed.
`serialized_executable` is already a snappy-compressed split `GpuExecutableProto` (see `WriteSplitGpuExecutable`). Compressing the outer container again saves no space (dropping it adds 0.06% on a 330 MB executable), but it costs an extra compression pass on write and a full decompression pass on load. Riegeli stores the compression type per chunk, so readers don't change and existing artifacts still load. Alternative considered: leave the inner proto uncompressed and compress the outer one instead. That would mean deciding per use case when to skip compression, which adds logic for little gain, since the outer proto is small apart from `serialized_executable`. PiperOrigin-RevId: 987497408
This commit is contained in:
committed by
TensorFlower Gardener
parent
1d4cc1bac9
commit
b08a6e3e6c
+7
-2
@@ -78,8 +78,13 @@ ExecutableAndOptionsProto GetProtoWithoutSerializedExecutable(
|
||||
absl::Status WriteSplitExecutableAndOptions(
|
||||
const ExecutableAndOptionsProto& executable_and_options,
|
||||
std::unique_ptr<riegeli::Writer> writer) {
|
||||
riegeli::RecordWriter record_writer(std::move(writer),
|
||||
GetGpuSplitProtoOptions());
|
||||
riegeli::RecordWriter record_writer(
|
||||
std::move(writer),
|
||||
// The bulk of this proto is `serialized_executable`, which is already a
|
||||
// compressed split proto. Compressing it again does not reduce the size
|
||||
// but adds a full (de)compression pass over the whole executable when
|
||||
// serializing and when loading it.
|
||||
GetSplitProtoRiegeliOptions(SplitProtoCompression::kNone));
|
||||
SplitProtoManifest manifest = BuildManifest();
|
||||
TF_RETURN_WITH_CONTEXT_IF_ERROR(
|
||||
WriteRecord(record_writer, manifest),
|
||||
|
||||
Reference in New Issue
Block a user