tools/telemetry/third_party/gsutilz/gslib/addlhelp/metadata.py

   1 # -*- coding: utf-8 -*-
   2 # Copyright 2012 Google Inc. All Rights Reserved.
   3 #
   4 # Licensed under the Apache License, Version 2.0 (the "License");
   5 # you may not use this file except in compliance with the License.
   6 # You may obtain a copy of the License at
   7 #
   8 #     http://www.apache.org/licenses/LICENSE-2.0
   9 #
  10 # Unless required by applicable law or agreed to in writing, software
  11 # distributed under the License is distributed on an "AS IS" BASIS,
  12 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13 # See the License for the specific language governing permissions and
  14 # limitations under the License.
  15 """Additional help about object metadata."""
  16
  17 from __future__ import absolute_import
  18
  19 from gslib.help_provider import HelpProvider
  20
  21 _DETAILED_HELP_TEXT = ("""
  22 <B>OVERVIEW OF METADATA</B>
  23   Objects can have associated metadata, which control aspects of how
  24   GET requests are handled, including Content-Type, Cache-Control,
  25   Content-Disposition, and Content-Encoding (discussed in more detail in
  26   the subsections below). In addition, you can set custom metadata that
  27   can be used by applications (e.g., tagging that particular objects possess
  28   some property).
  29
  30   There are two ways to set metadata on objects:
  31
  32   - at upload time you can specify one or more headers to associate with
  33     objects, using the gsutil -h option.  For example, the following command
  34     would cause gsutil to set the Content-Type and Cache-Control for each
  35     of the files being uploaded:
  36
  37       gsutil -h "Content-Type:text/html" \\
  38              -h "Cache-Control:public, max-age=3600" cp -r images \\
  39              gs://bucket/images
  40
  41     Note that -h is an option on the gsutil command, not the cp sub-command.
  42
  43   - You can set or remove metadata fields from already uploaded objects using
  44     the gsutil setmeta command. See "gsutil help setmeta".
  45
  46   More details about specific pieces of metadata are discussed below.
  47
  48
  49 <B>CONTENT TYPE</B>
  50   The most commonly set metadata is Content-Type (also known as MIME type),
  51   which allows browsers to render the object properly.
  52   gsutil sets the Content-Type automatically at upload time, based on each
  53   filename extension. For example, uploading files with names ending in .txt
  54   will set Content-Type to text/plain. If you're running gsutil on Linux or
  55   MacOS and would prefer to have content type set based on naming plus content
  56   examination, see the use_magicfile configuration variable in the gsutil/boto
  57   configuration file (See also "gsutil help config"). In general, using
  58   use_magicfile is more robust and configurable, but is not available on
  59   Windows.
  60
  61   If you specify a Content-Type header with -h when uploading content (like the
  62   example gsutil command given in the previous section), it overrides the
  63   Content-Type that would have been set based on filename extension or content.
  64   This can be useful if the Content-Type detection algorithm doesn't work as
  65   desired for some of your files.
  66
  67   You can also completely suppress content type detection in gsutil, by
  68   specifying an empty string on the Content-Type header:
  69
  70     gsutil -h 'Content-Type:' cp -r images gs://bucket/images
  71
  72   In this case, the Google Cloud Storage service will not attempt to detect
  73   the content type. In general this approach will work better than using
  74   filename extension-based content detection in gsutil, because the list of
  75   filename extensions is kept more current in the server-side content detection
  76   system than in the Python library upon which gsutil content type detection
  77   depends. (For example, at the time of writing this, the filename extension
  78   ".webp" was recognized by the server-side content detection system, but
  79   not by gsutil.)
  80
  81
  82 <B>CACHE-CONTROL</B>
  83   Another commonly set piece of metadata is Cache-Control, which allows
  84   you to control whether and for how long browser and Internet caches are
  85   allowed to cache your objects. Cache-Control only applies to objects with
  86   a public-read ACL. Non-public data are not cacheable.
  87
  88   Here's an example of uploading an object set to allow caching:
  89
  90     gsutil -h "Cache-Control:public,max-age=3600" cp -a public-read \\
  91            -r html gs://bucket/html
  92
  93   This command would upload all files in the html directory (and subdirectories)
  94   and make them publicly readable and cacheable, with cache expiration of
  95   one hour.
  96
  97   Note that if you allow caching, at download time you may see older versions
  98   of objects after uploading a newer replacement object. Note also that because
  99   objects can be cached at various places on the Internet there is no way to
 100   force a cached object to expire globally (unlike the way you can force your
 101   browser to refresh its cache).
 102
 103   Another use of the Cache-Control header is through the "no-transform" value,
 104   which instructs Google Cloud Storage to not apply any content transformations
 105   based on specifics of a download request, such as removing gzip
 106   content-encoding for incompatible clients.  Note that this parameter is only
 107   respected by the XML API. The Google Cloud Storage JSON API respects only the
 108   no-cache and max-age Cache-Control parameters.
 109
 110   Note that if you upload an object with a public-read ACL and don't include a
 111   Cache-Control header, it will be served with a Cache-Control header allowing
 112   the object to be cached for 3600 seconds. This will not happen if the object
 113   is uploaded with a non-public ACL and then changed to public. Moreover, if you
 114   upload an object with a public-read ACL and later change the ACL not to be
 115   public-read, the object will no longer be served with the default
 116   Cache-Control header noted above (so will be served as not cacheable).
 117
 118   For details about how to set the Cache-Control header see
 119   "gsutil help setmeta".
 120
 121
 122 <B>CONTENT-ENCODING</B>
 123   You can specify a Content-Encoding to indicate that an object is compressed
 124   (for example, with gzip compression) while maintaining its Content-Type.
 125   You will need to ensure that the files have been compressed using the
 126   specified Content-Encoding before using gsutil to upload them. Consider the
 127   following example for Linux:
 128
 129     echo "Highly compressible text" | gzip > foo.txt
 130     gsutil -h "Content-Encoding:gzip" -h "Content-Type:text/plain" \\
 131       cp foo.txt gs://bucket/compressed
 132
 133   Note that this is different from uploading a gzipped object foo.txt.gz with
 134   Content-Type: application/x-gzip because most browsers are able to
 135   dynamically decompress and process objects served with Content-Encoding: gzip
 136   based on the underlying Content-Type.
 137
 138   For compressible content, using Content-Encoding: gzip saves network and
 139   storage costs, and improves content serving performance. However, for content
 140   that is already inherently compressed (archives and many media formats, for
 141   instance) applying another level of compression via Content-Encoding is
 142   typically detrimental to both object size and performance and should be
 143   avoided.
 144
 145   Note also that gsutil provides an easy way to cause content to be compressed
 146   and stored with Content-Encoding: gzip: see the -z option in "gsutil help cp".
 147
 148
 149 <B>CONTENT-DISPOSITION</B>
 150   You can set Content-Disposition on your objects, to specify presentation
 151   information about the data being transmitted. Here's an example:
 152
 153     gsutil -h 'Content-Disposition:attachment; filename=filename.ext' \\
 154       cp -r attachments gs://bucket/attachments
 155
 156   Setting the Content-Disposition allows you to control presentation style
 157   of the content, for example determining whether an attachment should be
 158   automatically displayed vs should require some form of action from the user to
 159   open it.  See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1
 160   for more details about the meaning of Content-Disposition.
 161
 162
 163 <B>CUSTOM METADATA</B>
 164   You can add your own custom metadata (e.g,. for use by your application)
 165   to an object by setting a header that starts with "x-goog-meta", for example:
 166
 167     gsutil -h x-goog-meta-reviewer:jane cp mycode.java gs://bucket/reviews
 168
 169   You can add multiple differently named custom metadata fields to each object.
 170
 171
 172 <B>SETTABLE FIELDS; FIELD VALUES</B>
 173   You can't set some metadata fields, such as ETag and Content-Length. The
 174   fields you can set are:
 175
 176   - Cache-Control
 177   - Content-Disposition
 178   - Content-Encoding
 179   - Content-Language
 180   - Content-MD5
 181   - Content-Type
 182   - Any field starting with a matching Cloud Storage Provider
 183     prefix, such as x-goog-meta- (i.e., custom metadata).
 184
 185   Header names are case-insensitive.
 186
 187   x-goog-meta- fields can have data set to arbitrary Unicode values. All
 188   other fields must have ASCII values.
 189
 190
 191 <B>VIEWING CURRENTLY SET METADATA</B>
 192   You can see what metadata is currently set on an object by using:
 193
 194     gsutil ls -L gs://the_bucket/the_object
 195 """)
 196
 197
 198 class CommandOptions(HelpProvider):
 199   """Additional help about object metadata."""
 200
 201   # Help specification. See help_provider.py for documentation.
 202   help_spec = HelpProvider.HelpSpec(
 203       help_name='metadata',
 204       help_name_aliases=[
 205           'cache-control', 'caching', 'content type', 'mime type', 'mime',
 206           'type'],
 207       help_type='additional_help',
 208       help_one_line_summary='Working With Object Metadata',
 209       help_text=_DETAILED_HELP_TEXT,
 210       subcommand_help_text={},
 211   )