s3 copyObject does not work to update metadata

[doapp-ryanp]

copyObject() is the only way I know of to update and exsting s3 object's metadata. When updating an object to add the Cache-Control header I get the following error. I also tried adding 'Metadata'=>array('updated'=>'1') but get same error:

This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes

Its simple to re-produce:

    $s3->copyObject(array(
        'Bucket' => 'mybucket',
        'CopySource'=> 'mybucket/test.gz',
        'Key' => 'test.gz',
        'ACL' => CannedAcl::PUBLIC_READ,
        'CacheControl' => "max-age=60, public",
        'Metadata'=>array('updated'=>'1')
    ));

For what its worth, this bug also exists in the aws cli tool. see aws/aws-cli#652

[skyzyx]

You need to set 'MetadataDirective' => 'REPLACE' to do an in-place update.

http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.S3.S3Client.html#_copyObject

[doapp-ryanp]

So there is no way to just 'add' one metadata field? If I have 10 things (content type, encoding, expires, cache control, custom headers etc..) and I just want to add one how is this accomplished?

[skyzyx]

So there is no at way to just 'add' one metadata field?

There is. You need to tell S3 to copy the object over itself with new data. You do this by setting 'MetadataDirective' => 'REPLACE' as part of your call.

---

[…] how is this accomplished?

Something like this:

$s3->copyObject(array(
    'Bucket'          => 'mybucket',
    'CopySource'      => 'mybucket/test.gz',
    'Key'             => 'test.gz',
    'ACL'             => CannedAcl::PUBLIC_READ,
    'CacheControl'    => "max-age=60, public",
    'ContentType'     => 'text/plain',
    'ContentEncoding' => 'gzip',
    'Expires'         => '0', // 1 Jan 1970, 12:00am GMT
    'Metadata'        => array(
        'updated'        => '1', // x-amz-meta-updated
        'custom-header2' => '2', // x-amz-meta-custom-header2
        'custom-header3' => '3', // x-amz-meta-custom-header3
    ),
    'MetadataDirective' => 'REPLACE',
));

[skyzyx]

[…] I just want to add one […]

Here's the catch: you need to read the properties of the object you want to maintain, and re-apply them when you replace an object with itself (i.e., an in-place update) because otherwise S3 will either ignore them, or reset the values to default (e.g., the ContentType value).

S3 is kinda dumb that way, but if you understand that S3 treats a object-copy the same way that it treats a fresh upload, this approach makes sense. You must explicitly set the properties that you want on the new copy and/or updated object. S3 doesn't maintain any state in this case.

Make sense?

[doapp-ryanp]

Re: the catch - thats what I was saying (sorry I probably didn't explain well). To update an existing object and add just 1 metadata field, you need to read exising, merge new, then overwite.

IMO this really isn't an 'update'. Think of the case where another process is changing the Expires header after i've done the read and before i've done the merge and overwite. It'll wipe out the new Expires.

I get that s3 is not a DB, and now that I understand how it works, and that its not possible I can live with it.

Thanks for the quick response on this.

[jrochkind]

This seems to no longer be supported for v3 SDK? copy_object can no longer update metadata at all?

[Zaky7]

@jrochkind what we should do in v3 sdk to avoid This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata exception any idea?

[skyzyx]

I would strongly recommend opening a fresh issue rather than commenting on one that was closed in 2014.