How to run the cp command to upload local files to OSS - Object Storage Service

You can run the cp command to upload resources, such as local files, images, and videos, to Object Storage Service (OSS). The command is suitable for uploading large files. You can also include the --include and --exclude options in the command to batch upload files that meet the specified conditions.

Usage notes

To upload a local file to OSS, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.
This topic provides sample command lines that are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil command reference.

When you run the cp command to upload a local file, ossutil automatically selects an upload method based on the file size:
- If the size of the local file that you want to upload is smaller than the threshold for performing a resumable upload task, ossutil uses simple upload. The threshold is specified by the --bigfile-threshold option and the default size is 100 MB.
- If the size of the local file that you want to upload is greater than or equal to the threshold for performing a resumable upload task, ossutil uses resumable upload.
  If the upload task is interrupted, the uploaded data is stored as parts in the OSS bucket. If you no longer need the uploaded parts, we recommend that you delete them by using one of the following methods to avoid additional storage costs:
  - Manually delete parts. For more information, see rm.
  - Configure lifecycle rules to automatically delete parts. For more information, see Configure lifecycle rules.

Command syntax

./ossutil64 cp file_url cloud_url
[-r, --recursive]
[-f --force]
[-u --update]
[--maxupspeed <value>]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--snapshot-path <value>]
[--disable-crc64]
[--disable-dir-object]
[--payer <value>]
[--tagging <value>]
[-j, --job <value>]
[--parallel <value>]
[--start-time <value>]
[--end-time <value>]

The following table describes the parameters and options in the command syntax.

Parameter/Option	Description
file_url	The path of the local file. Examples: `/localfolder/examplefile.txt` in Linux and `D:\localfolder\examplefile.txt` in Windows.
cloud_url	The path in which you want to store the uploaded local file. Format: `oss://bucketname/objectname`. Example: `oss://examplebucket/examplefile.txt`.
-r, --recursive	The recursive operations. If you specify this option, ossutil performs operations on all objects in the bucket that meet the specified conditions. If you do not specify this option, ossutil performs operations only on the specified object.
-f --force	Specifies that the operation is forcibly performed without a prompt for confirmation.
-u, --update	Specifies that ossutil uploads files only when the destination objects do not exist or when the last modified time of the files is later than that of the destination objects.
--maxupspeed	The maximum upload speed. Unit: KB/s. Default value: 0. A value of 0 specifies that the upload speed is not limited.
--enable-symlink-dir	Specifies that ossutil uploads subdirectories in directories to which symbolic links point. By default, ossutil does not upload subdirectories in directories to which symbolic links point.
--disable-all-symlink	Specifies that ossutil ignores subdirectories and objects in directories to which symbolic links point.
--disable-ignore-error	Specifies that errors are not ignored during batch operations.
--only-current-dir	Specifies that ossutil uploads only objects in the current directory. Subdirectories in this directory and objects in the subdirectories are ignored.
--bigfile-threshold	The threshold for performing a resumable upload task. If the object size exceeds the threshold, resumable upload is used. Unit: bytes. Default value: 100 MB. Valid values: 0 to 9223372036854775807.
--part-size	The part size. Unit: bytes. By default, ossutil determines the part size based on the object size. Valid values: 1 to 9223372036854775807.
--checkpoint-dir	The directory in which information about the resumable upload task is stored. If the task fails, ossutil automatically creates a directory named `.ossutil_checkpoint` and saves the checkpoint information in the directory. If the task is successful, ossutil deletes the directory. If you specify a checkpoint directory, make sure that the directory can be deleted.
--encoding-type	The method that is used to encode the names of objects. Set the value to url. If you do not specify this option, the names of objects are not encoded.
--include	Specifies that all objects that meet specified conditions are matched. For more information about the syntax and examples, see Batch upload objects that meet specified conditions.
--exclude	Specifies that all objects that do not meet specified conditions are matched. For more information about the syntax and examples, see Batch upload objects that meet specified conditions.
--meta	The object metadata. Object metadata includes standard HTTP headers and user metadata. User metadata starts with `x-oss-meta-`. Format: `header:value#header:value` format. Example: `Cache-Control:no-cache#Content-Encoding:gzip`. For more information, see Manage object metadata.
--acl	The access control list (ACL) of the objects. Valid values: default: The ACL of the objects is the same as the ACL of the bucket in which the objects are stored. This is the default ACL setting for objects. private: Only the bucket owner can perform read and write operations on the objects in the bucket. Other users cannot access the objects in the bucket. public-read: Only the bucket owner can perform write operations on the objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. This may result in unrestricted access to the data in your bucket and unexpectedly high fees. If a user uploads prohibited data or information, your legitimate interests and rights may be infringed. We recommend that you do not set the object ACL to public-read except in special scenarios. public-read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. This may result in unrestricted access to the data in your bucket and unexpectedly high fees. Exercise caution when you set the ACL to public-read-write.
--snapshot-path	The directory in which the snapshots of uploaded objects are stored. In the subsequent upload task, ossutil reads the snapshot information from the specified directory and performs incremental upload.
--disable-crc64	Specifies that CRC-64 is disabled. By default, ossutil enables CRC-64 during data transmission.
--disable-dir-object	Specifies that no OSS object is generated for the directory during object upload.
--payer	The payer of the traffic and request fees. If you want the requester who accesses the resources in the specified path to pay for the traffic and request, set this option to requester.
--tagging	The tags of objects. Format: `TagkeyA=TagvalueA&TagkeyB=TagvalueB....`.
-j, --jobs	The number of concurrent tasks that can be performed across multiple objects. Valid values: 1 to 10000. Default value: 3.
--parallel	The number of concurrent tasks that can be performed on a single object. Valid values: 1 to 10000. If you do not specify this option, ossutil specifies a value based on the operation type and the object size.
--start-time	The beginning of the time range to filter objects by the last modified time. The value is a UNIX timestamp. Objects whose last modified time is earlier than the UNIX timestamp are ignored. Note Only ossutil V1.7.18 and later support this parameter. For more information about how to update ossutil, see update.
--end-time	The end of the time range to filter objects by the last modified time. The value is a UNIX timestamp. Objects whose last modified time is later than the UNIX timestamp are ignored. Note If both --start-time and --end-time are specified, the command applies only to objects whose last modified time is within the range specified by --start-time and --end-time. Only ossutil V1.7.18 and later support this parameter. For more information about how to update ossutil, see update.

If the default number of concurrent tasks does not meet your performance requirements, you can use the -j, --jobs and --parallel options in the preceding command syntax can be used to adjust upload performance. By default, ossutil calculates the value of --parallel based on the object size. When you batch upload large objects, the actual concurrency value is calculated by multiplying the --jobs value and the --parallel value.

We recommend that you adjust the concurrency value to a value smaller than 100 if the resources such as network bandwidth, memory, and CPUs of your ECS instance or server are limited. If resources, such as network bandwidth, memory, and CPUs, are left idle, you can increase the number of concurrent tasks.
If the number of concurrent tasks is excessively large, the upload performance of ossutil may decrease because threads switch and compete for resources. In addition, an end-of-file (EOF) error may occur. To prevent these issues, you can adjust the values of -j, --jobs and --parallel based on your available resources. To perform stress testing, specify small values for the two options before you incrementally increase them to the optimal values.

Sample environment

In this topic, local files or directories are uploaded from a Linux OS to OSS. You can modify the options in the examples based on your OS and environment. The following resources are used in examples:

Local file: examplefile.txt (a file in the root directory)
Local directory: localfolder (a subdirectory in the root directory)
Destination bucket: examplebucket
Directory in the destination bucket: desfolder

Simple upload

If you want to upload resources, such as local files, images, and videos, to OSS, you can use ossutil. The following sample command provides examples on how to use ossutil to upload data to OSS.

Upload a single local file
By default, if you do not specify an object name, OSS uses the name of the local file as the object name. If you specify an object name, OSS uses the name for the object.
```
./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/
```
Upload files from a local directory
You can add the -r option to the cp command to upload files from a local directory to the specified path.
```
./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/
```
Upload files whose last modified time is within the specified time range from a local directory
You can run the following command to upload files whose last modified time is within the range of 10:09:18 (UTC+8) October 31, 2023 to 12:55:58 (UTC+8) October 31, 2023 from a local directory to the specified OSS path.
```
./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --start-time 1698718158 --end-time 1698728158
```
Upload a local directory and the files in the directory
You can add the -r option to the cp command and include the name of a local directory in the OSS path to upload the local directory and the files in the directory to the specified OSS path. For the specified directory, OSS does not create a directory object. For a subdirectory in the specified directory, OSS creates a zero-byte object whose name ends with a forward slash (/). If you want to create a directory object for the local directory, run the mkdir command.
```
./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/localfolder/
```
Upload a single file and specify --meta
You can specify object metadata in the header:value#header:value... format by using the --meta option when you run the cp command to upload a file.
```
./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/examplefile.txt --meta=Cache-Control:no-cache#Content-Encoding:gzip
```
Upload a directory and skip existing objects
When you resume a failed batch upload task, you can specify the --update (abbreviated as -u) option to skip uploaded objects and perform an incremental upload. Example:
```
./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ -u
```

Upload a file to a bucket for which pay-by-requester is enabled

./ossutil64 cp localfolder/examplefile.txt oss://examplebucket/ --payer=requester

Upload only files in the current directory and ignore subdirectories

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --only-current-dir -r

Upload a directory without generating an object for the uploaded directory
In OSS, a directory is a zero-byte object whose name ends with a forward slash (/). If you specify the --disable-dir-object option in the cp command to upload a directory, OSS does not generate an object for the directory. However, you can view the directory in the OSS console. If you delete all objects in the directory, the directory is also deleted.
```
./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --disable-dir-object -r
```

Upload objects in the subdirectory to which a symbolic link points

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --enable-symlink-dir -r

Ignore the subdirectory to which a symbolic link points and all objects in this subdirectory during upload
```
./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ -r --disable-all-symlink
```

Configure the maximum upload speed

When you upload an object, you can set --maxupspeed to specify the maximum upload speed. Unit: KB/s. Examples:

Upload an object to OSS and set the maximum upload speed to 1 MB/s

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --maxupspeed 1024

Upload a directory to OSS and set the maximum upload speed to 1 MB/s

./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --maxupspeed 1024

Upload an object and configure tags for the object

When you run the cp command to upload an object, you can specify the --tagging option to configure tags for the object. If you configure multiple tags for the object, separate them with ampersands (&). Example:

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --tagging "abc=1&bcd=2&……"

For more information, see object-tagging.

Upload an object and specify the storage class of the object

When you run the cp command to upload a local file, you can specify the --meta option to specify the storage class of the object. ossutil supports the following storage classes:

Standard
IA
Archive

If you do not specify the storage class of the object in the command, OSS applies the storage class of the bucket to the object. For more information, see Overview.

Examples:

Upload an object and set the storage class of the object to IA

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta X-oss-Storage-Class:IA

Upload a directory and set the storage class of all objects in the directory to Standard

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --meta X-oss-Storage-Class:Standard -r

Upload an object and specify the ACL of the object

When you run the cp command to upload an object, you can set the --acl option to specify the ACL of the object. ossutil supports the following ACLs:

default
private
public-read
public-read-write

Examples:

Upload an object and set the ACL of the object to private

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --acl private

Upload an object and set the ACL of the object to public-read

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --acl public-read  -r

Upload an object and specify the encryption method of the object

When you upload an object, you can enable server-side encryption to store the object in the encrypted state. Examples:

Upload an object and set the encryption method of the object to SSE-OSS and the encryption algorithm to AES-256
```
./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:AES256
```
Upload an object and set the encryption method of the object to SSE-KMS without specifying a customer master key (CMK) ID
```
./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:KMS
```
When you set the encryption method to SSE-KMS, you are charged for the usage of CMK IDs. For more information, see Billing of KMS.

Upload an object and set the encryption method of the object to SSE-KMS with a specified CMK ID

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:KMS#x-oss-server-side-encryption-key-id:7bd6e2fe-cd0e-483e-acb0-f4b9e1******

For more information about server-side encryption, see Server-side encryption.

Generate snapshots for objects when you upload the objects

When you run the cp command to upload multiple objects at a time and specify --snapshot-path in the command, ossutil generates the snapshots for the uploaded objects in the specified directory to record the last modified time of the objects. In subsequent upload tasks, ossutil determines whether to skip existing objects based on their last modified time. If you specify --snapshot-path in the command, make sure that the objects in OSS are not modified by other users from the time when the objects were last uploaded. The --snapshot-path option is used to accelerate incremental uploads. Example:

./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --snapshot-path=path

Important

ossutil does not automatically delete snapshots stored in the directory specified by snapshot-path. To prevent snapshots from consuming excessive storage space, delete the snapshots that you no longer need from the directory specified by snapshot-path.
Additional overheads are required to read and write snapshot information. We recommend that you do not use this option in scenarios in which you have only a small number of objects to upload, the network condition is good, or other users need to perform operations on the objects. Instead, you can use --update to perform incremental uploads.
You can configure both --update and --snapshot-path in a cp command. ossutil determines whether to skip an object based on the snapshots stored in the directory specified by --snapshot-path. If no snapshots are generated for the object, ossutil determines whether to skip the object based on the value of --update.

Batch upload objects that meet specified conditions

When you run the cp command to upload multiple objects at a time and specify the --include and --exclude options in the command, ossutil uploads objects that meet the specified conditions.

The --include and --exclude options support the following formats:

Asterisk (*): matches any number of characters. For example, *.txt matches all TXT files.
Question mark (?): matches a single character. For example, abc?.jpg matches all JPG objects whose names are abc followed by a single character, such as abc1.jpg.
[sequence]: matches characters in a sequence. For example, abc[1-5].jpg matches JPG objects whose names are abc followed by a number in sequence [1-5]. The object names are abc1.jpg, abc2.jpg, abc3.jpg, abc4.jpg, and abc5.jpg.
[!sequence]: matches characters outside a sequence. For example, abc[!0-7].jpg matches objects other than abc0.jpg, abc1.jpg, abc2.jpg, abc3.jpg, abc4.jpg, abc5.jpg, abc6.jpg, and abc7.jpg.

A rule can contain multiple conditions specified by --include and --exclude. After these conditions are configured, ossutil evaluates the conditions from left to right to find the final matches. If the test.txt object exists in a directory for which conditions are specified, ossutil generates match results based on different matching rules.

Rule 1: --include "*test*" --exclude "*.txt". ossutil first evaluates the --include "*test*" condition and includes the test.txt object in the results because the object is a match. Then, ossutil evaluates the --exclude "*.txt" condition, finds that the test.txt object is a match because the object name contains ".txt", and then excludes the object from the results. The test.txt object is excluded from the final results.
Rule 2: --exclude "*.txt" --include "*test*". ossutil first evaluates the --exclude "*.txt" condition and excludes the test.txt object from the results because the object is a match. Then, ossutil evaluates the--include "*test*" condition, finds that the test.txt object is a match because the object name contains "test", and then includes the object in the results. The test.txt object is included in the final results.
Rule 3: --include "*test*" --exclude "*.txt" --include "te?t.txt". ossutil first evaluates the --include "*test*" condition, finds that the test.txt object is a match, and then includes the object in the results. Then, ossutil evaluates the --exclude "*.txt" condition, finds that the test.txt object is a match because the object name contains "txt", and excludes the object from the results. Finally, ossutil evaluates the --include "te?t.txt" condition, finds that the test.txt object is a match, and then includes the object in the results. The test.txt object is included in the final results.

Important

You cannot specify directories in --include and --exclude options. For example, --include "/usr/test/.jpg" is invalid.

Examples:

Upload all TXT objects from the specified directory

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --include "*.txt" -r

Upload all objects whose names contain abc and whose formats are not JPG or TXT from the specified directory

./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --include "*abc*" --exclude "*.jpg" --exclude "*.txt" -r

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.

For example, you can run the following command to upload a local file named exampleobject.txt to the destfolder directory of the examplebucket bucket that is located in the China (Shanghai) region and owned by another Alibaba Cloud account:

./ossutil64 cp exampleobject.txt oss://examplebucket/desfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about common options, see Common options.