Understand CURL and HTTP Headers

Published / by MasonCloud / Leave a Comment
Introduction

curl is a command line tool which is used to transfer data over the internet. It began as a project by Daniel Stenberg to transfer data over HTTP but has now evolved into a very robust tool that transfers data not just over HTTP but also FTP, TELNET, IMAP, and many more.In this guide we will cover the basics of curl and look at how we can use it to retrieve and send data, and also take a deep dive and look at HTTP headers.

Requirements

To follow this guide, you’ll need access to a Linux or Mac machine. You’ll need some familiarity with using the terminal to execute commands. If you aren’t familiar with using the terminal, feel free to take advantage of the resources available at Linux Academy to get up to speed. These resources are listed at the end of this guide.

Using curl

The best way to understand curl is to use it. This guide is an introduction-by-example. We’ll use curl to send requests to httpbin.org which is an HTTP client testing service. Let’s get started.

GET request

Fire up a terminal and execute the following command:

me@home:~$ curl www.httpbin.org

This command will fetch the HTML for the HTTPBin homepage. Notice that we’ve not mentioned the protocol or the HTTP method. They’ve defaulted to HTTP and GET, respectively.

HEAD request

Sometimes you may want to see just the headers that the server would return when it’s issued a GET request. In such a case, we’d use the HEAD method. The head request is issued with -I (capital i) or –head.

me@home:~$ curl -I www.httpbin.org    
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 01 Dec 2016 18:50:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12150
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: keep-alive
Verbose output

Sometimes, you may want to debug a problem because the curl command hasn’t returned what it was supposed to or just hasn’t worked at all. In such cases, you can use the verbose mode using -v or –verbose to get more output from curl. Let’s modify the previous HTTP request to become verbose.

me@home:~$ curl -Iv www.httpbin.org

Now there’s a lot more output in the terminal. Let’s see what it means.

Lines prefixed with an asterisk (*) show additional work curl has done. For example:

* Rebuilt URL to: www.httpbin.org/
*   Trying 23.22.14.18...
* Connected to www.httpbin.org (23.22.14.18) port 80 (#0)

The first line shows that curl added a slash at the end of the URL.

The second line shows that curl has resolved the URL to an IP address.

Finally, the third line shows that curl has connected to the URL on port 80.

Lines prefixed with a greater-than (>) sign show the data curl has sent to the server. For example:

> HEAD / HTTP/1.1
> Host: www.httpbin.org
> User-Agent: curl/7.47.0
> Accept: */*

Lines prefixed with a less-than (<) sign show the data curl has received from the server. For example:

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx
Server: nginx
Multiple requests

You can also send requests to multiple URLs by separating them by space. Here’s how:

me@home:~$ curl httpbin.org/status/418 httpbin.org/status/418

The above example will send two GET requests, both to the same URL, one after the other. You should see two little teapots printed in your terminal.

POST request

You can send a POST request by using the -d (or –data) flag. It͛s up to you to properly URL encode the data or specify it as JSON. We’ll see the examples for both of those. Here’s how you’d send URL encoded data as a part of the POST body

me@home:~$ curl -d 'name=john+doe&course=linux' httpbin.org/post

The response sent by HTTPBin will simply echo what you͛ve sent.

To send JSON:

me@home:~$ curl -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post

Notice that in both the examples, the data is enclosed in single quotes. Using double quotes is okay but it may cause problems if the body also contains double quotes like JSON. Using single quotes is the safer approach.

Setting headers

You can set headers by using the -H flag. In our previous example, we’ve sent JSON without setting the Content-Type header to application/json. Here’s the previous example modified to set the header:

me@home:~$ curl -H 'Content-Type: application/json' -d '{"name":"John Doe","course":"Linux"}' httpbin.org/post

You set the header as a key-value pair consisting of the header name followed by a colon and the value.

Posting form data

You can send form data by using the -F flag. Use this flag for every form field that you want to send. Here’s how you’d send form data:

me@home:~$ curl -F "name=John Doe" -F "course=linux"  httpbin.org/post

Notice that we’ve written the name as John Doe without a plus sign in between. The response received will have the Content-Type header set to multipart/form-data.

Uploading files

Using curl, you can upload files both as a part of a form field or to a ReST API endpoint. We’ll first create a file and then upload it both as a part of a form and to a ReST endpoint.

me@home:~$ echo "hello, world" >> file.txt     
me@home:~$ curl -F "file=@file.txt" httpbin.org/post

Since we’re using the -F flag, the file is uploaded as a part of the form. You specify the file to upload by an @ followed by the path to the file. Since our file is in the current directory, we just specify the file name.

Uploading to a ReST endpoint is similar. We just use the -d flag instead.

me@home:~$ curl -d @file.txt httpbin.org/post

Notice now that there are no quotes around file.txt. This is because we want the contents of the file to be the body of the POST request and not string “@file.txt” itself.

PUT request

ReST APIs use the PUT request to update an existing resource that the server manages. Making a PUT request with curl is similar to making a POST request. Here’s how you’d make a PUT request:

me@home:~$ curl -X PUT -d '{"name":"Jane Doe"}' httpbin.org/put
DELETE request

Similarly, ReST APIs use DELETE request to delete an existing resource. Here’s how to make a delete request:

me@home:~$ curl -X DELETE httpbin.org/delete
Basic authentication

Curl includes support for basic authentication. With basic authentication, you send your username and password as a part of the URL. There’s a two ways in which you can do this in curl. Here’s how:

me@home:~$ curl -u john:abc123 httpbin.org/basic-auth/john/abc123
me@home:~$ curl john:abc123@httpbin.org/basic-auth/john/abc123

In the first one you specify the username and password using the -u (short for –user) flag and curl appends this to the URL you provide. In the second case, you do it manually. Basic authentication does not encrypt your credentials in any way and is thus insecure unless used with HTTPS.

Setting cookies

You can set cookies using the -b (short for –cookie) flag. You specify the key and the value for the cookie with the flag. Here͛s an example showing how to set two cookies:

me@home:~$ curl --cookie "name=john;course=linux beginner" httpbin.org/cookies

Notice that cookies are separated by a semicolon and that you do not need to encode spaces as a plus sign.This brings us to the end of the guide on using curl. The next part of the guide will introduce you to HTTP headers.

HTTP headers

The HTTP protocol allows a client to send a request to a server. With every HTTP request and response, there are associated headers that provide some more information. Let’s make a HEAD request and see what we get:

me@home:~$ curl -Iv httpbin.org/get     
*   Trying 23.22.14.18...
* Connected to httpbin.org (23.22.14.18) port 80 (#0)
> HEAD /get HTTP/1.1
> Host: httpbin.org> User-Agent: curl/7.49.1
> Accept: */*
> 
< HTTP/1.1 200 OKHTTP/1.1 200 OK
< Server: nginxServer: nginx
< Date: Wed, 14 Dec 2016 16:23:11 GMTDate: Wed, 14 Dec 2016 16:23:11 GMT
< Content-Type: application/json
Content-Type: application/json
< Content-Length: 186
Content-Length: 186
< Connection: keep-alive
Connection: keep-alive
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
Access-Control-Allow-Credentials: true

In the output that’s printed, the following are headers: Host, User-Agent, Accept, Server, Content-Type, Content-Length, Connection, Access-Control-Allow-Origin, and Access-Control-Allow-Credentials.

Headers can be classified as request and response headers that contain information about the request and response, respectively, or as general headers that contain information about both request and response. There’s also another class of headers known as entity headers that contain more information about an entity, such as a file (i.e. it’s length and MIME-type). Here’s some of the most common headers and what they mean.

Authorization header

The Authorization header is used to provide authentication information such as bearer tokens.The server then uses this information to find out if the request should be processed further or not, depending on the validity of the authentication information provided. Example:

Authorization: Bearer 3beca038a248ff027d0445342fe285
Cache-Control header

The Cache-Control header decides how long applications, such as the browser, can keep a local copy of the data. This helps improve the efficiency since it helps avoid a round-trip to the server. Cache-Control has various directives that control various aspects of caching, such as the length of time data may be cached or if it shouldn’t be cached at all. Example:

Cache-Control: max-age=100
Expires header

The Expires header is also used for caching and specifies the date and time after which a particular cached resource is considered stale. When both Expires and Cache-Control header are set, the Cache-Control header takes higher priority. Example:

Expires: Wed, 21 Oct 2015 07:28:00 GMT
Connection header

The Connection header specifies whether a connection should last for the duration of the request or if it should stay open, allowing it to be reused for subsequent requests. The default value is “close” which creates a connection that only lasts for the duration of the request. Setting the header to “keep-alive” allows it to be persistent. Example:

Connection: keep-alive

Accept header

The Accept header tells the server what kind of content the client is expecting. The value of this header is the MIME type of the content. Example:

Accept: application/json
Cookie header

The cookie header contains all the cookies and their values. Cookies are used to identify the users, maintain sessions, and so on. Example:

Cookie: name=John;course=Linux
Content-Length header

The Content-Length header specifies the length of the content in bytes. Example:

Content-Length: 111020
Content-Type header

The Content-Type header tells the client the MIME-type of the content that it has received. Example:

Content-Type: application/json

There are a lot more headers than that can be covered in a simple guide, which brings us to the end of the guide on curl and HTTP headers. If you are eager to learn more about Linux command line, please check out the following section.

Burn ISO to DVD

Published / by MasonCloud / Leave a Comment
$ diskutil list
/dev/disk0 (internal, physical):
 #: TYPE NAME SIZE IDENTIFIER
 0: GUID_partition_scheme *251.0 GB disk0
 1: EFI EFI 209.7 MB disk0s1
 2: Apple_CoreStorage www.MasonCloud.com 250.1 GB disk0s2
 3: Apple_Boot Recovery HD 650.0 MB disk0s3

/dev/disk1 (internal, virtual):
 #: TYPE NAME SIZE IDENTIFIER
 0: hddrive +249.8 GB disk1
 Logical Volume on disk0s2
 300F37A7-6E6F-433C-8D23-1C224E67E88C
 Unlocked Encrypted

/dev/disk2 (external, physical):
 #: TYPE NAME SIZE IDENTIFIER
 0: FDisk_partition_scheme *32.4 GB disk2
 1: Windows_NTFS RedHat_v7.2 32.4 GB disk2s1

$ sudo dd if=/path-to-iso/rhel-server-7.2-x86_64-dvd.iso of=/dev/disk2 bs=1m
Password:

# Press CNTR+T inorder to show below output:
load: 1.90 cmd: dd 4045 running 0.00u 14.54s
1931+0 records in
1930+0 records out
2023751680 bytes transferred in 117.854438 secs (17171620 bytes/sec)
3856+0 records in
3856+0 records out
4043309056 bytes transferred in 232.334790 secs (17402943 bytes/sec)

Syntax of crontab

Published / by MasonCloud / Leave a Comment
Syntax of crontab (field description)

The syntax is:

1 2 3 4 5 /path/to/command arg1 arg2
OR
1 2 3 4 5 /root/backup.sh

Where,

1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
/path/to/command - Script or command name to schedule

Easy to remember format:

* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

Your cron job looks as follows for system jobs:

1 2 3 4 5 USERNAME /path/to/command arg1 arg2
OR
1 2 3 4 5 USERNAME /path/to/script.sh

Example: Run backup cron job script
If you wished to have a script named /root/backup.sh run every day at 3am, your crontab entry would look like as follows. First, install your cronjob by running the following command:
# crontab -e

Append the following entry:
0 3 * * * /root/backup.sh

Save and close the file.

More examples
To run /path/to/command five minutes after midnight, every day, enter:
5 0 * * * /path/to/command

Run /path/to/script.sh at 2:15pm on the first of every month, enter:
15 14 1 * * /path/to/script.sh

Run /scripts/phpscript.php at 10 pm on weekdays, enter:
0 22 * * 1-5 /scripts/phpscript.php

Run /root/scripts/perl/perlscript.pl at 23 minutes after midnight, 2am, 4am ..., everyday, enter:
23 0-23/2 * * * /root/scripts/perl/perlscript.pl

Run /path/to/unixcommand at 5 after 4 every Sunday, enter:
5 4 * * sun /path/to/unixcommand

How do I use operators?
An operator allows you to specifying multiple values in a field. There are three operators:

The asterisk (*) : This operator specifies all possible values for a field. For example, an asterisk in the hour time field would be equivalent to every hour or an asterisk in the month field would be equivalent to every month.
The comma (,) : This operator specifies a list of values, for example: "1,5,10,15,20, 25".
The dash (-) : This operator specifies a range of values, for example: "5-15" days , which is equivalent to typing "5,6,7,8,9,....,13,14,15" using the comma operator.
The separator (/) : This operator specifies a step value, for example: "0-23/" can be used in the hours field to specify command execution every other hour. Steps are also permitted after an asterisk, so if you want to say every two hours, just use */2.
How do I disable email output?
By default the output of a command or a script (if any produced), will be email to your local email account. To stop receiving email output from crontab you need to append >/dev/null 2>&1. For example:
0 3 * * * /root/backup.sh >/dev/null 2>&1

To mail output to particular email account let us say vivek@nixcraft.in you need to define MAILTO variable as follows:
MAILTO="vivek@nixcraft.in"
0 3 * * * /root/backup.sh >/dev/null 2>&1

See "Disable The Mail Alert By Crontab Command" for more information.

Task: List all your cron jobs
Type the following command:
# crontab -l
# crontab -u username -l

To remove or erase all crontab jobs use the following command:
# Delete the current cron jobs #
crontab -r

## Delete job for specific user. Must be run as root user ##
crontab -r -u username

Use special string to save time

Instead of the first five fields, you can use any one of eight special strings. It will not just save your time but it will improve readability.

Special string Meaning
@reboot Run once, at startup.
@yearly Run once a year, "0 0 1 1 *".
@annually (same as @yearly)
@monthly Run once a month, "0 0 1 * *".
@weekly Run once a week, "0 0 * * 0".
@daily Run once a day, "0 0 * * *".
@midnight (same as @daily)
@hourly Run once an hour, "0 * * * *".
Examples
Run ntpdate command every hour:
@hourly /path/to/ntpdate

Make a backup everyday:
@daily /path/to/backup/script.sh

More about /etc/crontab file and /etc/cron.d/* directories

/etc/crontab is system crontabs file. Usually only used by root user or daemons to configure system wide jobs. All individual user must must use crontab command to install and edit their jobs as described above. /var/spool/cron/ or /var/cron/tabs/ is directory for personal user crontab files. It must be backup with users home directory.

Understanding Default /etc/crontab

Typical /etc/crontab file entries:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
First, the environment must be defined. If the shell line is omitted, cron will use the default, which is sh. If the PATH variable is omitted, no default will be used and file locations will need to be absolute. If HOME is omitted, cron will use the invoking users home directory.

Additionally, cron reads the files in /etc/cron.d/ directory. Usually system daemon such as sa-update or sysstat places their cronjob here. As a root user or superuser you can use following directories to configure cron jobs. You can directly drop your scripts here. The run-parts command run scripts or programs in a directory via /etc/crontab file:

Directory Description
/etc/cron.d/ Put all scripts here and call them from /etc/crontab file.
/etc/cron.daily/ Run all scripts once a day
/etc/cron.hourly/ Run all scripts once an hour
/etc/cron.monthly/ Run all scripts once a month
/etc/cron.weekly/ Run all scripts once a week
How do I use above directories to put my own scripts or jobs?
Here is a sample shell script called clean.cache. This script is created to clean up cached files every 10 days. This script is directly created at /etc/cron.daliy/ directory. In other words create a text file called /etc/cron.daily/clean.cache as follows.

 #!/bin/bash
# A sample shell script to clean cached file from lighttpd web server
CROOT="/tmp/cachelighttpd/"

# Clean files every $DAYS
DAYS=10

# Web server username and group name
LUSER="lighttpd"
LGROUP="lighttpd"

# Okay, let us start cleaning as per $DAYS
/usr/bin/find ${CROOT} -type f -mtime +${DAYS} | xargs -r /bin/rm

# Failsafe
# if directory deleted by some other script just get it back
if [ ! -d $CROOT ]
then
 /bin/mkdir -p $CROOT
 /bin/chown ${LUSER}:${LGROUP} ${CROOT}
fi
Save and close the file. Set the permissions:
# chmod +x /etc/cron.daily/clean.cache

How do I backup installed cron jobs entries?

Simply type the following command to backup your cronjobs to a nas server mounted at /nas01/backup/cron/users.root.bakup directory:
# crontab -l > /nas01/backup/cron/users.root.bakup
# crontab -u userName -l > /nas01/backup/cron/users.userName.bakup