You can read the whole standard specification but the basic concept is simple: by writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots. It is best explained with an example: 
# /robots.txt file for http://www.excitech.co.uk
# mail marketing@excitech.co.uk for constructive criticism

User-agent: *
Disallow: /admin
Disallow: /connections
Disallow: /data
Disallow: /logging
Disallow: /industry/aec/Events/RIBA2003
Disallow: /em
Disallow: /useradmin
Disallow: /Templates
Disallow: /oldpages
Disallow: /Source