Here's the job. You have app.log, 12 lines long, and every line looks like 2026-10-05T09:00:04Z ERROR db connection timeout after 5000ms: a timestamp, a level, a component and a message. You need to know which component throws the most errors, and which error messages repeat, and you want the answer saved to a file. One line of Linux pipes and redirection does all of that. This guide builds that line one step at a time and then explains every operator it uses, plus the ones you'll need next: 2>&1, tee, xargs, pipefail and the rest. If cd, ls and cat are still new to you, start with command line basics for beginners.
I ran the Bash examples in Git Bash 5.2.37 on Windows 11 and in Ubuntu 24.04.5 (bash 5.2.21), and some of them in bash 5.3.20 as well. Where a block ran in only one of those, the text says which. The few commands I couldn't run are labelled as untested.
Build a log-analysis pipeline step by step
Linux pipes and redirection examples make more sense on real data than on echo hello. So build a pipeline the way you'd debug one: add a stage, look at the output, then add the next stage. These are the steps and the output each one produced on the sample log.
Step 1: find the error lines
set -o pipefail
grep ERROR app.log # 7 lines
grep -c ERROR app.log # 7The set -o pipefail line looks unnecessary right now. Step 6 shows why it's there.
Step 2: keep only the component column
grep ERROR app.log | awk '{print $3}'
# db api db auth db db api (one per line)The | sends grep's standard output into awk's standard input. awk splits each line on whitespace, so $3 is the component.
Step 3: count each component
grep ERROR app.log | awk '{print $3}' | sort | uniq -c
# 2 api / 1 auth / 4 db (uniq -c pads the counts with leading spaces)sort is required here. uniq only collapses duplicate lines that sit next to each other. When I skipped sort, I got six fragmented counts (1 db, 1 api, 1 db, 1 auth, 2 db, 1 api) instead of three groups.
Step 4: rank them
grep ERROR app.log | awk '{print $3}' | sort | uniq -c | sort -nr
# 4 db / 2 api / 1 authStep 5: top messages, shown on screen and saved to a file
grep ERROR app.log | awk '{ $1=$2=$3=""; sub(/^ +/, ""); print }' \
| sort | uniq -c | sort -nr | head -3 | tee error_summary.txt
# 3 connection timeout after 5000ms
# 2 upstream returned 502
# 1 invalid token user=17awk blanks the first three fields and strips the leading spaces left behind, so only the message is left. tee at the end prints the result and writes error_summary.txt at the same time. Two variations are worth keeping. The first does the per-component count in awk alone, and the second counts errors per minute:
awk '$2 == "ERROR" { n[$3]++ } END { for (c in n) print n[c], c }' app.log | sort -nr
# 4 db / 2 api / 1 auth
awk '$2=="ERROR" {print substr($1,1,16)}' app.log | sort | uniq -c # errors per minuteStep 6: make a typo fail loudly
grep ERROR app.lgo | awk '{print $3}' | sort | uniq -c; echo "exit=$? ${PIPESTATUS[*]}"
# exit=2 2 0 0 0The filename is misspelled, so grep fails with exit status 2. With pipefail set, the whole pipeline returns 2. Without it, the pipeline takes the status of the last command (uniq, which succeeded), so a script would see 0 and keep going. PIPESTATUS shows where the failure happened.
All six steps gave byte-identical output in Git Bash (GNU Awk 5.3.2) and Ubuntu (mawk 1.3.4). This grep, count and rank pattern is often the first thing to run when debugging production issues from logs.
Linux pipes and redirection cheat sheet
Here are the operators from the pipeline, plus the others you'll use. "Bash only" means dash (Ubuntu's /bin/sh) handles them differently, so use #!/usr/bin/env bash in scripts that rely on them. A dash in the last column means I didn't check that operator against POSIX.
| Operator | What it does | Portable? |
|---|---|---|
cmd > file | stdout to file, creating or truncating it | POSIX |
cmd >> file | stdout appended to file | POSIX |
cmd >| file | overwrite even when noclobber is on | POSIX |
cmd 2> file | stderr only to file | POSIX |
cmd > file 2>&1 | both streams to file (order matters) | POSIX |
cmd &> file / &>> file | both streams, overwrite / append | Bash only |
cmd 2>/dev/null | discard errors | POSIX |
cmd < file | file becomes stdin | POSIX |
<<EOT / <<< "str" | here-doc / here-string as stdin | for <<<, use Bash to be safe |
a | b | a's stdout into b's stdin | POSIX |
a |& b | stdout and stderr into b (2>&1 |) | Bash only |
| tee file | show output and save it | POSIX utility |
| xargs cmd | turn input lines into arguments | POSIX utility |
a ; b | run b no matter how a ended | - |
a && b / a || b | run b only if a succeeded / failed | - |
$(cmd) / <(cmd) | output as text / output as a filename | <() not in /bin/sh (bash, ksh, zsh) |
set -o pipefail | pipeline fails if any stage fails | POSIX since 2024, not in dash 0.5.12 |
stdin, stdout and stderr: the three streams
Everything in Linux pipes and redirection comes down to three file descriptors that every process starts with: stdin (0), stdout (1) and stderr (2). In a terminal all three point at the terminal, so normal output and error messages look the same until you redirect one of them. A redirection that starts with > and has no number means fd 1. One that starts with < means fd 0.
+---------+ --fd 1 (stdout)--> terminal, file, or next command's stdin
fd 0 (stdin) | cmd |
---------> +---------+ --fd 2 (stderr)--> terminal (not carried by |)To see the difference, run a command that writes to both streams:
echo hello > exists.txt
ls exists.txt missing.txt # both streams to terminal
ls exists.txt missing.txt > out.txt # error still on terminal
ls exists.txt missing.txt 2> err.txt # listing on terminal, error in file
ls exists.txt missing.txt > out.txt 2> err.txtThe error is ls: cannot access 'missing.txt': No such file or directory and the exit status is 2. With only > out.txt, the error still prints and out.txt holds only exists.txt. The last line puts the output and the errors in separate files.
Watch: Linux PIPES and REDIRECTS: Command Line Ninja Skillz by Shawn Powers
Shawn Powers' whiteboard explanation of the three streams (0:00 to 5:12) is a good visual introduction. Watch 7:41 in particular: he pipes a misspelled filename into less and gets a blank screen, because the error went to stderr. He fixes it with 2>&1 | less at 8:22. At 14:19 he shows that typing > less instead of | less just creates a file called less. One small correction: at 12:22 he says 2>&1 sends standard error "to standard input". It actually makes stderr a copy of standard output. His command is correct, and at 15:02 he explains it correctly.
The video is from the Shawn Powers channel and is based on LPI Linux Essentials objective 3.2. That objective is still part of the current 010-160 v1.6 exam, and nothing the video shows has changed in Bash 5.3.
Redirect stderr and stdout to a file
When you want both streams in one file, use > file 2>&1. Bash also has the shorter &>, which the Bash manual says is "semantically equivalent to >word 2>&1".
ls exists.txt missing.txt > both.txt 2>&1 # nothing on screen; both lines in file
ls exists.txt missing.txt 2>&1 > wrong.txt # error printed on screen; file has only exists.txt
ls exists.txt missing.txt &> amp.txt # same as > amp.txt 2>&1
ls exists.txt missing.txt &>> amp.txt # append both; file now 4 linesTested in Git Bash, Ubuntu and bash 5.3.
Bash redirection order: why the second line goes wrong
Bash processes redirections from left to right, and 2>&1 means "point fd 2 wherever fd 1 points right now". In > both.txt 2>&1, stdout already points at the file, so stderr follows it there. In 2>&1 > wrong.txt, stderr is copied to the terminal first, and only then does stdout move to the file. The Bash manual's Redirections page uses this same example with ls > dirlist 2>&1.
Throwing output away with /dev/null
ls exists.txt missing.txt 2>/dev/null # prints exists.txt, exit 2 kept
ls exists.txt missing.txt > /dev/null 2>&1 # silent, exit 2 kept
find / -name passwd 2>/dev/null # typical useThe first two lines were tested. Discarding output doesn't change the exit status, so if and && still behave correctly. The find / line is untested (I checked it against the Bash manual), but it's the most common real use of 2>: it hides the "Permission denied" noise.
>, >> and noclobber
> truncates the file to zero length before the command runs. One wrong > can wipe a log. set -o noclobber (also written set -C) blocks that:
echo one > log.txt; echo two >> log.txt
set -o noclobber
echo new > log.txt # bash: log.txt: cannot overwrite existing file (exit 1)
echo forced >| log.txt # exit 0, overwrites
echo appended >> log.txt # exit 0, append still allowed
set +o noclobberInput redirection: <, here-docs and here-strings
name=world
cat <<EOT
hello $name
EOT
cat <<'EOT'
hello $name
EOT
wc -w <<< "one two three"
wc -l < log.txtThis prints hello world, then hello $name, then 3, then 2. Quoting the delimiter ('EOT') turns off expansion, which is what you want when you write a script or config file that contains its own $ variables. <<- strips leading tabs, but not spaces.
Pipes: what they carry and how they fail
Linux pipe vs redirect
| connects a stream to another command. > and < connect a stream to a file. Mixing them up doesn't produce an error:
cat exists.txt > less # creates a file literally named "less"In both test environments this created a file named less containing hello. If a file named after a command shows up in your directory, this is how it got there.
Pipes only carry stdout (and how |& changes that)
ls exists.txt missing.txt | wc -l # error on screen, prints 1
ls exists.txt missing.txt 2>&1 | wc -l # 2
ls exists.txt missing.txt |& wc -l # 2
ls /nope 2>/dev/null |& wc -l # 1 <- |& runs after your 2>/dev/null
ls /nope 2>&1 2>/dev/null | wc -l # 0The fourth line catches people out. |& is shorthand for 2>&1 |, but its implicit 2>&1 runs after your own redirections, so it overrides your 2>/dev/null.
Catching failures: set -o pipefail and PIPESTATUS
false | true; echo $? # 0
ls missing.txt 2>/dev/null | sort | wc -l; echo "${PIPESTATUS[@]}" # 0 / 2 0 0
grep ERROR nosuchfile.log | wc -l; echo $? # grep error, 0, exit 0 (silent failure)
set -o pipefail
grep ERROR nosuchfile.log 2>/dev/null | wc -l; echo $? # 0, exit 2
(exit 3) | (exit 5) | true; echo $? # 5 (rightmost non-zero)
false | true | false; st=("${PIPESTATUS[@]}"); echo "${st[*]}"; echo "${PIPESTATUS[*]}" # 1 0 1 / 0
set -o pipefail; yes | head -1; echo "$? ${PIPESTATUS[*]}" # y / 141 141 0Tested in all three environments. The SIGPIPE line on the end was run on Ubuntu only. pipefail is off by default. With it on, a pipeline returns the status of the rightmost command that failed. Copy PIPESTATUS straight away, because the very next command overwrites it. The last line shows the catch. head exits after one line, yes is killed by SIGPIPE (128+13 = 141), and under set -eo pipefail your script dies even though nothing really went wrong. Watch for this with head, grep -q and other commands that stop reading early. Even so, CI scripts should start with set -o pipefail, because a failure that nobody sees is worse, especially when you build a self-healing CI/CD pipeline that has to notice failures. POSIX added pipefail in 2024, but dash 0.5.12 on Ubuntu 24.04 still replies sh: 1: set: Illegal option -o pipefail.
The tee command in Linux (and sudo tee)
echo first | tee t.txt # prints first, writes file
echo second | tee -a t.txt > /dev/null # append quietly
./script.sh 2>&1 | tee run.log # watch + save both streamstee copies its stdin to stdout and to each file you name. It overwrites those files unless you pass -a. The ./script.sh line is untested (checked against the coreutils manual), but I tested the same pattern with ls ... 2>&1 | tee combined.txt.
The most useful job for tee is writing to files you don't own:
sudo echo "x" > /etc/demo.conf # FAILS: bash: line 1: /etc/demo.conf: Permission denied
echo "x" | sudo tee -a /etc/demo.conf > /dev/null # works
sudo sh -c 'echo x >> /etc/demo.conf' # worksThe first line fails because your shell opens /etc/demo.conf before sudo even starts. sudo makes echo run as root, but the shell that performs the redirect is still you. In the second line, tee itself runs as root and opens the file. Tested on Ubuntu 24.04 with sudo 1.9.15p5 as a non-root user. In Git Bash, sudo is the Windows one, so don't test this there.
xargs vs pipe: data or arguments
Use a plain pipe when the next command reads stdin (grep, sort, wc). Use xargs when the next command wants arguments (rm, cp, mkdir, grep pattern FILE). These two lines show the difference:
echo "logs/app2.log" | cat # prints the name (pipe = data)
echo "logs/app2.log" | xargs cat # prints ERROR net (xargs = arguments)By default, xargs splits its input on blanks and newlines, so filenames with spaces break. Here's that in action:
mkdir logs
printf 'ok\nERROR disk\n' > "logs/app one.log"; printf 'ERROR net\n' > logs/app2.log
find logs -name '*.log' | xargs grep -c ERROR # breaks: grep: logs/app: No such file..., exit 123
find logs -name '*.log' -print0 | xargs -0 grep -c ERROR # logs/app one.log:1, logs/app2.log:1
find logs -name '*.log' | xargs -d '\n' grep -c ERROR # works (GNU only)
find logs -name '*.log' -print0 | xargs -0 -I {} echo "file: {}"
printf '%s\n' a b c d e | xargs -n 2 echo # a b / c d / e
printf '' | xargs echo "ran anyway:" # GNU runs once with no input
printf '' | xargs -r echo "should not run" # nothing
printf '2\n2\n2\n' | xargs -n 1 sleep # 6s
printf '2\n2\n2\n' | xargs -n 1 -P 3 sleep # 2-3sTested with findutils 4.10.0 (Git Bash) and 4.9.0 (Ubuntu). -print0 | xargs -0 is the safe default, and both halves are in POSIX since 2024. -r stops GNU xargs running the command once with no input. -P 3 brought the three sleeps down from 6 seconds to 2-3. Always pair -P with -n or -L; otherwise xargs may make just one call.
A pitfall: piping a file list into xargs rm
You'll see this pattern a lot. Here's what it does when one filename has a space:
touch "sample 1.txt" sample2.txt; printf '%s\n' "sample 1.txt" sample2.txt > files.txt
cat files.txt | xargs rm
# rm: cannot remove 'sample': No such file or directory
# rm: cannot remove '1.txt': No such file or directory (exit 123)sample2.txt is deleted and sample 1.txt survives. If a name had split into a word that matched some other file, that file would be the one deleted. Use newline-delimited input instead, and skip the cat as well:
xargs -d '\n' rm -- < files.txt # untested as written: verified against the findutils manual
tr '\n' '\0' < files.txt | xargs -0 rm -- # untested: verified against the findutils manualI tested -d '\n' itself in the previous block. -- stops a filename that starts with - from being read as an option.
Combine Linux commands in one line: ;, && and ||
mkdir d1 ; echo "after ; runs"
mkdir d1 2>/dev/null && echo "not printed" || echo "mkdir failed, || ran"
mkdir d2 && cd d2 && pwdThis prints after ; runs, then mkdir failed, || ran, then the path of d2. && and || have equal precedence and are evaluated left to right, so a && b || c is not if/else. c also runs when b fails. When that matters, use a real if.
Command substitution and process substitution
echo "Today is $(date +%F)"
echo "nested: $(basename "$(pwd)")"
printf 'b\na\nc\n' > a.txt; printf 'c\nb\nd\n' > b.txt
diff <(sort a.txt) <(sort b.txt) # 1d0 / < a / 3a3 / > d ; exit 1
echo <(true) # /dev/fd/63
x=1; y=${ x=2; echo out; }; echo "y=$y x=$x" # bash 5.3+: y=out x=2$(...) drops in the command's output as text, without trailing newlines. It nests cleanly, which backticks don't. <(...) drops in a filename (/dev/fd/63), so you can give diff two sorted streams without creating temporary files. There must be no space between < and (. The last line needs Bash 5.3 (I tested it on 5.3.20): ${ cmd; } runs in the current shell, so the change to x sticks. With $( ) it would be lost.
Running pipelines in the background
set -m # (only needed in scripts; interactive shells have job control)
sleep 30 & sleep 31 &
jobs # [1]- Running sleep 30 & / [2]+ Running sleep 31 &
kill %2
disown %1 # removed from job table; keeps running, won't get SIGHUP
nohup ./long.sh > long.log 2>&1 & # no nohup.out when you redirect
false & echo $? # 0 ; wait $! -> 1Tested on Ubuntu. Git Bash's job control was unreliable (disown %1 answered no such job), so practise this part in WSL or Linux. A bare nohup cmd on a terminal prints nohup: ignoring input and appending output to 'nohup.out'. Redirect it yourself and you choose the log file. & always returns 0, so use wait $! to get the real exit status. For long jobs over SSH, a terminal multiplexer such as tmux (3.7c is current) lets you reattach later. I haven't tested tmux here, and the same goes for disown -h, which keeps the job in the table but stops SIGHUP from reaching it.
On Windows: WSL, PowerShell and cmd.exe
Linux pipes and redirection work unchanged in a WSL distro, and nearly everything above also runs in Git Bash. PowerShell is different:
| Bash (WSL, Git Bash) | PowerShell | cmd.exe | |
|---|---|---|---|
| Discard | 2>/dev/null | 2>$null | 2>NUL |
2>&1 > file | errors stay on screen | both captured | errors stay on screen |
| All streams | &> file | *> file (streams 1-6) | > file 2>&1 (untested; same order rule as Bash) |
Input < | works | The '<' operator is reserved for future use. | works |
&& / || | works | PowerShell 7+ only; 5.1 gives a parse error | works |
| What a pipe carries | bytes | .NET objects (e.g. System.IO.FileInfo) | bytes |
Get-ChildItem exists.txt, missing.txt 2>&1 > both1.txt # BOTH captured (unlike bash)
Get-ChildItem missing.txt *> all.txt # all streams
Get-ChildItem missing.txt 2>$null # silent; $? = False
"hi" > out.txt # 5.1: FF FE 68 00 69 00 (UTF-16LE BOM); 7.4: 68 69 0A (UTF-8 no BOM)Tested on Windows PowerShell 5.1 and PowerShell 7.4.2. I didn't test 7.6.6, but the about_Redirection docs describe the same behaviour. The encoding line matters: the 5.1 docs page says > writes UTF-8 without a BOM, but 5.1 actually wrote UTF-16LE with a BOM. Files like that can break Linux tools that read them later.
Troubleshooting: messages and fixes
| Symptom | Cause | Fix |
|---|---|---|
Errors still print after > file | > only moves stdout | > file 2>&1 or &> file |
Errors on screen with 2>&1 > file | Redirection order | Put 2>&1 last |
A file called less appears | > used instead of | | cmd | less |
cannot overwrite existing file | noclobber is on | >| to force, or >> |
Permission denied with sudo echo > | Your shell opens the file | | sudo tee -a file > /dev/null |
| Pipeline exits 0 though grep failed | Status of last command only | set -o pipefail, check PIPESTATUS |
Exit 141 under pipefail | SIGPIPE from head or grep -q | Expect it, or don't use pipefail for that pipeline |
Illegal option -o pipefail | Script run by dash | #!/usr/bin/env bash |
&> leaves an empty file, or Syntax error: "&" unexpected | dash treats &> as & + > and rejects |& | Use Bash, or > f 2>&1 |
uniq -c counts are fragmented | Input not sorted | sort | uniq -c |
xargs exit 123, cannot remove 'sample' | Spaces split names | -print0 | xargs -0 or -d '\n' |
| xargs exit 127 | Command not found | Check the command name and PATH |
2>/dev/null |& still passes errors | |& applies its 2>&1 last | Use plain | |
FAQ
How do I redirect both stderr and stdout to a file?
cmd > file 2>&1 works in any POSIX shell. In Bash, cmd &> file does the same. To append, use >> file 2>&1 or &>> file. Don't write 2>&1 > file, which captures only stdout.
Why does the order of 2>&1 matter?
Redirections run left to right, and 2>&1 copies wherever fd 1 points at that moment. PowerShell is the exception: there 2>&1 > file captures both.
When do I need xargs instead of a pipe?
When the next command takes filenames or other values as arguments rather than reading stdin. For example, rm ignores stdin completely. Use find ... -print0 | xargs -0 so that spaces in names can't break it.
How do I make a pipeline fail if any command fails?
Run set -o pipefail in a Bash script, then read "${PIPESTATUS[@]}" immediately to see which stage failed. It won't work under #!/bin/sh on Ubuntu 24.04.
How do I write to a root-owned file with sudo?
echo "line" | sudo tee -a /etc/file > /dev/null, or sudo sh -c 'echo line >> /etc/file'.
Where to practise
The fastest way to make Linux pipes and redirection automatic is to solve puzzles that need them. The OverTheWire Bandit command line walkthrough uses sort | uniq, grep and 2>/dev/null constantly. If you only keep three Linux command line tips and tricks from this page, keep these: put 2>&1 last, start scripts with set -o pipefail, and give xargs null-delimited input. The full reference is the GNU Bash manual.
Last updated: 2026-10-06; tested on Git Bash 5.2.37 (Windows 11), Ubuntu 24.04.5 with bash 5.2.21, and bash 5.3.20; Windows parts on PowerShell 5.1 and 7.4.2.