"lts" version is older, more stable than "stable", for Long Term Support.

arch-chroot is a wrapper for chroot that mounts /proc, etc, sets up resolv.conf.
Its part of arch-install-scripts package and is in rescue images.

systemd
    App confile files usually /etc/*.conf.
    systemctl controls (systemd) services
        For listing, --no-pager switch seems to leave things very concise
          ellipsizing but stops non-elipsize truncation.  Maybe not?
        To eliminates annoying pause at bottom just don't user "less" as pager.
          export SYSTEMD_EDITOR=more
        systemctl status
        systemctl [mask|stop|start|restart|reload|status|enable|disable|is-enabled] unit
        Can use wildcards like: systemctl status '*.mount'
        For services you can identify as S or S.service (i.e. ".service" suffix
         is optional); for targets must say S.target.
        To restart network:  systemctl restart systemd-networkd
          (but for some reason sometimes the network iface (link) is down
          and you'll need to: ip link set wlp1s0 up
    Unit (and service?) definition files at
      /etc/systemd/system (admined incl. custom!) and /usr/lib/systemd/system  (package-managed).
      (/lib -> /usr/lib)
      Great reference:
        https://www.freedesktop.org/software/systemd/man/systemd.service.html
      With type simple can have one each ExecStartPre, ExecStart, ExecStartPost
        command which run synchronously as you would expect.
      NOT SHELL COMMMANDS or ENV operation.
      Can set vars with: Environment=N1='val 1' N2='val 2'.  Ref like ${this}.
      Only with type 'oneshot' can have multiple ExecStarts that run
       synchronously as you would want.
        Multiple` ExecStarts is entirely equiv to multiple " ; "-delimited
        commands in a single ExecStart.
      Single and double quoting work to pass more than one token as an argv.
        (TODO: verify that ${expansion} work both).
         But quotes must start and end with surrounding whitespace: " 'thing' "
      Does not support compound commands.  Workaround is to leverage quoting
        and shell -c:  sh -c 'dmesg | tac'
      StandardOutput=append:/path or truncate:\path work but
        ALL ExecStart* writings overwrite one another.  Worse than rewriting.
        By default stderr will go there too, but can specify StandardError separately.
      To keep something running, use Restart and RestartSec settings.
      To run something stateless like you would have thrown into /etc/rcinit or whatever that was,
        create a script like mine at ~/code-templates/archLinux/update-route53.service.
      GOTCHA!:  Remember after edit unit scripts:  systemctl daemon-reload
        For troubleshooting, it's useful to disable and reenable the service.
    To override an OOTB script, make a new file (or copy OOTB) of same name
      at /etc/systemd/system, then: sytemctl reenable <UNIT>
    SOME enabled script sym links at /etc/systemd/system/*.wants/*
      These get created/removed by execution of: systemctl {enable|disable}
      (/usr/lib/systemd/system/*.wants/* do not correspond to enabled units/services)
      But I do have 3 enabled services that do not have links anywhere that I can find:
        autovt@.service getty@.service sddm.service
    targets ~ runlevels
      Determine active:
      systemctl list-units --type=target  # Doesn't work for me.
                                          # THEY ALL SAY ACTIVE!!!
        Try this:  journalctl -b | grep "Reached target'
      Change active:
      systemctl isolate multi-user.target|graphical.target
      Change default:
      systemctl set-default [-f] multi-user.target|graphical.target
    systemd timers is a superset of cron and at functionality.
        "systemctl list-timers"
        *.timer files invoke same-basename *.service files.
        "systemd-run --on-active=... {command|--unit x.service}" like 'at'.
        Cron-like schedules by *.timer files with "OnCalendar=" setting.
        Monotonic timers is other type and have "OnTypeSec=" setting.
        Enable/start/etc. like *.services but must specify .timer suffix.
        OnCalendars schedule specifications.
            Can have multiple in a single *.timer file.
            Delimiters:  - then " " then : FOR SEGMENTS
                *==any, ..==contiguous range
            [DayOfWeek [[Year-Month-Day]] HH:MM[:SS[.subsec]]
            You usually only need to specify one value type and all others
             will default to * for less-specifics and 0s for more-specifics.
            There are shorthand values "*ly" that default the to 0s for
              the required sub-resolution literals.
    Journal replaces syslog:
        Can "strings" the binary log files, but I can't find them.
        Since last boot:  journalctl -[x]b
        -x:  Option adds message catalog context messages
        Tail:  journalctl -n | journalctl --lines=20
        Tail -f (implies -n too):  journalctl -f
        Write a log msg:  echo msg | systemd-cat -t yourIdentifier
        Purge unnecessary files:
          journalctl --vacuum-time=2d
          OR
          journalctl --vacuum-size=500M
    Customize an OOTB until
        Check customizations:  systemd-delta
        Modify preferred:
            add addition/override files /etc/systemd/system/<UNIT>.d/*.conf
            For non-oneshot settings, to replace instead of add to, must first:
                ExistingVar=
        Or replace:
            Add a /etc/systemd/system/* to override /usr/lib/systemd/system/*.[:w
        + systemctl daemon-reload
        Check on it:  systemctl cat <UNIT>

    Stopping tmpfs mounting:  systemctl mask tmp.mount
    Statuses:  enabled:  explicitly enabled;
               static: enabled by dependency;  (Can only 'mask' these, not disable them)
               indirect???  (only 4);
    ? "oneshot" type setting means scalar.  Otherwise list variable that takes
      a multiple settings.

virtual consoles / X switching
    CTRL+ALT+#  ==  chvt # (from shell)
    Session 1 is X

pacman  FOR INSTALLED OR INSTALLING PACKAGES
    Install yay manually from AUR (instructions elsewhere in this file), then do all system updates
      with yay instead of pacman:  yay --noconfirm -Syu
    package groups vs. meta package.
    metas update more dynamically, but they do not allow changes to package
      components (can't exclude anything).
    RM:  pacman --noconfirm -R[s] name  # -s + deps. only of this (safely)
    ADD: pacman --noconfirm -S name
    FORFILE: pacman -Q[q]o file/path  (*INSTALLED* pkg providing/containing)
    FILES: pacman -Q[q]l name  # seems no filtering option like rpm's for 
                               # documentation or executable files.
    All QUERY variants:  -q option switch makes quiet to skip version label
    QUERY available:  pacman -Ss[q] 'rePattern'  # Matched against name+descript
      available from a repos:  pacman -Sl[q] repo
      or:  paclist repo   # requires pacman-contrib package
    QUERY installed:  pacman -Q[q] preciseName   OR  pacman -Q[q]s patterns...
      limit to foreign or native (correspondingly):  pacman -Q[qmn]
    Available groups:  https://www.archlinux.org/groups/
    (As of Nov 2016, need workaround before -Syu will succeed:
        pacman -S --force ttf-dejavu   )
    List packages in group:  pacman -Q[q]g groupName
      ('pacman -Sg groupName' and 'pacman -Sgg" not working.  Perhaps only see
      native packages?)
    Update all:  yay --noconfirm -Syu
      If get *invalid or corrupted package* failure or 'singature from... is marginal trust',
      run this then retry:
        sudo pacman -Sy archlinux-keyring
    Upgrade single package:  pacman --noconfirm -S <PKG>
    [DUBIOUS:  Warns that there are real dependencies!
    Remove unused package dependencies:
        pacman -Qdt > /tmp/pacman.orphans
        gvim /tmp/pacman.orphans  # Remove keepers like zip, remove ' .+'
        pacman -Rsn $(</tmp/pacman.orphans)]
    Purge unnecessary files from, /var/cache/pacman/pkg/ and repos from /var/lib/pacman:
        # Be aware that this deletes even very last so can't do rollbacks!
        pacman --noconfirm -Sc
    If pacman operation gets interrupted and you get "unable to lock database",
     just remove file /var/lib/pacman/db.lck.
    You don't want to rebuild the database unless can't avoid it.  It's a very
     complicated and probably unreliable task.
    DOWNGRADING, i.e. backing out updates.  Just:  pacman -U older/pkg.tar.x
        Can check install dates with 'pacman -Qi | perl...' to find packages
        updated in certain time span.
    -F [-x] /file/path/substr  # Available.  -x interprets pattern as a regexp
        -Fy  # update the -F database
     -Qdtq  lists orphaned packages
    Resetting when get HTTP 401 errors and/or signature validation failures:
        https://bbs.archlinux.org/viewtopic.php?id=244976
        # gpg --refresh-keys
        # pacman-key --init && pacman-key --populate archlinux
        # pacman-key --refresh-keys
        # pacman -S archlinux-keyring
pactree lists dependencies of specified package

pkgfile FOR DETAILS OF NOT-INSTALLED packages
(similar to "pacman -F..."; use pacman -Ss to query INSTALLED packages).
    pkgfile -u  # as root
    pkgfile fileBase.name  # without path.  Or...
    pkgfile /absolute/path/to/file.name
Does not work for filename patterns or substrings.

expac
list packages by installation date:  expac --timefmt='%Y-%m-%d %T' '%l\t%n' | sort -n

yay installation (for managing AUR packages):
sudo pacman -S --needed base-devel git  # get dependencies
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si

AUR Manual installs:
Install 'yay' this way.
If get errors about GPG keys, allow then like:
  gpg --keyserver keyserver.ubuntu.com --recv-keys THEKEY
clone the source via: git clone URL
  Chrome:  https://aur.archlinux.org/google-chrome.git
  (If Git source not available then acquire *.tar.gz file and extract it).
cd into dir
execute: makepkg --noconfirm -sirc   # don't use --noconfirm if this will
  # replace a package because in that case the default answer is to NOT replace.
When done, just remove entire build dir since next time makepkg will have to
 rebuild everything from scratch regardless.

SDDM replaces KDDM
Can't log in as root (or any other system account) by default.

netctl  (good for wireless)
    Works with config files that you can name arbitrarily in /etc/netctl/.
        Can copy in examples from /etc/netctl/examples/*.
    Do sometimes need to run: ip link set wlp1s0 up
    wifi-menu -o
    netctl enable|disable|reenable|status|start <PROFILE>
      (Should only effect sym links in /etc/systemd/system/, but I find things
      don't work sometimes after changes unless I reenable).
    For dynamically changing at runtime need to enable asystemd units.
      See https://wiki.archlinux.org/index.php/Netctl#Special_systemd_units
    Troubleshoot netctl (though this has never been any help to me):
        journal -xn
        netctl status <PROFILE>

systemd-networkd  (good for ethernet)
https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdNetworkdSetup
    SETUP
        Enable and start:  systemd-networkd
    Config files:
      To reload config files do: systemctl restart systemd-networkd
      * /etc/systemd/network/  HIGHEST PRIORITY
        See my sample file ~/sharedhome/vmtemplates/admc/etc_systemd_network/*.
      * /usr/lib/systemd/network/
      * /usr/local/lib/systemd/network/
      * /run/systemd/network/  [ephemeral.  Present on some hosts!]
    These files allow #comments but only for entire lines.
    *.network  Match devices
    *.netdev  Create virtual devices matching an env.
    *.link  When a network device appears, udev looks up first matching .link file
    All files search for matching [Match] key/value criteria (default do match)
    IP aliases by just having multiple Address assignments in *.network files.
      LAST addr is one used for source addr by default.
    Check .../network/* file updates with:
      systemctl enable systemd-networkd
      systemd-resolve --status

systemd-resolved    I see no advantage.  Leave it off and use /etc/resolv.conf.
    I don't know how to really disable that system, so live with it:
    Use "resolvconf" system (man resolvconf and man resolvconf.conf).
    Add additional nameservers to "/etc/resolvconf.conf".
    Run "resolvconf -u" to regenerate /etc/resolv.conf, then check it.
    *.network  Match devices  ?????????
        Needs to have Domains=dom1,dom2  for normal resolv.conf
        (?? that is not format for the network file, which requires multiple
        "Domains=x" lines).
        Domains settings in *.network files is documented to be space-separated!
    !! I CAN NOT GET /etc/resolv.conf to update now!!!
       I update /etc/systemd/network/*.network with DNS settings,
       I restart systemd-networkd, run "resolvconf -u", and no change  to
        /etc/resolv.conf.
    resolvectl status   is useful
    resolvectl query a.name    # like "host" or "nslookup"
    I edited Domain= in /etc/systemd/resolved.conf and it had no effect
      on "host" or "resolvectl query" lookups.  If you edit the /etc/resolv.conf
      symc link then it just gets overwritten.
      Fuck it!  Just replace /etc/resolv.conf link with real copy and edit that!
    I can't get domain lookups configured sanely.  Somehow admc.com is now
     search domain on beyla (through /etc/resolv.conf) but I don't see it
     configured anywhere and /etc/resolv.conf never updates even when
     systemd-resolvd is running.  On my EC2 instance, whatever I set in
     /etc/systemd/resolvd.conf gets added right to /etc/resolv.conf even though
     [DHCP]UseDOmains=yes.


ip (definitely use for ethernet nics)
    ip link|addr|route ...
    ip addr  ~ ifconfig -a
    ip link set <IFACE> down
    ip addr add 192.168.101.11/24 dev enp2s6
    ip route add default via 192.168.100.1 dev enp2s6
    ip route del default via 192.168.101.246 dev enp2s25

Firewall
    ufw simple Firewall.  https://gist.github.com/kimus/9315140
    https://wiki.archlinux.org/index.php/simple_stateful_firewall
    https://wiki.archlinux.org/index.php/Internet_sharing

Network Device Renaming
I think only occurs with dhcp clients.
Controlled by /etc/udev/rules.d/*
Can also do with *.link or *.netdev files from systemd.networkd.

netstat replacements
ss (from iproute2 package)
  args very close for most things (but not for -r).
  This arg combo the same:  -lnt
ip route show
  replacement for "netstat -r"

lsblk -o SIZE,NAME,FSTYPE,LABEL,MOUNTPOINT  (I have alias "lsb")
Great alternative to "df" (if you don't need to know USAGE) and "mnt"


Team Games
Don't even try 3D games without an Intel GPU.
Enable multilib:
    /etc/pacman.conf uncomment:
        [multilib]
        Include = /etc/pacman.d/mirrorlist
    pacman --noconfirm -Syu
Install steam:  pacman -S steam  #  select lib32-mesa-libgl 
pacman --noconfirm -S lib32-libxt lib32-libxrandr lib32-curl
+ all the libs listed at
 https://wiki.archlinux.org/index.php/Steam/Troubleshooting#Libraries_for_x86_64
then start like:
    LD_PRELOAD='/usr/$LIB/libstdc++.so.6 /usr/$LIB/libgcc_s.so.1 /usr/$LIB/libxcb.so.1 /usr/$LIB/libgpg-error.so' /usr/bin/steam
  (paste this command from
  https://wiki.archlinux.org/index.php/Steam/Troubleshooting#Dynamic_linker )

Installing packages from pacman db /var/cache/pacman/pkg.
TO have yaourt  retain /var/cache/pacman/pkg/*, etc /etc/yaourtrc:
    EXPORT=X   # 2 to save /var/cache/pacman/pkg/*, default (0) to not retain.
After /var/cache/pacman/pkg/* and /etc/pacman.d/mirrorlist as desired:
    xargs -a Packages pacman -S --noconfirm --needed

Two ways to start ssh:
Normal:  sshd.socket
If especially high traffic:  sshd.service


Printing from vim.
Vim apparently uses the lp* system.
Even after Cups is working perfectly and has a default printer set,
lpr, etc. don't know about default and vim gives crazy PDF printing errors.
Set the default:
    lpstat -p  # determine print queues
    lpoptions -d <A_QUEUE>

Microphone:
Determine max sample rate for the hardware according to
https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting#Determine_sampling_rate_of_the_sound_card_.282.2F5.29
For the arecord/aplay commands, I have to specify the device like "-D hw:1"
Test with audacity.  Set audacity devices to pulse.

Commercial (VMware) VMware tools don't work.
Must use package open-vm-tools instead.  Service is 'vmtoolsd'.

named
For jail, I had to add this in addition to /srv/named/dev/random:
    mknod -m 0644 /srv/named/dev/urandom c 1 9
I also installed havegd and set a special VMware setting *.vmx setting but that
only moved randomness up to 222 entropy so don't know if this was necessary.

emergency recovery/restore
Boot on installer image, mount root partition, execute arch-chroot.
You DO NO NEED to set up /proc, /run filesystems, etc. because arch-chroot
does that for you.

Wireless
Beware some cards only use one MHz "band".  (Card now in beyla only does 2.4).
Follow http://wiki.archlinux.org/index.php/Wpa_supplicant
To avoid having to give switches to wpa_client,
    a) Use default socket location of /var/run/wpa_supplicant AND
    b) We'll assume we want to work with first wireless adapter found.
systemctl [status|start|enable] wpa_supplicant@<IF>.service
For security, comment out the '#psk=' lines in the *.conf file.
(If you go by other guides, don't depend on output of wpa_client scan
before running scan_results).
For some reason I have to work with network id 1 instead of 0 which
 guides and HOWTOs I find all use.
/etc/systemd/network/*.network files seem to be unnecessary FOR ME
 (but this likely due to 'Wireless and USB Tethering' issue described below.
My wireless cards are seen by 'lspci -v' as: 
  SMC2602W: SMC 2602 (IIRC)   (11 Mbps)
  TL-WN851ND:  Qualcomm Atheros AR9227  (300 Mbps)
Could be coincidence, but when I booted up with SMC card with a corded antenna
  I got mostly failures with speed-test attempts.
Device name may come from PCI slot used.

Wireless and USB Tethering Issue
USB tethering works perfectly as documented in ArchLinux docs.
  Device name comes from USB port used.  Documented tactic on giving static name.
My setup for eth device sets a gateway.  Don't want this route working when
doing tethering so I have set the ethernet default route metric > 1024.
I'll test that when I get new wireless adapter next week.
If that doesn't work then will have to disable the Gateway= setting in the
eth *.network file while doing tethering.
FYI '[DHCPv4] UseGateway=false' isn't working for me (but I don't need it).

Motherboard info available by
lspci -v
dmidecode  (under "Base Board Information")

VIM Config
I don't know what the fucks is up with vim setup of latest ArchLinux.
Directories.  /etc/vimrc somehow loads /usr/hare/vim/vimfiles/archlinux.vim
and that sets the backup directories to ~/.cache/vim/*, but it doesn't
set backup; and that's behavior seen with new accounts.
/usr/share/vim/vim80/vimrc_example.vim (sourced by ~/.vimrc) sets backup
conditionally but I don't understand the criteria.
Looks like ~/.vimrc not being honored at all or the backup settings there
are ineffective.

Kernel switch
Can install and remove kernels by simply installing/removing linux* packages.
    pacman --noconfirm -{S|R} linux*
Make sure to do this to upgrade grub:  grub-mkconfig -o /boot/grub/grub.cfg
See 'Which Kernel To Boot' section in this file if you have multiple kernels
  installed.
For some damned reason, the linux-ec2-lts kernel's preset config
  does not have standard setup to generate a initramfs fallback image in /boot.

Which Kernel To Boot
Kernels identified at both places below according to 0-based 'menentry' indexes
  from /boot/grub/grub.cfg.
Use grub-set-default to specify kernel to boot in /boot/grub/grubenv. NO EFFECT!
GOTCHA:  grub-set-default / grubenv does not work for me.
Why not /etc/default/grub GRUB_DEFAULT followed by:
    grub-mkconfig -o /boot/grub/grub.cfg

OpenJDK
java-rhono for some JavaScript support from the JDK.
As of 2020-11-11 ArchLinux officially supports only versions 7, 8, 11, 14.
"archlinux-java help" documents how to change default version.


Screen shot from CLI:  import outputfile.png  # then click crosshair on window
import -window root fullscreen.png
Useful optional switches:  -border -frame -pause secs 
(-border looks useful but browser windows, at least, have no border).

On ArchLinux for Gimp screen shots to work, just run like:
    dbus-launch gimp ...

If during pacman installs or updates you get failures like
    error:  X: signorea from "something" is unknown trust
    ... invalid or corrupted package)
then run some of these commands:
  gpg --keyserver keyserver.ubuntu.com --recv-keys THEKEY
  pacman-key --refresh-keys
  pacman --noconfirm -Sy archlinux-keyring && pacman --noconfirm -Syu

For some reason specify FS type of NTFS partitions as ntfs3.
This can't be autodetermined, so you must specify in mount commands and in fstab.

To update /etc/pacman/mirrorlist
If no /usr/bin/reflector then:  pacman --noconfirm -S reflector
reflector --country US --protocol https,http --score 20 --sort rate --save /etc/pacman.d/mirrorlist

Purging /var disk space
See 'pacman' section above about those artifacts.
Can wipe the password-attempt store file /var/log/btmp
Can vacuum journalctl logs (many files) via time or size:
    journalctl --vacuum-time=2d
    journalctl --vacuum-size=500M

Visibility of new disk devices on ArchLinux, without rebooting.  Use lsblk to see available devs.
    Archlinux doesn't have the /sys/.../*scsi* dirs, so do:
      udevadm trigger
      udevadm settle