Welcome to TiddlyWiki created by Jeremy Ruston; Copyright © 2004-2007 Jeremy Ruston, Copyright © 2007-2011 UnaMesa Association
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="700" height="460" poster="" data-setup="{}">
<source src="video/putty.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
{{Warning{Note: The connection port shown in the video must be changed from {{Monospaced{22}}} to {{Monospaced{2205}}}. }}}
<html>
<table border=0>
<tr border=0><TD align=center border=0>
<iframe src="https://www.google.com/calendar/embed?src=cl5tpjhd9f3bj9c52idttghtsg%40group.calendar.google.com&ctz=America/New_York" style="border: 0" width="1000" height="600" frameborder="0" scrolling="no"></iframe>
<BR><B>This calendar is authoritative should there be due date conflicts with other pages on the site</B>
</TD></TR>
</table>
</html>
A traditional class meeting on campus naturally allows for regular communication. This is beneficial by helping students better understand the material along with allowing the instructor to more easily gauge how everyone is doing in the class.
I would like to ensure hosting this course online does not deprive us of regular communication. Class participation will be worth 10% of your overall grade. Each week's class participation will be worth ''50 points'' total. Multiple posts each week will be necessary to receive full credit.
Posts for each week must be made by Sunday, 11:55 p.m. (EST) the following week in order to receive full credit. This allows one week to post questions about outstanding assignments and one week to post questions about labs after they have been returned. Organization is important - __please post lab questions on the discussion board for the week they were assigned__.
Joining group Zoom meetings will also earn class participation credit.
!! Participation:
You may ask questions, work collaboratively on assignments, and provide assistance to one another in Discord. You can also provide ideas or helpful resources that assisted you on your assignments. Credit may also be received for joining or participating in either regularly scheduled or ad-hoc group Zoom meetings.
!! Rubric for weekly class participation:
* 25 points - Actively participate in a group Zoom meeting
* 10-20 points - High quality posts which contain well-developed questions, actionable suggestions, or recommendations
* 15 points - Attend a group Zoom meeting
* 5-10 points - General comments regarding the assignments. No specific insights directly related to the problem or responses to questions which are not actionable.
!! Quality of Remarks:
You will be evaluated based on the quality of your participation by asking questions, providing constructive assistance, making recommendations related to our material, and making pertinent comments.
The discussion forum and Zoom meetings are a valuable component of learning since they allow you to see a variety of solutions and ideas just like you would in a classroom.
Generally, please do not post direct solutions to lab questions, especially unsolicited, before their due date. Doing so will not be awarded participation points. If someone is genuinely stuck on a problem and you'd like to help, guidance towards the solution is always a more beneficial place to start rather then just posting the answer. If you just post the answer, I cannot tell if someone understands the problem or simply copied your solution.
Please be sure to check out the [[Using Discord]] page to see more useful information.
!! Adding New Threads
Good organization is important. Create new threads in the weekly Discord channels in which the material was assigned. When naming your threads, use something descriptive in the name and not just the lab and question number. The highlighted thread is a model to follow and will make things easier to find as the number of posts grows. Be sure to scan for an existing thread relating to your topic before creating a new one. Usability is an important consideration in what you do. ''Not using descriptive thread titles is detrimental to usability, so that post will not receive full credit.''
[img[img/discussionBoards.png]]
(yes, this screenshot is from Blackboard, but the point still stands. The [[Using Discord]] page has more detail about Discord threads.)
Our class utilizes a [[DigitalOcean|https://www.digitalocean.com/products/droplets/]] droplet for the first half and a [[Hetzner bare-metal auction|https://www.hetzner.com/sb?ram_from=8&ram_to=10&ssd=true&cpu_from=8000&cpu_to=30000&price_from=50&price_to=90&search=Xeon+E5]] server to support the student lab environment during the second half of the semester.
!! ~DigitalOcean droplet
Our needs are very minimal for the first half of the semester; we only require a Linux shell server everyone can access to practice the commands and submit their work. ~CentOS is used as our Linux distribution.
A $5 per month ~DigitalOcean droplet (virtual machine) is more than enough. ~DigitalOcean droplets are great for small projects like this. We'll use it for two months then take a snapshot and destroy the droplet to save money. It'll then be brought back from the snapshot when it's needed again for the next semester.
!! Hetzner Bare-metal
About halfway through the semester we'll switch from being Unix users to administrators. Each student will be given a small collection of virtual machines to install and configure. Additional resources are required for this portion of the class since each student will require about 6 ~VMs. Instead of just a single VM to cover the entire class, we'll now need a full server. Hetzner auction servers have been a reliable, low-cost option for such short-term needs.
To provide enough resources for the entire class, I'll be looking for a server with the following minimum specs:
* 64gb RAM
* CPU with at least 6 cores at 3.4GHz. I'm currently using the ~E5-1650V2.
* 2x 256gb SSD (SSD drives are important for disk speeds. SATA are too slow)
A server with these specs comes to about 60,00 € per month. We'll need it for 2 months. The total infrastructure cost for this class per-semester is then about $150.
The server is initially provisioned by Hetzner with Debian Linux and [[Proxmox|https://www.proxmox.com/en/]] is then installed to act as our hypervisor. Proxmox runs on Debian and can either be installed from [[its own CD image|https://www.proxmox.com/en/downloads]] or the [[packages can be installed on an existing Debian system|https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster]]. We'll use the latter option here since Hetzner must install the original OS for us and they do not have an option for Proxmox.
After the Hetzner server and Proxmox are installed, ~VMs will be created for our class shell server and for internal monitoring. The class shell server will then be migrated from ~DigitalOcean. Student ~VMs will be created from templates.
At the end of the semester everyone will be given the option to download their ~VMs for use locally with ~VirtualBox. Student ~VMs will then be deleted, administrative ~VMs will be backed up to ~BackBlaze B2 storage for next time, then the server contract will be ended.
Other tools/services used:
* [[Fossil source code manager|https://fossil-scm.org/home/doc/trunk/www/index.wiki]] - Used to handle revision control for server configuration files and scripts
* [[SaltStack Infrastructure management|https://docs.saltproject.io/en/latest/]] - Used to orchestrate VM templates and manage infrastructure monitoring
* [[Naemon monitoring suite|https://www.naemon.org/]] - Used to monitor student ~VMs and provide feedback on completed/outstanding tasks
* [[Docker containerization|https://www.docker.com/]] - Used to rapidly deploy and isolate different services on the same VM in a way that can be easily repeated.
* [[BackBlaze B2 cloud storage|https://www.backblaze.com/b2/cloud-storage.html]] - Used to store management ~VMs and VM templates between semesters. Storage here costs $0.005 per Gb.
The combination of these tools allow for the Hetzner server to be quickly brought online when needed for the new semester, it's ~VMs and templates downloaded from ~BackBlaze B2 storage, and made ready to support our class.
Useful concepts:
* [[Infrastructure as code|https://en.wikipedia.org/wiki/Infrastructure_as_code]] - Rapidly provision servers, ~VMs, and Docker containers for individual services using ~APIs & orchestration tools with pre-made definition files instead of manually. Using this concept, our class lab server is brought up from bare metal to fully online and ready to support users in about 30 minutes with just a handful of commands.
/***
!! CollapseTiddlersPlugin
^^Author: Bradley Meck^^
^^Source: http://gensoft.revhost.net/Collapse.html^^
|ELS 2/24/2006: added fallback to "CollapsedTemplate if "WebCollapsedTemplate" is not found |
|ELS 2/6/2006: added check for 'readOnly' flag to use alternative "WebCollapsedTemplate" |
***/
config.commands.collapseTiddler = {
text: "fold",
tooltip: "Collapse this tiddler",
handler: function(event,src,title)
{
var e = story.findContainingTiddler(src);
if(e.getAttribute("template") != config.tiddlerTemplates[DEFAULT_EDIT_TEMPLATE]){
var t = (readOnly&&store.tiddlerExists("WebCollapsedTemplate"))?"WebCollapsedTemplate":"CollapsedTemplate";
if (!store.tiddlerExists(t)) { alert("Can't find 'CollapsedTemplate'"); return; }
if(e.getAttribute("template") != t ){
e.setAttribute("oldTemplate",e.getAttribute("template"));
story.displayTiddler(null,title,t);
}
}
}
}
config.commands.expandTiddler = {
text: "unfold",
tooltip: "Expand this tiddler",
handler: function(event,src,title)
{
var e = story.findContainingTiddler(src);
story.displayTiddler(null,title,e.getAttribute("oldTemplate"));
}
}
config.macros.collapseAll = {
handler: function(place,macroName,params,wikifier,paramString,tiddler){
createTiddlyButton(place,"collapse all","",function(){
story.forEachTiddler(function(title,tiddler){
if(tiddler.getAttribute("template") != config.tiddlerTemplates[DEFAULT_EDIT_TEMPLATE])
var t = (readOnly&&store.tiddlerExists("WebCollapsedTemplate"))?"WebCollapsedTemplate":"CollapsedTemplate";
if (!store.tiddlerExists(t)) { alert("Can't find 'CollapsedTemplate'"); return; }
story.displayTiddler(null,title,t);
})})
}
}
config.macros.expandAll = {
handler: function(place,macroName,params,wikifier,paramString,tiddler){
createTiddlyButton(place,"expand all","",function(){
story.forEachTiddler(function(title,tiddler){
var t = (readOnly&&store.tiddlerExists("WebCollapsedTemplate"))?"WebCollapsedTemplate":"CollapsedTemplate";
if (!store.tiddlerExists(t)) { alert("Can't find 'CollapsedTemplate'"); return; }
if(tiddler.getAttribute("template") == t) story.displayTiddler(null,title,tiddler.getAttribute("oldTemplate"));
})})
}
}
config.commands.collapseOthers = {
text: "focus",
tooltip: "Expand this tiddler and collapse all others",
handler: function(event,src,title)
{
var e = story.findContainingTiddler(src);
story.forEachTiddler(function(title,tiddler){
if(tiddler.getAttribute("template") != config.tiddlerTemplates[DEFAULT_EDIT_TEMPLATE]){
var t = (readOnly&&store.tiddlerExists("WebCollapsedTemplate"))?"WebCollapsedTemplate":"CollapsedTemplate";
if (!store.tiddlerExists(t)) { alert("Can't find 'CollapsedTemplate'"); return; }
if (e==tiddler) t=e.getAttribute("oldTemplate");
//////////
// ELS 2006.02.22 - removed this line. if t==null, then the *current* view template, not the default "ViewTemplate", will be used.
// if (!t||!t.length) t=!readOnly?"ViewTemplate":"WebViewTemplate";
//////////
story.displayTiddler(null,title,t);
}
})
}
}
<div><div class='toolbar' macro='toolbar -closeTiddler closeOthers +editTiddler permalink references jump newHere expandTiddler collapseOthers'></div>
<div class='title' macro='view title'></div></div>
Background: #fff
Foreground: #000
PrimaryPale: #8cf
PrimaryLight: #18f
PrimaryMid: #04b
PrimaryDark: #014
SecondaryPale: #ffc
SecondaryLight: #fe8
SecondaryMid: #db4
SecondaryDark: #841
TertiaryPale: #eee
TertiaryLight: #ccc
TertiaryMid: #999
TertiaryDark: #666
Error: #f88
! Material
!! Watch:
* Difference between virtual machines and containers: https://www.youtube.com/watch?v=cjXI-yxqGTI
* Brief Docker intro: https://www.youtube.com/watch?v=_dfLOzuIg2o
! Notes
As servers become more powerful, it is increasing useful to switch from a standard server installation to some form of virtualization. Virtualization allows us to run several separate instances of an operating system, or different operating systems, on the same physical server. Consolidating what would have been separate physical servers into one allows us to save on hardware, electrical and networking resources, and physical space. A reduced hardware footprint is also easier to maintain. This consolidation has allowed for substansial decrease of the phyiscal footprint occupied by modern datacenters. What previously occupied and entire room can be reduced to a single rack.
Virtualization is supporting this class. All of your virtual servers for our material are running on a single large server in a cloud datacenter, along with many other ~VMs for other purposes. When I first took a course similar to this one many years ago, every student was assigned several physical systems to complete our work. This hardware consumed an entire lab and the overhead to maintain the hardware consumed a great deal of our time.
Containers are an alternative to this type of full virtualization, or can be used in addition to it. Containers, as the name implies, contain a set of resources and isolate them from the rest of the system. Instead of a service, like apache, having full access to all resources on the operating system its running on, running Apache within a container will limit its scope to only the resources we decide it should have.
!! Install Docker
Docker is one popular container system, where images are published and can be pulled down for use. The default version of Docker available with ~CentOS is very old. Instead, we going to add an additional yum repository and install the most recent version.
Execute the following commands to install Docker
# {{Command{ yum install -y yum-utils git }}}
# {{Command{ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo }}}
# {{Command{ yum install docker-ce docker-compose }}}
!! Start the Docker service
# Configure your system to start the docker service on boot
# Start the docker service now
!! Docker examples
After reading the chapter and reviewing the videos posted above, open the https://docker-curriculum.com/ site and work through its examples.
* Perform these steps on your test VM
* create a scratch space within {{File{ /opt/ }}} when you get to the {{Command{ git clone }}} command. This will download files to your VM.
* Stop when you reach //Docker on AWS//
! Assignment
<<tiddler [[Lab X - Containerization with Docker]]>>
/***
|Name |DarkModePlugin|
|Description|This plugin introduces "dark mode" (changes styles) and switching it by the {{{darkMode}}} macro and operating system settings|
|Documentation|https://yakovl.github.io/TiddlyWiki_DarkModePlugin/|
|Version |1.3.3|
|Author |Yakov Litvin|
|Source |https://github.com/YakovL/TiddlyWiki_DarkModePlugin/blob/master/DarkModePlugin.js|
|License |[[MIT|https://github.com/YakovL/TiddlyWiki_YL_ExtensionsCollection/blob/master/Common%20License%20(MIT)]]|
!!!Demo
<<darkMode>>
<<darkMode label:"☀️/🌘">>
!!!Syntax
{{{
<<darkMode>> (<<switchNightMode>> also works, for backward compatibility)
<<darkMode label:"☀️/🌘">>
}}}
!!!Installation
Is as usual: import or copy the plugin with the {{{systemConfig}}} tag, reload. Note: for the plugin to work correctly, you should keep its name (DarkModePlugin).
!!!Optional configuration
When the dark mode is applied, the {{{darkMode}}} class is added to the {{{html}}} element. This allows to add ''styles for dark mode'' only, like this:
{{{
.darkMode code { color:red }
code { color: green }
}}}
Ordinary styles are applied to both modes, but {{{.darkMode}}} ones have higher precedence and "overwrite" the oridinary ones.
The palette applied for the dark mode can be ''customized'' by editing ColorPaletteDark (removing it restores the default values).
!!!Additional notes
Styles of some browser interface bits (like <html><button class="button" onclick='alert("this is known as an alert")'>alert</button</html>) are only affected by OS/browser's "dark mode"/theme, so for good UI it is recommended to switch OS dark mode (DarkModePlugin will follow). For Windows users, [[switching by hotkey|https://superuser.com/a/1724237/576393]] may be useful.
The plugin ''adds extra styles'' (see ~FollowDarkMode and ~FewerColors sections) which are not yet configurable.
The option {{{chkDarkMode}}} is now ''deprecated'': later it will be either removed or re-implemented.
!!!Code
***/
//{{{
config.macros.switchNightMode = // backward compatibility
config.macros.darkMode = {
pluginName: "DarkModePlugin",
optionName: "chkDarkMode",
getDarkPaletteText: function() {
return store.getTiddlerText(this.darkPaletteTitle)
},
// this helper may become more complex for custom themes
getMainPaletteTitle: function() {
return "ColorPalette"
},
lightPaletteTitle: "ColorPaletteLight",
darkPaletteTitle: "ColorPaletteDark",
// setDark, setLight, and applyAdjustments are "governed outside": they don't check or change the cookie-parameter
setDark: function() {
var paletteTitle = this.getMainPaletteTitle()
var lightPaletteTiddler = new Tiddler(this.lightPaletteTitle)
lightPaletteTiddler.text = store.getTiddlerText(paletteTitle) || "shadow"
store.saveTiddler(lightPaletteTiddler)
var darkPaletteTiddler = new Tiddler(paletteTitle)
darkPaletteTiddler.text = this.getDarkPaletteText()
// attach the tiddler, recalc slices, invoke notifiers
store.saveTiddler(darkPaletteTiddler)
this.applyAdjustments(true)
},
setLight: function() {
var paletteTitle = this.getMainPaletteTitle()
var lightPaletteText = store.getTiddlerText(this.lightPaletteTitle)
if(!lightPaletteText || lightPaletteText === "shadow")
store.removeTiddler(paletteTitle) // to recalc slices of ColorPalette
else
store.saveTiddler(paletteTitle, paletteTitle, lightPaletteText)
store.deleteTiddler(this.lightPaletteTitle)
this.applyAdjustments(false)
},
applySectionCSS: function(sectionName) {
var sectionText = store.getRecursiveTiddlerText(this.pluginName + "##" + sectionName, "", 1)
var css = sectionText.replace(/^\s*{{{((?:.|\n)*?)}}}\s*$/, "$1")
return setStylesheet(css, sectionName)
},
applyAdjustments: function(isDarkMode) {
if(isDarkMode) {
jQuery('html').addClass('darkMode')
this.applySectionCSS("FollowDarkMode")
this.applySectionCSS("~FewerColors")
} else {
jQuery('html').removeClass('darkMode')
removeStyleSheet("FollowDarkMode")
removeStyleSheet("~FewerColors")
}
},
// "governance" methods
isDarkMode: function() {
return !!store.fetchTiddler(this.lightPaletteTitle)
},
switchMode: function() {
var me = config.macros.darkMode
config.options[me.optionName] = !config.options[me.optionName]
config.options[me.optionName] ? me.setDark() : me.setLight()
// "baking" doesn't work yet..
if(saveOption)
saveOption(me.optionName)
else
saveOptionCookie(me.optionName)
refreshColorPalette()
},
followOsMode: function(followLight) {
// old browsers may fail to detect
var isOsDarkModeDetected = window.matchMedia &&
window.matchMedia('(prefers-color-scheme: dark)').matches
if(isOsDarkModeDetected && !this.isDarkMode()) {
config.options[this.optionName] = false
this.switchMode()
}
if(!isOsDarkModeDetected && this.isDarkMode() && followLight) {
config.options[this.optionName] = true
this.switchMode()
}
},
restoreSavedMode: function() {
if(!this.isDarkMode()) return
// TODO: check if styles are really missing (avoid applying twice)
this.applyAdjustments(true)
config.options[this.optionName] = true
},
handler: function(place, macroName, params, wikifier, paramString, tiddler) {
var pParams = paramString.parseParams("anon", null, true, false, true)
var label = getParam(pParams, "label", "switch")
var tooltip = ""
createTiddlyButton(place, label, tooltip, this.switchMode, 'button darkModeSwitcher')
}
}
// We avoid using .init to support installation via SharedTiddlersPlugin, TiddlerInFilePlugin, and reinstalling via CookTiddlerPlugin.
// This also helps to avoid extra refreshing.
;(function(macro) {
// Save the palette as shadow so that one can cusomize it
config.shadowTiddlers[macro.darkPaletteTitle] =
store.getTiddlerText(macro.pluginName + "##DarkModeColorPalette")
// Set dark mode on start if OS dark mode is set or dark mode was saved previously
macro.followOsMode(false)
macro.restoreSavedMode()
// install only once
if(!config.extensions.DarkModePlugin) {
// prevent sites to ask about unsaved changes after switching mode
config.extensions.DarkModePlugin = {
orig_confirmExit: confirmExit
}
window.confirmExit = function() {
if(readOnly) return
return config.extensions.DarkModePlugin.orig_confirmExit ?
config.extensions.DarkModePlugin.orig_confirmExit() : undefined
}
// Detect OS mode change, apply
if(window.matchMedia) window.matchMedia('(prefers-color-scheme: dark)')
.addEventListener('change', function(event) { macro.followOsMode(true) })
}
})(config.macros.darkMode)
//}}}
/***
!!!FollowDarkMode
{{{
input, select, textarea {
color:[[ColorPalette::Foreground]];
background-color:[[ColorPalette::Background]];
}
.darkMode {
color-scheme: dark;
}
}}}
!!!~FewerColors
{{{
.title, h1, h2, h3, h4, h5, h6 {
color: [[ColorPalette::PrimaryDark]];
}
::selection {
background: [[ColorPalette::TertiaryMid]];
}
}}}
!!!DarkModeColorPalette
Background: #000
Foreground: #ddd
~PrimaryPale: #730
~PrimaryLight: #e70
~PrimaryMid: #fb4
~PrimaryDark: #feb
~SecondaryPale: #003
~SecondaryLight: #017
~SecondaryMid: #24b
~SecondaryDark: #7be
~TertiaryPale: #111
~TertiaryLight: #333
~TertiaryMid: #666
~TertiaryDark: #999
Error: #f44
!!!
***/
<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::EditToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class="editor">Title</div><div class='editor' macro='edit title'></div>
<div class="editor">Tags</div><div class='editor' macro='edit tags'></div><div class='editorFooter'><span macro='message views.editor.tagPrompt'></span><span macro='tagChooser'></span></div>
<div macro='annotations'></div>
<div class='editor' macro='edit text'></div>
<!--}}}-->
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="984" height="768" poster="" data-setup="{}">
<source src="video/FoxyProxy.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
!! Problem Reports:
If you have a problem, please send me a report I can work with. I need details of the problem, what you tried, steps you took to diagnose it, documentation you reviewed, screenshots, logs, etc. If you send me something vague like "//X command doesn't work//" with no supporting details, there may not be much I can do for you and I will wait for you to follow up your message with meaningful information.
The level of assistance I provide will be proportionate to your effort to troubleshoot and supply details. If you do nothing to troubleshoot and send me little information to work with, you should then expect that much effort put into a response.
!! Time management & workload expectations:
SUNY Poly, as well as most others, [[requires 42.5 hours of work per credit hour|https://www.suny.edu/sunypp/documents.cfm?doc_id=168]]. A four-credit course will thus require 170 hours over the course of our 16 week term, or 10.5 hours per week. Going to college full time is effectively a full time job. I will be expecting that time commitment each week.
Waiting until the last minute to complete, or even worse, begin, the lab assignments will not be a recipe for success. Review the tasks early so you have plenty of time to research the problems, seek help in the discussion boards, and get up to speed if you are behind on any prerequisite material.
!! Grading:
All course deliverables will be collected as PDF documents. Graded copies of these PDF documents will be returned to you containing my annotations. If you have questions regarding your grade or my comments, please contact me via email.
My grading is more traditional. Meeting the bare minimum does not yield an A. A high grade will require intellectual curiosity, problem-solving abilities, thorough responses, and good presentation. Meeting the minimum requirements with standard quality will result in a ''B''. An ''A'' grade is reserved for exceptional work.
Letter grades will be assigned as follows:
| !Percent | !Grade |
| 95% ≥ | A |
| 90% ≥ | A- |
| 87% ≥ | B+ |
| 84% ≥ | B |
| 79% ≥ | B- |
| 77% ≥ | C+ |
| 74% ≥ | C |
| 69% ≥ | C- |
| 67% ≥ | D+ |
| 63% ≥ | D |
| ≤ 62% | F |
!NCS 205 Course Notes
[[Getting Started|Week 0]] - Administrative Tasks & course intro
Aug 25 [[Week 1, Part 1]] - Unix Introduction
Aug 27 [[Week 1, Part 2]] - The filesystem
/%
Jan 27 [[Week 2, Part 1]] - Exploring the system
Jan 29 [[Week 2, Part 2]] - Manipulating Files & Directories
Feb 3 [[Week 3, Part 1]] - Links & File Globbing
Feb 5 [[Week 3, Part 2]] - Home Directories & Shell documentation
Feb 10 [[Week 4, Part 1]] - File Permissions
Feb 12 [[Week 4, Part 2]] - Streams & Redirection, Introduction to filters
Feb 17 [[Week 5, Part 1]] - Filters Continued (awk, sed, & tr)
Feb 19 [[Week 5, Part 2]] - Working with grep
Feb 24 [[Week 6, Part 1]] - Catch up and review
Feb 26 [[Week 6, Part 2]] -
Mar 3 [[Week 7, Part 1]] - I/O practice & Quoting
Mar 5 [[Week 7, Part 2]] - Process management & Job control
March 10 - 14 : Spring Break!
Mar 17 [[Week 8, Part 1]] - Substitution
Mar 19 [[Week 8, Part 2]] - Text Editors & Shell Scripting Intro
Mar 24 [[Week 9, Part 1]] - Shell Scripting
Mar 31 [[Week 10, Part 1]] - Version Control
Apr 2 [[Week 10, Part 2]] - Shell Scripting 2
Extra Material: [[Regular Expressions|Week B]] - Fits into the semester here if you'd like to review this extra-credit content.
Extra Material: [[The Environment|Week C]] - Fits about here too. Also extra-credit.
Important background material - [[Working more efficiently with GNU screen & SSH keys]] and [[Tunnels & Proxies with SSH]]
Apr 14 [[Week 12, Part 1]] - Basic networking & SSH
Apr 16 [[Week 12, Part 2]] - Expanding our systems: Working with apt, installing software from package and source
Apr 21 [[Week 13, Part 1]] - Web services
Apr 23 [[Week 13, Part 2]] - Time & Logging
Apr 28 [[Week 14, Part 1]] - DNS
Apr 30 Week 14, Part 2 - DNS Continued
May 5 [[Week 15, Part 1]] - Crypto, Securing communications, & Scheduled tasks
%/
/% May 7 - [[Wrapping up]] %/
!!!Agendas for pages in italics are tentative
/%
Extra Credit Material:
- We don't have time to fit this in, but it's good stuff to know:
* [[Week A|Week 9, Part 1]] - With the {{Command{vi}}} material in the Assignment section
* [[Week B]] - Regular Expressions
* [[Week C]] - The Environment
/%
* [[Week E]] - Storage Systems & LVM
* [[Week F]] - Network File System (NFS)
* [[Week G]] - Backups & disaster recovery
* [[Week H]] - Linux Firewalls
* [[Virtualization & Containers]]
%/
[img[https://www.ncs205.net/img/1x1.png]]
<html>
<font size="-2">Last Updated: 250823 13:41</font>
</html>
/***
To use, add {{{[[Styles HorizontalMainMenu]]}}} to your StyleSheet tiddler, or you can just paste the CSS in directly. See also HorizontalMainMenu and PageTemplate.
***/
/*{{{*/
#topMenu br {display:none; }
#topMenu { background: #39a; }
#topMenu { float: left; }
#topMenu { width: 90%; }
#topMenu { padding: 2px 0 2px 0; }
#topMenu .button, #topMenu .tiddlyLink { padding-left:1em; padding-right:1em; color:white; font-size:115%;}
#displayArea { margin: 1em 15.7em 0em 1em; }
#rightMenu {
float: right;
background: #39a;
width: 10%;
padding: 2px 0 2px 0;
}
#rightMenu .button, #rightMenu .tiddlyLink { padding-left:1em; padding-right:1em; color:white; font-size:115%;}
/* just in case want some QuickOpenTags in your topMenu */
#topMenu .quickopentag { padding:0px; margin:0px; border:0px; }
#topMenu .quickopentag .tiddlyLink { padding-right:1px; }
#topMenu .quickopentag .button { padding-left:1px; border:0px; }
/*}}}*/
!![[Lab 51 - Bring test and www online]]
Assigned [[Week 12, Part 1]]
* Set a root password for your test and www ~VMs so you are able to log into them via SSH.
* Ensure your test and www ~VMs are online and joined to the lab network.
** The notes above will help you configure networking
*** Use the hostname {{Monospaced{''www.//username//.ncs205.net''}}} and second IP address in your range for your web server.
** [[Virtual Machines]] - VM information (Linked on the top menu bar)
** Also complete and submit the [[Lab 51|labs/lab51.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated.
* Connect to your ~VMs via SSH from the class shell server
The Proxmox virtual console is a means to access the ~VMs for initial troubleshooting and in case something goes wrong. Once your ~VMs are online, all work should be done via SSH login.
You cannot log into these ~VMs directly since they are on private IP addresses (192.168.x.x) behind the class router. They can only be accessed by first connecting to the class shell server from home. Use putty (or another SSH client) to connect to the class shell server and then use the {{Command{ssh}}} command to connect to your ~VMs.
{{Warning{Be sure to keep your ~VMs online and do not power them down, else it'll look like the work hasn't been completed when it comes time for grading.}}}
!![[Lab 52 - VM updates & software installation]]
Assigned [[Week 12, Part 2]]
!!! On both ~VMs:
* Update the OS and currently installed software
* Install the following packages via {{Command{ apt}}}: {{Monospaced{man wget telnet rsync openssh-client bzip2 ncat}}}
* Also complete and submit the [[Lab 52|labs/lab52.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated.
These packages will also need to be installed on all future ~VMs. Make a note of it in your documentation.
!![[Lab 53 - Web Server]]
Assigned [[Week 12, Part 2]]
!!! Lab Tasks
<<<
1. Install apache and PHP on your web server
* Directions for this are above
2. Become familiar with apache and its configuration.
* Check out the config files within {{File{/etc/apache2/}}}
* The file {{File{/etc/apache2/apache2.conf}}} is the main configuration file
* The //Mastering Ubuntu Server// Chapter 14 (Serving Web Content) may also be a useful resource.
3. Change the Apache {{Monospaced{''~DocumentRoot''}}} directory to {{File{/opt/work/htdocs}}}
* Create the directory {{File{/opt/work/htdocs}}} on your web server VM
* Identify the configuration file which has this configuration directive
* Make a backup of the configuration file before you change it
** Always take a backup of a configuration file before making changes. See the note below. This way you'll have a known-good copy to refer to if there's any problems.
* Update the Apache configuration lines necessary to make this change
* You may also need to make another configuration change to grant permission to the new directory. Search the configuration files for the old ~DocumentRoot path to find the section.
* Don't forget to restart Apache after changing its configuration file
4. Download the new {{File{index.html}}} file from my web server at 192.168.12.25 to your new Apache {{Monospaced{''~DocumentRoot''}}} directory
* The file {{File{index.html}}} is the default web page delivered to a client (eg: your web browser). This file must exist in the correct location with correct permissions so your web server can provide content.
5. Ensure your web server is providing the correct website. The new site should be 6 lines long and include ''Welcome to ~NCS205!'' in the body.
* Check the apache access logs and ensure you are now seeing a proper ''200'' status code for the web requests.
<<<
{{Warning{''Warning:'' It's always wise to make a backup of a configuration file before making changes. The easiest way to do so is to copy the file with a timestamp appended to the new file name, for example: {{Command{cp httpd.conf httpd.conf.210322-1522.bak}}}. This captures the date & time in a way that's easily sortable. The {{Command{diff}}} command can compare the config file to a backup, showing lines which differ between the two. Example: {{Command{diff httpd.conf httpd.conf.210322-1522.bak}}}
}}}
!!! Lab Deliverable
* Also complete and submit the [[Lab 53|labs/lab53.pdf]] worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated.
!![[Lab 54 - Set up MediaWiki]]
Assigned [[Week 13, Part 1]]
Complete the steps on this page to install and configure ~MediaWiki
Install [[MediaWiki|http://www.mediawiki.org/wiki/MediaWiki]] and customize it to your tastes.
* Install ~MariaDB
** Add a wiki user and database
* Download the ~MediaWiki source tarball
** Extract its contents to {{File{/opt/work/htdocs/}}}
** Rename the extracted directory to ''wiki''
* Install required Wiki dependency packages
* Set up a tunnel or proxy to access your wiki
** You can access it by IP address until DNS is online: http://your_IP/wiki/
** Be sure to replace //your_IP// with proper values.
* Configure ~MediaWiki to fully bring it online
* Be sure you can view the wiki after uploading the {{File{~LocalSettings.php}}} file. It should look something like this:
[img[img/wiki.png]]
* Also complete and submit the [[Lab 54|labs/lab54.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated.
!! [[Lab 55 - Bring core VM online]]
Assigned [[Week 13, Part 2]]
<<<
Bring new core VM online:
* Hostname = {{Monospaced{core.//username//.ncs205.net}}}
* Use the third IP address in your range
* Apply outstanding updates and ensure your VM is running the latest available kernel
** A system reboot may be necessary if the kernel was also updated
* Also complete and submit the [[Lab 55|labs/lab55.pdf]] verification worksheet. This submitted worksheet will indicate your VM is online and ready to be evaluated for this lab.
Install additional software:
* Standard packages, as previously discussed and recorded in your class notes
* DNS server software. The package is {{Monospaced{bind9}}}
<<<
!! [[Lab 56 - Time]]
Assigned [[Week 13, Part 2]]
!!! Modify Hosts file:
Add a record similar to the following to the {{File{/etc/hosts}}} file on all of your ~VMs. Do not remove any lines which may already be in the file.
<<<
{{Monospaced{192.168.12.26 core core.merantn.ncs205.net ntp.merantn.ncs205.net loghost ntp}}}
<<<
* The IP address should be the address for your core VM
* Replace my username with yours
* These steps won't work if that line is missing or incorrect.
* Read about the {{File{/etc/hosts}}} file in the Mastering Ubuntu Server textbook starting on page 213 (//Understanding Linux name resolution//). This file is necessary because we don't have DNS running yet.
!!! Set your timezone
The default timezone for our systems is UTC. We can easily show this with the {{Command{date}}} command:
{{{
root@core:~# date
Mon Nov 18 21:37:10 UTC 2024
}}}
Use one of the commands above to change the system timezone on your ~VMs from UTC to US Eastern Time.
!!! Install NTP services and syncronize time:
Install the {{Monospaced{chrony}}} and {{Monospaced{ntpdate}}} packages on all ~VMs
On your core VM: Configure {{Monospaced{chronyd}}} ({{File{/etc/chrony/chrony.conf}}}) as a time server:
* Synchronize time from the lab ntp server instead of the pool servers - use {{Monospaced{ntp.ncs205.net}}}
* Allow your block of 8 IP addresses to communicate with the NTP service running on your core VM
* Add an allow directives to allow the naemon server full access:
** {{Monospaced{allow 192.168.12.15}}}
** See my config below for examples
On your test & www ~VMs (and future ~VMs): Configure {{Monospaced{chronyd}}} ({{File{/etc/chrony/chrony.conf}}}) as a client:
* Synchronize time from the ntp service on your core VM instead of the pool NTP servers
** Use the hostname {{Monospaced{ntp.//username//.ncs205.net}}} instead of IP addresses
** This hostname should resolve due to the entry you just added to the {{File{/etc/hosts}}} file. Test it with {{Command{ping}}}.
My Configs for reference, with comments removed (click the yellow box to expand them):
* Be sure to change host names and IP addresses appropriately:
* +++[My NTP Server]
{{Monospaced{core# }}} {{Command{ grep '^[a-z]' /etc/chrony/chrony.conf }}}
{{{
confdir /etc/chrony/conf.d
pool ntp.ncs205.net iburst maxsources 4
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
allow 192.168.12.24/29
allow 192.168.12.15
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapsectz right/UTC
}}}
===
* +++[My NTP Clients]
{{Monospaced{www# }}} {{Command{ grep '^[a-z]' /etc/chrony/chrony.conf }}}
{{{
pool ntp.merantn.ncs205.net iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
allow 192.168.12.24/29
allow 192.168.12.15
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony
}}}
===
!!! Start & enable Chrony
All ~VMs (current and future):
* Set chronyd to start on boot on all ~VMs
* Start the chronyd service now on all ~VMs
!!!! Verify it is working and time is being synchronized properly:
My core VM, an NTP server:
{{{
root@core:/etc/chrony# chronyc sources -v
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current best, '+' = combined, '-' = not combined,
| / 'x' = may be in error, '~' = too variable, '?' = unusable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.12.15 2 6 377 39 -20us[ -49us] +/- 6454us
}}}
My www server, an NTP client:
It took a few minutes after starting the services for the clock to synchronize. An unsynchronized client will begin with ^? instead of ^*
{{{
[root@www ~]# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? core 2 6 377 39 -20us[ -49us] +/- 6454us
}}}
Eventually it did and I saw this:
{{{
[root@www ~]# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* core 3 6 17 8 +1844ns[ +17us] +/- 5296us
}}}
* Also complete and submit the [[Lab 56|labs/lab56.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!!! Troubleshooting
Time synchronization with ntpd doesn't happen immediately. The service needs some time to build trust in its upstream time provider so that it will use it as a time source. Be sure to allow at least a 30 minute delay after starting the services for this trust to be established before reporting issues.
If you are having difficulty getting time to synchronize, the following commands may help direct you to the root cause:
* {{Command{systemctl status -l chronyd}}}
* {{Command{chronyc servers}}}
* {{Command{chronyc clients}}}
* {{Command{cat /etc/hosts}}}
* {{Command{ps aux | grep chronyd}}}
Any requests for help in the discussion boards should include output from the above commands for both your NTP server and any impacted NTP clients along with their chronyd configuration file. A copy/paste of the text into the discussion boards is easier to work with and highlight issues than simple screenshots. Be sure to include the command & shell prompt with any output included. ''Do not'' only include output without the shell prompt and the command which obtained that output.
!! [[Lab 57 - Logging]]
Assigned [[Week 13, Part 2]]
!!! Modify Hosts file:
Be sure a record similar to the following exists in the file {{File{/etc/hosts}}} file on all of your ~VMs. This should have been completed in the previous lab.
<<<
{{Monospaced{192.168.12.26 core core.merantn.ncs205.net ntp.merantn.ncs205.net loghost ntp}}}
<<<
* The IP address should be the address for your core VM
* Replace my username with yours
* These steps won't work if that line is missing or incorrect.
!!! syslog:
* Ensure the {{Monospaced{rsyslog}}} service is installed on your ~VMs.
* It would be wise to make configuration file backups before proceeding with changes
!!!! core VM:
* configure rsyslog to receive log information from other hosts
On your core VM, find these lines at the top of the file {{File{/etc/rsyslog.conf}}}:
{{{
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
}}}
Remove the comments from the bottom 2 lines (the first is actually a comment and should remain so).
!!!! www VM:
* configure rsyslog to also send log information to the core VM
On your www VM, append the following lines to your {{File{/etc/rsyslog.conf}}}:
{{{
action(type="omfwd"
Target="loghost" Port="514" Protocol="tcp")
}}}
* ''loghost'' is an alias for our core VM.
** Its handy to use aliases like this in case we need to move our log destination. We can then easily change the alias to point to a different system. This isn't so convenient when our systems are defined in the {{File{/etc/hosts}}} file, but easy once DNS is in place.
Experiment with logging. Investigate the logging commands and the log files within the directory {{File{/var/log/}}}.
!!! Activate the configuration file changes
What step must come after making changes to configuration files?
!!! Also complete and submit the [[Lab 57|labs/lab57.pdf]] verification worksheet. This submitted worksheet will indicate your VM is online and ready to be evaluated for this lab.
!! [[Lab 58 - Working with logs]]
Assigned [[Week 13, Part 2]]
This lab provides a brief refresher on working with text files and an introduction to using your syslog logs to investigate issues on your systems.
Automated brute-force password attacks are a common attack vector on the open internet. An attacker will scan large blocks of IP addresses or the entire internet trying to log into a service using common usernames and passwords. SSH is a common service to attack since most Unix/Linux systems are running it by default.
There are two primary types of brute-force password attacks:
* [[Password Guessing|https://attack.mitre.org/techniques/T1110/001/]] - //Adversaries with no prior knowledge of legitimate credentials within the system or environment may guess passwords to attempt access to accounts. Without knowledge of the password for an account, an adversary may opt to systematically guess a password using a repetitive or iterative mechanism. If usernames are also unknown, common usernames may be also be guessed such as root or standard first names.//
* [[Password Spraying|https://attack.mitre.org/techniques/T1110/003/]] - //Adversaries may use a single or small list of commonly used passwords against many different accounts to attempt to acquire valid account credentials. Password spraying uses one password (e.g. 'Password01'), or a small list of commonly used passwords, that may match the complexity policy of the domain.//
Tools like [[hydra|https://www.kali.org/tools/hydra/]] and [[ncrack|https://nmap.org/ncrack/]] along with password lists like [[RockYou|https://github.com/zacheller/rockyou]] ([[RockYou Background|https://en.wikipedia.org/wiki/RockYou#Data_breach]]) facilitate these types of attacks.
A few security best-practices are implemented in our lab environment to thwart these types of attacks:
* Our lab ~VMs are protected behind the class router and cannot be accessed from the internet, so they cannot be remotely scanned by attackers. This is why you have to access everything through the class shell server - a single choke point is much easier to monitor and control than a wide attack surface.
* The class shell server listens on an alternate SSH port so attackers cannot easily find it. This also keeps down noise in the logs so real events are easier to see.
* We use Campus usernames, which are cryptic, instead of common names that can be easily guessed.
* The administrator account common to all Unix/Linux systems, {{Monospaced{root}}}, cannot log in to the class shell server via SSH. Everyone with privileges must first log in as a normal user and then switch to {{Monospaced{root}}}. This also improves accountability for authorized users.
* Countermeasures are configured to detect and block rapid failed login attempts to the class shell server. Even if someone finds our alternate port, they won't be allowed to just beat on the server with failed login attempts.
//Lateral movement// refers to the technique used by attackers to progressively move through a network, gaining access to different systems or resources. Once attackers have initially breached a network, they aim to expand their influence, moving laterally from one system to another, seeking valuable information or increasing their control over the network. An attacker will try to compromise additional systems from within the network that they could not reach from the outside.
Your lab ~VMs cannot be attacked from the internet, but they can easily be attacked from within our lab network. A brute force SSH attack was launched against your core VM. Identify the system which launched the attack and any impacted user accounts.
!!! Complete and submit [[Lab 58|labs/lab58.pdf]].
* This lab will be due ''November 26''.
!! [[Lab 59 - Bind config and zones]]
Assigned [[Week 14, Part 1]]
Our end goal is for everyone to configure their own authoritative name servers for their lab networks. I own the domain ncs205.net and manage its authoritative name server. I am delegating control of the subdomain //username//.ncs205.net to each of you. You will set up the master DNS server for this zone on your core VM. The DNS server ns5.ncs205.net is running within the lab network. It is a slave for your //username//.ncs205.net zone. After you make changes to your DNS server, you will signal to this slave that you have records ready for it. It will then perform a zone transfer to obtain your DNS records. Once this zone transfer is complete, the DNS records for your //username//.ncs205.net domain will be globally available.
Begin to configure the DNS server on your core VM
* Also complete [[Lab 59|labs/lab59.pdf]] and submit this PDF after your DNS server is online. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!!! Configuring Bind
Lets stand up our authoritative name server
On your core VM: Install the packages: {{Monospaced{bind9}}}
Its configuration files are stored within the directory {{File{/etc/bind/}}}. We'll be interested primarily with two files:
* {{File{named.conf.local}}} - We'll store our access control lists (~ACLs) & zone definitions here
* {{File{named.conf.options}}} - This file contains our options for the bind service
{{Warning{''Note:'' You'll see my username and IP addresses in the configurations below. Be sure to replace my values with yours.}}}
Set these options within the //options// section of {{File{named.conf.options}}}. Do not remove any other options which are already defined.
{{{
listen-on port 53 { any; };
allow-query { any; };
allow-recursion { ncs205; };
forwarders { 192.168.12.10; };
}}}
!!! Access control lists:
Add access control lists to the top of the {{File{named.conf.local}}} file. This ~ACL will limit who can query your name server.
{{{
acl "ncs205" {
127.0.0.1;
localhost;
192.168.12.24/29;
};
}}}
This is just an example. ''24'' is my starting IP address. Replace ''24'' with your starting IP address.
!!! Forwarders:
The name server will forward any query it can't answer locally (from authoritative zone data or from cache) to the forwarder.
Forwarders are queried in the listed order until an answer is received. The forwarding IP address above is the name server for our lab ~VMs.
(This was already set above)
!!! Starting bind
It's always a good idea to verify your configuration before restarting the service. DNS is a critical service. If you issue a service restart and your configuration is invalid, you'll experience downtime while you sort out the problem.
* Verify the configuration: {{Command{named-checkconf /etc/bind/named.conf}}}
** No output will be returned if the configuration file is valid. Correct any errors which were displayed.
* Start the {{Monospaced{bind9}}} service now.
** It may have started automatically after installing the package. If so, be sure to restart it to activate your configuration changes.
** Be sure to reload the service whenever there's a configuration change.
* Set this service to also start at system boot, if it's not already.
Verify the service is working. You should be able to look up a DNS record by querying your new DNS server. The +noall +answer options provide an abbreviated output and do not always need to be used.
{{{
[root@core ~]# dig www.google.com @localhost +noall +answer
www.google.com. 126 IN A 142.250.185.164
}}}
It might take a few minutes after the service is started/restarted to be able to query external records. It might not work right away. And it might return a different record than the one above. But it should return //something//.
!!! Defining zones:
Lets put our zone definitions at the bottom of {{File{named.conf.local}}}. Be sure to replace my username with yours:
!!!! Forward zone:
This configuration block creates and defines your forward DNS zone:
{{{
zone "merantn.ncs205.net" {
type master;
file "/etc/bind/master/merantn.ncs205.net.fwd";
};
}}}
!!! Creating Zone files:
* Create your forward and reverse zone files. Be sure to include:
** A default TTL
*** This must be the first line of the zone file. See below for an example.
** A SOA record
** NS records for your primary and slave DNS servers
** A and PTR records for each of your hosts (so far we have 3 ~VMs: test, web, and core)
*** Be sure to note that A and PTR records do not belong in the same zone file
** CNAME records for host name __directory__, __ntp__, and __loghost__ pointing to your core VM
Here is my complete forward zone file. Save this to {{File{/etc/bind/master///username//.ncs205.net.fwd}}}
- You may need to create the {{File{master}}} directory.
- Replace usernames and IP addresses as necessary.
{{{
$TTL 5m
@ IN SOA ns1.merantn.ncs205.net. hostmaster.merantn.ncs205.net. (
2024112200 ; serial number
1d ; refresh
5d ; retry
2w ; expire
30m ; minimum
)
IN NS ns1.merantn.ncs205.net.
IN NS ns5.ncs205.net.
ns1 IN A 192.168.12.26
test IN A 192.168.12.24
www IN A 192.168.12.25
core IN A 192.168.12.26
loghost IN CNAME core
ntp IN CNAME core
directory IN CNAME core
}}}
!!! Allow zone transfers to the slave
We need to grant permission for the slave DNS server to perform a zone transfer against your server.
On your core vm, add a new ACL to the top of {{File{named.conf.local}}}, after the current ACL block:
{{{
acl "slaves" {
127.0.0.1; // allow transfer from localhost for testing
192.168.12.10; // allow the secondary authoritative name server to perform zone transfers
192.168.12.x; // allow your core system to preform zone transfers. Replace x with the IP address of your core VM
};
}}}
and add this statement to the options config file:
{{{
allow-transfer { slaves; };
}}}
You can verify your zone configuration with {{Command{named-checkzone //zonename// //filename//}}}
- eg: {{Command{named-checkzone merantn.ncs205.net /etc/bind/master/merantn.ncs205.net.fwd}}}
- If you make changes that break your zone file and restart the service, you'll have a critical outage. Verify your configs first!
Restart bind to reload the configuration and test your server:
Be sure to replace my username with yours.
{{{
[root@core ~]# dig www.merantn.ncs205.net @localhost
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> www.merantn.ncs205.net @localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50280
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: d68528867302de350100000067438f6172b0706f104044f7 (good)
;; QUESTION SECTION:
;www.merantn.ncs205.net. IN A
;; ANSWER SECTION:
www.merantn.ncs205.net. 300 IN A 192.168.12.25
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(localhost) (UDP)
;; WHEN: Sun Nov 24 15:41:05 EST 2024
;; MSG SIZE rcvd: 95
}}}
!!! Configure the slave (I already did this for each of you)
On ns5.ncs205.net (the slave I manage), I need to add a configuration block similar to the following for each student network in the class. This will configure that server to become a slave for your zones:
{{{
zone "merantn.ncs205.net" {
type slave;
file "slave/ncs205.net/merantn";
masters { 192.168.12.26; };
};
}}}
{{Warning{''Warning:'' This zone block above is just for your information. Do not place it into your zone file. It's purely to demonstrate what needs to be added to ns5, the bridge between your DNS service on your core VM and the outside world.}}}
!!! Verification:
1. Once your zone file is published, you can check the directory {{File{/opt/pub/ncs205/zones/}}} on the class shell server. You should see a file in that directory that matches your username.
2. You should be able to run the dig command on the shell server for one of your hosts, eg: {{Command{dig www.merantn.ncs205.net}}}
3. You should be able to run a DNS lookup on your home system for one of your hosts, eg:
4. The system log {{File{/var/log/messages}}} on your core VM may contain useful ~DNS-related messages.
Testing from my home Windows PC:
{{{
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\nick>nslookup www.merantn.ncs205.net
Server: one.one.one.one
Address: 1.1.1.1
Non-authoritative answer:
Name: www.merantn.ncs205.net
Address: 192.168.12.25
}}}
The SOA record is also good to check and observe the serial number. By looking at this SOA record, I can see that the date is for today, and this is the only update so far today. These are the values I expect.
This command performs a check of your SOA record against an external DNS server:
{{{
root@core:~# dig SOA merantn.ncs205.net @1.1.1.1
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> SOA merantn.ncs205.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38285
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 18 (Prohibited)
;; QUESTION SECTION:
;merantn.ncs205.net. IN SOA
;; ANSWER SECTION:
merantn.ncs205.net. 300 IN SOA ns1.merantn.ncs205.net. hostmaster.merantn.ncs205.net. 2024112200 86400 432000 1209600 1800
;; Query time: 46 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Fri Nov 22 15:50:57 EST 2024
;; MSG SIZE rcvd: 104
}}}
If all of these check out, congrats - your DNS zones are now globally accessible!
!!! DNS client configuration
Once your DNS server is working, modify the netplan configuration on each of your ~VMs to use your new nameserver and expand the search domains
* We want to query our nameserver first and the lab DNS server second
* We want to search our domain first
Here's my nameservers section in {{File{/etc/netplan/00-installer-config.yaml}}}:
{{{
nameservers:
addresses: [192.168.12.26, 192.168.12.10]
search: ['merantn.ncs205.net']
}}}
Validate with {{Command{resolvectl status}}}:
{{{
root@core:~# resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (ens18)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.12.26
DNS Servers: 192.168.12.26 192.168.12.10
DNS Domain: merantn.ncs205.net
}}}
The //search// keyword is a list of domains to add to unqualified host names. If you don't specify a fully qualified host name, then those domains will be appended in order until a match is found. This lets us save some typing and refer to our hosts by their unqualified name instead of having to type out the full domain name each time.
Unqualified: Just the host name, eg: //core//
Fully qualified domain name: //core.merantn.ncs205.net//
You can access your ~VMs by host name from anywhere within the lab network once you have DNS set up.
The //ncs205.net// domain is also in the search string on the class shell server. This means you can easily access your ~VMs from the shell server by entering //host//.//username//.
For example:
{{{
[merantn@shell ~]$ cat /etc/resolv.conf
nameserver 192.168.12.10
search ncs205.net
[merantn@shell ~]$ ssh core.merantn -l root
root@core.merantn's password:
Last login: Sun Nov 24 13:15:11 2024 from 192.168.12.10
[root@core ~]#
}}}
{{Note{''Note'': The class shell server runs Alma Linux, so it uses the {{File{/etc/resolv.conf}}} file for configuring DNS instead of the netplan configuration like our other lab ~VMs.}}}
!! Misc DNS topics
''Don't forget to update your serial number after making changes!''
- Notifications are broadcast to slave NS only if the serial number increments
- Records may be stale if you forget to increment the serial
You can use the command {{Command{rndc reload}}} to refresh your DNS zones after updates are made.
!! [[Lab 60 - SSL Certificates]]
Assigned [[Week 15, Part 2]]
Follow the directions above to add SSL encryption support to your web server.
Also complete and submit the [[Lab 60|labs/lab60.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!! [[Lab 60 - Scheduled Tasks]]
Assigned [[Week 13, Part 1]]
!!! SSL Certificate renewal
Our lab web server SSL certificate will expire in 90 days. This isn't much of an issue for us because class will have ended and this server will be decommissioned by then.
But if we were doing this for real, renewing that SSL certificate would be a task we would need to account for. SSL certificates created with the {{Command{acme.sh}}} tool can be renewed by running the command with the {{Monospaced{--cron}}} option. Any certs in its configuration will be checked for upcoming expiration and automatically renewed if they are about to expire.
We don't want to worry about running this manually and potentially forgetting about it. We can instead use cron to run this command for us at a set interval.
!!! Cron:
Create the following cron task on your www VM:
* Schedule {{Command{acme.sh --cron}}} to run every other day at 6pm.
* Save your scheduled job to the file {{File{/etc/cron.d/acme}}}
Also complete and submit the [[Lab 60|labs/lab60.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
* ''Note:'' Lab 60 is a late addition to the semester, so it's getting a high number.
!! [[Lab 61 - Scheduled Tasks]]
Assigned [[Week 15, Part 2]]
!!! SSL Certificate renewal
Our lab web server SSL certificate will expire in 90 days. This isn't much of an issue for us because class will have ended and this server will be decommissioned by then.
But if we were doing this for real, renewing that SSL certificate would be a task we would need to account for. SSL certificates created with the {{Command{acme.sh}}} tool can be renewed by running the command with the {{Monospaced{--cron}}} option. Any certs in its configuration will be checked for upcoming expiration and automatically renewed if they are about to expire.
We don't want to worry about running this manually and potentially forgetting about it. We can instead use cron to run this command for us at a set interval.
!!! Cron:
# Install the package for the {{Monospaced{postfix}}} mail server
** You will be prompted to configure the service.
### Configuration type: ''5. Local Only''
### System mail name: ''//username//.ncs205.net''
### Recipient for root and postmaster: ''root''
** Ensure the service is set to start on boot and running now.
# Create the following cron task on your www VM:
** Schedule {{Command{acme.sh --cron}}} to run every other day at 6pm.
** Save your scheduled job to the file {{File{/etc/cron.d/acme}}}
Also complete and submit the [[Lab 61|labs/lab61.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!! [[Lab 62 - VM Lockdown - Secure your VMs]]
Assigned [[Week 16]]
!!! Add user accounts to all ~VMs
Add two local user accounts to your ~VMs
* First account - set the user name to your campus username
** UID = 1001
** GID = 100
** Set a valid password
** Create a home directory within {{File{/home/}}}
** Copy the environment configuration files from {{File{/etc/skel/}}} to the new home directory
* Second account - username = {{Monospaced{merantn}}}
** UID = 7289
** GID = 100
** Shell must be set to bash
** Create a home directory within {{File{/home/}}}
** Copy the environment configuration files from {{File{/etc/skel/}}} to the new home directory
** Copy my SSH public key (see below) to the user's {{File{~/.ssh/authorized_keys}}} file
*** You will likely need to create the directory and file
*** Be sure you understand how SSH key-based authentication works.
** Use this password hash:
{{{
$6$nmUnix22$FCHlRIf.MFckb664yGEMGIC09cxfIk6NO/6fz/ou5EBbLQuo5.J0.szsg7aRswSIvxVjPGYWhiQ2XKD62eg4Y0
}}}
{{Warning{''Warning:'' Many students seem to have difficulty setting my hash correctly. These lab verification worksheets also provide a good opportunity to validate your work. Pay close attention to the output of Question 5 to ensure my hash is properly displayed.}}}
* Verify permissions:
** Both user's home directories and all files below them must be owned by the user and GID 100
** The user's home directory must have proper directory permissions - it must not be writable by the group or others for proper SSH function.
/%
* Verify ~SELinux
** ~SELinux must be disabled for SSH public key authentication to function properly
** Edit {{File{/etc/selinux/config}}} and change {{Monospaced{enforcing}}} to {{Monospaced{disabled}}} on line #7 to disable ~SELinux on system startup
** Execute {{Command{setenforce 0}}} to disable ~SELinux for the current boot
** This may already have been completed.
%/
My SSH public key
{{{
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBVLQcFklXcim/xylMML4QnLy4iuzrdgOUWivktOAlNX merantn@shell.ncs205.net
}}}
{{Note{''Note:'' You can test logins to your ~VMs using my user account by creating your own SSH keypair and adding your SSH public key to the {{File{~/.ssh/authorized_keys}}} file in my home directory on your VM. See the directions in [[Working more efficiently with GNU screen & SSH keys]] for how to create an SSH keypair. The {{File{authorized_keys}}} file can contain multiple public keys. Any of the corresponding private keys will be accepted for login.}}}
!!! Disable direct root login via SSH on all ~VMs
# Adjust the sshd configuration to disable direct root logins. All users must first login as a regular, unprivileged user and then elevate privileges.
** Look for the {{Monospaced{~PermitRootLogin}}} configuration option in {{File{/etc/ssh/sshd_config}}}
# Adjust PAM to require wheel group membership in order to su to root
** Look in {{File{/etc/pam.d/su}}}
** Create the {{Monospaced{wheel}}} group on your VMs with GID 75
# Don't forget to add both user accounts to the wheel group
{{Warning{''Warning:'' When messing with authentication, it's always wise to verify everything works before logging out. Open a new putty window, ssh in, and elevate up to a root prompt before disconnecting from your original putty session. Otherwise, if you log out and something is broken, you may have difficulty accessing the system.}}}
!!! Lock {{Monospaced{lab}}} account
Generic user accounts should be avoided as much as possible. Users should always log in with an account unique to them and their own credentials. The {{Monospaced{lab}}} user account is no longer needed now that we have our own user accounts. Lock the {{Monospaced{lab}}} user account once you are confident you are fully able to access your VMs with your account.
* Do not remove the account. It must still exist for historical reasons but configured so no one can log in with it.
* Document steps taken in the lab verification worksheet.
!!! Verification Worksheet
Also complete and submit the [[Lab 62|labs/lab62.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!![[Lab 63 - sudo]]: apache configuration access
Assigned [[Week 16]]
We would like to provide the webmaster the ability to update the apache configuration and restart the service on the web server virtual machine without granting full root level access. The {{Command{sudo}}} and {{Command{sudoedit}}} utilities can be use to accomplish this.
!!! Create a webmaster user on your web server VM
* username = wes
* uid = 2000
* gid = 100
* Fully configure the environment for this user
* Use the password hash {{{$6$IamWes205$JGDoWDKSXqdeHfFXYrPvG6vZQayMPMaCA8p/NlMFf1.Pu.mnHHh18H38B7HOxt2X3Z5dwiyS9jwYYXPHZJamt0}}}
!!! Create a new group for the webmasters
* group name = webmaster
* gid = 1010
* add wes to this group
!!! Configure {{Command{sudo}}} / {{Command{sudoedit}}} to:
# Grant the user ''wes'' the ability to edit the primary apache configuration file
# Grant the user ''wes'' the ability to execute the {{Command{apachectl}}} command as root.
{{Warning{''Warning:'' Be sure you understand why {{Command{sudoedit}}} is used for modifying root-owned files instead of just {{Command{sudo}}} followed by an editor, eg: {{Command{sudo vi /etc/httpd/conf/httpd.conf}}}. }}}
!!! Verification Worksheet
Also complete and submit the [[Lab 63|labs/lab63.pdf]] verification worksheet. This submitted worksheet will indicate your VM is ready to be evaluated for this lab.
!! [[Lab 64 - Enable Two-Factor Authentication]]
Assigned [[Week 16]]
Passwords are increasingly proven to be insufficient as the sole means of authentication. Passwords are too easily phished, captured via shoulder surfing or key loggers, or stolen from data breeches. We also generally do a poor job of selecting passwords. Password effectiveness is greatly reduced due to reuse across multiple sites and the selection of poor quality, weak passwords. Strong, secure passwords should be unique and contain a minimum of 12 random characters across the full alphabetic, numeric, and symbol character space. This then makes them difficult to remember.
These shortcomings can be mitigated with the use of multifactor authentication. Utilizing a hardware token is ideal. Google recently [[made the news|https://krebsonsecurity.com/2018/07/google-security-keys-neutralized-employee-phishing/]] for introducing hardware tokens for their employees to access corporate resources with great success. The Google-made [[Titan Security Key|https://cloud.google.com/titan-security-key/]] is now available for general purchase. [[YubiKeys|https://www.yubico.com/store/]] are another popular alternative for general use. Such keys can easily be used to add multi-factor authentication to operating system logins, services, or web sites after these systems are enabled to support hardware tokens.
Soft tokens are available as a free alternative to hardware tokens. A soft token is a desktop or mobile application which generates a one-time pin which can be entered along with a password to prove identity. Instead of a token on your keychain, your desktop or phone becomes "something you have". Multi-factor authentication should be used for any services where a higher level of security is warranted due to an increased exposure to attack.
Google Authenticator ([[Android|https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en_US]] or [[Apple|https://itunes.apple.com/us/app/google-authenticator/id388497605?mt=8]]) is a popular soft token with wide support. The Google Authenticator can be used as a second factor for ssh authentication to Linux servers.
If not for our class's virtual lab infrastructure protecting us from the outside world, our class ~VMs would all otherwise be exposed to the internet and open to attack. Any externally accessible server with such an increased exposure to attack would necessitate the deployment of multi-factor authentication.
!!! We will now implement two-factor authentication using Google Authenticator for access to our core VM.
{{Note{''Note:'' Two-factor authentication with the Google Authenticator will be set up for your regular user, not the root account.}}}
* Ensure your user account exists on your core VM and you are able to authenticate with a password.
** None of this will work if your user account is not fully functional
* Get started by installing the Google Authenticator app on your phone.
* We must next generate a barcode or key to add to the Google Authenticator App.
** Log in to your core VM via SSH and elevate to root
*** Install the {{Monospaced{libpam-google-authenticator}}} package on your core VM
** Exit the root login and log in to your core VM as a regular user
*** Run the command {{Command{google-authenticator}}} to initialize the token
*** Answer ''y'' to the question: ''Do you want authentication tokens to be time-based (y/n)''
You will be presented with a QR code to scan from the Google Authenticator app on your phone along with a secret key and a series of emergency scratch codes. The secret key can be used to add this account to the Google Authenticator in case you are unable to scan the barcode. Emergency scratch codes should be stored somewhere safe and are used to authenticate in case you lose your phone.
** Save the secret Key. We'll need it later.
Next, on your phone, launch the Google Authenticator app and choose the option to scan a barcode or enter a key and provide the appropriate input.
[img[img/googleauth1.png]]
Return to your VM and answer the remaining questions:
{{Monospaced{Do you want me to update your "/home/merantn/.google_authenticator" file? (y/n) ''y''}}}
{{Monospaced{Do you want to disallow multiple uses of the same authentication token? This restricts you to one login about every 30s, but it increases your chances to notice or even prevent man-in-the-middle attacks (y/n) ''y''}}}
{{Monospaced{By default, a new token is generated every 30 seconds by the mobile app. In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server. Do you want to do so? (y/n) ''n''}}}
{{Monospaced{If the computer that you are logging into isn't hardened against brute-force login attempts, you can enable rate-limiting for the authentication module. By default, this limits attackers to no more than 3 login attempts every 30s. Do you want to enable rate-limiting? (y/n) ''n''}}}
{{Warning{''Warning:'' Answering no to the last question is a poor security choice. If we were implementing this in a production environment we would answer yes to enable rate-limiting. We are only answering no because we are testing something new and do not want to lock ourselves out in the process.}}}
The file {{File{~/.google_authenticator}}} will contain your 2FA configuration.
You should now have the Google Authenticator app installed on your phone and an account configured for use. Next we must configure the operating system to require this second form of authentication for SSH logins. We will not modify the configuration for Console logins, so if things go wrong we can always log in through the Proxmox console to fix it.
!!! Configure the core server to require two-factor authentication
* Escalate to root privileges
* Edit the file {{File{/etc/pam.d/sshd}}} and add the following line to the bottom:
{{{
auth required pam_google_authenticator.so nullok
}}}
* Edit the file {{File{/etc/ssh/sshd_config}}} and search for //~ChallengeResponseAuthentication// . Ensure the value is set to ''yes''. You may need to add it if it does not already exist.
{{{
ChallengeResponseAuthentication yes
}}}
{{Warning{''Warning:'' If the //~ChallengeResponseAuthentication// configuration setting does not already exist in the file and you are adding it anew, be sure to place it near the top of the file. Appending it at the bottom of the file will cause issues.}}}
* Save and close the file then restart the sshd service.
Finally, ''without logging out'', attempt to log in to your core VM from itself. Launch the Google Authenticator App to generate a new token. When making changes to remote connection services, we do not want to log out until we can verify those changes are functioning properly. If we disconnect and something went wrong, we might end up locked out!
[img[img/googleauth2.jpg]]
Logging in with two-factor authentication:
{{{
[merantn@core ~]$ ssh localhost -l merantn
Password:
Verification code:
Last login: Sat Dec 7 00:22:40 2024 from localhost
[merantn@core ~]$
}}}
With the Google Authenticator changes in place, I'm prompted for my password as usual along with the verification code from the Authenticator App. ''Note:'' Each code is valid only once to prevent replay attacks. Once you log in, you may need to wait up to 30 seconds for a new code to be generated before you can log in again.
!!! Verification Worksheet
Also complete and submit the [[Lab 64|labs/lab64.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
Mastery of this subject material will only come with practice. To that end, this will be a very hands-on and practical course. Expect graded lab assignments regularly to provide ample practice with the assigned material. Properly completing lab assignments on time is necessary to receive a good grade for this course. Not competing lab assignments at all will likely result in a failing grade.
Any deliverables will be collected for review no sooner than their due date. Late assignments may be accepted, subject to time-dependent grade penalty of up to 50%. Presentation of submitted assignments will also impact grade.
{{Note{''Note:'' It is much better to have correct work submitted late than obviously wrong or incomplete work submitted on time. If you're having trouble with some of the material and need more time, please let me know and we can discuss adjusting due dates. Submitting poor quality work to meet a due date is not a wise professional strategy and will receive harsher grading.}}}
!! Submitting Homework Assignments
Homework assignments are to be uploaded to the class shell server using a file transfer program like ~WinSCP and saved to the directory {{File{/opt/pub/ncs205/submit/}}}. I will then grade/annotate your work and return the files to you for review. Most homework assignments will be PDF forms to complete. Download the lab PDF and open it in [[Acrobat Reader|https://get.adobe.com/reader/]]. ''Be careful using the PDF viewer in your web browser''. Some browsers will not properly save the file and you will upload a blank document. Grades will be posted to the file {{File{.grades.txt}}} in your home directory.
After downloading the PDF assignment and opening the file in [[Acrobat Reader|https://get.adobe.com/reader/]], add your name to the top, fill in your responses, then save & close the file. It would be wise to reopen the PDF in Acrobat Reader to make sure everything saved correctly before uploading to the server. You should be in the habit of verifying your work before submitting it.
Files must be named appropriately so we don't have filename collisions among everyone's uploaded files. Rename your PDF document following this naming convention: {{File{''ncs205-lab#-username.pdf''}}}
* replace # with the lab number
* replace username with your campus username
* Example lab file name ready to submit: {{File{ncs205-lab1-jdoe2.pdf}}}
Uploaded labs ''must'' contain your name at the top of the document and their file names ''must'' follow this file name format __exactly__ in order to be graded. This includes case - all letters must be lowercase. The Unix operating systems are case sensitive, so {{File{~NCS205-lab1-jdoe2.pdf}}} is a different file than {{File{ncs205-lab1-jdoe2.pdf}}}. The former would not be accepted for review.
{{Warning{''Warning:'' The Microsoft Windows operating system hides file extensions by default. This is a terrible setting for a security practitioner and should be disabled. A common mistake is to fail to take this into account and upload files with a double extension, such as {{File{ncs205-lab1-jdoe2.pdf.pdf}}}. This file would not be named correctly and thus not accepted for review.}}}
!! How to upload your lab assignments:
--A video will be posted here demonstrating the process in the coming days.-- Please let me know if you have trouble figuring this out.
!! Late Penalties
Point penalties for late lab assignments will be assessed as follows:
|!Penalty|!Condition|
| 0 |Sneak it in past the due date but before I grade the labs|
| 10% |Submitted after the batch has been graded|
| 20% |Submitted after graded labs have been returned|
| 30% |Submitted after we've reviewed a lab|
| 40% |Submitted after I've posted a review video or we've held an online meeting to discuss a lab.|
{{Warning{''Note:'' Labs 1 through 25 will not be accepted after the last date to Withdraw from the course unless prior approval is obtained.}}}
!! Common point deductions
!!! {{Command{cat}}} abuse
It is common for new students to abuse the {{Command{cat}}} command and use it unnecessarily with command strings like {{Command{ cat //file// | grep //string// }}}. In this example, the {{Command{grep}}} command will accept a filename argument and should be represented as {{Command{grep //string// //file//}}}. This seems to be a common bad habit to break. Abusing the {{Command{cat}}} command and using it where it provides no value will result in a 10% point penalty per lab.
!! The grading workflow
# You upload a completed lab PDF to {{File{/opt/pub/ncs205/submit/}}} on the class shell server
# Every hour a script will collect new lab submissions which are properly named and copy them to the grading queue, {{File{/opt/pub/ncs205/queue/}}}.
## An accepted lab will be moved from the {{File{submit/}}} directory to the directory {{File{/opt/pub/ncs205/submit/collected/}}}
## Any improperly named files will not be accepted and remain in the {{File{submit/}}} directory for one week. They will then be moved to {{File{/opt/pub/ncs205/submit/invalid/}}}
## ''Note:'' The collection script may occasionally be manually executed between the scheduled hourly runs
# The grading queue will be synchronized to my tablet for review.
# Any annotations will be recorded and synchronized back to the shell server, saved to the directory {{File{/opt/pub/ncs205/graded/}}}.
# Grades are recorded in my gradebook.
# A script will then be executed to move graded labs ready to be returned to your directory within {{File{/opt/pub/ncs205/returned/}}}. You may download them from this directory to see their grades and my annotations.
## This script will also update the file {{File{~/.grades.txt}}} in your home directory. This file contains your grades for the semester.
The directories {{File{/opt/pub/ncs205/queue/}}} and {{File{/opt/pub/ncs205/graded/}}} are staging directories in the workflow pipeline. You can view the contents of these directories but cannot write to them. Your access is only so you can have full visibility on where your labs reside in the workflow.
tl;dr: You upload new labs to {{File{/opt/pub/ncs205/submit/}}} and retrieve graded copies from within {{File{/opt/pub/ncs205/returned/}}}.
!! Extra Credit Labs
Extra material which was written for other courses or removed from this course is available for extra credit. Extra credit material will be posted to the main page of the class website and their labs will be denoted with a letter in their number, for example Lab A1. Submit these labs as you normally would with the letter portion of the lab number represented in uppercase.
Extra credit labs will be graded on the same 10-point scale as regular labs and will be tracked in their own group. At the end of the semester, 10% of the extra credit group grade will be applied to your final course average. This bonus cannot exceed one minor-level grade boost. For example, a B may become a B+ but cannot become an A-.
Any instance of academic dishonesty will result in loss of eligibility of extra credit.
!![[Lab E1 - Bring Files VM online]]
Assigned [[Week E]]
Bring your files VM online:
* A new VM was added for you
* Assign it the 4th IP address in your range
* Add the hostname {{Monospaced{''files.//username//.ncs205.net''}}} to the file {{File{/etc/hostname}}}
* Add an A record to your DNS zone for this new VM
* Reboot the VM to ensure all network settings were properly applied
* Install the standard software packages
* Apply any outstanding updates
* Configure NTP to synchronize time against your core VM and ensure time is fully synchronized
* Apply the steps in [[Lab 62 - VM Lockdown - Secure your VMs]] to harden this VM.
!!! Verification Worksheet
Also complete and submit the [[Lab E1|labs/labE1.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!![[Lab E2 - Logical Volume Manager]]
Assigned [[Week E]]
Complete the steps in the [[Lab E2 Instructions|labs/labE2-instructions.pdf]] PDF on your files VM to become familiar with the Linux logical volume manager.
Add additional filesystems to your core VM server
* See the last page in the [[Lab E Instructions|labs/labE-instructions.pdf]]
* Complete the [[Lab E2 Deliverable|labs/labE2.pdf]] and submit this PDF to {{File{/opt/pub/ncs205/submit/}}} on the class shell server
This lab will involve restarting your file server VM. Be sure the necessary services are configured to start on boot and ~SELinux and firewalld are properly configured.
!![[Lab E3 - Storage Expansion]]
Assigned [[Week E]]
Some systems need additional storage beyond what was initially provisioned. Here, we have a file server VM that was created with an additional disk. We now need to make that additional disk available to the operating system for storing additional data.
Perform the following steps on your files VM.
!!! Observe available storage devices
The {{Command{lsblk}}} command is a quick way to visualize all storage devices available to a system. Here, we can see that there are two unallocated drives - {{File{vdb}}} and {{File{vdc}}}. We'll use {{File{vdb}}} for this lab and leave {{File{vdc}}} alone.
{{{
[root@files ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 16G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 15G 0 part
├─centos-root 253:0 0 13.4G 0 lvm /
└─centos-swap 253:1 0 1.6G 0 lvm [SWAP]
vdb 252:16 0 2G 0 disk
vdc 252:32 0 2G 0 disk
}}}
!!! Create vdb1 Partition
It's generally preferred to create partitions on the drives instead of using the bare device. Partitions are logical divisions of the physical disk that will then hold the filesystem. Here, we're going to devote the entire disk to a single partition and a single filesystem. Creating multiple partitions on a disk allow it to hold separate filesystems. In some instances, physically dividing groups of files into separate filesystems is preferred. One example is logs. If you have a system, such as a webserver, that may generate a lot of logs, it's wise to store those logs on their own filesystem. If everything is stored on the same filesystem, excessive logs could fill the disk and interfere with the database's ability to store new data.
Refer to the //Storage Layers// diagram. We'll be following the path on the left from Storage Devices to Partitions to Filesystems.
[img[img/storage-layers.jpg]]
Duplicate this interaction with the {{Command{parted}}} command to create a new disk label and new partition. The first {{Command{print}}} command shows the disk is currently bare.
{{{
[root@files ~]# parted /dev/vdb
GNU Parted 3.1
Using /dev/vdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Error: /dev/vdb: unrecognised disk label
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 2147MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
(parted)
}}}
{{{
(parted) mklabel gpt
(parted) mkpart
Partition name? []? storage
File system type? [ext2]? xfs
Start? 1
End? 100%
(parted) quit
Information: You may need to update /etc/fstab.
[root@files ~]#
}}}
Now run {{Command{ lsblk }}} to verify the new partition was created. It's always wise to add verification steps as you proceed instead of just blindly assuming everything is working as it should. If you compare this output to the one above, you'll see that the {{File{ vdb1 }}} partition has been created.
{{{
[root@files ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 16G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 15G 0 part
├─centos-root 253:0 0 13.4G 0 lvm /
└─centos-swap 253:1 0 1.6G 0 lvm [SWAP]
vdb 252:16 0 2G 0 disk
└─vdb1 252:17 0 2G 0 part
vdc 252:32 0 2G 0 disk
}}}
!!! Create the filesystem
We can see from the {{Command{ lsblk }}} command that the new partition, {{File{vdb1}}}, has been successfully created. Now we must put a filesystem on it. Partitions are the physical divisions of a disk. Filesystems are the data structures the operating system interacts with in order to store files.
{{{
[root@files ~]# mkfs.xfs /dev/vdb1
meta-data=/dev/vdb1 isize=512 agcount=4, agsize=130944 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=523776, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
}}}
!!! Create the mount point
A mount point is a representation of the filesystem that we can interact with. On Windows systems, mount points are generally drive letters, like C:\ or D:\. In the Unix/Linux world, everything is one big filesystem tree. Linux mount points are directories on that tree. We identify a directory to mount our new filesystem to, and then any interaction with that directory and all items within it will be directed to our new disk volume. Here, we want to make our new disk available to the system at the directory {{File{ /opt/storage/ }}}.
We first need to ensure the new mount point exists:
{{{
[root@files ~]# mkdir /opt/storage
}}}
!!! Edit the filesystem table
The ''f''ile''s''ystem ''tab''le, {{File{/etc/fstab}}}, is the configuration file which specifies which disk volumes are mounted at system startup. Add your new disk volume to the file so it is mounted on boot.
Here's a copy of my {{File{ /etc/fstab }}} file. The last line is the one you need to copy to yours. Each line contains:
* the physical volume, {{File{ /dev/vdb1 }}}
* the mount point, {{File{/opt/storage }}}
* the filesystem type, {{Monospaced{ xfs }}}
* any special mount options. Here, just the {{Monospaced{ defaults }}}
* a binary value ({{Monospaced{0}}} or {{Monospaced{1}}}) to indicate whether the filesystem should be backed up. This is largely deprecated.
* the order in which filesystem checks ({{Command{fsck}}} command) should be run. A value of {{Monospaced{ 0 }}} disables these checks
{{{
[root@files ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Mar 13 00:03:20 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=f68b9069-7271-48de-b968-00d62e825144 /boot xfs defaults 0 0
/dev/mapper/centos-swap swap swap defaults 0 0
/dev/vdb1 /opt/storage xfs defaults 0 0
}}}
!!! Mount the new filesystem
Changes to the {{File{/etc/fstab}}} file should be tested and filesystems mounted with the {{Command{ mount -a }}} command. This will catch any errors in the file. If there is an error mounting a filesystem on system startup, the OS will not fully load and your only option will be to fix the problem on console. This can be a nasty surprise if you don't have easy access to console.
The {{Command{df -h}}} command adds a verification step that the filesystem is fully mounted and accessible. The old proverb //trust, but verify// must apply to everything you do.
{{{
[root@files ~]# mount -a
[root@files ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 232M 0 232M 0% /dev
tmpfs 244M 120K 244M 1% /dev/shm
tmpfs 244M 29M 215M 12% /run
tmpfs 244M 0 244M 0% /sys/fs/cgroup
/dev/mapper/centos-root 14G 2.4G 12G 18% /
/dev/vda1 1014M 228M 787M 23% /boot
tmpfs 49M 0 49M 0% /run/user/7289
/dev/vdb1 2.0G 33M 2.0G 2% /opt/storage
}}}
!!! Verification worksheet
You should now have successfully added a new storage volume to your files server VM. Complete and submit the [[Lab E3|labs/labE3.pdf]] verification worksheet when you are ready for review.
!![[Lab E4 - Monitoring disk usage with Nagios]]
Assigned [[Week E]]
Naemon infrastructure monitoring checks are performed through a series of plugins. Naemon plugins are shell scripts or binary executables which perform their specific check and return an exit status and their results to the Naemon service.
Currently the majority of our naemon checks are for services running on our ~VMs. The Naemon plugins interact with these services in a normal fashion and records its findings for display on the web interface. Occasionally it is necessary to examine a system internally, beyond the reach of our standard external plugins. The Naemon Remote Plugin Executor (NRPE) is a service which can be run on a system being monitored to provide a central Naemon server the ability to execute internal plugins and examine the target system from the inside. The Naemon server will communicate with NRPE and request it run a local plugin residing on the target system. NRPE will then return the results of that plugin to the Naemon monitoring service.
Certain checks, such as those examining the amount of free space available on a system, can only be performed via NRPE and a plugin residing on the system being checked. Monitoring the amount of free space in your log and home directory partitions is of special importance in order to prevent log information loss or user data loss.
Install NRPE and the necessary naemon plugin on each of your production ~VMs. Configure NRPE to grant our Naemon server permission to access it and configure the disk check plugin to monitor the {{File{/var/log}}} and {{File{/home}}} filesystems.
!!! 1. Install the EPEL repository
Install the Extra Packages for Enterprise Linux (EPEL) yum repository. The extra Nagios utilities and plugins are not available via the standard yum repo that ships with ~CentOS.
{{Command{yum install epel-release}}}
!!! 2. Install the nrpe service and nagios disk check plugin via yum. Note: Nagios & Naemon are two different monitoring tools that are comparable with each other and share the same plugins.
{{Command{yum install nrpe nagios-plugins-disk}}}
!!! 3. Edit the {{File{/etc/nagios/nrpe.cfg}}} config file:
Search for the ''allowed_hosts'' configuration directive and grant the local system, the class shell server, and the Naemon server permission to access it:
{{{
allowed_hosts=127.0.0.1,192.168.12.10,192.168.12.15
}}}
Add these three lines to the end of the command definitions (towards the bottom), creating disk check commands for {{File{/}}}, {{File{/var/log/}}}, and {{File{/home/}}}. The Naemon server will execute these configured commands via the NRPE service.
{{{
command[check_disk_root]=/usr/lib64/nagios/plugins/check_disk -w 15% -c 8% -p /
command[check_disk_log]=/usr/lib64/nagios/plugins/check_disk -w 15% -c 8% -p /var/log
command[check_disk_home]=/usr/lib64/nagios/plugins/check_disk -w 15% -c 8% -p /home
}}}
!!! 4. Start the NRPE service now and on boot:
Start on boot: {{Command{ systemctl enable nrpe }}}
Start now: {{Command{ systemctl start nrpe }}}
Verify: {{Command{ systemctl status nrpe }}}
!!! 5. Install the NRPE Nagios plugin
This is the plugin used by the Naemon server to call remote commands via NRPE. Normally this plugin is only installed on the Naemon server. We're installing in on our VMs for testing.
{{Command{ yum install nagios-plugins-nrpe }}}
!!! 6. Test
Execute the plugin to test your NRPE instance. The string returned is what would be reported back to Naemon and what will be displayed on the Naemon web interface.
{{Command{ /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk_root }}}
{{Command{ /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk_log }}}
{{Command{ /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk_home }}}
!!! 7. Adjust the firewall
You should be running a host-based firewall on your ~VMs. Don't forget to update the ~FirewallD lab worksheet and adjust the firewall on your ~VMs to allow traffic to the NRPE service. This service listens on TCP/5666.
!!! Verification Worksheet
Also complete and submit the [[Lab E4|labs/labE4.pdf]] verification worksheet. This submitted worksheet will indicate your ~VMs are ready to be evaluated for this lab.
!! [[Lab H - Host-based Firewalls]]
Assigned [[Week H]]
!!! Implement a host-based firewall on your ~VMs
* Complete the [[Lab H|labs/labH.pdf]] worksheet and upload it to the class shell server
* Take note of the state of your services in Naemon. It's always a good idea to have a known baseline of what things look like before making network changes. Taking a screenshot may be helpful.
** If something is down after you make changes and you don't know what things looked like before, you won't know if your change was the reason for the outage.
* Enable the firewalld service so it starts on boot and start the service now
* Request a scan of your services on Naemon. Take note of any changes to the alarms.
* Add the firewall rules you identified in the Lab H PDF.
* Recheck your services in Naemon and ensure all new alarms have cleared.
!![[Lab X - Containerization with Docker]]
Assigned [[Week X, Part 1]]
Defining infrastructure as code is another advantage of Docker containerization. With this concept, the services we need to run on our server are all clearly defined in a single configuration file. Once defined, these services can be brought online with just a few commands. Defining our infrastructure in this fashion makes it much easier to document our server configuration, track our infrastructure changes over time, and replicate our configuration elsewhere.
Recall the work we previously did to install the database and wiki. Several commands were executed to install and start the services, configure Apache, create the database accounts, download & unpack the wiki, and bring everything online for use. Now, a single infrastructure definition file will contain our requirements to run the wiki and database in separate Docker containers and make them available for use.
Complete the following steps on your www VM:
!!! 1. Create our working directory
Create the directory {{File{/opt/docker/}}} to use as the storage location for our files. All work will be performed within this directory.
!!! 2. Define the infrastructure
A {{File{docker-compose.yml}}} file contains the definition for our services and is another way to launch and maintain Docker containers. This file also fully documents the requirements and changes being made to support these services. Complex {{File{docker-compose.yml}}} files can be used to define an entire server running several different services within containers. Should we need to reinstall or rebuild the underlying OS, our containers and all dependencies are fully documented. It would just be a matter of copying our {{File{docker-compose.yml}}} file and all required data volumes to the new system in order to quickly replicate it.
Create the {{File{/opt/docker/docker-compose.yml}}} file containing the following contents. Adjust the file to set your own database username and password.
{{{
[root@www docker]# cat docker-compose.yml
version: '3'
networks:
wiki_network:
services:
mediawiki:
image: mediawiki
container_name: mediawiki
ports:
- 8080:80
links:
- mariadb
networks:
- wiki_network
volumes:
- /var/www/html/images
# After initial setup, download LocalSettings.php to the wiki directory, remove the below comment, and restart the container
# - ./wiki/LocalSettings.php:/var/www/html/LocalSettings.php
restart: always
mariadb:
image: mariadb
container_name: mariadb
networks:
- wiki_network
volumes:
- ./mariadb:/var/lib/mysql
environment:
# See https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/DefaultSettings.php for other possible settings
MYSQL_DATABASE: mediawiki
MYSQL_USER: wikiuser
MYSQL_PASSWORD: example_password
MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
restart: always
}}}
{{Warning{''Warning:'' Indentation within the docker-compose file is very specific. Like Python, indentation is used to establish nesting of configuration items. The indentation levels of this file must be properly preserved.}}}
!!! 3. Create database directory
As you can see from the definition above, the database files are stored in a local directory easily accessible to us. This also makes it easier if we wanted to back up just these database files.
Create the directory {{File{/opt/docker/mariadb/}}}. Any storage must be saved outside of the container, either to a local directory similar to this one or to a Docker volume. Storing any files which would changed, like database files, outside of the container ensures that data is available if the container was recreated. If the database files were stored within the container, all data would then be lost of the container was rebuilt or upgraded to a newer version.
Local storage is defined with the volumes tag, as seen in the example above.
!!! 4. Start containers
Run the command {{Command{docker-compose up -d}}} to bring our containers online. Be sure you're currently in the {{File{/opt/docker/}}} directory.
!!! 5. Validate images
The containers should be downloaded from the Docker hub and should be visible:
{{{
[root@www docker]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mariadb latest 1de5905a6164 4 days ago 410MB
mediawiki latest c8ce33ea98e9 2 weeks ago 809MB
}}}
!!! 6. Validate running containers
We should also see the new containers fully online.
{{{
[root@www docker]# docker-compose ps
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------
mariadb docker-entrypoint.sh mariadbd Up 3306/tcp
mediawiki docker-php-entrypoint apac ... Up 0.0.0.0:8080->80/tcp,:::8080->80/tcp
}}}
!!! 7. Observe database files:
Run the command {{Command{ls -l mariadb}}} to view the database files.
!!! 8. Obtain the generated root password
Our database service is configured to generate a unique root password the first time the container starts up. Check the Docker logs to find the new root password and save this value in case we need it later.
This command demonstrates finding my database root password. Yours will be different:
{{{
[root@www docker]# docker logs mariadb 2>&1 | grep ROOT
2022-05-01 01:45:43+00:00 [Note] [Entrypoint]: GENERATED ROOT PASSWORD: izp#st9p7`a_+Y<@:xIc&v=lEF`NG~%G
}}}
!!! 9. Validate database connection
Before proceeding, it would be prudent to ensure you are able to access the ~MariaDB database running inside the container. Execute the following command to log in to the database.
* {{Monospaced{ ''-u'' }}} refers to the value set in the {{Monospaced{~MYSQL_USER}}} environment variable in your {{File{docker-compose.yml}}} file.
* The final argument, {{Monospaced{//mediawiki//}}}, is the name of the database you specified in the {{Monospaced{~MYSQL_DATABASE}}} variable.
* The password you are prompted to enter will be the value set in the {{Monospaced{~MYSQL_PASSWORD}}} variable.
{{{
[root@www docker]# docker exec -ti mariadb mysql -u wikiuser -p mediawiki
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.7.3-MariaDB-1:10.7.3+maria~focal mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [mediawiki]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mediawiki |
+--------------------+
2 rows in set (0.002 sec)
MariaDB [mediawiki]> exit;
}}}
!!! 10. Configure wiki
Your services should now be online with a copy of the ~MediaWiki and required database running in containers. Access your wiki and configure it, similar to how we did it last time. The URL {{Monospaced{http://192.168.12.''X'':8080}}} will load your new containerized wiki. Replace the ''X'' with the last octet for your web server VM.
Once the site is loaded, go through the wiki configuration as you did last time and generate the {{File{~LocalSettings.php}}} file.
* Be sure to set {{{Database Host}}} to {{{mariadb}}} when prompted for the database information
!!! 11. Add {{File{~LocalSettings.php}}} configuration file
a. Create the directory {{File{/opt/docker/wiki/}}} on your www VM and upload your new {{File{~LocalSettings.php}}} file to it.
b. Uncomment the following line in your new {{File{docker-compose.yml}}} file:
This will make the wiki configuration file available within the container
{{{
# - ./wiki/LocalSettings.php:/var/www/html/LocalSettings.php
}}}
c. Restart the wiki container to activate the volume change. This command must be executed from within the {{File{/opt/docker/}}} directory.
{{Command{docker-compose up -d mediawiki}}}
!!! 12. Validate your new, containerized wiki
Access your wiki again through the tunnel to ensure everything is set up correctly and it is online for use.
!!! 13. Submit the deliverable
* Complete [[Lab 55|labs/lab55.pdf]] and submit this PDF to {{File{/opt/pub/ci233/submit/}}} on the class shell server
** This lab is due Saturday
| !Character | !Shortcut | !Most Useful |
| ~CTRL-C |Send interrupt signal to a running command (abort)| * |
|~|Clear entered command line text|
| ~CTRL-A |Move cursor to beginning of command line| * |
| ~CTRL-E |Move cursor to end of command line| * |
| ~CTRL-L |Clear Screen; move cursor to top to screen| * |
| ~ALT-B |Move one word backward on command line|
| ~ALT-F |Move one word forward on command line|
| ~CTRL-U |Erase line to left|
| ~CTRL-K |Erase line to the right|
| ~CTRL-W |Erase a word to left on command line| * |
| ~ALT-D |Erase a word to right on command line|
| ~CTRL-Y |Paste previously erased text|
| ~CTRL-D |Send EOF signal, ending input| * |
|~|Erase character under cursor| * |
|~|Log out (when no other text is on the command line)| * |
| ~Shift-INS |Paste clipboard at cursor| * |
| ~Shift-PgUp |Scroll window up|
| ~Shift-PgDn |Scroll window down|
| Tab |Auto-complete command or file name| * |
| Up Arrow |Previous Command| * |
| Down Arrow |Next command| * |
| Page Up |Previous command search| * |
| Page Down |Next command search| * |
{{Note{''Note:'' The above key sequences were listed with uppercase letters for clarity. It is not necessary to also press the shift key.}}}
!! Tab Completion
The tab key will auto-complete commands or file names, pausing when it reaches a decision point.
If I type the letters ''ad'' on the command line and press tab, the shell will autocomplete it to the string ''add'' before it reaches a decision point and cannot proceed without input. If I press tab twice it will then show me the options I have to complete the command:
<<<
[root@shell data]# add
addgnupghome addpart addr2line adduser
<<<
If I press the letter p and then tab again, the shell will know which command I'm looking for and auto-complete the command ''addpart''
The same auto-completion can be used for files. The path to the networking configuration file on Linux systems is rather long. Try this scenario on the class shell server:
* Type {{Command{cat /etc/sysco}}} and press ''tab''. The shell should autocomplete that to {{Command{cat /etc/sysconfig/}}}.
* We're at a decision point since there are many different ways we could proceed. Type: {{Command{netw}}} and press tab. The shell will autocomplete that to {{Command{cat /etc/sysconfig/network}}}.
* Press the {{Command{-}}} key and press tab again. The shell will autocomplete that to {{Command{cat /etc/sysconfig/network-scripts/}}}.
* Type {{Command{ifcfg-eth}}} and press tab twice. We are presented with the available options.
* Type {{Command{0}}} and hit enter to view the network configuration file.
Using tab helped me identify the available files and reduced the amount of letters I needed to type to view the file. It's slow at first, but once you get used to it greatly speeds up the speed and efficiency of using the shell and reduces the amount of information you have to remember.
!! Command recall
The page up and page down keys can be used to scroll through the recently used commands. This isn't universal; the shell needs to be configured to support it, but its supported by most systems out of the box.
If you have a long command string that wasn't used very recently, rather then press the up arrow several times to find it, you can enter the first few letters of that command and then ~Page-Up. The shell will cycle through your recent commands which began with those letters.
For example, a few days ago I ran the command {{Command{fail2ban-client status sshd-root}}} to see how many systems were trying to break into the class shell server. Rather then type out that entire command (or have to remember it), if I enter the first few letters {{Command{fai}}} and then press ~Page-Up, the shell will search backward in my command history and bring me right to it. If I used the up arrow, I'd first have to scroll through the hundreds of commands I may have entered since then.
!! Copy/Paste
In putty and most other terminal emulators, highlighting text with the mouse will copy it to the clipboard. Clicking the right mouse button will paste text from the clipboard into the terminal at the position of the cursor. If you are connecting from a Linux host like Kali instead of Windows, clicking the middle mouse button or scroll wheel will paste text to the terminal. ~Shift-Insert will also paste text from the clipboard into the terminal.
// //''Name:'' Calendar plugin
// //''Version:'' 0.1.0
// //''Author:'' SteveRumsby
// //''Syntax:''
// //<< {{{listTags tag //sort// //prefix//}}} >>
// //''Description:''
// //Generate a list of tiddlers tagged with the given tag.
// //If both //sort// and //prefix// are omitted the list is sorted in increasing order of title, with one tiddler per line.
// //If //sort// is specified the list is sorted in increasing order of the given tiddler property. Possible properties are: title. modified, modifier.
// //If //prefix// is specified the given string is inserted before the tiddler title. The insertion happens before the text is wikified. This can be used to generated bulleted or numbered lists.
// //''Examples:''
// //<< {{{listTags usage}}} >> - generate a plain list of all tiddlers tagged with tag //usage//, sorted by title
// //<< {{{listTags usage modified}}} >> - the same list, with most recently modified tiddlers last
// //<< {{{listTags usage title #}}} >> - generate a numbered list if tiddlers tagged with //usage//, sorted by title
// //''Code section:''
version.extensions.listTags = {major: 0, minor: 1, revision: 0, date: new Date(2005, 6,16)};
config.macros.listTags = {
text: "Hello"
};
config.macros.listTags.handler = function(place,macroName,params)
{
var tagged = store.getTaggedTiddlers(params[0], params[1]);
var string = "";
for(var r=0;r<tagged.length;r++)
{
if(params[2]) string = string + params[2] + " ";
string = string + "[[" + tagged[r].title + "]]\n";
}
wikify(string, place, null, null);
}
<<defaultHome>> [[Notebook]] [[Virtual Machines]] [[Outline]] [[Calendar]]
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="1572" height="724" poster="" data-setup="{}">
<source src="video/naemon.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
/***
''NestedSlidersPlugin for TiddlyWiki version 1.2.x and 2.0''
^^author: Eric Shulman
source: http://www.TiddlyTools.com/#NestedSlidersPlugin
license: [[Creative Commons Attribution-ShareAlike 2.5 License|http://creativecommons.org/licenses/by-sa/2.5/]]^^
Quickly make any tiddler content into an expandable 'slider' panel, without needing to create a separate tiddler to contain the slider content. Optional syntax allows ''default to open'', ''custom button label/tooltip'' and ''automatic blockquote formatting.''
You can also 'nest' these sliders as deep as you like (see complex nesting example below), so that expandable 'tree-like' hierarchical displays can be created. This is most useful when converting existing in-line text content to create in-line annotations, footnotes, context-sensitive help, or other subordinate information displays.
For more details, please click on a section headline below:
++++!!!!![Configuration]>
Debugging messages for 'lazy sliders' deferred rendering:
<<option chkDebugLazySliderDefer>> show debugging alert when deferring slider rendering
<<option chkDebugLazySliderRender>> show debugging alert when deferred slider is actually rendered
===
++++!!!!![Usage]>
When installed, this plugin adds new wiki syntax for embedding 'slider' panels directly into tiddler content. Use {{{+++}}} and {{{===}}} to delimit the slider content. Additional optional syntax elements let you specify
*default to open
*cookiename
*heading level
*floater (with optional CSS width value)
*mouse auto rollover
*custom label/tooltip/accesskey
*automatic blockquote
*deferred rendering
The complete syntax, using all options, is:
//{{{
++++(cookiename)!!!!!^width^*[label=key|tooltip]>...
content goes here
===
//}}}
where:
* {{{+++}}} (or {{{++++}}}) and {{{===}}}^^
marks the start and end of the slider definition, respectively. When the extra {{{+}}} is used, the slider will be open when initially displayed.^^
* {{{(cookiename)}}}^^
saves the slider opened/closed state, and restores this state whenever the slider is re-rendered.^^
* {{{!}}} through {{{!!!!!}}}^^
displays the slider label using a formatted headline (Hn) style instead of a button/link style^^
* {{{^width^}}} (or just {{{^}}})^^
makes the slider 'float' on top of other content rather than shifting that content downward. 'width' must be a valid CSS value (e.g., "30em", "180px", "50%", etc.). If omitted, the default width is "auto" (i.e., fit to content)^^
* {{{*}}}^^
automatically opens/closes slider on "rollover" as well as when clicked^^
* {{{[label=key|tooltip]}}}^^
uses custom label/tooltip/accesskey. {{{=key}}} and {{{|tooltip}}} are optional. 'key' is must be a ''single letter only''. Default labels/tootips are: ">" (more) and "<" (less), with no default access key assignment.^^
* {{{">"}}} //(without the quotes)//^^
automatically adds blockquote formatting to slider content^^
* {{{"..."}}} //(without the quotes)//^^
defers rendering of closed sliders until the first time they are opened. //Note: deferred rendering may produce unexpected results in some cases. Use with care.//^^
//Note: to make slider definitions easier to read and recognize when editing a tiddler, newlines immediately following the {{{+++}}} 'start slider' or preceding the {{{===}}} 'end slider' sequence are automatically supressed so that excess whitespace is eliminated from the output.//
===
++++!!!!![Examples]>
simple in-line slider:
{{{
+++
content
===
}}}
+++
content
===
----
use a custom label and tooltip:
{{{
+++[label|tooltip]
content
===
}}}
+++[label|tooltip]
content
===
----
content automatically blockquoted:
{{{
+++>
content
===
}}}
+++>
content
===
----
all options combined //(default open, cookie, heading, sized floater, rollover, label/tooltip/key, blockquoted, deferred)//
{{{
++++(testcookie)!!!^30em^*[label=Z|click or press Alt-Z to open]>...
content
===
}}}
++++(testcookie)!!!^30em^*[label=Z|click or press Alt-Z to open]>...
content
===
----
complex nesting example:
{{{
+++^[get info...=I|click for information or press Alt-I]
put some general information here, plus a floating slider with more specific info:
+++^10em^[view details...|click for details]
put some detail here, which could include a rollover with a +++^25em^*[glossary definition]explaining technical terms===
===
===
}}}
+++^[get info...=I|click for information or press Alt-I]
put some general information here, plus a floating slider with more specific info:
+++^10em^[view details...|click for details]
put some detail here, which could include a rollover with a +++^25em^*[glossary definition]explaining technical terms===
===
===
----
nested floaters
>menu: <<tiddler NestedSlidersExample>>
(see [[NestedSlidersExample]] for definition)
----
===
!!!!!Installation
<<<
import (or copy/paste) the following tiddlers into your document:
''NestedSlidersPlugin'' (tagged with <<tag systemConfig>>)
<<<
!!!!!Revision History
<<<
''2006.05.11 - 1.9.0'' added optional '^width^' syntax for floating sliders and '=key' syntax for setting an access key on a slider label
''2006.05.09 - 1.8.0'' in onClickNestedSlider(), when showing panel, set focus to first child input/textarea/select element
''2006.04.24 - 1.7.8'' in adjustSliderPos(), if floating panel is contained inside another floating panel, subtract offset of containing panel to find correct position
''2006.02.16 - 1.7.7'' corrected deferred rendering to account for use-case where show/hide state is tracked in a cookie
''2006.02.15 - 1.7.6'' in adjustSliderPos(), ensure that floating panel is positioned completely within the browser window (i.e., does not go beyond the right edge of the browser window)
''2006.02.04 - 1.7.5'' add 'var' to unintended global variable declarations to avoid FireFox 1.5.0.1 crash bug when assigning to globals
''2006.01.18 - 1.7.4'' only define adjustSliderPos() function if it has not already been provided by another plugin. This lets other plugins 'hijack' the function even when they are loaded first.
''2006.01.16 - 1.7.3'' added adjustSliderPos(place,btn,panel,panelClass) function to permit specialized logic for placement of floating panels. While it provides improved placement for many uses of floating panels, it exhibits a relative offset positioning error when used within *nested* floating panels. Short-term workaround is to only adjust the position for 'top-level' floaters.
''2006.01.16 - 1.7.2'' added button property to slider panel elements so that slider panel can tell which button it belongs to. Also, re-activated and corrected animation handling so that nested sliders aren't clipped by hijacking Slider.prototype.stop so that "overflow:hidden" can be reset to "overflow:visible" after animation ends
''2006.01.14 - 1.7.1'' added optional "^" syntax for floating panels. Defines new CSS class, ".floatingPanel", as an alternative for standard in-line ".sliderPanel" styles.
''2006.01.14 - 1.7.0'' added optional "*" syntax for rollover handling to show/hide slider without requiring a click (Based on a suggestion by tw4efl)
''2006.01.03 - 1.6.2'' When using optional "!" heading style, instead of creating a clickable "Hn" element, create an "A" element inside the "Hn" element. (allows click-through in SlideShowPlugin, which captures nearly all click events, except for hyperlinks)
''2005.12.15 - 1.6.1'' added optional "..." syntax to invoke deferred ('lazy') rendering for initially hidden sliders
removed checkbox option for 'global' application of lazy sliders
''2005.11.25 - 1.6.0'' added optional handling for 'lazy sliders' (deferred rendering for initially hidden sliders)
''2005.11.21 - 1.5.1'' revised regular expressions: if present, a single newline //preceding// and/or //following// a slider definition will be suppressed so start/end syntax can be place on separate lines in the tiddler 'source' for improved readability. Similarly, any whitespace (newlines, tabs, spaces, etc.) trailing the 'start slider' syntax or preceding the 'end slider' syntax is also suppressed.
''2005.11.20 - 1.5.0'' added (cookiename) syntax for optional tracking and restoring of slider open/close state
''2005.11.11 - 1.4.0'' added !!!!! syntax to render slider label as a header (Hn) style instead of a button/link style
''2005.11.07 - 1.3.0'' removed alternative syntax {{{(((}}} and {{{)))}}} (so they can be used by other
formatting extensions) and simplified/improved regular expressions to trim multiple excess newlines
''2005.11.05 - 1.2.1'' changed name to NestedSlidersPlugin
more documentation
''2005.11.04 - 1.2.0'' added alternative character-mode syntax {{{(((}}} and {{{)))}}}
tweaked "eat newlines" logic for line-mode {{{+++}}} and {{{===}}} syntax
''2005.11.03 - 1.1.1'' fixed toggling of default tooltips ("more..." and "less...") when a non-default button label is used
code cleanup, added documentation
''2005.11.03 - 1.1.0'' changed delimiter syntax from {{{(((}}} and {{{)))}}} to {{{+++}}} and {{{===}}}
changed name to EasySlidersPlugin
''2005.11.03 - 1.0.0'' initial public release
<<<
!!!!!Credits
<<<
This feature was implemented by EricShulman from [[ELS Design Studios|http:/www.elsdesign.com]] with initial research and suggestions from RodneyGomes, GeoffSlocock, and PaulPetterson.
<<<
!!!!!Code
***/
//{{{
version.extensions.nestedSliders = {major: 1, minor: 9, revision: 0, date: new Date(2006,5,11)};
//}}}
//{{{
// options for deferred rendering of sliders that are not initially displayed
if (config.options.chkDebugLazySliderDefer==undefined) config.options.chkDebugLazySliderDefer=false;
if (config.options.chkDebugLazySliderRender==undefined) config.options.chkDebugLazySliderRender=false;
// default styles for 'floating' class
setStylesheet(".floatingPanel { position:absolute; z-index:10; padding:0.5em; margin:0em; \
background-color:#eee; color:#000; border:1px solid #000; text-align:left; }","floatingPanelStylesheet");
//}}}
//{{{
config.formatters.push( {
name: "nestedSliders",
match: "\\n?\\+{3}",
terminator: "\\s*\\={3}\\n?",
lookahead: "\\n?\\+{3}(\\+)?(\\([^\\)]*\\))?(\\!*)?(\\^(?:[^\\^\\*\\[\\>]*\\^)?)?(\\*)?(\\[[^\\]]*\\])?(\\>)?(\\.\\.\\.)?\\s*",
handler: function(w)
{
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart)
{
// location for rendering button and panel
var place=w.output;
// default to closed, no cookie, no accesskey
var show="none"; var title=">"; var tooltip="show"; var cookie=""; var key="";
// extra "+", default to open
if (lookaheadMatch[1])
{ show="block"; title="<"; tooltip="hide"; }
// cookie, use saved open/closed state
if (lookaheadMatch[2]) {
cookie=lookaheadMatch[2].trim().slice(1,-1);
cookie="chkSlider"+cookie;
if (config.options[cookie]==undefined)
{ config.options[cookie] = (show=="block") }
if (config.options[cookie])
{ show="block"; title="<"; tooltip="hide"; }
else
{ show="none"; title=">"; tooltip="show"; }
}
// parse custom label/tooltip/accesskey: [label=X|tooltip]
if (lookaheadMatch[6]) {
title = lookaheadMatch[6].trim().slice(1,-1);
var pos=title.indexOf("|");
if (pos!=-1) { tooltip = title.substr(pos+1,title.length); title=title.substr(0,pos); }
if (title.substr(title.length-2,1)=="=") { key=title.substr(title.length-1,1); title=title.slice(0,-2); }
if (pos==-1) tooltip += " "+title; // default tooltip: "show/hide <title>"
}
// create the button
if (lookaheadMatch[3]) { // use "Hn" header format instead of button/link
var lvl=(lookaheadMatch[3].length>6)?6:lookaheadMatch[3].length;
var btn = createTiddlyElement(createTiddlyElement(place,"h"+lvl,null,null,null),"a",null,null,title);
btn.onclick=onClickNestedSlider;
btn.setAttribute("href","javascript:;");
btn.setAttribute("title",tooltip);
}
else
var btn = createTiddlyButton(place,title,tooltip,onClickNestedSlider);
btn.sliderCookie = cookie; // save the cookiename (if any) in the button object
btn.keyparam=key; // save the access key letter ("" if none)
if (key.length) {
btn.setAttribute("accessKey",key); // init access key
btn.onfocus=function(){this.setAttribute("accessKey",this.keyparam);}; // **reclaim** access key on focus
}
// "non-click" MouseOver open/close slider
if (lookaheadMatch[5]) btn.onmouseover=onClickNestedSlider;
// create slider panel
var panelClass=lookaheadMatch[4]?"floatingPanel":"sliderPanel";
var panel=createTiddlyElement(place,"div",null,panelClass,null);
panel.style.display = show;
if (lookaheadMatch[4] && lookaheadMatch[4].length>2) panel.style.width=lookaheadMatch[4].slice(1,-1); // custom width
panel.button = btn; // so the slider panel know which button it belongs to
btn.sliderPanel=panel;
// render slider (or defer until shown)
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
if ((show=="block")||!lookaheadMatch[8]) {
// render now if panel is supposed to be shown or NOT deferred rendering
w.subWikify(lookaheadMatch[7]?createTiddlyElement(panel,"blockquote"):panel,this.terminator);
// align slider/floater position with button
adjustSliderPos(place,btn,panel,panelClass);
}
else {
var src = w.source.substr(w.nextMatch);
var endpos=findMatchingDelimiter(src,"+++","===");
panel.setAttribute("raw",src.substr(0,endpos));
panel.setAttribute("blockquote",lookaheadMatch[7]?"true":"false");
panel.setAttribute("rendered","false");
w.nextMatch += endpos+3;
if (w.source.substr(w.nextMatch,1)=="\n") w.nextMatch++;
if (config.options.chkDebugLazySliderDefer) alert("deferred '"+title+"':\n\n"+panel.getAttribute("raw"));
}
}
}
}
)
// TBD: ignore 'quoted' delimiters (e.g., "{{{+++foo===}}}" isn't really a slider)
function findMatchingDelimiter(src,starttext,endtext) {
var startpos = 0;
var endpos = src.indexOf(endtext);
// check for nested delimiters
while (src.substring(startpos,endpos-1).indexOf(starttext)!=-1) {
// count number of nested 'starts'
var startcount=0;
var temp = src.substring(startpos,endpos-1);
var pos=temp.indexOf(starttext);
while (pos!=-1) { startcount++; pos=temp.indexOf(starttext,pos+starttext.length); }
// set up to check for additional 'starts' after adjusting endpos
startpos=endpos+endtext.length;
// find endpos for corresponding number of matching 'ends'
while (startcount && endpos!=-1) {
endpos = src.indexOf(endtext,endpos+endtext.length);
startcount--;
}
}
return (endpos==-1)?src.length:endpos;
}
//}}}
//{{{
window.onClickNestedSlider=function(e)
{
if (!e) var e = window.event;
var theTarget = resolveTarget(e);
var theLabel = theTarget.firstChild.data;
var theSlider = theTarget.sliderPanel
var isOpen = theSlider.style.display!="none";
// if using default button labels, toggle labels
if (theLabel==">") theTarget.firstChild.data = "<";
else if (theLabel=="<") theTarget.firstChild.data = ">";
// if using default tooltips, toggle tooltips
if (theTarget.getAttribute("title")=="show")
theTarget.setAttribute("title","hide");
else if (theTarget.getAttribute("title")=="hide")
theTarget.setAttribute("title","show");
if (theTarget.getAttribute("title")=="show "+theLabel)
theTarget.setAttribute("title","hide "+theLabel);
else if (theTarget.getAttribute("title")=="hide "+theLabel)
theTarget.setAttribute("title","show "+theLabel);
// deferred rendering (if needed)
if (theSlider.getAttribute("rendered")=="false") {
if (config.options.chkDebugLazySliderRender)
alert("rendering '"+theLabel+"':\n\n"+theSlider.getAttribute("raw"));
var place=theSlider;
if (theSlider.getAttribute("blockquote")=="true")
place=createTiddlyElement(place,"blockquote");
wikify(theSlider.getAttribute("raw"),place);
theSlider.setAttribute("rendered","true");
}
// show/hide the slider
if(config.options.chkAnimate)
anim.startAnimating(new Slider(theSlider,!isOpen,e.shiftKey || e.altKey,"none"));
else
theSlider.style.display = isOpen ? "none" : "block";
// if showing panel, set focus to first 'focus-able' element in panel
if (theSlider.style.display!="none") {
var ctrls=theSlider.getElementsByTagName("*");
for (var c=0; c<ctrls.length; c++) {
var t=ctrls[c].tagName.toLowerCase();
if (t=="input" || t=="textarea" || t=="select")
{ ctrls[c].focus(); break; }
}
}
if (this.sliderCookie && this.sliderCookie.length)
{ config.options[this.sliderCookie]=!isOpen; saveOptionCookie(this.sliderCookie); }
// align slider/floater position with target button
adjustSliderPos(theSlider.parentNode,theTarget,theSlider,theSlider.className);
return false;
}
// hijack animation handler 'stop' handler so overflow is visible after animation has completed
Slider.prototype.coreStop = Slider.prototype.stop;
Slider.prototype.stop = function() { this.coreStop(); this.element.style.overflow = "visible"; }
// adjust panel position based on button position
if (window.adjustSliderPos==undefined) window.adjustSliderPos=function(place,btn,panel,panelClass) {
if (panelClass=="floatingPanel") {
var left=0;
var top=btn.offsetHeight;
if (place.style.position!="relative") {
var left=findPosX(btn);
var top=findPosY(btn)+btn.offsetHeight;
var p=place; while (p && p.className!='floatingPanel') p=p.parentNode;
if (p) { left-=findPosX(p); top-=findPosY(p); }
}
if (left+panel.offsetWidth > getWindowWidth()) left=getWindowWidth()-panel.offsetWidth-10;
panel.style.left=left+"px"; panel.style.top=top+"px";
}
}
function getWindowWidth() {
if(document.width!=undefined)
return document.width; // moz (FF)
if(document.documentElement && ( document.documentElement.clientWidth || document.documentElement.clientHeight ) )
return document.documentElement.clientWidth; // IE6
if(document.body && ( document.body.clientWidth || document.body.clientHeight ) )
return document.body.clientWidth; // IE4
if(window.innerWidth!=undefined)
return window.innerWidth; // IE - general
return 0; // unknown
}
//}}}
* [[Class Syllabus|syllabus/NCS205Syllabus2509.pdf]]
* [[General SOPs]]
* [[Lab Assignments]]
* [[Class Participation]]
** [[Using Discord]]
* [[Shell script submission requirements]]
* [[Shell scripting best practices]]
* [[Material Sections]]
/% * [[Term Project]] - For NCS 515 %/
/% ** [[Using Blackboard]] %/
Other helpful material to make things easier:
* [[Working more efficiently with GNU screen & SSH keys]]
* [[Tunnels & Proxies with SSH]]
!!Handouts
[[Command line summary handout|handouts/UnixCommandSummary.pdf]]
[[Substitution Handout|handouts/SubstitutionHandout.pdf]] (from tcsh man page)
[[ASCII Chart|handouts/ascii-chart.gif]]
[[Shell Metacharacter Table|handouts/ShellMetacharacterTable.pdf]]
[[Regular expression metacharacters]]
* [[Metacharacter Handout|handouts/Metacharacters.pdf]] - Metacharacters and how they differ in the shell & regular expression contexts.
[[vi diagram handout|handouts/viDiagram.pdf]]
[[awk handout|handouts/awkHandout.pdf]]
!!Reference Material
[[Class technology stack]] - Mostly my notes for setting up our class servers
[[UNIX in a Nutshell|http://books.google.com/books?id=YkNiiLupct4C&dq=unix+in+a+nutshell&printsec=frontcover&source=bn&hl=en&ei=aKlWS43lJJCOlQeW3rSCBA&sa=X&oi=book_result&ct=result&resnum=5&ved=0CCIQ6AEwBA#v=onepage&q=&f=false]] - Google books
[[The Linux Command Line (No Starch Press)|http://www.merantn.net/reference/TLCL-19.01.pdf]]
[[UNIX Toolbox|https://merantn.net/docs/unixtoolbox.xhtml]]
[[Shell scripting notes]]
[[Table of Commands]]
** [[less quick reference]]
[[Linux Shortcuts]]
/***
|Name|OpenTopPlugin|
|Created by|SaqImtiaz|
|Location|http://lewcid.googlepages.com/lewcid.html#OpenTopPlugin|
|Version|0.1|
|Requires|~TW2.x|
!!!Description:
Open new tiddlers at the top of the screen.
!!!Code
***/
//{{{
Story.prototype.coreLewcidDisplayTiddler=Story.prototype.displayTiddler ;
Story.prototype.displayTiddler =
function(srcElement,title,template,unused1,unused2,animate,slowly)
{
var srcElement=null;
if (document.getElementById(this.idPrefix + title))
{story.closeTiddler(title);}
this.coreLewcidDisplayTiddler(srcElement,title,template,unused1,unused2,animate,slowly);
window.scrollTo(0,0);
}
//}}}
<<option chkSaveBackups>> SaveBackups
<<option chkAutoSave>> AutoSave
<<option chkRegExpSearch>> RegExpSearch
<<option chkCaseSensitiveSearch>> CaseSensitiveSearch
<<option chkAnimate>> EnableAnimations
<<darkMode label:"☀️/🌘">>
----
Also see AdvancedOptions
/%
[[Week 3, Part 2]] - Home Directories & Shell documentation
[[Week 3, Part 1]] - Links & File Globbing
* Links:
** Read
*** Chapter 3, pp 23 & 24
*** Chapter 4, pp 33 & 34
** Watch:
*** Links - https://www.youtube.com/watch?v=lW_V8oFxQgA
** Complete: [[Lab 8|labs/lab8.pdf]]
* File Globbing:
** Read: Chapter 4, pp 25-27 (Wildcards)
** Watch:
*** File Globbing: https://www.youtube.com/watch?v=QIysdjpiLcA
*** Brace Expansion: https://www.youtube.com/watch?v=LGzSnVYS2J4
** Complete: [[Lab 9|labs/lab9.pdf]] & [[Lab 10|labs/lab10.pdf]]
----
[[Week 2, Part 2]] - Manipulating Files & Directories
* Review: Complete [[Lab 5|labs/lab5.pdf]]
* Read Chapter 4 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
** You can gloss over the parts about wildcards (pp 26-27) for now. We'll come back to them later.
** Focus on becoming familiar with the available commands.
* Watch:
** Creating and Deleting files and directories: https://www.youtube.com/watch?v=91FhiTyEaCU
** Moving and copying files: https://www.youtube.com/watch?v=GKEGNdNIQrw
* Complete [[Lab 6|labs/lab6.pdf]] & [[Lab 7|labs/lab7.pdf]]
[[Week 2, Part 1]] - Exploring the system
* Read Chapter 3 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
* Complete [[Lab 3|labs/lab3.pdf]] & [[Lab 4|labs/lab4.pdf]]
----
[[Week 1, Part 2]] - The Filesystem
* Read Chapter 2 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]] - Navigation
* Complete [[Lab 2|labs/Lab2.pdf]]
[[Week 1, Part 1]] - Unix Intro
* Read Chapter 1 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]] - What is the Shell?
* Complete [[Lab 1|labs/Lab1.pdf]]
%/
<!--{{{-->
<div class='header' macro='gradient vert #000 #069'>
<div id='topTitle' class='headerShadow'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div id='topTitle' class='headerForeground'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div id='topMenu' refresh='content' tiddler='MainMenu'></div>
<div id='rightMenu' refresh='content' tiddler='RightMenu'></div>
</div>
<div id='sidebar'>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->
function onClickDefaultHome(e) {
story.closeAllTiddlers();
config.options.txtDefaultTiddlers = "";
saveOptionCookie('txtDefaultTiddlers');
var start = store.getTiddlerText("DefaultTiddlers");
if(start)
story.displayTiddlers(null,start.readBracketedList());
}
config.macros["defaultHome"] = {label: "Home", prompt: "Show the default tiddlers", title: "Home"};
config.macros.defaultHome.handler = function(place) {
createTiddlyButton(place,this.label,this.prompt,onClickDefaultHome);
}
config.macros.listTags = { text: "Hello" };
config.macros.listTags.handler = function(place,macroName,params)
{
var tagged = store.getTaggedTiddlers(params[0]);
var ul = createTiddlyElement(place,"ul",null,null,"");
for(var r=0;r<tagged.length;r++)
{
var li = createTiddlyElement(ul,"li",null,null,"");
createTiddlyLink(li,tagged[r].title,true);
}
}
Type the text for 'Plugins'
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="466" height="448" poster="" data-setup="{}">
<source src="video/PuttyProxy.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
/***
|Name:|QuickOpenTagPlugin|
|Description:|Changes tag links to make it easier to open tags as tiddlers|
|Version:|3.0.1 ($Rev: 3861 $)|
|Date:|$Date: 2008-03-08 10:53:09 +1000 (Sat, 08 Mar 2008) $|
|Source:|http://mptw.tiddlyspot.com/#QuickOpenTagPlugin|
|Author:|Simon Baird <simon.baird@gmail.com>|
|License:|http://mptw.tiddlyspot.com/#TheBSDLicense|
***/
//{{{
config.quickOpenTag = {
dropdownChar: (document.all ? "\u25bc" : "\u25be"), // the little one doesn't work in IE?
createTagButton: function(place,tag,excludeTiddler) {
// little hack so we can do this: <<tag PrettyTagName|RealTagName>>
var splitTag = tag.split("|");
var pretty = tag;
if (splitTag.length == 2) {
tag = splitTag[1];
pretty = splitTag[0];
}
var sp = createTiddlyElement(place,"span",null,"quickopentag");
createTiddlyText(createTiddlyLink(sp,tag,false),pretty);
var theTag = createTiddlyButton(sp,config.quickOpenTag.dropdownChar,
config.views.wikified.tag.tooltip.format([tag]),onClickTag);
theTag.setAttribute("tag",tag);
if (excludeTiddler)
theTag.setAttribute("tiddler",excludeTiddler);
return(theTag);
},
miniTagHandler: function(place,macroName,params,wikifier,paramString,tiddler) {
var tagged = store.getTaggedTiddlers(tiddler.title);
if (tagged.length > 0) {
var theTag = createTiddlyButton(place,config.quickOpenTag.dropdownChar,
config.views.wikified.tag.tooltip.format([tiddler.title]),onClickTag);
theTag.setAttribute("tag",tiddler.title);
theTag.className = "miniTag";
}
},
allTagsHandler: function(place,macroName,params) {
var tags = store.getTags(params[0]);
var filter = params[1]; // new feature
var ul = createTiddlyElement(place,"ul");
if(tags.length == 0)
createTiddlyElement(ul,"li",null,"listTitle",this.noTags);
for(var t=0; t<tags.length; t++) {
var title = tags[t][0];
if (!filter || (title.match(new RegExp('^'+filter)))) {
var info = getTiddlyLinkInfo(title);
var theListItem =createTiddlyElement(ul,"li");
var theLink = createTiddlyLink(theListItem,tags[t][0],true);
var theCount = " (" + tags[t][1] + ")";
theLink.appendChild(document.createTextNode(theCount));
var theDropDownBtn = createTiddlyButton(theListItem," " +
config.quickOpenTag.dropdownChar,this.tooltip.format([tags[t][0]]),onClickTag);
theDropDownBtn.setAttribute("tag",tags[t][0]);
}
}
},
// todo fix these up a bit
styles: [
"/*{{{*/",
"/* created by QuickOpenTagPlugin */",
".tagglyTagged .quickopentag, .tagged .quickopentag ",
" { margin-right:1.2em; border:1px solid #eee; padding:2px; padding-right:0px; padding-left:1px; }",
".quickopentag .tiddlyLink { padding:2px; padding-left:3px; }",
".quickopentag a.button { padding:1px; padding-left:2px; padding-right:2px;}",
"/* extra specificity to make it work right */",
"#displayArea .viewer .quickopentag a.button, ",
"#displayArea .viewer .quickopentag a.tiddyLink, ",
"#mainMenu .quickopentag a.tiddyLink, ",
"#mainMenu .quickopentag a.tiddyLink ",
" { border:0px solid black; }",
"#displayArea .viewer .quickopentag a.button, ",
"#mainMenu .quickopentag a.button ",
" { margin-left:0px; padding-left:2px; }",
"#displayArea .viewer .quickopentag a.tiddlyLink, ",
"#mainMenu .quickopentag a.tiddlyLink ",
" { margin-right:0px; padding-right:0px; padding-left:0px; margin-left:0px; }",
"a.miniTag {font-size:150%;} ",
"#mainMenu .quickopentag a.button ",
" /* looks better in right justified main menus */",
" { margin-left:0px; padding-left:2px; margin-right:0px; padding-right:0px; }",
"#topMenu .quickopentag { padding:0px; margin:0px; border:0px; }",
"#topMenu .quickopentag .tiddlyLink { padding-right:1px; margin-right:0px; }",
"#topMenu .quickopentag .button { padding-left:1px; margin-left:0px; border:0px; }",
"/*}}}*/",
""].join("\n"),
init: function() {
// we fully replace these builtins. can't hijack them easily
window.createTagButton = this.createTagButton;
config.macros.allTags.handler = this.allTagsHandler;
config.macros.miniTag = { handler: this.miniTagHandler };
config.shadowTiddlers["QuickOpenTagStyles"] = this.styles;
store.addNotification("QuickOpenTagStyles",refreshStyles);
}
}
config.quickOpenTag.init();
//}}}
| !Symbol | !Meaning | !Escape | !Not supported by |
| ^ |Start of line| | |
| $ |End of line| | |
| [ ] |Character Classes (match any one character listed) | | |
|~|Characters may be specified singly or in ranges| | |
| [^ ] |Negated character class (match any one character not listed| | |
| ? |Optional item. Match 0 or 1. | | sed |
| ( ) |Alternation (match any one of the sub-expressions)| | |
|~|Grouping| | |
|~|Capture backreference Access with \//n//| * | |
| {{{|}}} |Or. Match either expression it separates. Use with ( )| | |
| . |Any single character| | |
| + |Repetition: 1 or more. | | sed |
| * |Repetition: 0 or more| | |
| { } |Defined range of matches (bounds) {//min//,//max//} or {//min//,} or {//exactly//}| * | |
| \ |Suppress normal behavior of a metacharacter| | |
|~|Access a backreference: \//n//| | |
| \< |Match start of word.| * | bsd sed |
| \> |Match end of word.| * | bsd sed |
| !Symbol | !File Globbing | !Regex | !Regex Equivalent |
| ? |Exactly 1|0 or 1| . |
| { } |Sets|# of matches| ( ) |
/***
|Name:|RenameTagsPlugin|
|Description:|Allows you to easily rename or delete tags across multiple tiddlers|
|Version:|3.0 ($Rev: 5501 $)|
|Date:|$Date: 2008-06-10 23:11:55 +1000 (Tue, 10 Jun 2008) $|
|Source:|http://mptw.tiddlyspot.com/#RenameTagsPlugin|
|Author:|Simon Baird <simon.baird@gmail.com>|
|License|http://mptw.tiddlyspot.com/#TheBSDLicense|
Rename a tag and you will be prompted to rename it in all its tagged tiddlers.
***/
//{{{
config.renameTags = {
prompts: {
rename: "Rename the tag '%0' to '%1' in %2 tidder%3?",
remove: "Remove the tag '%0' from %1 tidder%2?"
},
removeTag: function(tag,tiddlers) {
store.suspendNotifications();
for (var i=0;i<tiddlers.length;i++) {
store.setTiddlerTag(tiddlers[i].title,false,tag);
}
store.resumeNotifications();
store.notifyAll();
},
renameTag: function(oldTag,newTag,tiddlers) {
store.suspendNotifications();
for (var i=0;i<tiddlers.length;i++) {
store.setTiddlerTag(tiddlers[i].title,false,oldTag); // remove old
store.setTiddlerTag(tiddlers[i].title,true,newTag); // add new
}
store.resumeNotifications();
store.notifyAll();
},
storeMethods: {
saveTiddler_orig_renameTags: TiddlyWiki.prototype.saveTiddler,
saveTiddler: function(title,newTitle,newBody,modifier,modified,tags,fields,clearChangeCount,created) {
if (title != newTitle) {
var tagged = this.getTaggedTiddlers(title);
if (tagged.length > 0) {
// then we are renaming a tag
if (confirm(config.renameTags.prompts.rename.format([title,newTitle,tagged.length,tagged.length>1?"s":""])))
config.renameTags.renameTag(title,newTitle,tagged);
if (!this.tiddlerExists(title) && newBody == "")
// dont create unwanted tiddler
return null;
}
}
return this.saveTiddler_orig_renameTags(title,newTitle,newBody,modifier,modified,tags,fields,clearChangeCount,created);
},
removeTiddler_orig_renameTags: TiddlyWiki.prototype.removeTiddler,
removeTiddler: function(title) {
var tagged = this.getTaggedTiddlers(title);
if (tagged.length > 0)
if (confirm(config.renameTags.prompts.remove.format([title,tagged.length,tagged.length>1?"s":""])))
config.renameTags.removeTag(title,tagged);
return this.removeTiddler_orig_renameTags(title);
}
},
init: function() {
merge(TiddlyWiki.prototype,this.storeMethods);
}
}
config.renameTags.init();
//}}}
Type the text for 'Resources'
<<toggleSideBar "" "Toggle Sidebar" hide>>
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="858" height="480" poster="" data-setup="{}">
<source src="video/ssh.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
!! Grading
Shell scripting labs will follow a more traditional grading approach where only meeting the objectives of the script will receive a B grade, or 8.5 / 10. A grade beyond that will require exceeding the minimum expectations.
15% of shell scripting lab grades will be reserved for style, efficiency, and completeness. /% For example, if the %/
| !Grade | !Quality |
| A |Exceptional - Exceeds expectations|
| B |Average - Meets expectations|
| C |Satisfactory - meets some expectations|
| D |Poor - minimally meets expectations|
| F |Does not meet minimal expectations|
!! Requirements
The following procedure must be followed for submitting shell scripting labs. Improperly submitted scripting labs will not be accepted.
The end goal of this process is to submit a single PDF containing these three components:
''a.'' Original lab assignment sheet as a cover page
''b.'' Your shell scripts
''c.'' A demonstration of your scripts
''1.'' Create the directory ~/bin/. Save all lab shell scripts in this directory with the naming convention ''ncs205-lab//xx//-q//yy//.sh'' where ''//xx//'' is the lab number and ''//yy//'' is the question number. It would make things easier for you if you always use two digits for //xx// and //yy//.
I may refer to the script files if I need to execute/test any of your scripts.
''2.'' A proper shebang must be added as the first line of your shell scripts.
''3.'' The following header must be placed at the top of each script file, immediately after the shebang:
{{{
# File name:
# Author:
# Date Written:
# Assignment:
# Purpose:
# Description:
#
#
#
}}}
The //Purpose// field should contain a brief, one-line summary of what your script is accomplishing. The Description field should contain more detailed information regarding how it is accomplishing that goal or any additional information helpful to understand the function of your script.
''4.'' Make use of comments throughout your script to document and convey what you're doing.
Long lines should be nicely wrapped with carriage returns. Cut long lines at about column 60. (makes it easier to read and print)
* You can escape the newline with a {{Command{''\''}}} to continue a long line of commands on the next line. For example:
{{{
dig axfr ${user}.ncs205.net @ns1.${user}.ncs205.net | \
grep -v ^\; | sort | md5sum | cut -d " " -f 1
}}}
{{Note{''Note:'' The remaining two steps are for labs which are //only// scripts and do not contain input boxes}}}
''5.'' Use the {{Command{script}}} command to launch a recording shell, saving the output to {{File{~/bin/labxx.raw}}} where //xx// is the lab number. Demonstrate execution of your scripts within this recording shell.
* Execute {{Command{script ~/bin/labxx.raw}}} to start the recording shell, saving output to the filename specified as the first command line argument
* Run your scripts. Everything you type and all output will be recorded in the file {{File{~/bin/labxx.raw}}}.
* Be sure you do not have excessive errors in the recording. Pressing the backspace key will be recorded as a separate keystroke and make your demonstration harder to read.
* Type {{Command{exit}}} to terminate the recording shell.
* If you examine the {{File{~/bin/labxx.raw}}}, you will see it contains a lot of control characters. The {{Command{ ansifilter }}} command will remove them.
** {{Command{ ansifilter -o ~/bin/labxx.out ~/bin/labxx.raw }}}
''6.'' Create a PDF of your scripts to save to the {{File{/opt/pub/ncs205/submit/}}} directory:
* The comments below explain what's going on.
* The first paragraph only explains the {{Command{enscript}}} command. The second paragraph contains the two commands you'll need to execute to submit your lab.
{{{
# enscript is a great tool for formatting documents about to be printed or saved as a PDF.
# The following command string will gather your labs and the output from the demo of your scripts, apply some
# document formatting, and display PostScript on STDOUT.
# The -o - option for enscript instructs the command to sent its output to STDOUT instead of saving it to a file
enscript -2jr -C -o - ~/bin/ncs205-labxx-q??.sh ~/bin/labxx.out
# PostScript is mostly the language of printers and isn't as useful on modern PCs. Instead of working with
# native PostScript or displaying STDOUT to the screen, let's convert it to PDF and save to a file.
# Caution! Only run this command when you are ready to submit your scripts.
# *** These are the command you will execute to submit your scripting labs ***
enscript -2jr -C -o - ~/bin/ncs205-labxx-q??.sh ~/bin/labxx.out | ps2pdf - ~/bin/ncs205-labxx-username.pdf
# Note: The - in the above ps2pdf command instructs the command to obtain its input from STDIN.
# The next command will combine the lab assignment PDF as a cover page with the PDF you just created containing your scripts,
# saving the output to the class submit directory. This is the PDF you are submitting for my review.
cpdf /opt/pub/ncs205/labs/labxx.pdf ~/bin/ncs205-labxx-username.pdf -o /opt/pub/ncs205/submit/ncs205-labxx-username.pdf
# Be sure to follow the standard lab naming scheme and change the xx and username to proper values
# The nice thing about using standard naming conventions is it makes everything easy to script.
# Rather than have to search for these commands for every scripting lab you need to submit, you might as well make a dynamic script out of it.
# (Hint: This will be a future assignment. It'll be more useful to you if you start working on it now.)
}}}
''7.'' Preview your submitted PDF
> Download the PDF saved to the {{File{submit/}}} directory to check your work. If you skip this important step and submit a PDF for review that does not contain your scripts, you will either receive no credit for the lab or a late penalty for resubmitting.
{{Note{''Note:'' The video below demonstrates the deprecated {{Command{a2ps}}} command. The new process instead uses {{Command{enscript}}} which is a drop-in replacement. The video has not yet been updated to reflect this change}}}
<html>
<center>
<video id="my-video" class="video-js" controls preload="auto" width="1000" height="662" poster="" data-setup="{}">
<source src="video/scripts.mp4" type='video/mp4'>
<p class="vjs-no-js">
To view this video please enable JavaScript, and consider upgrading to a web browser that
<a href="http://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
</p>
</video>
<script src="https://vjs.zencdn.net/7.8.2/video.min.js"></script>
</center>
</html>
The following general best practices will make it much easier for you to write your scripts and help ensure they're correct. Good practices will also help others understand what your scripts are doing.
!! Items to include within your script:
1. Start your script with a proper shebang.
<<<
The first line of your script, the shebang (shell bang because it starts with a {{Monospace{#!}}}) must contain the shell interpreter to use. Since these scripts are otherwise just text files, this line indicates what language your script was written in. Bourne shell scripts should have this on the first line: {{Monospace{''#!/bin/sh''}}}
<<<
2. Provide clarity
<<<
You may not be the only one using your script. Others may have to look at the code to troubleshoot or make modifications later. Or you may have to come back years later to decipher what you did and what your past self was thinking. Good usability should be built into everything you do. Be sure your code is well laid out and clear to follow. If you make it so I have a hard time understanding your logic and workflow with these simple scripts when I know the objective, then others will surely have difficulty understanding more complex ones later.
<<<
3. Add proper header information to your script
<<<
Identifying the script author, creation date, and purpose at the top of your script is important. I encounter far too many scripts written by others at my day job which lack this detail. Who do I go to if there's issues with this script? How old is this? Is it ancient and not valid anymore as processes have changed?
<<<
4. Add comments to explain what you are doing and the purpose of each section
<<<
Comments should be utilized to explain what you are doing or your methodology if the command itself does not make it clear. Simple and obvious commands and processes may be self-documenting. Others should have comments to explain them. It's also helpful to cut your comments at about 60 characters so they don't wrap to the next line in the terminal or PDF copy.
Comments should be concise and professional. Unnecessary verbosity can cause your meaning to be lost.
<<<
5. Cut long lines
<<<
Very long command lines can be cut and extended to the next line to help readability. The backslash can be used to escape the newline if it's the last character in the line. By escaping the newline, your long command string can then continue on the next line.
Consider these two examples. Notice the use of the backslash in the long command string.
{{{
# Combine lab sheet as a cover page with PDF containing shell
# scripts, saving output to submit directory.
cpdf /opt/pub/ncs205/labs/labxx.pdf ~/bin/ncs205-labxx-username.pdf -o /opt/pub/ncs205/submit/ncs205-labxx-username.pdf
}}}
{{{
# Combine lab sheet as a cover page with PDF containing shell
# scripts, saving output to submit directory.
cpdf /opt/pub/ncs205/labs/labxx.pdf ~/bin/ncs205-labxx-username.pdf \
-o /opt/pub/ncs205/submit/ncs205-labxx-username.pdf
}}}
<<<
6. Use meaningful variable names
<<<
Choose descriptive variable names to make your code self-documenting. If your variable should contain a file name, the name of the variable should be something line {{Monospace{''//filename//''}}}, not something generic like {{Monospace{''//var//''}}}. Also avoid using one-letter variable names (e.g., {{Monospace{x}}} or {{Monospace{i}}} when possible.
<<<
7. Use double quotes with variables (when you can)
<<<
Enclose variables in double quotes to handle spaces and special characters correctly, especially when using variables within if-statements. There may be times you cannot use quotes, but these are the exception and not the norm.
<<<
8. Properly indent your code
<<<
Follow a consistent and readable indentation style to make your code visually appealing and logic easier to follow. Any control structures should be properly indented.
Consider these two small examples. The first with no indentation and the second with proper indentation. The issue becomes more pronounced as control structures become nested and larger.
{{{
if [ -n "$note" ]
then
echo $note
fi
}}}
{{{
if [ -n "$note" ]
then
echo $note
fi
}}}
<<<
9. Avoid hardcoding values
<<<
Minimize hardcoding values in your script. Use variables or configuration files to store and manage settings. For example, if you have to refer to a file later in your script, perhaps defining a variable at the top of the script would make the script easier to maintain?
<<<
10. Avoid code duplication
<<<
Avoid using the same block of code in multiple places within your script. If you find that you have done this, then look at ways to either adjust the script's logic or approach to remove the code reuse. We aren't covering functions here, but proper use of functions can typically help with this in larger scripts.
<<<
!! Script validation and debugging
1. Debugging statements may be helpful
<<<
You can run your script with {{Monospace{bash -x}}} to see the actual lines that are being executed in your script, after any substitutions occur. This can be a great tool for identifying why something isn't working out the way it should.
{{{
bash -x scriptname.sh
}}}
Debugging {{Monospace{echo}}} statements throughout your script can help show any intermediate values as they're being used. This way we can more easily visualize what these variables contain as the script is executing.
{{{
echo DEBUG: $mdate : $mtime : $oneday : $threeday
}}}
Just remember to remove or comment them before submitting the script
<<<
2. Test your script
<<<
Write test cases to ensure your script works as expected, especially when making changes or updates. Be sure to test different scenarios and test for failure. Don't just provide what's expected. What happens if unexpected input is provided.
We won't have all the tools to handle this when we start our shell scripting work, but we should by the end.
<<<
3. Use version control or make backups
<<<
Keep your scripts in a version control system (e.g., Git) to track changes and easily roll back to an old version if you need to.
Even if you don't use a full version control system, always have a back-out plan if a change doesn't work. Maybe back up the file before making changes so you can revert or comment out the old line until you're sure its no longer needed. My convention is to append a timestamp of the backup to the filename in the format - ~YYMMDD-HHMM. For example, 230320-2244 for 10:44pm on Mar 20. This timestamp then also sorts well in a directory listing. The {{Command{diff}}} command can be used to compare different versions of a file.
The same concept works well for configuration files in our next class material.
<<<
*Shell scripting quick reference: http://www.math.uga.edu/~caner/08reu/shell.html
*Awk one liners: http://www.catonmat.net/blog/wp-content/uploads/2008/09/awk1line.txt
*Sed one liners: http://www.catonmat.net/blog/wp-content/uploads/2008/09/sed1line.txt
<<search>><<closeAll>><<collapseAll>><<expandAll>><<permaview>><<newTiddler>><<saveChanges>><<slider chkSliderOptionsPanel OptionsPanel "options »" "Change TiddlyWiki advanced options">><<slider chkSliderContents [[TabContents]] 'contents »' 'contents'>>
/*{{{*/
#sidebar {
color: #000;
background: transparent;
}
#sidebarOptions {
background: #fff;
}
#sidebarOptions .button {
color: #999;
}
#sidebarOptions .button:hover {
color: #000;
background: #fff;
border-color:white;
}
#sidebarOptions .button:active {
color: #000;
background: #fff;
}
#sidebarOptions .sliderPanel {
background: transparent;
}
#sidebarOptions .sliderPanel A:hover {
color: #000;
background: #fff;
}
#sidebarOptions .sliderPanel A:active {
color: #000;
background: #fff;
}
.sidebarSubHeading {
color: #000;
}
#sidebarOptions .sliderPanel .tabSelected{
border: 1px solid #ccc;
background-color: #fff;
margin: 0px;
padding-top: 5px;
padding-bottom: 0px;
padding-left: 2px;
padding-right: 2px;
-moz-border-radius-topleft: 1em;
-moz-border-radius-topright: 1em;}
#sidebarOptions .sliderPanel .tabUnselected{
border: 1px solid #ccc;
background-color: #eee;
margin: 0px;
padding-top: 5px;
padding-bottom: 0px;
padding-left: 2px;
padding-right: 2px;
-moz-border-radius-topleft: 1em;
-moz-border-radius-topright: 1em;}
#sidebarTabs .tabContents .tiddlyLink:hover {
background: #fff;
color: #000;
}
#sidebarTabs .tabContents {
color: #000;
}
#sidebarTabs .button {
color: #666;
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 2px solid #ccc;
border-right: 2px solid #ccc;
}
#sidebarTabs .tabContents .button:hover {
color: #000;
background: #fff;
}
.tagging, .tagged {
padding: 0.5em;
background-color: #eee;
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 3px solid #ccc;
border-right: 3px solid #ccc;
-moz-border-radius: 1em; }
/*}}}*/
NCS 205 - Introduction to Linux - SUNY Polytechnic Institute
[[HorizontalMainMenuStyles]]
[[SideBarStyles]]
[[TagglyTaggingStyles]]
/*{{{*/
body { background: #eee; }
h1 {font-size:2.0em; }
h2 { color: #000; background: transparent; text-decoration: underline; }
h3 { margin: 0.0em; color: #000; background: transparent; }
h4,h5 { color: #000; background: transparent; }
h1 {
margin: 4px 0 4px 0;
padding: 5px;
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::PrimaryPale]];
}
ul {
margin-top: 0;
margin-bottom: 0;
}
.headerShadow {
padding: 1.0em; }
.headerForeground {
padding: 1.0em; }
.selected .tagging, .selected .tagged {
padding: 0.5em;
background-color: #eee;
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 3px solid #ccc;
border-right: 3px solid #ccc;
-moz-border-radius: 1em; }
.shadow .title {
color: #999; }
.siteTitle {
font-size: 2.5em; }
.siteSubtitle {
font-size: 1.0em; }
.subtitle {
font-size: 0.8em;
}
.tagging, .tagged {
padding: 0.5em;
background-color: #eee;
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 3px solid #ccc;
border-right: 3px solid #ccc;
-moz-border-radius: 1em; }
.tiddler {
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 3px solid #ccc;
border-right: 3px solid #ccc;
margin: 0.5em;
background:#fff;
padding: 0.5em;
-moz-border-radius: 1em; }
.darkMode .tiddler {
border-top: 1px solid #ccc;
border-left: 1px solid #ccc;
border-bottom: 3px solid #ccc;
border-right: 3px solid #ccc;
margin: 0.5em;
background: #000;
padding: 0.5em;
-moz-border-radius: 1em; }
.title {
color:black;
font-size: 1.5em; }
.tabSelected{
padding-top: 0.0em;
padding-left: 0.5em;
padding-right: 0.5em;
-moz-border-radius-topleft: 0.5em;
-moz-border-radius-topright: 0.5em;}
.tabUnselected {
padding-top: 0.0em;
padding-left: 0.5em;
padding-right: 0.5em;
-moz-border-radius-topleft: 0.5em;
-moz-border-radius-topright: 0.5em;}
.tabContents {
margin: 0px;
padding-top: 0px;
padding-bottom: 0px;
padding-left: 2px;
padding-right: 2px;
-moz-border-radius: 1em; }
.viewer .listTitle {
list-style-type: none;
}
.viewer pre {
background-color: #f8f8ff;
border-color: #ddf; }
#messageArea { background-color:#bde; border-color:#8ab; border-width:4px; border-style:dotted; font-size:90%; }
#messageArea .button { text-decoration:none; font-weight:bold; background:transparent; border:0px; }
#messageArea .button:hover {background: #acd;}
/*}}}*/
/*{{{*/
.Command{color: fuchsia;font-size: 10pt;font-family: Courier, monospace;margin-left: 2px;margin-right: 2px;}
.Commandi{color: fuchsia;font-size: 10pt;font-family: Courier, monospace;margin-left: 20px;margin-right: 2px;}
.File{color: #4c7fbc;font-size: 10pt;font-family: Courier, monospace;margin-left: 2px;margin-right: 2px; font-weight:bold;}
.Remove{background-color: orange}
.Host{color: #0f9791;font-size: 10pt;font-family: Courier, monospace;margin-left: 2px;margin-right: 2px; font-weight:bold;}
.Note{display:block;background-color:#e9ffdb;border:1px solid darkgreen;margin: 0 2em 0 2em;padding:5px 5px 5px 5px;}
.Warning{display:block;background-color:#ffee88; border:2px solid darkorange;margin: 0 2em 0 2em;padding:5px 5px 5px 5px;}
.Monospaced{font-size: 10pt;font-family: Courier, monospace;margin-left: 2px;margin-right: 2px;}
.Commands{background-color:#F0F0FF; font-size: 10pt;font-family: Courier, monospace;margin-left: 2px;margin-right: 2px;padding:5px 5px 5px 5px;}
/*}}}*/
.HideSideBarButton {margin-left: 3em;}
.viewer div.centeredTable {
text-align: center;
}
.viewer div.centeredTable table {
margin: 0 auto;
text-align: left;
}
.viewer table.borderless,
.viewer table.borderless * {
border: 0;
}
/*{{{*/
body {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
a {color:[[ColorPalette::PrimaryMid]];}
a:hover {background-color:[[ColorPalette::PrimaryMid]]; color:[[ColorPalette::Background]];}
a img {border:0;}
h1,h2,h3,h4,h5,h6 {color:[[ColorPalette::SecondaryDark]]; background:transparent;}
h1 {border-bottom:2px solid [[ColorPalette::PrimaryLight]];} */
h2,h3 {border-bottom:1px solid [[ColorPalette::TertiaryLight]];}
.button {color:[[ColorPalette::PrimaryDark]]; border:1px solid [[ColorPalette::Background]];}
.button:hover {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::SecondaryLight]]; border-color:[[ColorPalette::SecondaryMid]];}
.button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::SecondaryDark]];}
.header {background:[[ColorPalette::PrimaryMid]];}
.headerShadow {color:[[ColorPalette::Foreground]];}
.headerShadow a {font-weight:normal; color:[[ColorPalette::Foreground]];}
.headerForeground {color:[[ColorPalette::Background]];}
.headerForeground a {font-weight:normal; color:[[ColorPalette::PrimaryPale]];}
.tabSelected{color:[[ColorPalette::PrimaryDark]];
background:[[ColorPalette::TertiaryPale]];
border-left:1px solid [[ColorPalette::TertiaryLight]];
border-top:1px solid [[ColorPalette::TertiaryLight]];
border-right:1px solid [[ColorPalette::TertiaryLight]];
}
.tabUnselected {color:[[ColorPalette::Background]]; background:[[ColorPalette::TertiaryMid]];}
.tabContents {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::TertiaryPale]]; border:1px solid [[ColorPalette::TertiaryLight]];}
.tabContents .button {border:0;}
#sidebar {}
#sidebarOptions input {border:1px solid [[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel {background:[[ColorPalette::PrimaryPale]];}
#sidebarOptions .sliderPanel a {border:none;color:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:hover {color:[[ColorPalette::Background]]; background:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:active {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::Background]];}
.wizard {background:[[ColorPalette::PrimaryPale]]; border:1px solid [[ColorPalette::PrimaryMid]];}
.wizard h1 {color:[[ColorPalette::PrimaryDark]]; border:none;}
.wizard h2 {color:[[ColorPalette::Foreground]]; border:none;}
.wizardStep {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];
border:1px solid [[ColorPalette::PrimaryMid]];}
.wizardStep.wizardStepDone {background:[[ColorPalette::TertiaryLight]];}
.wizardFooter {background:[[ColorPalette::PrimaryPale]];}
.wizardFooter .status {background:[[ColorPalette::PrimaryDark]]; color:[[ColorPalette::Background]];}
.wizard .button {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryLight]]; border: 1px solid;
border-color:[[ColorPalette::SecondaryPale]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryPale]];}
.wizard .button:hover {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Background]];}
.wizard .button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::Foreground]]; border: 1px solid;
border-color:[[ColorPalette::PrimaryDark]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryDark]];}
.wizard .notChanged {background:transparent;}
.wizard .changedLocally {background:#80ff80;}
.wizard .changedServer {background:#8080ff;}
.wizard .changedBoth {background:#ff8080;}
.wizard .notFound {background:#ffff80;}
.wizard .putToServer {background:#ff80ff;}
.wizard .gotFromServer {background:#80ffff;}
#messageArea {border:1px solid [[ColorPalette::SecondaryMid]]; background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]];}
#messageArea .button {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::SecondaryPale]]; border:none;}
.popupTiddler {background:[[ColorPalette::TertiaryPale]]; border:2px solid [[ColorPalette::TertiaryMid]];}
.popup {background:[[ColorPalette::TertiaryPale]]; color:[[ColorPalette::TertiaryDark]]; border-left:1px solid [[ColorPalette::TertiaryMid]]; border-top:1px solid [[ColorPalette::TertiaryMid]]; border-right:2px solid [[ColorPalette::TertiaryDark]]; border-bottom:2px solid [[ColorPalette::TertiaryDark]];}
.popup hr {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::PrimaryDark]]; border-bottom:1px;}
.popup li.disabled {color:[[ColorPalette::TertiaryMid]];}
.popup li a, .popup li a:visited {color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:active {background:[[ColorPalette::SecondaryPale]]; color:[[ColorPalette::Foreground]]; border: none;}
.popupHighlight {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
.listBreak div {border-bottom:1px solid [[ColorPalette::TertiaryDark]];}
.tiddler .defaultCommand {font-weight:bold;}
.shadow .title {color:[[ColorPalette::TertiaryDark]];}
.title {color:[[ColorPalette::SecondaryDark]];}
.subtitle {color:[[ColorPalette::TertiaryDark]];}
.toolbar {color:[[ColorPalette::PrimaryMid]];}
.toolbar a {color:[[ColorPalette::TertiaryLight]];}
.selected .toolbar a {color:[[ColorPalette::TertiaryMid]];}
.selected .toolbar a:hover {color:[[ColorPalette::Foreground]];}
.tagging, .tagged {border:1px solid [[ColorPalette::TertiaryPale]]; background-color:[[ColorPalette::TertiaryPale]];}
.selected .tagging, .selected .tagged {background-color:[[ColorPalette::TertiaryLight]]; border:1px solid [[ColorPalette::TertiaryMid]];}
.tagging .listTitle, .tagged .listTitle {color:[[ColorPalette::PrimaryDark]];}
.tagging .button, .tagged .button {border:none;}
.footer {color:[[ColorPalette::TertiaryLight]];}
.selected .footer {color:[[ColorPalette::TertiaryMid]];}
.sparkline {background:[[ColorPalette::PrimaryPale]]; border:0;}
.sparktick {background:[[ColorPalette::PrimaryDark]];}
.error, .errorButton {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Error]];}
.warning {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryPale]];}
.lowlight {background:[[ColorPalette::TertiaryLight]];}
.zoomer {background:none; color:[[ColorPalette::TertiaryMid]]; border:3px solid [[ColorPalette::TertiaryMid]];}
.imageLink, #displayArea .imageLink {background:transparent;}
.annotation {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border:2px solid [[ColorPalette::SecondaryMid]];}
.viewer .listTitle {list-style-type:none; margin-left:-2em;}
.viewer .button {border:1px solid [[ColorPalette::SecondaryMid]];}
.viewer blockquote {border-left:3px solid [[ColorPalette::TertiaryDark]];}
.viewer table, table.twtable {border:2px solid [[ColorPalette::TertiaryDark]];}
.viewer th, .viewer thead td, .twtable th, .twtable thead td {background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::Background]];}
.viewer td, .viewer tr, .twtable td, .twtable tr {border:1px solid [[ColorPalette::TertiaryDark]];}
.viewer pre {border:1px solid [[ColorPalette::SecondaryLight]]; background:[[ColorPalette::SecondaryPale]];}
.viewer code {color:[[ColorPalette::SecondaryDark]];}
.viewer hr {border:0; border-top:dashed 1px [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::TertiaryDark]];}
.highlight, .marked {background:[[ColorPalette::SecondaryLight]];}
.editor input {border:1px solid [[ColorPalette::PrimaryMid]];}
.editor textarea {border:1px solid [[ColorPalette::PrimaryMid]]; width:100%;}
.editorFooter {color:[[ColorPalette::TertiaryMid]];}
#backstageArea {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::TertiaryMid]];}
#backstageArea a {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstageArea a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; }
#backstageArea a.backstageSelTab {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
#backstageButton a {background:none; color:[[ColorPalette::Background]]; border:none;}
#backstageButton a:hover {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstagePanel {background:[[ColorPalette::Background]]; border-color: [[ColorPalette::Background]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]];}
.backstagePanelFooter .button {border:none; color:[[ColorPalette::Background]];}
.backstagePanelFooter .button:hover {color:[[ColorPalette::Foreground]];}
#backstageCloak {background:[[ColorPalette::Foreground]]; opacity:0.6; filter:'alpha(opacity:60)';}
/*}}}*/
/*{{{*/
body {
background: [[ColorPalette::Background]];
color: [[ColorPalette::Foreground]];
}
a{
color: [[ColorPalette::PrimaryMid]];
}
a:hover{
background: [[ColorPalette::PrimaryMid]];
color: [[ColorPalette::Background]];
}
a img{
border: 0;
}
h1,h2,h3,h4,h5 {
color: [[ColorPalette::SecondaryDark]];
background: [[ColorPalette::PrimaryPale]];
}
.button {
color: [[ColorPalette::PrimaryDark]];
border: 1px solid [[ColorPalette::Background]];
}
.button:hover {
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::SecondaryLight]];
border-color: [[ColorPalette::SecondaryMid]];
}
.button:active {
color: [[ColorPalette::Background]];
background: [[ColorPalette::SecondaryMid]];
border: 1px solid [[ColorPalette::SecondaryDark]];
}
.header {
background: [[ColorPalette::PrimaryMid]];
}
.headerShadow {
color: [[ColorPalette::Foreground]];
}
.headerShadow a {
font-weight: normal;
color: [[ColorPalette::Foreground]];
}
.headerForeground {
color: [[ColorPalette::Background]];
}
.headerForeground a {
font-weight: normal;
color: [[ColorPalette::PrimaryPale]];
}
.tabSelected{
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::TertiaryPale]];
border-left: 1px solid [[ColorPalette::TertiaryLight]];
border-top: 1px solid [[ColorPalette::TertiaryLight]];
border-right: 1px solid [[ColorPalette::TertiaryLight]];
}
.tabUnselected {
color: [[ColorPalette::Background]];
background: [[ColorPalette::TertiaryMid]];
}
.tabContents {
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::TertiaryPale]];
border: 1px solid [[ColorPalette::TertiaryLight]];
}
.tabContents .button {
border: 0;}
#sidebar {
}
#sidebarOptions input {
border: 1px solid [[ColorPalette::PrimaryMid]];
}
#sidebarOptions .sliderPanel {
background: [[ColorPalette::PrimaryPale]];
}
#sidebarOptions .sliderPanel a {
border: none;
color: [[ColorPalette::PrimaryMid]];
}
#sidebarOptions .sliderPanel a:hover {
color: [[ColorPalette::Background]];
background: [[ColorPalette::PrimaryMid]];
}
#sidebarOptions .sliderPanel a:active {
color: [[ColorPalette::PrimaryMid]];
background: [[ColorPalette::Background]];
}
.wizard {
background: [[ColorPalette::SecondaryLight]];
border-top: 1px solid [[ColorPalette::SecondaryMid]];
border-left: 1px solid [[ColorPalette::SecondaryMid]];
}
.wizard h1 {
color: [[ColorPalette::SecondaryDark]];
}
.wizard h2 {
color: [[ColorPalette::Foreground]];
}
.wizardStep {
background: [[ColorPalette::Background]];
border-top: 1px solid [[ColorPalette::SecondaryMid]];
border-bottom: 1px solid [[ColorPalette::SecondaryMid]];
border-left: 1px solid [[ColorPalette::SecondaryMid]];
}
.wizard .button {
color: [[ColorPalette::Background]];
background: [[ColorPalette::PrimaryMid]];
border-top: 1px solid [[ColorPalette::PrimaryLight]];
border-right: 1px solid [[ColorPalette::PrimaryDark]];
border-bottom: 1px solid [[ColorPalette::PrimaryDark]];
border-left: 1px solid [[ColorPalette::PrimaryLight]];
}
.wizard .button:hover {
color: [[ColorPalette::PrimaryLight]];
background: [[ColorPalette::PrimaryDark]];
border-color: [[ColorPalette::PrimaryLight]];
}
.wizard .button:active {
color: [[ColorPalette::Background]];
background: [[ColorPalette::PrimaryMid]];
border-top: 1px solid [[ColorPalette::PrimaryLight]];
border-right: 1px solid [[ColorPalette::PrimaryDark]];
border-bottom: 1px solid [[ColorPalette::PrimaryDark]];
border-left: 1px solid [[ColorPalette::PrimaryLight]];
}
#messageArea {
border: 1px solid [[ColorPalette::SecondaryDark]];
background: [[ColorPalette::SecondaryMid]];
color: [[ColorPalette::PrimaryDark]];
}
#messageArea .button {
padding: 0.2em 0.2em 0.2em 0.2em;
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::Background]];
}
.popup {
background: [[ColorPalette::PrimaryLight]];
border: 1px solid [[ColorPalette::PrimaryMid]];
}
.popup hr {
color: [[ColorPalette::PrimaryDark]];
background: [[ColorPalette::PrimaryDark]];
border-bottom: 1px;
}
.popup li.disabled {
color: [[ColorPalette::PrimaryMid]];
}
.popup li a, .popup li a:visited {
color: [[ColorPalette::TertiaryPale]];
border: none;
}
.popup li a:hover {
background: [[ColorPalette::PrimaryDark]];
color: [[ColorPalette::Background]];
border: none;
}
.tiddler .defaultCommand {
font-weight: bold;
}
.shadow .title {
color: [[ColorPalette::TertiaryDark]];
}
.title {
color: [[ColorPalette::SecondaryDark]];
}
.subtitle {
color: [[ColorPalette::TertiaryDark]];
}
.toolbar {
color: [[ColorPalette::PrimaryMid]];
}
.tagging, .tagged {
border: 1px solid [[ColorPalette::TertiaryPale]];
background-color: [[ColorPalette::TertiaryPale]];
}
.selected .tagging, .selected .tagged {
background-color: [[ColorPalette::TertiaryLight]];
border: 1px solid [[ColorPalette::TertiaryMid]];
}
.tagging .listTitle, .tagged .listTitle {
color: [[ColorPalette::PrimaryDark]];
}
.tagging .button, .tagged .button {
border: none;
}
.footer {
color: [[ColorPalette::TertiaryLight]];
}
.selected .footer {
color: [[ColorPalette::TertiaryMid]];
}
.sparkline {
background: [[ColorPalette::PrimaryPale]];
border: 0;
}
.sparktick {
background: [[ColorPalette::PrimaryDark]];
}
.error, .errorButton {
color: [[ColorPalette::Foreground]];
background: [[ColorPalette::Error]];
}
.warning {
color: [[ColorPalette::Foreground]];
background: [[ColorPalette::SecondaryPale]];
}
.cascade {
background: [[ColorPalette::TertiaryPale]];
color: [[ColorPalette::TertiaryMid]];
border: 1px solid [[ColorPalette::TertiaryMid]];
}
.imageLink, #displayArea .imageLink {
background: transparent;
}
.viewer .listTitle {list-style-type: none; margin-left: -2em;}
.viewer .button {
border: 1px solid [[ColorPalette::SecondaryMid]];
}
.viewer blockquote {
border-left: 3px solid [[ColorPalette::TertiaryDark]];
}
.viewer table {
border: 2px solid [[ColorPalette::TertiaryDark]];
}
.viewer th, thead td {
background: [[ColorPalette::SecondaryMid]];
border: 1px solid [[ColorPalette::TertiaryDark]];
color: [[ColorPalette::Background]];
}
.viewer td, .viewer tr {
border: 1px solid [[ColorPalette::TertiaryDark]];
}
.viewer pre {
border: 1px solid [[ColorPalette::SecondaryLight]];
background: [[ColorPalette::SecondaryPale]];
}
.viewer code {
color: [[ColorPalette::SecondaryDark]];
}
.viewer hr {
border: 0;
border-top: dashed 1px [[ColorPalette::TertiaryDark]];
color: [[ColorPalette::TertiaryDark]];
}
.highlight, .marked {
background: [[ColorPalette::SecondaryLight]];
}
.editor input {
border: 1px solid [[ColorPalette::PrimaryMid]];
}
.editor textarea {
border: 1px solid [[ColorPalette::PrimaryMid]];
width: 100%;
}
.editorFooter {
color: [[ColorPalette::TertiaryMid]];
}
/*}}}*/
/*{{{*/
* html .tiddler {height:1%;}
body {font-size:.75em; font-family:arial,helvetica; margin:0; padding:0;}
h1,h2,h3,h4,h5,h6 {font-weight:bold; text-decoration:none;}
h1,h2,h3 {padding-bottom:1px; margin-top:1.2em;margin-bottom:0.3em;}
h4,h5,h6 {margin-top:0em;margin-bottom:0em;}
h1 {font-size:1.35em;}
h2 {font-size:1.25em;}
h3 {font-size:1.1em;}
h4 {font-size:1em;}
h5 {font-size:.9em;}
hr {height:1px;}
a {text-decoration:none;}
dt {font-weight:bold;}
ol {list-style-type:decimal;}
ol ol {list-style-type:lower-alpha;}
ol ol ol {list-style-type:lower-roman;}
ol ol ol ol {list-style-type:decimal;}
ol ol ol ol ol {list-style-type:lower-alpha;}
ol ol ol ol ol ol {list-style-type:lower-roman;}
ol ol ol ol ol ol ol {list-style-type:decimal;}
.txtOptionInput {width:11em;}
#contentWrapper .chkOptionInput {border:0;}
.externalLink {text-decoration:underline;}
.indent {margin-left:3em;}
.outdent {margin-left:3em; text-indent:-3em;}
code.escaped {white-space:nowrap;}
.tiddlyLinkExisting {font-weight:bold;}
.tiddlyLinkNonExisting {font-style:italic;}
/* the 'a' is required for IE, otherwise it renders the whole tiddler in bold */
a.tiddlyLinkNonExisting.shadow {font-weight:bold;}
#mainMenu .tiddlyLinkExisting,
#mainMenu .tiddlyLinkNonExisting,
#sidebarTabs .tiddlyLinkNonExisting {font-weight:normal; font-style:normal;}
#sidebarTabs .tiddlyLinkExisting {font-weight:bold; font-style:normal;}
.header {position:relative;}
.header a:hover {background:transparent;}
.headerShadow {position:relative; padding:4.5em 0em 1em 1em; left:-1px; top:-1px;}
.headerForeground {position:absolute; padding:4.5em 0em 1em 1em; left:0px; top:0px;}
.siteTitle {font-size:3em;}
.siteSubtitle {font-size:1.2em;}
#mainMenu {position:absolute; left:0; width:10em; text-align:right; line-height:1.6em; padding:1.5em 0.5em 0.5em 0.5em; font-size:1.1em;}
#sidebar {position:absolute; right:3px; width:16em; font-size:.9em;}
#sidebarOptions {padding-top:0.3em;}
#sidebarOptions a {margin:0em 0.2em; padding:0.2em 0.3em; display:block;}
#sidebarOptions input {margin:0.4em 0.5em;}
#sidebarOptions .sliderPanel {margin-left:1em; padding:0.5em; font-size:.85em;}
#sidebarOptions .sliderPanel a {font-weight:bold; display:inline; padding:0;}
#sidebarOptions .sliderPanel input {margin:0 0 .3em 0;}
#sidebarTabs .tabContents {width:15em; overflow:hidden;}
.wizard {padding:0.1em 1em 0em 2em;}
.wizard h1 {font-size:2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizard h2 {font-size:1.2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizardStep {padding:1em 1em 1em 1em;}
.wizard .button {margin:0.5em 0em 0em 0em; font-size:1.2em;}
.wizardFooter {padding:0.8em 0.4em 0.8em 0em;}
.wizardFooter .status {padding:0em 0.4em 0em 0.4em; margin-left:1em;}
.wizard .button {padding:0.1em 0.2em 0.1em 0.2em;}
#messageArea {position:fixed; top:2em; right:0em; margin:0.5em; padding:0.5em; z-index:2000; _position:absolute;}
.messageToolbar {display:block; text-align:right; padding:0.2em 0.2em 0.2em 0.2em;}
#messageArea a {text-decoration:underline;}
.tiddlerPopupButton {padding:0.2em 0.2em 0.2em 0.2em;}
.popupTiddler {position: absolute; z-index:300; padding:1em 1em 1em 1em; margin:0;}
.popup {position:absolute; z-index:300; font-size:.9em; padding:0; list-style:none; margin:0;}
.popup .popupMessage {padding:0.4em;}
.popup hr {display:block; height:1px; width:auto; padding:0; margin:0.2em 0em;}
.popup li.disabled {padding:0.4em;}
.popup li a {display:block; padding:0.4em; font-weight:normal; cursor:pointer;}
.listBreak {font-size:1px; line-height:1px;}
.listBreak div {margin:2px 0;}
.tabset {padding:1em 0em 0em 0.5em;}
.tab {margin:0em 0em 0em 0.25em; padding:2px;}
.tabContents {padding:0.5em;}
.tabContents ul, .tabContents ol {margin:0; padding:0;}
.txtMainTab .tabContents li {list-style:none;}
.tabContents li.listLink { margin-left:.75em;}
#contentWrapper {display:block;}
#splashScreen {display:none;}
#displayArea {margin:1em 17em 0em 14em;}
.toolbar {text-align:right; font-size:.9em;}
.tiddler {padding:1em 1em 0em 1em;}
.missing .viewer,.missing .title {font-style:italic;}
.title {font-size:1.6em; font-weight:bold;}
.missing .subtitle {display:none;}
.subtitle {font-size:1.1em;}
.tiddler .button {padding:0.2em 0.4em;}
.tagging {margin:0.5em 0.5em 0.5em 0; float:left; display:none;}
.isTag .tagging {display:block;}
.tagged {margin:0.5em; float:right;}
.tagging, .tagged {font-size:0.9em; padding:0.25em;}
.tagging ul, .tagged ul {list-style:none; margin:0.25em; padding:0;}
.tagClear {clear:both;}
.footer {font-size:.9em;}
.footer li {display:inline;}
.annotation {padding:0.5em; margin:0.5em;}
* html .viewer pre {width:99%; padding:0 0 1em 0;}
.viewer {line-height:1.4em; padding-top:0.5em;}
.viewer .button {margin:0em 0.25em; padding:0em 0.25em;}
.viewer blockquote {line-height:1.5em; padding-left:0.8em;margin-left:2.5em;}
.viewer ul, .viewer ol {margin-left:0.5em; padding-left:1.5em;}
.viewer table, table.twtable {border-collapse:collapse; margin:0.8em 1.0em;}
.viewer th, .viewer td, .viewer tr,.viewer caption,.twtable th, .twtable td, .twtable tr,.twtable caption {padding:3px;}
table.listView {font-size:0.85em; margin:0.8em 1.0em;}
table.listView th, table.listView td, table.listView tr {padding:0px 3px 0px 3px;}
.viewer pre {padding:0.5em; margin-left:0.5em; font-size:1.2em; line-height:1.4em; overflow:auto;}
.viewer code {font-size:1.2em; line-height:1.4em;}
.editor {font-size:1.1em;}
.editor input, .editor textarea {display:block; width:100%; font:inherit;}
.editorFooter {padding:0.25em 0em; font-size:.9em;}
.editorFooter .button {padding-top:0px; padding-bottom:0px;}
.fieldsetFix {border:0; padding:0; margin:1px 0px 1px 0px;}
.sparkline {line-height:1em;}
.sparktick {outline:0;}
.zoomer {font-size:1.1em; position:absolute; overflow:hidden;}
.zoomer div {padding:1em;}
* html #backstage {width:99%;}
* html #backstageArea {width:99%;}
#backstageArea {display:none; position:relative; overflow: hidden; z-index:150; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageToolbar {position:relative;}
#backstageArea a {font-weight:bold; margin-left:0.5em; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageButton {display:none; position:absolute; z-index:175; top:0em; right:0em;}
#backstageButton a {padding:0.1em 0.4em 0.1em 0.4em; margin:0.1em 0.1em 0.1em 0.1em;}
#backstage {position:relative; width:100%; z-index:50;}
#backstagePanel {display:none; z-index:100; position:absolute; width:90%; margin:0em 3em 0em 3em; padding:1em 1em 1em 1em;}
.backstagePanelFooter {padding-top:0.2em; float:right;}
.backstagePanelFooter a {padding:0.2em 0.4em 0.2em 0.4em;}
#backstageCloak {display:none; z-index:20; position:absolute; width:100%; height:100px;}
.whenBackstage {display:none;}
.backstageVisible .whenBackstage {display:block;}
/*}}}*/
/***
!Sections in this Tiddler:
*Generic rules
**Links styles
**Link Exceptions
*Header
*Main menu
*Sidebar
**Sidebar options
**Sidebar tabs
*Message area
*Popup
*Tabs
*Tiddler display
**Viewer
**Editor
*Misc. rules
!Generic Rules /%==============================================%/
***/
/*{{{*/
body {
font-size: .75em;
font-family: arial,helvetica;
position: relative;
margin: 0;
padding: 0;
}
h1,h2,h3,h4,h5 {
font-weight: bold;
text-decoration: none;
padding-left: 0.4em;
}
h1 {font-size: 1.5em;}
h2 {font-size: 1.25em;}
h3 {font-size: 1.1em;}
h4 {font-size: 1em;}
h5 {font-size: .9em;}
hr {
height: 1px;
}
a{
text-decoration: none;
}
ol { list-style-type: decimal }
ol ol { list-style-type: lower-alpha }
ol ol ol { list-style-type: lower-roman }
ol ol ol ol { list-style-type: decimal }
ol ol ol ol ol { list-style-type: lower-alpha }
ol ol ol ol ol ol { list-style-type: lower-roman }
ol ol ol ol ol ol ol { list-style-type: decimal }
/*}}}*/
/***
''General Link Styles'' /%-----------------------------------------------------------------------------%/
***/
/*{{{*/
.externalLink {
text-decoration: underline;
}
/* the 'a' is required for IE, otherwise it renders the whole tiddler a bold */
a.tiddlyLinkNonExisting.shadow {
font-weight: bold;
}
/*}}}*/
/***
''Exceptions to common link styles'' /%------------------------------------------------------------------%/
***/
/*{{{*/
#mainMenu .tiddlyLinkExisting,
#mainMenu .tiddlyLinkNonExisting,
#sidebarTabs .tiddlyLinkExisting,
#sidebarTabs .tiddlyLinkNonExisting{
font-weight: normal;
font-style: normal;
}
/*}}}*/
/***
!Header /%==================================================%/
***/
/*{{{*/
.header {
position: relative;
}
.header a:hover {
background: transparent;
}
.headerShadow {
position: relative;
padding: 4.5em 0em 1em 1em;
left: -1px;
top: -1px;
}
.headerForeground {
position: absolute;
padding: 4.5em 0em 1em 1em;
left: 0px;
top: 0px;
}
.siteTitle {
font-size: 3em;
}
.siteSubtitle {
font-size: 1.2em;
padding: 0em 0em 0em 2em;
}
/*}}}*/
/***
!Main menu /%==================================================%/
***/
/*{{{*/
#mainMenu {
position: absolute;
left: 0;
width: 10em;
text-align: right;
line-height: 1.6em;
padding: 1.5em 0.5em 0.5em 0.5em;
font-size: 1.1em;
}
/*}}}*/
/***
!Sidebar rules /%==================================================%/
***/
/*{{{*/
#sidebar {
position: absolute;
right: 3px;
width: 16em;
font-size: .9em;
}
/*}}}*/
/***
''Sidebar options'' /%----------------------------------------------------------------------------------%/
***/
/*{{{*/
#sidebarOptions {
padding-top: 0.3em;
}
#sidebarOptions a {
margin: 0em 0.2em;
padding: 0.2em 0.3em;
display: block;
}
#sidebarOptions input {
margin: 0.4em 0.5em;
}
#sidebarOptions .sliderPanel {
margin-left: 1em;
padding: 0.5em;
font-size: .85em;
}
#sidebarOptions .sliderPanel a {
font-weight: bold;
display: inline;
padding: 0;
}
#sidebarOptions .sliderPanel input {
margin: 0 0 .3em 0;
}
/*}}}*/
/***
''Sidebar tabs'' /%-------------------------------------------------------------------------------------%/
***/
/*{{{*/
#sidebarTabs .tabContents {
width: 15em;
overflow: hidden;
}
/*}}}*/
/***
!Message area /%==================================================%/
***/
/*{{{*/
#messageArea {
position:absolute; top:0; right:0; margin: 0.5em; padding: 0.5em;
}
*[id='messageArea'] {
position:fixed !important; z-index:99;}
.messageToolbar {
display: block;
text-align: right;
}
#messageArea a{
text-decoration: underline;
}
/*}}}*/
/***
!Popup /%==================================================%/
***/
/*{{{*/
.popup {
font-size: .9em;
padding: 0.2em;
list-style: none;
margin: 0;
}
.popup hr {
display: block;
height: 1px;
width: auto;
padding: 0;
margin: 0.2em 0em;
}
.popup li.disabled {
padding: 0.2em;
}
.popup li a{
display: block;
padding: 0.2em;
}
/*}}}*/
/***
!Tabs /%==================================================%/
***/
/*{{{*/
.tabset {
padding: 1em 0em 0em 0.5em;
}
.tab {
margin: 0em 0em 0em 0.25em;
padding: 2px;
}
.tabContents {
padding: 0.5em;
}
.tabContents ul, .tabContents ol {
margin: 0;
padding: 0;
}
.txtMainTab .tabContents li {
list-style: none;
}
.tabContents li.listLink {
margin-left: .75em;
}
/*}}}*/
/***
!Tiddler display rules /%==================================================%/
***/
/*{{{*/
#displayArea {
margin: 1em 17em 0em 14em;
}
.toolbar {
text-align: right;
font-size: .9em;
visibility: hidden;
}
.selected .toolbar {
visibility: visible;
}
.tiddler {
padding: 1em 1em 0em 1em;
}
.missing .viewer,.missing .title {
font-style: italic;
}
.title {
font-size: 1.6em;
font-weight: bold;
}
.missing .subtitle {
display: none;
}
.subtitle {
font-size: 0.8em;
}
/* I'm not a fan of how button looks in tiddlers... */
.tiddler .button {
padding: 0.2em 0.4em;
}
.tagging {
margin: 0.5em 0.5em 0.5em 0;
float: left;
display: none;
}
.isTag .tagging {
display: block;
}
.tagged {
margin: 0.5em;
float: right;
}
.tagging, .tagged {
font-size: 0.9em;
padding: 0.25em;
}
.tagging ul, .tagged ul {
list-style: none;margin: 0.25em;
padding: 0;
}
.tagClear {
clear: both;
}
.footer {
font-size: .9em;
}
.footer li {
display: inline;
}
/***
''The viewer is where the tiddler content is displayed'' /%------------------------------------------------%/
***/
/*{{{*/
* html .viewer pre {
width: 99%;
padding: 0 0 1em 0;
}
.viewer {
line-height: 1.4em;
padding-top: 0.5em;
}
.viewer .button {
margin: 0em 0.25em;
padding: 0em 0.25em;
}
.viewer blockquote {
line-height: 1.5em;
padding-left: 0.8em;
margin-left: 2.5em;
}
.viewer ul, .viewer ol{
margin-left: 0.5em;
padding-left: 1.5em;
}
.viewer table {
border-collapse: collapse;
margin: 0.8em 1.0em;
}
.viewer th, .viewer td, .viewer tr,.viewer caption{
padding: 3px;
}
.viewer pre {
padding: 0.5em;
margin-left: 0.5em;
font-size: 1.2em;
line-height: 1.4em;
overflow: auto;
}
.viewer code {
font-size: 1.2em;
line-height: 1.4em;
}
/*}}}*/
/***
''The editor replaces the viewer in the tiddler'' /%------------------------------------------------%/
***/
/*{{{*/
.editor {
font-size: 1.1em;
}
.editor input, .editor textarea {
display: block;
width: 100%;
font: inherit;
}
.editorFooter {
padding: 0.25em 0em;
font-size: .9em;
}
.editorFooter .button {
padding-top: 0px; padding-bottom: 0px;}
.fieldsetFix {border: 0;
padding: 0;
margin: 1px 0px 1px 0px;
}
/*}}}*/
/***
!Misc rules /%==================================================%/
***/
/*{{{*/
.sparkline {
line-height: 1em;
}
.sparktick {
outline: 0;
}
.zoomer {
font-size: 1.1em;
position: absolute;
padding: 1em;
}
.cascade {
font-size: 1.1em;
position: absolute;
overflow: hidden;
}
/*}}}*/
/*{{{*/
@media print {
#mainMenu, #sidebar, #messageArea, .toolbar, #backstageButton, #backstageArea, #toolbar, #topMenu, #rightMenu {display: none !important;}
#header, #headerShadow {display: none !important;}
.siteSubtitle {display: none !important;}
.siteTitle { font-size: 1.5em; }
#displayArea {margin: 1em 1em 0em;}
noscript {display:none;} /* Fixes a feature in Firefox 1.5.0.2 where print preview displays the noscript content */
}
/*}}}*/
Type the text for 'Styles'
<<tabs txtMainTab Timeline Timeline TabTimeline All 'All tiddlers' TabAll Tags 'All tags' TabTags More 'More lists' TabMore>>
The following command table should prove useful for this course. This is not an extensive list of commands you will need to know / become familiar with.
| ! Cover | ! Command | !Description |
|>|>|bgcolor(#a0ffa0): ''Basic Commands'' |
| x | echo |output command arguments to the terminal|
| x | cd |change directories|
| x | pwd |display current working directory|
| x | ls |list files|
| x | cp |copy files|
| x | rm |remove files|
| x | mv |move files|
| x | mkdir |create directory|
| x | rmdir |remove directory|
| x | touch |create an empty file with default permissions|
| x | ln |create link|
| x | man |view man pages|
| x | chmod |set permissions for a file|
| | chgrp |set group for a file|
|>|>|bgcolor(#a0ffa0): ''Display Text / Editors'' |
| x | less |display text output one page at a time|
| | pico |easy to use text editor|
| | nano |GNU clone of pico|
| x | vi |advanced unix text editor|
| | ex |line oriented version of vi|
| | vim |vi improved|
| x | vimtutor |learn how to use the vim editor|
|>|>|bgcolor(#a0ffa0): ''Filters'' |
| x | cat |concatenate and print files|
| x | grep |pattern matching filter|
| x | egrep |extended regular expression pattern matching filter|
| x | head |display first lines of a file|
| x | tail |display the last part of a file|
| x | cut |cut out selected portions of each line of a file|
| x | fold |fold long lines for finite width output device|
| x | sort |sort lines of text files|
| x | uniq |report or filter out repeated lines in a file|
| x | wc |word, line, character, and byte count|
| x | tr |translate characters|
| | paste |merge lines of input|
| | nl |line numbering filter|
| x | sed |stream editor|
| x | awk |pattern-directed scanning and processing language|
| x | tee |duplicate standard input to a file|
| | strings |print the strings of printable characters in (binary) files|
| | cmp |compare two files|
| | diff |compare files line by line|
| | comm |select or reject lines common to two files|
|>|>|bgcolor(#a0ffa0): ''System Commands'' |
| x | script |save copy of terminal session|
| | source |read a .file|
| x | rehash |recompute hash table of where commands are located|
| x | which |scan path for a program and return its location (or definition of an alias)|
| x | df |display free disk space|
| x | du |disk usage (-s display each file, -k 1K blocks , -h = human readable|
| x | find |walk a file hierarchy in search of files|
| | locate |find filenames quickly based on pre-generated file database|
| | hostname |print name of current host system|
| x | uptime |show how long system has been running|
| x | uname |display information about the system|
| | xargs |construct argument list(s) and execute utility|
| | quota |display disk usage and limits|
| | crontab |schedule commands for automated execution on regular intervals|
| | at |schedule a job for later execution|
|>|>|bgcolor(#a0ffa0): ''Process Management / Job Control'' |
| x | ps |process status|
| x | top |display and update information about the top cpu processes|
| x | kill |terminate or signal a process|
| x | jobs |display all jobs|
| x | fg |continue background jobs in the foreground|
| x | bg |continue suspended job in the background|
| x | stop |suspend job running in the background|
| | suspend |suspend the current running shell|
|>|>|bgcolor(#a0ffa0): ''User Information'' |
| x | w |display who is logged in and what they are doing|
| x | id |return user identity|
| x | groups |show group memberships|
| | users |list usernames of current logged in users|
| | who |display who is on the system|
| | whoami |display effective user id|
| x | finger |user information lookup program|
| x | last |indicate last logins of users and ttys|
|>|>|bgcolor(#a0ffa0): ''Misc commands useful for shell scripting'' |
| x | clear |clear the screen|
| x | read //var// |prompt the user to enter information, saving to //var//|
| x | date |display the current date and time with optional formatting. see strftime manpage|
| x | test |condition evaluation utility. Linked to [ See test manpage.|
| x | expr |evaluate an expression|
| x | jot |print sequential or random numbers|
| | sleep //n// |pause execution for //n// seconds|
| | stat |display extended file status/information|
| | stty |set the options for a terminal device interface|
| | basename |return the file name portion of a path|
| | dirname |return the directory name portion of a path|
| | fstat |List open files or determine whether specified file is open|
| x | exit [//n//] |log out or quit a script with the option exit status of //n//|
|>|>|bgcolor(#a0ffa0): ''Networking / Communication'' |
| x | ssh |~OpenSSH SSH client|
| x | scp |secure copy (remote file copy program)|
| | rsync |a fast, versatile, remote (and local) file-copying tool|
| | telnet |user interface to the TELNET protocol. also useful for testing connectivity to arbitrary ports|
| | talk / ytalk |talk to another user|
| | write |send a message to another user|
| | mesg |display (do not display) messages from other users|
| | host |DNS lookup utility|
| | nslookup |query Internet name servers interactively|
| | traceroute |print the route packets take to network host|
| | ping |send ICMP ~ECHO_REQUEST packets to network hosts|
| | lynx / links |character mode WWW browser|
|>|>|bgcolor(#a0ffa0): ''Text Formatting & Printing'' |
| x | lpr |command line print utility|
| x | lpq |print spool queue examination program|
| x | lprm |remove jobs from the line printer spooling queue|
| x | pdf2ps |Ghostscript PDF to ~PostScript translator|
| x | a2ps |format files for printing on a ~PostScript printer|
|>|>|bgcolor(#a0ffa0): ''Working with files'' |
| x | file |display file type|
| x | tar |manipulate file archive files|
| x | gzip |compression tool using ~Lempel-Ziv coding|
| x | gunzip |decompression tool using ~Lempel-Ziv coding|
| x | bzip2 |a block-sorting file compressor|
| x | bunzip2 |a block-sorting file decompressor|
| | split |split a file into pieces|
| x | md5 / md5sum |calculate a message-digest fingerprint (checksum) for a file (freebsd / linux)|
| | srm |securely remove files or directories|
| | rsync |a fast, versatile, remote (and local) file-copying tool|
/***
|Name|TagglyListPlugin|
|Created by|SimonBaird|
|Location|http://simonbaird.com/mptw/#TagglyListPlugin|
|Version|1.1.2 25-Apr-06|
|Requires|See TagglyTagging|
!History
* 1.1.2 (25-Apr-2006) embedded TagglyTaggingStyles. No longer need separated tiddler for styles.
* 1.1.1 (6-Mar-2006) fixed bug with refreshAllVisible closing tiddlers being edited. Thanks Luke Blanshard.
***/
/***
!Setup and config
***/
//{{{
version.extensions.TagglyListPlugin = {
major: 1, minor: 1, revision: 2,
date: new Date(2006,4,25),
source: "http://simonbaird.com/mptw/#TagglyListPlugin"
};
config.macros.tagglyList = {};
config.macros.tagglyListByTag = {};
config.macros.tagglyListControl = {};
config.macros.tagglyListWithSort = {};
config.macros.hideSomeTags = {};
// change this to your preference
config.macros.tagglyListWithSort.maxCols = 6;
config.macros.tagglyList.label = "Tagged as %0:";
// the default sort options. set these to your preference
config.macros.tagglyListWithSort.defaults = {
sortBy:"title", // title|created|modified
sortOrder: "asc", // asc|desc
hideState: "show", // show|hide
groupState: "nogroup", // nogroup|group
numCols: 1
};
// these tags will be ignored by the grouped view
config.macros.tagglyListByTag.excludeTheseTags = [
"systemConfig",
"TiddlerTemplates"
];
config.macros.tagglyListControl.tags = {
title:"sortByTitle",
modified: "sortByModified",
created: "sortByCreated",
asc:"sortAsc",
desc:"sortDesc",
hide:"hideTagged",
show:"showTagged",
nogroup:"noGroupByTag",
group:"groupByTag",
cols1:"list1Cols",
cols2:"list2Cols",
cols3:"list3Cols",
cols4:"list4Cols",
cols5:"list5Cols",
cols6:"list6Cols",
cols7:"list7Cols",
cols8:"list8Cols",
cols9:"list9Cols"
}
// note: should match config.macros.tagglyListControl.tags
config.macros.hideSomeTags.tagsToHide = [
"sortByTitle",
"sortByCreated",
"sortByModified",
"sortDesc",
"sortAsc",
"hideTagged",
"showTagged",
"noGroupByTag",
"groupByTag",
"list1Cols",
"list2Cols",
"list3Cols",
"list4Cols",
"list5Cols",
"list6Cols",
"list7Cols",
"list8Cols",
"list9Cols"
];
//}}}
/***
!Utils
***/
//{{{
// from Eric
function isTagged(title,tag) {
var t=store.getTiddler(title); if (!t) return false;
return (t.tags.find(tag)!=null);
}
// from Eric
function toggleTag(title,tag) {
var t=store.getTiddler(title); if (!t || !t.tags) return;
if (t.tags.find(tag)==null) t.tags.push(tag);
else t.tags.splice(t.tags.find(tag),1);
}
function addTag(title,tag) {
var t=store.getTiddler(title); if (!t || !t.tags) return;
t.tags.push(tag);
}
function removeTag(title,tag) {
var t=store.getTiddler(title); if (!t || !t.tags) return;
if (t.tags.find(tag)!=null) t.tags.splice(t.tags.find(tag),1);
}
// from Udo
Array.prototype.indexOf = function(item) {
for (var i = 0; i < this.length; i++) {
if (this[i] == item) {
return i;
}
}
return -1;
};
Array.prototype.contains = function(item) {
return (this.indexOf(item) >= 0);
}
//}}}
/***
!tagglyList
displays a list of tagged tiddlers.
parameters are sortField and sortOrder
***/
//{{{
// not used at the moment...
function sortedListOfOtherTags(tiddler,thisTag) {
var list = tiddler.tags.concat(); // so we are working on a clone..
for (var i=0;i<config.macros.hideSomeTags.tagsToHide.length;i++) {
if (list.find(config.macros.hideSomeTags.tagsToHide[i]) != null)
list.splice(list.find(config.macros.hideSomeTags.tagsToHide[i]),1); // remove hidden ones
}
for (var i=0;i<config.macros.tagglyListByTag.excludeTheseTags.length;i++) {
if (list.find(config.macros.tagglyListByTag.excludeTheseTags[i]) != null)
list.splice(list.find(config.macros.tagglyListByTag.excludeTheseTags[i]),1); // remove excluded ones
}
list.splice(list.find(thisTag),1); // remove thisTag
return '[[' + list.sort().join("]] [[") + ']]';
}
function sortHelper(a,b) {
if (a == b) return 0;
else if (a < b) return -1;
else return +1;
}
config.macros.tagglyListByTag.handler = function (place,macroName,params,wikifier,paramString,tiddler) {
var sortBy = params[0] ? params[0] : "title";
var sortOrder = params[1] ? params[1] : "asc";
var result = store.getTaggedTiddlers(tiddler.title,sortBy);
if (sortOrder == "desc")
result = result.reverse();
var leftOvers = []
for (var i=0;i<result.length;i++) {
leftOvers.push(result[i].title);
}
var allTagsHolder = {};
for (var i=0;i<result.length;i++) {
for (var j=0;j<result[i].tags.length;j++) {
if (
result[i].tags[j] != tiddler.title // not this tiddler
&& config.macros.hideSomeTags.tagsToHide.find(result[i].tags[j]) == null // not a hidden one
&& config.macros.tagglyListByTag.excludeTheseTags.find(result[i].tags[j]) == null // not excluded
) {
if (!allTagsHolder[result[i].tags[j]])
allTagsHolder[result[i].tags[j]] = "";
allTagsHolder[result[i].tags[j]] += "**[["+result[i].title+"]]\n";
if (leftOvers.find(result[i].title) != null)
leftOvers.splice(leftOvers.find(result[i].title),1); // remove from leftovers. at the end it will contain the leftovers...
}
}
}
var allTags = [];
for (var t in allTagsHolder)
allTags.push(t);
allTags.sort(function(a,b) {
var tidA = store.getTiddler(a);
var tidB = store.getTiddler(b);
if (sortBy == "title") return sortHelper(a,b);
else if (!tidA && !tidB) return 0;
else if (!tidA) return -1;
else if (!tidB) return +1;
else return sortHelper(tidA[sortBy],tidB[sortBy]);
});
var markup = "";
if (sortOrder == "desc") {
allTags.reverse();
}
else {
// leftovers first...
for (var i=0;i<leftOvers.length;i++)
markup += "*[["+leftOvers[i]+"]]\n";
}
for (var i=0;i<allTags.length;i++)
markup += "*[["+allTags[i]+"]]\n" + allTagsHolder[allTags[i]];
if (sortOrder == "desc") {
// leftovers last...
for (var i=0;i<leftOvers.length;i++)
markup += "*[["+leftOvers[i]+"]]\n";
}
wikify(markup,place);
}
config.macros.tagglyList.handler = function (place,macroName,params,wikifier,paramString,tiddler) {
var sortBy = params[0] ? params[0] : "title";
var sortOrder = params[1] ? params[1] : "asc";
var numCols = params[2] ? params[2] : 1;
var result = store.getTaggedTiddlers(tiddler.title,sortBy);
if (sortOrder == "desc")
result = result.reverse();
var listSize = result.length;
var colSize = listSize/numCols;
var remainder = listSize % numCols;
var upperColsize;
var lowerColsize;
if (colSize != Math.floor(colSize)) {
// it's not an exact fit so..
lowerColsize = Math.floor(colSize);
upperColsize = Math.floor(colSize) + 1;
}
else {
lowerColsize = colSize;
upperColsize = colSize;
}
var markup = "";
var c=0;
var newTaggedTable = createTiddlyElement(place,"table");
var newTaggedBody = createTiddlyElement(newTaggedTable,"tbody");
var newTaggedTr = createTiddlyElement(newTaggedBody,"tr");
for (var j=0;j<numCols;j++) {
var foo = "";
var thisSize;
if (j<remainder)
thisSize = upperColsize;
else
thisSize = lowerColsize;
for (var i=0;i<thisSize;i++)
foo += ( "*[[" + result[c++].title + "]]\n"); // was using splitList.shift() but didn't work in IE;
var newTd = createTiddlyElement(newTaggedTr,"td",null,"tagglyTagging");
wikify(foo,newTd);
}
};
/* snip for later.....
//var groupBy = params[3] ? params[3] : "t.title.substr(0,1)";
//var groupBy = params[3] ? params[3] : "sortedListOfOtherTags(t,tiddler.title)";
//var groupBy = params[3] ? params[3] : "t.modified";
var groupBy = null; // for now. groupBy here is working but disabled for now.
var prevGroup = "";
var thisGroup = "";
if (groupBy) {
result.sort(function(a,b) {
var t = a; var aSortVal = eval(groupBy); var aSortVal2 = eval("t".sortBy);
var t = b; var bSortVal = eval(groupBy); var bSortVal2 = eval("t".sortBy);
var t = b; var bSortVal2 = eval(groupBy);
return (aSortVal == bSortVal ?
(aSortVal2 == bSortVal2 ? 0 : (aSortVal2 < bSortVal2 ? -1 : +1)) // yuck
: (aSortVal < bSortVal ? -1 : +1));
});
}
if (groupBy) {
thisGroup = eval(groupBy);
if (thisGroup != prevGroup)
markup += "*[["+thisGroup+']]\n';
markup += "**[["+t.title+']]\n';
prevGroup = thisGroup;
}
*/
//}}}
/***
!tagglyListControl
Use to make the sort control buttons
***/
//{{{
function getSortBy(title) {
var tiddler = store.getTiddler(title);
var defaultVal = config.macros.tagglyListWithSort.defaults.sortBy;
if (!tiddler) return defaultVal;
var usetags = config.macros.tagglyListControl.tags;
if (tiddler.tags.contains(usetags["title"])) return "title";
else if (tiddler.tags.contains(usetags["modified"])) return "modified";
else if (tiddler.tags.contains(usetags["created"])) return "created";
else return defaultVal;
}
function getSortOrder(title) {
var tiddler = store.getTiddler(title);
var defaultVal = config.macros.tagglyListWithSort.defaults.sortOrder;
if (!tiddler) return defaultVal;
var usetags = config.macros.tagglyListControl.tags;
if (tiddler.tags.contains(usetags["asc"])) return "asc";
else if (tiddler.tags.contains(usetags["desc"])) return "desc";
else return defaultVal;
}
function getHideState(title) {
var tiddler = store.getTiddler(title);
var defaultVal = config.macros.tagglyListWithSort.defaults.hideState;
if (!tiddler) return defaultVal;
var usetags = config.macros.tagglyListControl.tags;
if (tiddler.tags.contains(usetags["hide"])) return "hide";
else if (tiddler.tags.contains(usetags["show"])) return "show";
else return defaultVal;
}
function getGroupState(title) {
var tiddler = store.getTiddler(title);
var defaultVal = config.macros.tagglyListWithSort.defaults.groupState;
if (!tiddler) return defaultVal;
var usetags = config.macros.tagglyListControl.tags;
if (tiddler.tags.contains(usetags["group"])) return "group";
else if (tiddler.tags.contains(usetags["nogroup"])) return "nogroup";
else return defaultVal;
}
function getNumCols(title) {
var tiddler = store.getTiddler(title);
var defaultVal = config.macros.tagglyListWithSort.defaults.numCols; // an int
if (!tiddler) return defaultVal;
var usetags = config.macros.tagglyListControl.tags;
for (var i=1;i<=config.macros.tagglyListWithSort.maxCols;i++)
if (tiddler.tags.contains(usetags["cols"+i])) return i;
return defaultVal;
}
function getSortLabel(title,which) {
// TODO. the strings here should be definable in config
var by = getSortBy(title);
var order = getSortOrder(title);
var hide = getHideState(title);
var group = getGroupState(title);
if (which == "hide") return (hide == "show" ? "−" : "+"); // 0x25b8;
else if (which == "group") return (group == "group" ? "normal" : "grouped");
else if (which == "cols") return "cols±"; // ±
else if (by == which) return which + (order == "asc" ? "↓" : "↑"); // ↑ ↓
else return which;
}
function handleSortClick(title,which) {
var currentSortBy = getSortBy(title);
var currentSortOrder = getSortOrder(title);
var currentHideState = getHideState(title);
var currentGroupState = getGroupState(title);
var currentNumCols = getNumCols(title);
var tags = config.macros.tagglyListControl.tags;
// if it doesn't exist, lets create it..
if (!store.getTiddler(title))
store.saveTiddler(title,title,"",config.options.txtUserName,new Date(),null);
if (which == "hide") {
// toggle hide state
var newHideState = (currentHideState == "hide" ? "show" : "hide");
removeTag(title,tags[currentHideState]);
if (newHideState != config.macros.tagglyListWithSort.defaults.hideState)
toggleTag(title,tags[newHideState]);
}
else if (which == "group") {
// toggle hide state
var newGroupState = (currentGroupState == "group" ? "nogroup" : "group");
removeTag(title,tags[currentGroupState]);
if (newGroupState != config.macros.tagglyListWithSort.defaults.groupState)
toggleTag(title,tags[newGroupState]);
}
else if (which == "cols") {
// toggle num cols
var newNumCols = currentNumCols + 1; // confusing. currentNumCols is an int
if (newNumCols > config.macros.tagglyListWithSort.maxCols || newNumCols > store.getTaggedTiddlers(title).length)
newNumCols = 1;
removeTag(title,tags["cols"+currentNumCols]);
if (("cols"+newNumCols) != config.macros.tagglyListWithSort.defaults.groupState)
toggleTag(title,tags["cols"+newNumCols]);
}
else if (currentSortBy == which) {
// toggle sort order
var newSortOrder = (currentSortOrder == "asc" ? "desc" : "asc");
removeTag(title,tags[currentSortOrder]);
if (newSortOrder != config.macros.tagglyListWithSort.defaults.sortOrder)
toggleTag(title,tags[newSortOrder]);
}
else {
// change sortBy only
removeTag(title,tags["title"]);
removeTag(title,tags["created"]);
removeTag(title,tags["modified"]);
if (which != config.macros.tagglyListWithSort.defaults.sortBy)
toggleTag(title,tags[which]);
}
store.setDirty(true); // save is required now.
story.refreshTiddler(title,false,true); // force=true
}
config.macros.tagglyListControl.handler = function (place,macroName,params,wikifier,paramString,tiddler) {
var onclick = function(e) {
if (!e) var e = window.event;
handleSortClick(tiddler.title,params[0]);
e.cancelBubble = true;
if (e.stopPropagation) e.stopPropagation();
return false;
};
createTiddlyButton(place,getSortLabel(tiddler.title,params[0]),"Click to change sort options",onclick,params[0]=="hide"?"hidebutton":"button");
}
//}}}
/***
!tagglyListWithSort
put it all together..
***/
//{{{
config.macros.tagglyListWithSort.handler = function (place,macroName,params,wikifier,paramString,tiddler) {
if (tiddler && store.getTaggedTiddlers(tiddler.title).length > 0)
// todo make this readable
wikify(
"<<tagglyListControl hide>>"+
(getHideState(tiddler.title) != "hide" ?
'<html><span class="tagglyLabel">'+config.macros.tagglyList.label.format([tiddler.title])+' </span></html>'+
"<<tagglyListControl title>><<tagglyListControl modified>><<tagglyListControl created>><<tagglyListControl group>>"+(getGroupState(tiddler.title)=="group"?"":"<<tagglyListControl cols>>")+"\n" +
"<<tagglyList" + (getGroupState(tiddler.title)=="group"?"ByTag ":" ") + getSortBy(tiddler.title)+" "+getSortOrder(tiddler.title)+" "+getNumCols(tiddler.title)+">>" // hacky
// + \n----\n" +
//"<<tagglyList "+getSortBy(tiddler.title)+" "+getSortOrder(tiddler.title)+">>"
: ""),
place,null,tiddler);
}
config.macros.tagglyTagging = { handler: config.macros.tagglyListWithSort.handler };
//}}}
/***
!hideSomeTags
So we don't see the sort tags.
(note, they are still there when you edit. Will that be too annoying?
***/
//{{{
// based on tags.handler
config.macros.hideSomeTags.handler = function(place,macroName,params,wikifier,paramString,tiddler) {
var theList = createTiddlyElement(place,"ul");
if(params[0] && store.tiddlerExists[params[0]])
tiddler = store.getTiddler(params[0]);
var lingo = config.views.wikified.tag;
var prompt = tiddler.tags.length == 0 ? lingo.labelNoTags : lingo.labelTags;
createTiddlyElement(theList,"li",null,"listTitle",prompt.format([tiddler.title]));
for(var t=0; t<tiddler.tags.length; t++)
if (!this.tagsToHide.contains(tiddler.tags[t])) // this is the only difference from tags.handler...
createTagButton(createTiddlyElement(theList,"li"),tiddler.tags[t],tiddler.title);
}
//}}}
/***
!Refresh everything when we save a tiddler. So the tagged lists never get stale. Is this too slow???
***/
//{{{
function refreshAllVisible() {
story.forEachTiddler(function(title,element) {
if (element.getAttribute("dirty") != "true")
story.refreshTiddler(title,false,true);
});
}
story.saveTiddler_orig_mptw = story.saveTiddler;
story.saveTiddler = function(title,minorUpdate) {
var result = this.saveTiddler_orig_mptw(title,minorUpdate);
// refreshAllVisible();
return result;
}
store.removeTiddler_orig_mptw = store.removeTiddler;
store.removeTiddler = function(title) {
this.removeTiddler_orig_mptw(title);
// refreshAllVisible();
}
config.shadowTiddlers.TagglyTaggingStyles = "/***\nTo use, add {{{[[TagglyTaggingStyles]]}}} to your StyleSheet tiddler, or you can just paste the CSS in directly. See also ViewTemplate, EditTemplate and TagglyTagging.\n***/\n/*{{{*/\n.tagglyTagged li.listTitle { display:none;}\n.tagglyTagged li { display: inline; font-size:90%; }\n.tagglyTagged ul { margin:0px; padding:0px; }\n.tagglyTagging { padding-top:0.5em; }\n.tagglyTagging li.listTitle { display:none;}\n.tagglyTagging ul { margin-top:0px; padding-top:0.5em; padding-left:2em; margin-bottom:0px; padding-bottom:0px; }\n\n/* .tagglyTagging .tghide { display:inline; } */\n\n.tagglyTagging { vertical-align: top; margin:0px; padding:0px; }\n.tagglyTagging table { margin:0px; padding:0px; }\n\n\n.tagglyTagging .button { display:none; margin-left:3px; margin-right:3px; }\n.tagglyTagging .button, .tagglyTagging .hidebutton { color:#aaa; font-size:90%; border:0px; padding-left:0.3em;padding-right:0.3em;}\n.tagglyTagging .button:hover, .hidebutton:hover { background:#eee; color:#888; }\n.selected .tagglyTagging .button { display:inline; }\n\n.tagglyTagging .hidebutton { color:white; } /* has to be there so it takes up space. tweak if you're not using a white tiddler bg */\n.selected .tagglyTagging .hidebutton { color:#aaa }\n\n.tagglyLabel { color:#aaa; font-size:90%; }\n\n.tagglyTagging ul {padding-top:0px; padding-bottom:0.5em; margin-left:1em; }\n.tagglyTagging ul ul {list-style-type:disc; margin-left:-1em;}\n.tagglyTagging ul ul li {margin-left:0.5em; }\n\n.editLabel { font-size:90%; padding-top:0.5em; }\n/*}}}*/\n";
refreshStyles("TagglyTaggingStyles");
//}}}
// // <html>▸▾−±</html>
Type the text for 'TagglyTagging'
/***
|Name:|TagglyTaggingPlugin|
|Description:|tagglyTagging macro is a replacement for the builtin tagging macro in your ViewTemplate|
|Version:|3.3.1 ($Rev: 6100 $)|
|Date:|$Date: 2008-07-27 01:42:07 +1000 (Sun, 27 Jul 2008) $|
|Source:|http://mptw.tiddlyspot.com/#TagglyTaggingPlugin|
|Author:|Simon Baird <simon.baird@gmail.com>|
|License:|http://mptw.tiddlyspot.com/#TheBSDLicense|
!Notes
See http://mptw.tiddlyspot.com/#TagglyTagging
***/
//{{{
merge(String.prototype,{
parseTagExpr: function(debug) {
if (this.trim() == "")
return "(true)";
var anyLogicOp = /(!|&&|\|\||\(|\))/g;
var singleLogicOp = /^(!|&&|\|\||\(|\))$/;
var spaced = this.
// because square brackets in templates are no good
// this means you can use [(With Spaces)] instead of [[With Spaces]]
replace(/\[\(/g," [[").
replace(/\)\]/g,"]] ").
// space things out so we can use readBracketedList. tricky eh?
replace(anyLogicOp," $1 ");
var expr = "";
var tokens = spaced.readBracketedList(false); // false means don't uniq the list. nice one JR!
for (var i=0;i<tokens.length;i++)
if (tokens[i].match(singleLogicOp))
expr += tokens[i];
else
expr += "tiddler.tags.contains('%0')".format([tokens[i].replace(/'/,"\\'")]); // fix single quote bug. still have round bracket bug i think
if (debug)
alert(expr);
return '('+expr+')';
}
});
merge(TiddlyWiki.prototype,{
getTiddlersByTagExpr: function(tagExpr,sortField) {
var result = [];
var expr = tagExpr.parseTagExpr();
store.forEachTiddler(function(title,tiddler) {
if (eval(expr))
result.push(tiddler);
});
if(!sortField)
sortField = "title";
result.sort(function(a,b) {return a[sortField] < b[sortField] ? -1 : (a[sortField] == b[sortField] ? 0 : +1);});
return result;
}
});
config.taggly = {
// for translations
lingo: {
labels: {
asc: "\u2191", // down arrow
desc: "\u2193", // up arrow
title: "title",
modified: "modified",
created: "created",
show: "+",
hide: "-",
normal: "normal",
group: "group",
commas: "commas",
sitemap: "sitemap",
numCols: "cols\u00b1", // plus minus sign
label: "Tagged as '%0':",
exprLabel: "Matching tag expression '%0':",
excerpts: "excerpts",
descr: "descr",
slices: "slices",
contents: "contents",
sliders: "sliders",
noexcerpts: "title only",
noneFound: "(none)"
},
tooltips: {
title: "Click to sort by title",
modified: "Click to sort by modified date",
created: "Click to sort by created date",
show: "Click to show tagging list",
hide: "Click to hide tagging list",
normal: "Click to show a normal ungrouped list",
group: "Click to show list grouped by tag",
sitemap: "Click to show a sitemap style list",
commas: "Click to show a comma separated list",
numCols: "Click to change number of columns",
excerpts: "Click to show excerpts",
descr: "Click to show the description slice",
slices: "Click to show all slices",
contents: "Click to show entire tiddler contents",
sliders: "Click to show tiddler contents in sliders",
noexcerpts: "Click to show entire title only"
},
tooDeepMessage: "* //sitemap too deep...//"
},
config: {
showTaggingCounts: true,
listOpts: {
// the first one will be the default
sortBy: ["title","modified","created"],
sortOrder: ["asc","desc"],
hideState: ["show","hide"],
listMode: ["normal","group","sitemap","commas"],
numCols: ["1","2","3","4","5","6"],
excerpts: ["noexcerpts","excerpts","descr","slices","contents","sliders"]
},
valuePrefix: "taggly.",
excludeTags: ["excludeLists","excludeTagging"],
excerptSize: 50,
excerptMarker: "/%"+"%/",
siteMapDepthLimit: 25
},
getTagglyOpt: function(title,opt) {
var val = store.getValue(title,this.config.valuePrefix+opt);
return val ? val : this.config.listOpts[opt][0];
},
setTagglyOpt: function(title,opt,value) {
if (!store.tiddlerExists(title))
// create it silently
store.saveTiddler(title,title,config.views.editor.defaultText.format([title]),config.options.txtUserName,new Date(),"");
// if value is default then remove it to save space
return store.setValue(title,
this.config.valuePrefix+opt,
value == this.config.listOpts[opt][0] ? null : value);
},
getNextValue: function(title,opt) {
var current = this.getTagglyOpt(title,opt);
var pos = this.config.listOpts[opt].indexOf(current);
// a little usability enhancement. actually it doesn't work right for grouped or sitemap
var limit = (opt == "numCols" ? store.getTiddlersByTagExpr(title).length : this.config.listOpts[opt].length);
var newPos = (pos + 1) % limit;
return this.config.listOpts[opt][newPos];
},
toggleTagglyOpt: function(title,opt) {
var newVal = this.getNextValue(title,opt);
this.setTagglyOpt(title,opt,newVal);
},
createListControl: function(place,title,type) {
var lingo = config.taggly.lingo;
var label;
var tooltip;
var onclick;
if ((type == "title" || type == "modified" || type == "created")) {
// "special" controls. a little tricky. derived from sortOrder and sortBy
label = lingo.labels[type];
tooltip = lingo.tooltips[type];
if (this.getTagglyOpt(title,"sortBy") == type) {
label += lingo.labels[this.getTagglyOpt(title,"sortOrder")];
onclick = function() {
config.taggly.toggleTagglyOpt(title,"sortOrder");
return false;
}
}
else {
onclick = function() {
config.taggly.setTagglyOpt(title,"sortBy",type);
config.taggly.setTagglyOpt(title,"sortOrder",config.taggly.config.listOpts.sortOrder[0]);
return false;
}
}
}
else {
// "regular" controls, nice and simple
label = lingo.labels[type == "numCols" ? type : this.getNextValue(title,type)];
tooltip = lingo.tooltips[type == "numCols" ? type : this.getNextValue(title,type)];
onclick = function() {
config.taggly.toggleTagglyOpt(title,type);
return false;
}
}
// hide button because commas don't have columns
if (!(this.getTagglyOpt(title,"listMode") == "commas" && type == "numCols"))
createTiddlyButton(place,label,tooltip,onclick,type == "hideState" ? "hidebutton" : "button");
},
makeColumns: function(orig,numCols) {
var listSize = orig.length;
var colSize = listSize/numCols;
var remainder = listSize % numCols;
var upperColsize = colSize;
var lowerColsize = colSize;
if (colSize != Math.floor(colSize)) {
// it's not an exact fit so..
upperColsize = Math.floor(colSize) + 1;
lowerColsize = Math.floor(colSize);
}
var output = [];
var c = 0;
for (var j=0;j<numCols;j++) {
var singleCol = [];
var thisSize = j < remainder ? upperColsize : lowerColsize;
for (var i=0;i<thisSize;i++)
singleCol.push(orig[c++]);
output.push(singleCol);
}
return output;
},
drawTable: function(place,columns,theClass) {
var newTable = createTiddlyElement(place,"table",null,theClass);
var newTbody = createTiddlyElement(newTable,"tbody");
var newTr = createTiddlyElement(newTbody,"tr");
for (var j=0;j<columns.length;j++) {
var colOutput = "";
for (var i=0;i<columns[j].length;i++)
colOutput += columns[j][i];
var newTd = createTiddlyElement(newTr,"td",null,"tagglyTagging"); // todo should not need this class
wikify(colOutput,newTd);
}
return newTable;
},
createTagglyList: function(place,title,isTagExpr) {
switch(this.getTagglyOpt(title,"listMode")) {
case "group": return this.createTagglyListGrouped(place,title,isTagExpr); break;
case "normal": return this.createTagglyListNormal(place,title,false,isTagExpr); break;
case "commas": return this.createTagglyListNormal(place,title,true,isTagExpr); break;
case "sitemap":return this.createTagglyListSiteMap(place,title,isTagExpr); break;
}
},
getTaggingCount: function(title,isTagExpr) {
// thanks to Doug Edmunds
if (this.config.showTaggingCounts) {
var tagCount = config.taggly.getTiddlers(title,'title',isTagExpr).length;
if (tagCount > 0)
return " ("+tagCount+")";
}
return "";
},
getTiddlers: function(titleOrExpr,sortBy,isTagExpr) {
return isTagExpr ? store.getTiddlersByTagExpr(titleOrExpr,sortBy) : store.getTaggedTiddlers(titleOrExpr,sortBy);
},
getExcerpt: function(inTiddlerTitle,title,indent) {
if (!indent)
indent = 1;
var displayMode = this.getTagglyOpt(inTiddlerTitle,"excerpts");
var t = store.getTiddler(title);
if (t && displayMode == "excerpts") {
var text = t.text.replace(/\n/," ");
var marker = text.indexOf(this.config.excerptMarker);
if (marker != -1) {
return " {{excerpt{<nowiki>" + text.substr(0,marker) + "</nowiki>}}}";
}
else if (text.length < this.config.excerptSize) {
return " {{excerpt{<nowiki>" + t.text + "</nowiki>}}}";
}
else {
return " {{excerpt{<nowiki>" + t.text.substr(0,this.config.excerptSize) + "..." + "</nowiki>}}}";
}
}
else if (t && displayMode == "contents") {
return "\n{{contents indent"+indent+"{\n" + t.text + "\n}}}";
}
else if (t && displayMode == "sliders") {
return "<slider slide>\n{{contents{\n" + t.text + "\n}}}\n</slider>";
}
else if (t && displayMode == "descr") {
var descr = store.getTiddlerSlice(title,'Description');
return descr ? " {{excerpt{" + descr + "}}}" : "";
}
else if (t && displayMode == "slices") {
var result = "";
var slices = store.calcAllSlices(title);
for (var s in slices)
result += "|%0|<nowiki>%1</nowiki>|\n".format([s,slices[s]]);
return result ? "\n{{excerpt excerptIndent{\n" + result + "}}}" : "";
}
return "";
},
notHidden: function(t,inTiddler) {
if (typeof t == "string")
t = store.getTiddler(t);
return (!t || !t.tags.containsAny(this.config.excludeTags) ||
(inTiddler && this.config.excludeTags.contains(inTiddler)));
},
// this is for normal and commas mode
createTagglyListNormal: function(place,title,useCommas,isTagExpr) {
var list = config.taggly.getTiddlers(title,this.getTagglyOpt(title,"sortBy"),isTagExpr);
if (this.getTagglyOpt(title,"sortOrder") == "desc")
list = list.reverse();
var output = [];
var first = true;
for (var i=0;i<list.length;i++) {
if (this.notHidden(list[i],title)) {
var countString = this.getTaggingCount(list[i].title);
var excerpt = this.getExcerpt(title,list[i].title);
if (useCommas)
output.push((first ? "" : ", ") + "[[" + list[i].title + "]]" + countString + excerpt);
else
output.push("*[[" + list[i].title + "]]" + countString + excerpt + "\n");
first = false;
}
}
return this.drawTable(place,
this.makeColumns(output,useCommas ? 1 : parseInt(this.getTagglyOpt(title,"numCols"))),
useCommas ? "commas" : "normal");
},
// this is for the "grouped" mode
createTagglyListGrouped: function(place,title,isTagExpr) {
var sortBy = this.getTagglyOpt(title,"sortBy");
var sortOrder = this.getTagglyOpt(title,"sortOrder");
var list = config.taggly.getTiddlers(title,sortBy,isTagExpr);
if (sortOrder == "desc")
list = list.reverse();
var leftOvers = []
for (var i=0;i<list.length;i++)
leftOvers.push(list[i].title);
var allTagsHolder = {};
for (var i=0;i<list.length;i++) {
for (var j=0;j<list[i].tags.length;j++) {
if (list[i].tags[j] != title) { // not this tiddler
if (this.notHidden(list[i].tags[j],title)) {
if (!allTagsHolder[list[i].tags[j]])
allTagsHolder[list[i].tags[j]] = "";
if (this.notHidden(list[i],title)) {
allTagsHolder[list[i].tags[j]] += "**[["+list[i].title+"]]"
+ this.getTaggingCount(list[i].title) + this.getExcerpt(title,list[i].title) + "\n";
leftOvers.setItem(list[i].title,-1); // remove from leftovers. at the end it will contain the leftovers
}
}
}
}
}
var allTags = [];
for (var t in allTagsHolder)
allTags.push(t);
var sortHelper = function(a,b) {
if (a == b) return 0;
if (a < b) return -1;
return 1;
};
allTags.sort(function(a,b) {
var tidA = store.getTiddler(a);
var tidB = store.getTiddler(b);
if (sortBy == "title") return sortHelper(a,b);
else if (!tidA && !tidB) return 0;
else if (!tidA) return -1;
else if (!tidB) return +1;
else return sortHelper(tidA[sortBy],tidB[sortBy]);
});
var leftOverOutput = "";
for (var i=0;i<leftOvers.length;i++)
if (this.notHidden(leftOvers[i],title))
leftOverOutput += "*[["+leftOvers[i]+"]]" + this.getTaggingCount(leftOvers[i]) + this.getExcerpt(title,leftOvers[i]) + "\n";
var output = [];
if (sortOrder == "desc")
allTags.reverse();
else if (leftOverOutput != "")
// leftovers first...
output.push(leftOverOutput);
for (var i=0;i<allTags.length;i++)
if (allTagsHolder[allTags[i]] != "")
output.push("*[["+allTags[i]+"]]" + this.getTaggingCount(allTags[i]) + this.getExcerpt(title,allTags[i]) + "\n" + allTagsHolder[allTags[i]]);
if (sortOrder == "desc" && leftOverOutput != "")
// leftovers last...
output.push(leftOverOutput);
return this.drawTable(place,
this.makeColumns(output,parseInt(this.getTagglyOpt(title,"numCols"))),
"grouped");
},
// used to build site map
treeTraverse: function(title,depth,sortBy,sortOrder,isTagExpr) {
var list = config.taggly.getTiddlers(title,sortBy,isTagExpr);
if (sortOrder == "desc")
list.reverse();
var indent = "";
for (var j=0;j<depth;j++)
indent += "*"
var childOutput = "";
if (depth > this.config.siteMapDepthLimit)
childOutput += indent + this.lingo.tooDeepMessage;
else
for (var i=0;i<list.length;i++)
if (list[i].title != title)
if (this.notHidden(list[i].title,this.config.inTiddler))
childOutput += this.treeTraverse(list[i].title,depth+1,sortBy,sortOrder,false);
if (depth == 0)
return childOutput;
else
return indent + "[["+title+"]]" + this.getTaggingCount(title) + this.getExcerpt(this.config.inTiddler,title,depth) + "\n" + childOutput;
},
// this if for the site map mode
createTagglyListSiteMap: function(place,title,isTagExpr) {
this.config.inTiddler = title; // nasty. should pass it in to traverse probably
var output = this.treeTraverse(title,0,this.getTagglyOpt(title,"sortBy"),this.getTagglyOpt(title,"sortOrder"),isTagExpr);
return this.drawTable(place,
this.makeColumns(output.split(/(?=^\*\[)/m),parseInt(this.getTagglyOpt(title,"numCols"))), // regexp magic
"sitemap"
);
},
macros: {
tagglyTagging: {
handler: function (place,macroName,params,wikifier,paramString,tiddler) {
var parsedParams = paramString.parseParams("tag",null,true);
var refreshContainer = createTiddlyElement(place,"div");
// do some refresh magic to make it keep the list fresh - thanks Saq
refreshContainer.setAttribute("refresh","macro");
refreshContainer.setAttribute("macroName",macroName);
var tag = getParam(parsedParams,"tag");
var expr = getParam(parsedParams,"expr");
if (expr) {
refreshContainer.setAttribute("isTagExpr","true");
refreshContainer.setAttribute("title",expr);
refreshContainer.setAttribute("showEmpty","true");
}
else {
refreshContainer.setAttribute("isTagExpr","false");
if (tag) {
refreshContainer.setAttribute("title",tag);
refreshContainer.setAttribute("showEmpty","true");
}
else {
refreshContainer.setAttribute("title",tiddler.title);
refreshContainer.setAttribute("showEmpty","false");
}
}
this.refresh(refreshContainer);
},
refresh: function(place) {
var title = place.getAttribute("title");
var isTagExpr = place.getAttribute("isTagExpr") == "true";
var showEmpty = place.getAttribute("showEmpty") == "true";
removeChildren(place);
addClass(place,"tagglyTagging");
var countFound = config.taggly.getTiddlers(title,'title',isTagExpr).length
if (countFound > 0 || showEmpty) {
var lingo = config.taggly.lingo;
config.taggly.createListControl(place,title,"hideState");
if (config.taggly.getTagglyOpt(title,"hideState") == "show") {
createTiddlyElement(place,"span",null,"tagglyLabel",
isTagExpr ? lingo.labels.exprLabel.format([title]) : lingo.labels.label.format([title]));
config.taggly.createListControl(place,title,"title");
config.taggly.createListControl(place,title,"modified");
config.taggly.createListControl(place,title,"created");
config.taggly.createListControl(place,title,"listMode");
config.taggly.createListControl(place,title,"excerpts");
config.taggly.createListControl(place,title,"numCols");
config.taggly.createTagglyList(place,title,isTagExpr);
if (countFound == 0 && showEmpty)
createTiddlyElement(place,"div",null,"tagglyNoneFound",lingo.labels.noneFound);
}
}
}
}
},
// todo fix these up a bit
styles: [
"/*{{{*/",
"/* created by TagglyTaggingPlugin */",
".tagglyTagging { padding-top:0.5em; }",
".tagglyTagging li.listTitle { display:none; }",
".tagglyTagging ul {",
" margin-top:0px; padding-top:0.5em; padding-left:2em;",
" margin-bottom:0px; padding-bottom:0px;",
"}",
".tagglyTagging { vertical-align: top; margin:0px; padding:0px; }",
".tagglyTagging table { margin:0px; padding:0px; }",
".tagglyTagging .button { visibility:hidden; margin-left:3px; margin-right:3px; }",
".tagglyTagging .button, .tagglyTagging .hidebutton {",
" color:[[ColorPalette::TertiaryLight]]; font-size:90%;",
" border:0px; padding-left:0.3em;padding-right:0.3em;",
"}",
".tagglyTagging .button:hover, .hidebutton:hover, ",
".tagglyTagging .button:active, .hidebutton:active {",
" border:0px; background:[[ColorPalette::TertiaryPale]]; color:[[ColorPalette::TertiaryDark]];",
"}",
".selected .tagglyTagging .button { visibility:visible; }",
".tagglyTagging .hidebutton { color:[[ColorPalette::Background]]; }",
".selected .tagglyTagging .hidebutton { color:[[ColorPalette::TertiaryLight]] }",
".tagglyLabel { color:[[ColorPalette::TertiaryMid]]; font-size:90%; }",
".tagglyTagging ul {padding-top:0px; padding-bottom:0.5em; margin-left:1em; }",
".tagglyTagging ul ul {list-style-type:disc; margin-left:-1em;}",
".tagglyTagging ul ul li {margin-left:0.5em; }",
".editLabel { font-size:90%; padding-top:0.5em; }",
".tagglyTagging .commas { padding-left:1.8em; }",
"/* not technically tagglytagging but will put them here anyway */",
".tagglyTagged li.listTitle { display:none; }",
".tagglyTagged li { display: inline; font-size:90%; }",
".tagglyTagged ul { margin:0px; padding:0px; }",
".excerpt { color:[[ColorPalette::TertiaryDark]]; }",
".excerptIndent { margin-left:4em; }",
"div.tagglyTagging table,",
"div.tagglyTagging table tr,",
"td.tagglyTagging",
" {border-style:none!important; }",
".tagglyTagging .contents { border-bottom:2px solid [[ColorPalette::TertiaryPale]]; padding:0 1em 1em 0.5em;",
" margin-bottom:0.5em; }",
".tagglyTagging .indent1 { margin-left:3em; }",
".tagglyTagging .indent2 { margin-left:4em; }",
".tagglyTagging .indent3 { margin-left:5em; }",
".tagglyTagging .indent4 { margin-left:6em; }",
".tagglyTagging .indent5 { margin-left:7em; }",
".tagglyTagging .indent6 { margin-left:8em; }",
".tagglyTagging .indent7 { margin-left:9em; }",
".tagglyTagging .indent8 { margin-left:10em; }",
".tagglyTagging .indent9 { margin-left:11em; }",
".tagglyTagging .indent10 { margin-left:12em; }",
".tagglyNoneFound { margin-left:2em; color:[[ColorPalette::TertiaryMid]]; font-size:90%; font-style:italic; }",
"/*}}}*/",
""].join("\n"),
init: function() {
merge(config.macros,this.macros);
config.shadowTiddlers["TagglyTaggingStyles"] = this.styles;
store.addNotification("TagglyTaggingStyles",refreshStyles);
}
};
config.taggly.init();
//}}}
/***
InlineSlidersPlugin
By Saq Imtiaz
http://tw.lewcid.org/sandbox/#InlineSlidersPlugin
// syntax adjusted to not clash with NestedSlidersPlugin
// added + syntax to start open instead of closed
***/
//{{{
config.formatters.unshift( {
name: "inlinesliders",
// match: "\\+\\+\\+\\+|\\<slider",
match: "\\<slider",
// lookaheadRegExp: /(?:\+\+\+\+|<slider) (.*?)(?:>?)\n((?:.|\n)*?)\n(?:====|<\/slider>)/mg,
lookaheadRegExp: /(?:<slider)(\+?) (.*?)(?:>)\n((?:.|\n)*?)\n(?:<\/slider>)/mg,
handler: function(w) {
this.lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = this.lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart ) {
var btn = createTiddlyButton(w.output,lookaheadMatch[2] + " "+"\u00BB",lookaheadMatch[2],this.onClickSlider,"button sliderButton");
var panel = createTiddlyElement(w.output,"div",null,"sliderPanel");
panel.style.display = (lookaheadMatch[1] == '+' ? "block" : "none");
wikify(lookaheadMatch[3],panel);
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
},
onClickSlider : function(e) {
if(!e) var e = window.event;
var n = this.nextSibling;
n.style.display = (n.style.display=="none") ? "block" : "none";
return false;
}
});
//}}}
/*{{{*/
/* created by TagglyTaggingPlugin */
.tagglyTagging { padding-top:0.5em; }
.tagglyTagging li.listTitle { display:none; }
.tagglyTagging ul {
margin-top:0px; padding-top:0.5em; padding-left:2em;
margin-bottom:0px; padding-bottom:0px;
}
.tagglyTagging { vertical-align: top; margin:0px; padding:0px; }
.tagglyTagging table { margin:0px; padding:0px; }
.tagglyTagging .button { visibility:hidden; margin-left:3px; margin-right:3px; }
.tagglyTagging .button, .tagglyTagging .hidebutton {
color:[[ColorPalette::TertiaryLight]]; font-size:90%;
border:0px; padding-left:0.3em;padding-right:0.3em;
}
.tagglyTagging .button:hover, .hidebutton:hover,
.tagglyTagging .button:active, .hidebutton:active {
border:0px; background:[[ColorPalette::TertiaryPale]]; color:[[ColorPalette::TertiaryDark]];
}
.selected .tagglyTagging .button { visibility:visible; }
.tagglyTagging .hidebutton { color:[[ColorPalette::Background]]; }
.selected .tagglyTagging .hidebutton { color:[[ColorPalette::TertiaryLight]] }
.tagglyLabel { color:[[ColorPalette::TertiaryMid]]; font-size:90%; }
.tagglyTagging ul {padding-top:0px; padding-bottom:0.5em; margin-left:1em; }
.tagglyTagging ul ul {list-style-type:disc; margin-left:-1em;}
.tagglyTagging ul ul li {margin-left:0.5em; }
.editLabel { font-size:90%; padding-top:0.5em; }
.tagglyTagging .commas { padding-left:1.8em; }
/* not technically tagglytagging but will put them here anyway */
.tagglyTagged li.listTitle { display:none; }
.tagglyTagged li { display: inline; font-size:90%; }
.tagglyTagged ul { margin:0px; padding:0px; }
.excerpt { color:[[ColorPalette::TertiaryDark]]; }
.excerptIndent { margin-left:4em; }
div.tagglyTagging table,
div.tagglyTagging table tr,
td.tagglyTagging
{border-style:none!important; }
.tagglyTagging .contents { border-bottom:2px solid [[ColorPalette::TertiaryPale]]; padding:0 1em 1em 0.5em;
margin-bottom:0.5em; }
.tagglyTagging .indent1 { margin-left:3em; }
.tagglyTagging .indent2 { margin-left:4em; }
.tagglyTagging .indent3 { margin-left:5em; }
.tagglyTagging .indent4 { margin-left:6em; }
.tagglyTagging .indent5 { margin-left:7em; }
.tagglyTagging .indent6 { margin-left:8em; }
.tagglyTagging .indent7 { margin-left:9em; }
.tagglyTagging .indent8 { margin-left:10em; }
.tagglyTagging .indent9 { margin-left:11em; }
.tagglyTagging .indent10 { margin-left:12em; }
.tagglyNoneFound { margin-left:2em; color:[[ColorPalette::TertiaryMid]]; font-size:90%; font-style:italic; }
/*}}}*/
{{Note{''Note:'' NCS 515 tracks with this course and requires a semester project utilizing Unix/Linux. A term project is ''not'' a requirement for NCS 205.}}}
!! Semester Project
A related term project will be a required component of this course, representing 25% of your final grade, and will be due at end of the semester during finals week.
You will be required to submit a project proposal, first draft, and final report. Project reports will only be accepted if the proposal has been approved.
Final reports must be in the dual-column ACM SIG conference format. Templates can be found on the [[ACM Website|https://www.acm.org/publications/proceedings-template]]. Not following the ACM format will result in a 10% grade deduction.
!!! Due Dates:
* Project Proposal: Third week of classes
* First Draft: Last week of classes
* Final version: Finals week
Actual due dates will be posted to the class calendar
!!! Project Proposal
Your proposal should be of professional quality. It should minimally state what you intend to do, how you intend to accomplish your goal, and what the success criteria will be. It should also include a timeline of milestones that you intend to complete to reach your end goal.
A project proposal grade will be posted for tracking but will not be factored into your overall grade for this course. Approval and acceptance of your project proposal must be obtained before proceeding.
!!! Final Report
Each student will also be expected to submit a paper that outlines the work they performed on their project. The paper MUST conform to the following guidlines:
* The paper must be formatted using the ACM ~SIG-Alternate format. Templates for MS Word and ~LateX can be found here: http://www.acm.org/publications/proceedings-template
* Be sure to review the ACM Author Guidelines that are also available on that page to ensure you are properly formatting your paper.
* The paper must include references. These references must be properly cited throughout the document according to the ACM guidelines.
** If you used any online references they should also be referenced as ONLINE resources. This includes any technical documents you followed, white papers, as well as links to homepages of tools that you used for your project. Please use the proper ACM guidelines for citing Online references.
** All references should be listed in the bibliography in ACM format.
* You are required to have the following sections in your paper:
** Abstract
** Introduction
** Background Information
*** General info on the topic
*** Background information necessary to prep the user on the topic (pre-req knowledge)
*** include experimentation product information and capabilities
** Procedures
*** What did you actually do? Should be written so someone can pick up your document and replicate the work easily. (ie. your grandmother)
** Issues and Resolutions
*** What did not work? How did you fix it?
** Conclusions
** References
All papers should be reviewed for spelling, grammar errors, and proper sentence structure. Points will be deducted if any such problems are found. Treat these papers as something you are writing for possible submission for publication.
Do not fill your papers with screenshots and images. Use images and screenshots vary sparingly and only to make a point. Tables as well as charts and graphs are acceptable as long as they make sense and support the information you are trying to convey.
! The vi editor
!! vi Intro
!!! Background:
* ex, vi, vim
** {{Command{ex}}} = line oriented text editor (for printed output / slow displays / modems)
*** demonstrate ''c'' (change) and ''i'' (insert) commands. Go to a line number, use command with ''.'' to return to prompt.
** {{Command{vi}}} = screen oriented instead of line oriented
*** Different modes - either entering text or executing commands
*** Commands are either {{Command{vi}}} commands or {{Command{ex}}} commands.
** {{Command{ex}}} & {{Command{vi}}} are different interfaces to the same program
** {{Command{ex}}} & {{Command{vi}}} began with original unix versions, over 30 years ago
** {{Command{vi}}} is now the standard unix text editor
** {{Command{vim}}} = vi Improved - extra commands and functionality
!!! Using vi:
* Opening a document for editing loads it into a buffer, which is the in-memory text of a file.
** Any changes are made to the buffer and not saved to the file until the //write// command is provided.
* There are two Modes:
** Command mode - where you provide commands to the editor
*** These may be either {{Command{vi}}} or {{Command{ex}}} commands
** Input mode - where you can interact with the content of the file
*** You'll typically see the string ''-- INSERT --'' in the bottom-left corner when you're in Input Mode
*** Leave input mode by pressing ESC
* vi commands (command mode) contain an operator (what to do) and scope (what to do it on)
** Examples:
*** {{Monospaced{''d$''}}} - delete (d) all text from the cursor to the end of the line ($ typically means end of line)
*** {{Monospaced{''dw''}}} - delete (d) the current word
*** {{Monospaced{''d5w''}}} - delete (d) the current and next 4 (5) words (w)
*** {{Monospaced{''d2d''}}} - delete (d) the current and next 1 (2) line (d)
*** {{Monospaced{''cw''}}} - change (c) the next word (w), placing you in input mode
*** {{Monospaced{''ct:''}}} - change (c) all characters until (t) the next colon (:)
* Searching with ''/'' and ''?''
** Search down with the ''/'' key
** Search up with the ''?'' key
*** After you type either ''/'' and ''?'', you cursor will move to the bottom-left corner and you will be prompted to enter a search string. Press enter to begin the search.
** Repeat your last search with ''n''
!! Using ex commands in vi
The {{Command{vi}}} editor is a the ''vi''sual screen-oriented front-end for the {{Command{ex}}} line-oriented text editor. {{Command{ex}}} was one of the original Unix text editors from the days where text files could only be displayed and edited one line at a time. It wasn't yet possible to display a full screen of text. The ''vi''sual functionality was supported after technology evolved to support full-screen document editing. {{Command{vi}}} also supports the original {{Command{ex}}} commands for manipulating a document. These commands bring a great deal of power to the editor and make solving complex tasks rather simple.
* Press the : (colon) key to enter {{Command{ex}}} command mode when you are no in Input mode. Your cursor will move to the bottom left corner.
* {{Command{ex}}} commands will be displayed on the bottom status line. Press ~CTRL-C to cancel the command and return to vi mode.
* Syntax: {{Monospaced{'' :[address]command ''}}}
** {{Monospaced{'' :[address] ''}}} is an optional component which allows you to specify which lines to act upon.
!!! Valid address formats
* Addresses may be addressed singly:
** {{Monospaced{''.''}}} - represents current line (default if no address is specified)
** {{Monospaced{''//n//''}}} - a specific line number
** {{Monospaced{''$''}}} - last line in the file
* or as a range:
**{{Monospaced{''%''}}} - Whole file
** {{Monospaced{''address1,address2''}}} - from address1 to address2.
** Also includes +//n// and -//n// to include the next or previous //n// lines
* Examples:
** {{Monospaced{'':12,20d''}}} - delete lines 12 to 20
** {{Monospaced{'':.,+5''}}} - current and next five lines
** {{Monospaced{'':10,$''}}} - lines 10 through the end of the file
** {{Monospaced{'':$-2,$''}}} - last three lines (last line and two previous)
!!! Most useful ex commands
* ''d'' - delete lines
** {{Monospaced{'':10d''}}} - delete line 10
** {{Monospaced{'' :1,10d ''}}} - delete lines 1 to 10
* ''e'' - edit
** {{Monospaced{'':e! ''}}} - reopen current file, discarding changes
* ''s'' - substitute
**{{Monospaced{'' :s/one/two/ ''}}} - change first instances of one to two on the current line
**{{Monospaced{'' :%s/one/two/ ''}}} - change first instance of one to two on all lines in the document
**{{Monospaced{'' :%s/one/two/g ''}}} - change all instances of one to two on all lines in the document
**{{Monospaced{'' :.,+5s/one/two/g ''}}} - change all instances of one to two on current and next 5 lines.
* ''g'' - globally execute specified commands on lines containing a particular pattern
** {{Monospaced{'' :g/stuff/d ''}}} - delete all lines containing the string stuff
** {{Monospaced{'' :g/lpd-errs/s/^/#/ ''}}} - add a comment to the beginning of the line on all lines containing the string lpd-errors
** {{Monospaced{'' :10,20/g/stuff/d ''}}} - remove lines between lines 10 and 20 that contain the string delete
----
!! More info
*vi handouts: [[vi Diagram|handouts/viDiagram.pdf]] & [[Old Handout|handouts/viHandout.pdf]]
*{{Command{vimtutor}}} command
*http://www.gentoo.org/doc/en/vi-guide.xml
*[[UNIX Command summary|handouts/UnixCommandSummary.pdf]] back page
http://docstore.mik.ua/orelly/unix/unixnut/ch09_01.htm
! Using the compilers
Also a simple exercise to get more practice editing text files with vi
{{Command{gcc}}} & {{Command{g++}}}
Use {{Command{gcc}}} for compiling C code and {{Command{g++}}} for compiling C++ code. Source code file extensions must either be .c or .cpp
{{Command{gcc -o //name_of_executable// source.c}}}
{{Command{g++ -o //name_of_executable// source.cpp}}}
//name_of_executable// = executable file to create after compiling your source code, instead of using the default a.out
{{{
#include <stdio.h>
main()
{
printf("Hello World in C\n\n");
}
}}}
{{{
#include <iostream>
using namespace std;
int main()
{
cout << "Hello World!" << endl;
return 0;
}
}}}
! Assignments
!! Read :
- Chapter 12 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
!! Complete:
- [[Lab 27|labs/lab27.pdf]] & [[Lab 28|labs/lab28.pdf]]
- These labs are optional for additional vi practice and will be accepted for extra credit.
/***
|Name|ToggleSideBarMacro|
|Created by|SaqImtiaz|
|Location|http://lewcid.googlepages.com/lewcid.html#ToggleSideBarMacro|
|Version|1.0|
|Requires|~TW2.x|
!Description:
Provides a button for toggling visibility of the SideBar. You can choose whether the SideBar should initially be hidden or displayed.
!Demo
<<toggleSideBar "Toggle Sidebar">>
!Usage:
{{{<<toggleSideBar>>}}} <<toggleSideBar>>
additional options:
{{{<<toggleSideBar label tooltip show/hide>>}}} where:
label = custom label for the button,
tooltip = custom tooltip for the button,
show/hide = use one or the other, determines whether the sidebar is shown at first or not.
(default is to show the sidebar)
You can add it to your tiddler toolbar, your MainMenu, or where you like really.
If you are using a horizontal MainMenu and want the button to be right aligned, put the following in your StyleSheet:
{{{ .HideSideBarButton {float:right;} }}}
!History
*23-07-06: version 1.0: completely rewritten, now works with custom stylesheets too, and easier to customize start behaviour.
*20-07-06: version 0.11
*27-04-06: version 0.1: working.
!Code
***/
//{{{
config.macros.toggleSideBar={};
config.macros.toggleSideBar.settings={
styleHide : "#sidebar { display: none;}\n"+"#contentWrapper #displayArea { margin-right: 1em;}\n"+"",
styleShow : " ",
arrow1: "«",
arrow2: "»"
};
config.macros.toggleSideBar.handler=function (place,macroName,params,wikifier,paramString,tiddler)
{
var tooltip= params[1]||'toggle sidebar';
var mode = (params[2] && params[2]=="hide")? "hide":"show";
var arrow = (mode == "hide")? this.settings.arrow1:this.settings.arrow2;
var label= (params[0]&¶ms[0]!='.')?params[0]+" "+arrow:arrow;
var theBtn = createTiddlyButton(place,label,tooltip,this.onToggleSideBar,"button HideSideBarButton");
if (mode == "hide")
{
(document.getElementById("sidebar")).setAttribute("toggle","hide");
setStylesheet(this.settings.styleHide,"ToggleSideBarStyles");
}
};
config.macros.toggleSideBar.onToggleSideBar = function(){
var sidebar = document.getElementById("sidebar");
var settings = config.macros.toggleSideBar.settings;
if (sidebar.getAttribute("toggle")=='hide')
{
setStylesheet(settings.styleShow,"ToggleSideBarStyles");
sidebar.setAttribute("toggle","show");
this.firstChild.data= (this.firstChild.data).replace(settings.arrow1,settings.arrow2);
}
else
{
setStylesheet(settings.styleHide,"ToggleSideBarStyles");
sidebar.setAttribute("toggle","hide");
this.firstChild.data= (this.firstChild.data).replace(settings.arrow2,settings.arrow1);
}
return false;
}
setStylesheet(".HideSideBarButton .button {font-weight:bold; padding: 0 5px;}\n","ToggleSideBarButtonStyles");
//}}}
|~ViewToolbar|closeTiddler closeOthers editTiddler > fields syncing permalink references jump|
|~EditToolbar|+saveTiddler -cancelTiddler deleteTiddler|
{{Note{This video is a nice demo and overview on how this all works. It may be helpful to review it before proceeding. https://www.youtube.com/watch?v=XFJ6_BYno08}}}
!! Defeating firewalls with SSH to access protected resources
Knowing how to more fully use SSH and it's tunneling and proxy capabilities to defeat firewalls or access private IP addresses over the internet is an excellent skill for a security practitioner to have! There are two methods we can use with SSH to defeat firewalls and access these resources:
A. Dynamic application-level port forwarding (SOCKS proxy)
<<<
Specifies local "dynamic" application-level port forwarding. This works by allocating a socket to listen to a port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and the application protocol is then used to determine where to connect to from the remote machine.
<<<
B. Port forwarding
<<<
Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side. This works by allocating a socket to listen to either a TCP port on the local side, optionally bound to the specified bind_address, or to a Unix socket. Whenever a connection is made to the local port or socket, the connection is forwarded over the secure channel, and a connection is made to either host port hostport, or the Unix socket remote_socket, from the remote machine.
<<<
Method A. functions as a traditional application-level proxy. You would configure your application (eg: web browser) to proxy all connections through the tunnel. Method B. creates a 1:1 connection: a TCP port on your local PC is tunneled through the SSH connection to a specific IP address and TCP port on the other side. This method is best when there is no option to configure a proxy in your application.
We're going to use method ''A'' for accessing internal web resources behind our class router. This grants us the most flexibility since our browser allows us to configure an application-level proxy.
Before you begin, open your web browser and load the page http://ifconfig.me. Take note of the IP address displayed. We will compare this to the IP address you receive after everything is set up.
!!! A. Establishing a SOCKS proxy with SSH
A proxy is a middle man, passing on network requests to their destination on your behalf.
A SOCKS proxy (socket secure) is a protocol to route packets between a client and a server through an intermediate proxy. This is used (typically for web traffic) when the client is not able to communicate with the server directly, but the client can communicate with the proxy system and the chosen proxy can communicate with the server. Some sites set up a proxy for web traffic as a means to enforce policy, monitor traffic, and block direct connections to web sites.
Here, your home PC cannot access your web server VM or the Naemon monitoring server but the class shell server can. We'll use the class shell server to proxy your browser's web connections and be the middleman for your web requests. This diagram illustrates the overall goal. We see your proxy connection traveling through the encrypted SSH tunnel to the class shell server. Web requests are then made from the perspective of the class shell server.
[img[img/proxy.png]]
SSH can be used to establish a SOCKS proxy. This functionality is available from putty or the command line ~OpenSSH
''1.'' To set up the Proxy on your home PC, complete either ''a)'' or ''b)'', depending on your OS:
''a)'' If your home OS is Mac or Unix: This command will create an encrypted proxy tunnel between your PC and the specified host, in this case our class shell server. Traffic connecting to your PC on port 8118 will then pass through this proxy. Execute a similar command on your home computer. You may also need to update the username.
<<<
Set up SOCKS proxy: {{Command{ssh -p 2205 -D 8118 lab.ncs205.net}}}
<<<
''b)'' Follow these steps when connecting with Putty from your home Windows PC:
<<<
* Expand the Connection / SSH menu
* Select Tunnels
* Enter ''8118'' in the Source port box
* Select ''Dynamic''
* Click Add
* Connect to a remote host (the class shell server) as normal
<<<
* [[This video|Putty Proxy]] demonstrates configuring Putty to add the dynamic tunnel.
''2.'' Your browser must be configured to pass traffic through the encrypted proxy.
I use the ~FoxyProxy extension to easily toggle between proxy settings in my browser. It can also be configured to automatically send only selected sites through the proxy.
* Firefox:
** [[Firefox Extension|https://addons.mozilla.org/en-US/firefox/addon/foxyproxy-standard/]]
** ~NCS205 settings file for use in Firefox: [[FoxyProxy-ncs205.json|https://www.ncs205.net/media/FoxyProxy-ncs205.json]]
* Chrome:
** [[Chrome Extension|https://chrome.google.com/webstore/detail/foxyproxy-standard/gcknhkkoolaabfmlnjonogaaifnjlfnp]]
** I don't have a version of the config for Chrome; you're on your own for now and will need to configure it manually.
** Be sure to check the //Proxy DNS// option
Install the browser extension, import the settings file, and enable the proxy.
* [[This video|Firefox Proxy]] demonstrates using Firefox with the proxy to access an internal website
!!! B. Verification
Verification should be built into everything you configure. Now that your proxy is established, let's verify it is functioning correctly and web connections from Firefox are flowing through the class infrastructure. Load the page http://ifconfig.me again in your browser and observe the IP address. It should have changed from the original value you observed and instead contain the public IP address of the class shell server. Next, run the command {{Command{curl ifconfig.me}}} on the class shell server. The IP address in your browser and displayed on the command line should match. This will confirm your traffic is now properly going through the proxy.
With the class server acting as a middle man, you can now load internal resources in this web browser which would have otherwise been blocked from the outside world.
!!! C. Naemon infrastructure monitoring
[[Naemon|https://www.naemon.org/]] is a tool which continuously monitors resources to provide a high level view of the health of an environment. I'm running a Naemon server to monitor your ~VMs and use it to assist with grading your labs. You can also use it to monitor the state of your systems and correct any issues it discovers.
Naemon is running on the internal class network and is not directly accessible from the outside world. You will need to bypass the router and use the class shell server as a proxy in order to reach it.
Once the proxy is configured in your browser, navigate to the URL '' http://head.ncs205.net/ ''. The username is ''ncs205'' and password is ''naemon''.
This video contains a brief [[Naemon Introduction]].
{{Note{Naemon status checks run every two hours. If you fix a problem, you will either need to wait up to two hours for the recheck or force Naemon to recheck.}}}
{{Warning{Warning: Naemon checks are not a replacement for your own sound testing and verification. They may return false positives and negatives. Not every possibility can be evaluated. They are only a troubleshooting and status aid; not a definitive determination that something is correct. I will still perform manual testing for most of your labs that Naemon cannot fully evaluate.}}}
!! Asking for help
* Title your posts appropriately. Use something descriptive in the name and not just the lab and question number. A subject like @@Lab 17, #2 - Incorrectly discarding data@@ is far more helpful than something generic like ''Lab 17''.
* When asking for help in blackboard, be sure to include relevant supporting information. You'll receive faster responses if you provide everything someone needs to help you.
** If you're asking about a lab question, include that question in your post so everyone doesn't need to first look at the lab.
** Did you receive an error from a command? Be sure to include the error and the command you ran. The shell prompt will also include helpful information:
*** The host you're running the command on
*** The user you're running the command as
*** A portion of the current working directory. Including the full output of the {{Command{pwd}}} command would be helpful too
*** The exact command string you're running.
** Don't forget to include any relevent log information and troubleshooting steps you've already taken. You're more likely to get help if you start the process and can describe what you've already done to troubleshoot.
* Be sure to review everything for typos first. Too many posts to Blackboard asking for help will be for problems caused by typos. Save some time and check your typing first.
* Screenshots are helpful too. Pictures are worth a thousand words.
!! Posting Screenshots
When posting screenshots, use the Insert Local Files(circled) in Blackboard. Don't attach a file. It's much easier work with embedded images than ones that need to be opened in a new tab.
@@display:block;text-align:center;[img[img/screenshots.png]]@@
!! Pasting in terminal output
Everyone should be using the Blackboard discussion boards during the course of the semester and will likely need to paste in output from the command line at some point.
Aesthetics and readability should be considered in everything you produce. We can make our post easier to read with a couple additional steps.
''1.'' Paste your copied text from the terminal where you would like it to appear. Finish typing out your message. Before sending, change the formatting for the portions you pasted from the terminal.
''2.'' Select the text you pasted in and change the paragraph type to Formatted Code. This will remove the double spacing.
''3.'' Select the text you pasted in and choose the font ''Courier New''. All commands and text copied from the terminal should be written with a monospaced font like Courier New to make spacing uniform between the characters and show that what you're typing is a command or output from one.
''4.'' Select the command you executed to get the output and change it to bold. This makes it easier to identify the command that was used from the output returned. Including the shell prompt and executed command provides important context.
''5.'' If appropriate, use the highlighter to draw attention to any parts you're talking about. Be sure to first change the color to a brighter one.
@@display:block;text-align:center;[img[img/blackboard4.png]]@@
You'll finally be left with something that is much easier to read. You're more likely to get a response to your forum post if you provide all necessary information in a way that's easy to work with. Pasting text like this is preferable to just posting a screenshot. If you paste in the text, someone can quote it in a reply and easily highlight relevant parts.
@@display:block;text-align:center;[img[img/blackboard3.png]]@@
/%
----
avoid Blackboard's text mangling and
If you paste copied text from the terminal, blackboard will turn it into a mangled mess:
than the Blackboard mangled mess
@@display:block;text-align:center;[img[img/blackboard0.png]]@@
''1.'' Insert a few blank lines where you want to put the pasted text. These blank lines will make it easier to add additional text after inserting your pasted text from the terminal
''2.'' Choose the HTML editor from the Toolbar
@@display:block;text-align:center;[img[img/blackboard1.png]]@@
''4.'' Add a {{Command{<pre>}}} HTML tag before your pasted text and a {{Command{</pre>}}} tag after it. This will prevent the mangled formatting and preserve all spacing, just as you see it in the terminal.
@@display:block;text-align:center;[img[img/blackboard2.png]]@@
''5.'' Click update. You should now see your copied text nicely formatted in Blackboard.
''6.'' Select the text you pasted in and choose the font ''Courier New''. All commands and text copied from the terminal should be written with a monospaced font like Courier New to make spacing uniform and highlight what you're typing is a command or output from one.
''7.'' Highlight the command you executed to get the output and change it to bold. This makes it easier to identify the command that was used from the output returned.
''8.'' If appropriate, use the highlighter to draw attention to any parts you're talking about. Be sure to first change the color to a brighter one.
@@display:block;text-align:center;[img[img/blackboard4.png]]@@
You'll finally be left with something that is much easier to read than the Blackboard mangled mess. You're more likely to get a response to your forum post if it is easier to read. Pasting text like this is preferable to just posting a screenshot. If you paste in the text, someone can quote it in a reply and easily highlight relevant parts.
@@display:block;text-align:center;[img[img/blackboard3.png]]@@ %/
!! Using Discord
We'll need to keep Discord organized in order to keep it useful. Get in the habit of this now, because you'll have these same issues later in the workplace. The concepts are very similar to what we need to do on Slack in the corporate world.
!!! There are four types of channels:
# //administrative// - Administrative questions about the class like grading, due dates, and general technical support issues. Not for course content.
# //misc-chatter// - Conversation not related to this class
# //resources// - Posts about general course notes and resources that might be helpful for others
# //week#// - The weekly course content discussions. These are Forum channels. Create a new post in this channel regarding material that was ''//assigned//'' in this week.
** For example, if you have a question about a week 1 lab, create a post in the week 1 channel even if we're now in week 2.
** Only posts in these weekly channels will be evaluated for a grade
!! Asking for help
* ''Required reading'': https://www.nohello.com/
** This is such an issue in corporate communication that there's a website dedicated to it. Discord for this class should be treated as business comms.
** Don't waste time with just an empty //hello// without asking your question. It could cause hours or even days to be wasted.
* Use threads for your questions to help keep things organized.
** See below for an example on using threads
** Title your threads appropriately. Use something descriptive in the name and not just the lab and question number. A subject like @@Lab 17, #2 - Incorrectly discarding data@@ is far more helpful than something generic like ''Lab 17''.
** Organization and usability is important in everything you do.
* When asking for help, be sure to include relevant supporting information. You'll receive faster responses if you provide everything someone needs to help you.
** If you're asking about a lab question, including that question in your post is helpful so everyone doesn't need to first look at the lab to know what you're talking about.
** Send us what you're seeing, don't just describe it. A picture is worth a thousand words. Screenshots often capture additional detail omitted from a text description.
*** Did you receive an error from a command? Be sure to include the error and the command you ran.
*** The shell prompt will also include helpful information, such as:
**** The host you're running the command on
**** The user you're running the command as
**** A portion of the current working directory. Including the full output of the {{Command{pwd}}} command might be helpful too
**** The exact command string you're running.
** Don't forget to include any relevant log information, configuration lines, and troubleshooting steps you've already taken. You're more likely to get help if you start the process and can describe what you've already done to troubleshoot.
* Be sure to review everything for typos first. Too many posts asking for help will be for problems caused by typos. Save some time and check your typing first.
* If you solve your problem while you're waiting for help, be sure to post an update. Don't let someone else waste their time helping you when you no longer need it.
!! Using code blocks
* Be sure all code, commands, and output is enclosed within a code block. This will make it easier to identify commands and prevent Discord from interpreting special characters.
* Single commands can be put inside of a code block by enclosing your command in backticks.
* A series of lines can be put inside of a code block by putting three backticks at the start of the first line and three backticks at the end of the last line.
* When possible, sending text in code blocks is better then just sending a screenshot. Text sent in a screenshot cannot be copy/pasted for any testing
* A full list of Markdown formatting options is available in the [[Discord help docs|https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline-]]
Example of using single line code block:
[img[img/discord-code3.png]]
Example of using multi-line code block:
[img[img/discord-code1.png]]
Results of using code blocks:
[img[img/discord-code2.png]]
!! Using threads
Threads in Discord will help keep the weekly channels and conversations organized. Create a new thread for each question you're asking.
----
[img[img/discord1.png]]
# Click on the week number for the material you would like to discuss
# Click on the threads icon up top
----
[img[img/discord2.png]]
# Enter your thread name
# Enter your question in the Starter Message followed by any supporting information in additional posts.
# Click on //Create Thread//
----
[img[img/discord3.png]]
# To join a thread, click on the //# Message// link. Your thread will open to the right
# Post any additional messages within the thread to the right.
----
[img[img/discord4.png]]
The available threads will appear under the weekly channel. You can click on the thread title to easily join the conversation.
Also notice the excellent use of replies here. Sending a message as a reply will notify your recipient they have a new message.
----
[img[img/discord5.png]]
If you would like to follow an interesting thread, right click on the thread message area and choose //Join Thread//.
----
[img[img/discord6.png]]
After joining a thread, it will appear on the left side of your screen under the channel for the week number. This will make it easier to find later.
<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::ViewToolbar]]'></div>
<div class='title' macro='view title'><span class="miniTag" macro="miniTag"></span></div>
<div class='subtitle'>Updated <span macro='view modified date [[MMM DD, YYYY]]'></span> (<span macro='message views.wikified.createdPrompt'></span> <span macro='view created date [[MMM DD, YYYY]]'></span>)<BR><BR></div>
<div class='viewer' macro='view text wikified'></div>
<div class="tagglyTagging" macro="tagglyTagging"><BR><BR></div>
<div class="tagglyTagged" macro="hideSomeTags"></div>
<div class='tagClear'></div>
<!--}}}-->
@@ This will be used in the second half of the semester @@
/%Proxmox hypervisor: https://lab.ncs205.net/
Subnet: 192.168.12.0/23
Gateway: 192.168.12.1
DNS: 192.168.12.10
!! IP Address Assignments:
| !Last octet | !Host Name | !Description |
| n | test |Testing|
| n+1 | www |Web Server|
| n+2 | core |DNS, syslog, ntp|
| n+3 | files |Storage Server|
| n+4 |>| Unused |
| n+5 |>|~|
| n+6 |>|~|
| n+7 | ||
* The fully-qualified hostname for your VM is //host//.//username//.ncs205.net where //host// is in the second column in the table above.
* Your VM IP addresses should be in the form 192.168.12.//n// where //n// is the first IP address you have been assigned in the table below. Increment the value of //n// as necessary for additional ~VMs.
** Do not deviate from the provided IP addresses. These IP addresses will be checked for grading. If you use other ~IPs you will not receive credit for the labs and may conflict with other students.
| !Start IP | !Username |
| 24 | merantn |
| 32 | abukars |
| 40 | aslamt |
| 48 | ayaza |
| 56 | azzarem |
| 64 | candelp |
| 72 | castelw |
| 80 | chenr |
| 88 | conleyrj |
| 96 | foxcl1 |
| 104 | fullersc1 |
| 112 | ganems |
| 120 | garramh |
| 128 | hallje |
| 136 | hewittb |
| 144 | mackc |
| 152 | marvinc |
| 160 | meyersml1 |
| 168 | oduwae |
| 176 | omercet |
| 184 | palapha |
| 192 | rojasd1 |
| 200 | warda |
! Lab network topology
[img[img/topo.png]]
%/
/% awk -v ip=32 '{print "| " ip " | " $1 " |"; ip+=8}' user2009.txt %/
! Introduction to NCS 205
!!! Expectations:
Mostly outlined in the [[syllabus|syllabus/NCS205Syllabus2509.pdf]], but to recap:
* ''Honesty & Integrity'' - Cheating generally results in a failing ''course'' grade.
** This course is in a security program. If you cannot be trusted, you do not belong here.
* ''Motivation & practice'' - You must be motivated to practice the work in order to pick up the material.
** Here's a good article discussing [[productive struggle|http://maateachingtidbits.blogspot.com/2017/11/the-role-of-failure-and-struggle-in.html]] that roughly outlines how I'm teaching this course.
* ''Graded Homework'' - Almost everything will be graded.
* ''Don't fall behind'' - Else the workload will bury you.
** Please let me know early if you're starting to run into trouble.
This class will also use Linux as a vehicle for reinforcing good soft skills. You will be expected to:
* Provide clear and thorough explanations
* Ask questions when help is needed
* Be an active participant in your learning
** Using a whitewater rafting analogy - I'll be the guide and you're our paddlers. I'll chart the path, but you need to get us there.
!!! Class Resources
* Required Textbooks:
** First half of the semester - [[The Linux Command Line|http://linuxcommand.org/tlcl.php]] (free download)
** Second half of the semester - [[Mastering Ubuntu Server, Fourth Edition|https://www.packtpub.com/en-us/product/mastering-ubuntu-server-9781803234243]]
* Class website: https://www.ncs205.net/
** The class website will be our primary resource for course content
** Each content page is generally divided into three sections:
### the content assignment (what to read or watch),
### my notes about the content
### the deliverables for that content
** The search function is your friend. Click the << in the top-right corner to open the side menu bar.
* Brightspace
** [[Brightspace|https://mylearning.suny.edu/d2l/login]] will be used only for announcements and initial class kickoff information.
* Discord will be used for regular class communication
** An invite to our server has been posted to Brightspace
** Discord participation will be [[evaluated as well|Class Participation]].
* Grades
** Grades will be tracked in a text file within your home directory on the class shell server: {{File{.grades.txt}}}
** Be sure to note the dot at the start of the file name
** A command like {{Command{cat ~/.grades.txt}}} will display your grades.
!!! Class Cadence
* A week's worth of new material will be posted to the class website Sunday evening in two parts.
** Unless stated otherwise, part 1 assignments will be due by end of day Wednesday
** Part 2 assignments will be due by end of day Saturday.
** An [[assignment calendar|Calendar]] can be found on our class website in the menu bar above.
* Carnegie credit hour
** A Carnegie credit hour is defined as 50 minutes of lecture and 2 hours of prep/homework for each traditional course credit hour
** This requirement is defined in [[SUNY Policy|https://www.suny.edu/sunypp/documents.cfm?doc_id=168]]
** Translated to our online class, this means we are expected to perform approximately 12 hours of instructional activity per week
** This is hard to gauge in an online class. Please let me know if you feel we are regularly exceeding that.
!!! Extra Help
Several options exist if you are stuck and would like some extra help.
* Post your question or problem to the class Discord server
** Be sure to post to the channel for the week the material was assigned
* Ad-hoc online meetings via Zoom. Let me know if you'd like to schedule one.
* Regularly scheduled weekly Zoom meetings. We can offer these if there is interest
* Weekly office hours. We can schedule these once the semester gets settled in.
{{Warning{
This class will test your skills as a student; ''being a good student will be important in order to successfully complete this course''. This will not be one where you can do the bare minimum and skate by with a good grade. Good ''time management'' and ''study skills'' will be critical. ''If you neglect the material you will likely not successfully complete the course.''
Everything we do this semester will look back on previous work. If you're rushing through and not retaining it, you will surely pay for it later. Having a keen eye for detail, paying attention to the directions, and taking the time to practice and retain the material will make for a much smoother semester.
}}}
!! Accessing the class shell server
The class shell server is an always-on system we will connect to in order to practice the class assignments and submit homework. There are two ways we will access the system - from the command line for entering commands or through a file transfer utility for uploading files.
!!! Connection Tools
* Access the shell (command line) with either:
** [[PuTTY for Windows|http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html]] (Download the latest version of the 64-bit MSI installer)
** [[PuTTY for Mac|https://www.ssh.com/ssh/putty/mac/]]
** Mac, Linux, or Windows Subsystem for Linux: You can also use the command-line SSH. Launch your terminal and run the command {{Command{ssh -p 2205 //username//@lab.ncs205.net}}}.
* Transfer files between the server and your local system:
** Windows: [[WinSCP|https://winscp.net/eng/download.php]]
** Mac: scp/sftp on the command line or any SFTP client like [[FileZilla|https://filezilla-project.org/]]
* Portable versions exist for these applications. This is convenient if you are using campus ~PCs that do not have the tools installed. You may download and run them from a flash drive or your home directory in the lab.
** [[PuTTY|http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html]] - Download and run putty.exe
** [[WinSCP|https://winscp.net/eng/download.php]]: Download the portable package
!!! Logging in
* Use one of the tools above to log in to ''lab.ncs205.net'' on port ''2205''
* Log in with your campus username
* Your initial password will be posted to the main //Content// page in Brightspace.
* Change your password after logging in.
** Run the {{Command{passwd}}} command to change your password
** ''Any accounts still using the default password will be locked on Saturday, September 7.''
This short video will walk you through downloading ~PuTTY, a unix remote access client, and connecting to the system for command line access. Be sure to change the connection details to match the information above.
Video: [[Accessing the shell server]]
/% Download: ~PuTTY - [[installer|https://the.earth.li/~sgtatham/putty/latest/w64/putty-64bit-0.70-installer.msi]] or [[exe|http://the.earth.li/~sgtatham/putty/latest/win64/putty.exe]] %/
!! Working on the command line
Console
* Console is considered the interface with a system as though you are physically sitting at its monitor and keyboard. This lets us interact with the system before the operating system loads
* A virtual console is available for ~VMs or through a lights-out management utility such as a Dell iDRAC.
Remote access
* Remote access to a Linux system such as our class shell server can also be obtained through a remote access service like SSH (Secure ~SHell).
* SSH is the standard command-line remote access interface for Unix/Linux systems. It allows us to interact via a SSH client, much like how your web browser is a client to a web server.
* Our class shell server is a traditional timeshare server. It's always available; we don't power it off.
Shells
* The shell is our interface with the command line. It's a program that takes input from the user, passes it on to the system to process, and returns any output back to you.
!!! Navigating our lab server's filesystem:
* Directory paths
** Directory paths enable us to have a hierarchy of directories and keep our files organized
** Similar to the command line on Windows
** The path separator is a forward slash on Unix/Linux systems - {{File{''/''}}}
** Change directories with the {{Command{cd}}} command
*** eg: {{Command{cd /opt/pub/ncs205/submit}}}
** List the contents of the directory with the {{Command{ls}}} command
** List the contents of the directory in long format with the {{Command{ls -l}}} command
*** Displaying the contents of a directory in long format is always preferred so you can easily see all information about the files
* Some directories of interest:
** {{File{/home/}}} - User home directories **typically** reside below this directory tree. This is just a standard convention - home directories can be anywhere on the system.
*** A user home directory is a space where each user can save their files.
** {{File{/opt/pub/ncs205/submit/}}} - Lab/Homework assignments are uploaded to this directory
** {{File{/opt/pub/ncs205/returned/}}} - Graded homework assignments are stored in this directory for you to download
** {{File{/opt/pub/ncs205/data/}}} - Data files for labs are stored here
** {{File{/tmp/}}} - Temporary scratch space
!!! Executing commands
* Structure of a command string:
** {{Command{''command'' [options] [arguments]}}}
** options and arguments may be optional or required depending on the command
** In Unix command documentation, an item within the square brackets is an optional component. Some commands will also require arguments and some will not.
*** The documentation for each command will outline its requirements.
* Viewing files
** Display a file: {{Command{cat //filename//}}}
** Display a file one page at a time: {{Command{less //filename//}}}
** Edit a text file: {{Command{nano //filename//}}} ''-or-'' {{Command{vi //filename//}}}
!!! Other useful commands
* The UNIX manual - {{Command{man}}}
** If you want to learn more about a command, check out its manpage.
** For example, {{Command{man ls}}} will display detail about the {{Command{ls}}} command
!!! Using Discord
* Class discussion in Discord will make up 10% of your total course grade this semester.
* How this part is graded is discussed in the [[Class Participation]] page
* [[Using Discord]] contains some tips for how to post
!!! Working efficiently
* View your previously executed commands with the {{Command{history}}} command
* Tab completion - Press the tab key to autocomplete commands or file paths
* Up / Down arrows - search up and down through your command history
* Page Up / Page Down - Use these keys to search through your command history for the last commands which begin with a given string
** For example, typing {{Command{ls}}} and then pressing Page Up will jump you to the last command string you executed which started with {{Command{ls}}}.
* The [[Linux Shortcuts]] page will have some more useful items.
!! Submitting homework assignments
See the [[Lab Assignments]] page for details
! Material
!! Read:
* Chapter 1 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
! Operating system basics
!! The core components of a Unix/Linux operating system are:
* Kernel - The main control program of the computer with handles process control, resource management, and interfaces with the hardware.
* File system - Organizes location of items on the system. Everything is shown as a file in the Unix asdas world.
* Shell - The primary interaction between the user and the system on the command line. The shell receives and interprets commands entered by users and passes them on to the kernel to execute.
!! Secondary components:
These are not part of the core OS, but necessary to do useful things with the system
* Basic Utilities - Many are from the GNU project
** System - Tools an administrator would use: mount, dd, fsck
** Userland - Tools regular users would use: file system tools (cd, ls, mkdir), text editors (vi, pico), filters (grep, cut, sed), process tools (ps, kill)
* Development environment - compilers and script interpreters
* System Documentation - man pages, info docs, etc.
* Window Managers - Desktop editions typically add a graphical environment, such as Gnome, KDE, or XFCE
* Larger Applications - Word processor, image editor, web browsers, etc
* Specialized utilities - (For example, tools that come with a focused distro like Kali)
!! UNIX is an OS that supports:
* Multi-tasking - foreground and background processes
* ~Multi-User - Multiple users may access the system at the same time
** Privilege separation - There are system administrators (root user) and regular users. Regular users are able to be isolated from each other
* Time sharing - Share computing resources among many users
* Portability - Can be run on different types of hardware systems/architectures (~PCs, servers, game systems, phones, embedded systems, etc)
!! Types and history of Unix/Linux
* This all started at Bell Labs as a research project
** 1969 Bell labs - AT&T Unics (Uniplexed Information & Computing Service)
** Unics became UNIX when multiuser support was added
** As part of a 1958 antitrust agreement with the government, AT&T could not go into the computer business and charge for software. They had to give free licenses to anyone who asked.
** Source code distributed to researchers at universities allowing them to modify and extend the OS
** Early Unix editions were numbered 1 - 10 based on the edition of the printed manual. It was still mostly for research and development purposes.
* 1978 BSD UNIX (Berkeley Software Distribution)
** Grad students at Berkeley modified and extended the AT&T code
** They bundled and released their add-ons for use at other universities
** Early development slowed due to licensing issues with AT&T and their lawsuits. AT&T wanted to monopolize and monetize this space.
** This project would eventually fork into three different projects: ~FreeBSD (1993), ~OpenBSD (1993), & ~NetBSD (1995)
*** The ~BSDs slowly fell out of favor as Linux gained more ground
* GNU project & the origins of Linux:
** GNU Project (GNU's not Unix) - Richard Stallman (1983) - Wanted to create a totally free OS unencumbered by commercial or licensing issues. The project started with the utilities.
*** FSF : Free Software Foundation (1985)
*** At MIT, he saw many MIT software developers get picked off by companies and sign restrictive non-disclosure agreements.
*** Many companies were now restricting access to Unix source code to limit modification and redistribution, facilitate hardware lock-in and push towards expanded commercialization.
*** This group believed software should be free to run, change, copy and modify so users are the ones in control, free from corporate control, and better software would then develop.
*** This led to the GNU license and brought a philosophy of freedom (freedom (speech), not price (beer) ). Users could access the source code and make changes, but companies could still charge for support.
*** The FSF kernel (GNU/Hurd) was taking too long to develop, though all other components (ie: the utilities) were complete.
** 1991 - The Linux Kernel:
** Linus Torvalds, a Finnish grad student, started working on a kernel for fun after getting impatient for a totally free kernel to work with due to all of the legal battles
*** Minix was a popular academic option. It was good for academics but not allowed for professional use
*** And Minix also required a fee and had a restrictive license
*** BSD was still somewhat encumbered by AT&T licensing issues and legal problems stalling development
*** Linux only had a kernel, not a complete operating system.
* Linux distributions:
** The GNU Project had utilities but no kernel. Linus Torvalds had a kernel but no utilities.
** So Linus provided the kernel (Linux kernel) to accompany FSF GNU utilities and components to make a Linux OS
** Different distributions (eg: Fedora, Gentoo, Debian, Ubuntu, etc) combine the Linux kernel, FSF utilities, and other applications in different ways and focus on different types of users.
*** Such as server distributions, desktop distributions, live distributions, or specialized distributions
** Early success was due to freedom - many other programmers were able to contribute code and ideas
* Early Commercial Distributions: (AIX, HPUX)
** Several commercial Unix distributions existed and only found in large enterprises. These were more popular decades ago and have lost ground to Linux.
A good article about the history of Unix/Linux: [[Did Linux Kill Commercial Unix|https://www.howtogeek.com/440147/did-linux-kill-commercial-unix/]]
! Interacting with the system
There are two ways to interact with a system: through a graphical interface (GUI) or the command line (CLI). Most of our work this semester will be conducted through the CLI.
!! Graphical User Interface (GUI)
* A desktop environment on top of OS. This is just another application and not baked into the OS like Windows
* Examples of GUI Desktop managers:
** Gnome
** KDE
** XFCE
* These window managers are much better for multitasking and necessary if you want to use graphical applications
* Some useful hotkeys:
**~Alt-F2 - Run a command
**~CTRL-ALT-Arrows - Change virtual desktops
**~CTRL-ALT-BKSP - Restart the window manager (if enabled)
** Navigating the menus - Much like what you're used to on either Windows or Mac
** Mouse: Highlight to copy, middle button to paste
*** This is the standard Unix/Linux way to copy/paste instead of having to press a key to do it. In putty, highlighting text copies it to the clipboard and clicking the right mouse button will paste to the terminal
!! Command Line Interface (CLI)
* Can be accessed from within the GUI, eg: the terminal program
* Or Console, which is accessed when you're sitting down at the keyboard and monitor on a system and not running a graphical environment
* Or through a virtual console (~CTRL-ALT-F[2-9]). Unix systems run many virtual consoles which can be accessed to run other tasks.
* Or accessed remotely, such as via SSH. We will be accessing our class shell server remotely to complete our work.
* Unix/Linux is primarily a Multi-user environment. Many users can easily log in concurrently and work simultaneously.
** About accounts:
*** Home directory - Every user has a home directory where they can store files
*** User ID - The ID number assigned to your account. It's these numbers which identify you as a user. Names, like user names or host names, are for people. The machines use the numbers.
*** Group ID - Users may belong to groups for shared resources. Everyone in this class is a member of the ''ncs205'' group and can access this class's resources
*** Who am I? - List information about your user account: {{Command{id}}}
*** Who are you? - List information about other user's accounts: {{Command{id //username//}}}
*** Who is connected? - Show the users who are currently logged in: {{Command{w}}} or {{Command{who}}}
!! The Shell
The shell is our command processor that provides:
* An interpreter - it reads and interprets commands from the user,
** displays the shell prompt and waits for input
*** Case matters here! - The commands {{Command{id}}} and {{Command{ID}}} are two different commands. One will work and one does not exist. The same goes for any other file names.
** user interface for entering and processing commands from the user
** then works with the kernel to execute your commands
* A programming interface
** the shell is also a script interpreter for executing shell scripts
** a shell script is just a collection of commands you could execute in sequence on the command line
** This makes it much easier to automate or run a lot of commands at once
!! Different Shells
Different shells for different things: bourne, bash, csh, tcsh, korn
[>img[img/shell.jpg]]
The Shell is what users interact with on the command line. It receives and interprets commands.
[[Two main families|img/shells.jpg]] - bourne and ~C-Shell
Thompson shell, original unix shell, ends with AT&T 6th edition and replaced by the modern branches:
* Bourne Shell ({{Command{sh}}})
**written to replace limited abilities of original shell
**Oldest and most primitive
**Korn shell ({{Command{korn}}}) - Closed shell from Bell Labs
***Built to be a vast improvement over the bourne shell
***Adopted in future editions of AT&T Unix (8-10th editions)
***Became popular with commercial users as a higher end, more powerful shell, especially as a programming language
**Bash ({{Command{bash}}}) - FSF - ''B''ourne ''a''gain ''sh''ell
***Extends bourne shell while being free to distribute
***Free software, community supported, part of the GNU toolset.
*~C-Shell ({{Command{csh}}}) - Created by Bill Joy for the Berkeley Software Distribution (BSD) unix.
** Based on C Programming language. scripting commands are based on C statements
** BSD License, couldn't distribute freely
** TCSH ({{Command{tcsh}}}) -
*** Enhancement of the C-shell while being free from licenses
*** In public domain for academic users
* There's a few newer shells which are beginning to gain popularity
Which to use:
Shells are split into three camps: {{Command{bash}}} for Linux, {{Command{tcsh}}} in BSD branch, and {{Command{korn}}} for commercial distributions (IBM AIX and ~HP-UX)
What we'll be using this semester:
Interactive use: bash, since we're doing everything in Linux
Shell scripting: bourne for portability/compatibility or bash for extended features.
We can see available shells on a system with: {{Command{cat /etc/shells}}}
The shell is just a regular program, so anyone can design their own shell. You can also execute it by its command name to run a different shell.
!! Working with the shell
* Commands are entered at the shell prompt
* They have a standard syntax: {{Monospaced{''command_name'' [options] [arguments]}}}
** Command - what action to take
** Options - modify how the action is applied or how the command does its job
** Arguments - Provide additional info to the command, such as object identifiers, text strings, or file names
** Some options can have their own arguments (option arguments) to provide additional information for that option
*** For example, {{Command{mkdir -m 755 ~/open/}}} to create the directory named {{File{open}}} within your home directory with different starting permissions. Here, the {{Monospaced{755}}} is an argument to the {{Monospaced{-m}}} option.
** The components in a command string ''must'' always be separated by some form of whitespace
*** The command string {{Command{ls -l /tmp/}}} is correct where all three options are properly separated by whitespace. Whereas the command {{Command{ls-l/tmp/}}} is an invalid command that does not exist on the system. Notice the lack of whitespace in the second example.
** In documentation, brackets around a component show that component is optional and not required by the command. Consider these two examples:
*** {{Monospaced{grep [OPTIONS] PATTERN [FILE...]}}} - only the //PATTERN// argument is required.
*** {{Monospaced{ls [OPTION]... [FILE]...}}} - the command may be executed without specifying any options or arguments
**** The ellipsis (three dots) denote additional items can be added. For example, multiple files can be specified for the {{Command{grep}}} or {{Command{ls}}} commands.
** Example commands: {{Command{ls}}}, {{Command{date}}}, {{Command{cal}}}, {{Command{who}}}
* Many commands have default options or arguments
** {{Command{date}}} - by default, show the current date and time. Different options can be specified to alter the format the date is displayed in
** {{Command{cal}}} - by default, show a calendar for the current month. {{Command{cal 2024}}} will display the entire year
** {{Command{cd}}} - by default, change directory to the user's home directory. Specifying an argument will change to the specified directory instead.
* Combining options
** several options can be combined together, for example: {{Command{ ls -lrt }}} to display the contents of the current directory in long listing format, sorted by modification date, with the most recently accessed files at the bottom. There's three different options combined here.
* {{Monospaced{ - }}} vs {{Monospaced{ --word }}} options (eg: the {{Command{cut}}} command)
** Some options can be specified with a single dash and letter, eg {{Monospaced{ -d }}}
** Or with two dashes and a word, eg {{Monospaced{ --delimiter }}}
* Autocompletion - Enter the first few characters of a file or command and the shell will complete the rest. Press the tab key to assist with autocompletion
** This video contains example: https://www.youtube.com/watch?v=5mOHSBFuSy4
* Scrolling through previously executed commands:
** The keyboard up/down arrows can be used to cycle through previous commands
** Page-up can search more efficiently: type the first few letters of a previously executed command and press page-up to return to the last command which began with those characters. Continue pressing page-up to scroll through the list.
* The ~CTRL-C combo will generally cancel a running command
* You can also group multiple commands together with a {{Monospace{'';''}}}
** For example: {{Command{date ; cal}}} will run these two commands together
! Finding more information / UNIX Documentation
[>img[https://imgs.xkcd.com/comics/rtfm.png][https://xkcd.com/293/]]
* The unix manual / man pages are a great resource. Usage: {{Command{man [section] //command//}}}
** Example: {{Command{ man ls }}}
** Navigation:
*** move up and down the pages with space, {{Monospace{f}}}, and {{Monospace{b}}}
*** search down with {{Monospace{/}}}, up with {{Monospace{?}}}, or use {{Monospace{n}}} for the next match
*** {{Monospace{g}}} moves to the top of the page and {{Monospace{G}}} to the bottom
*** {{Monospace{q}}} to quit the manual
*** {{Monospace{h}}} for help with using the manual
* Man page chapters:
** Synopsis - Overview of the command, listing options and requirements. Optional items are contained within {{Monospace{''[ brackets ]''}}}
** Description - Description of actions performed by the command and detail information about all of the supported options
** Examples - Examples of common usage
**See Also - Other man pages to read for related information
**Check manpages for look, chmod, cut
*{{Command{man -k //keyword//}}} - search the unix manual
*Manual sections:
**1 - Commands
**2 - System Calls
**3 - Library Functions
**4 - Special Files
**5 - File Formats
**6 - Games
**7 - Misc Info
**8 - System administration
{{Note{''Note:'' When working with Unix documentation, items in ''[''brackets'']'' are optional.}}}
! Assignment
!! Read Chapter 1 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
- Complete [[Lab 1|labs/lab1.pdf]]
''Warning:'' Do not complete these lab assignments ~PDFs within your web browser. Download the files and open them in [[Acrobat Reader|https://get.adobe.com/reader/]] or a similar PDF document reader. Web browsers to not save form input appropriately and your responses will likely be lost.
{{Warning{It's wise to preview the completed PDF document in Acrobat Reader to verify your responses before uploading to the class shell server.}}}
Be sure to read the the instructions for submitting assignments and information about the labs in the [[Lab Assignments]] page. Assignments will only be accepted if they are properly submitted according to the instructions.
! Material
!! Read:
* Chapter 2 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
! UNIX Files
Navigating the filesystem is at the core of working with the Unix command line. Explore our shell server using the {{Command{cd}}}, {{Command{ls}}}, and {{Command{pwd}}} commands. Files for our class and your labs can be found within the {{File{/opt/pub/ncs205/}}} directory. Use the material introduced in this chapter to explore the filesystem on the shell server, especially the directories within {{File{/opt/pub/ncs205/}}}.
Everything on a Linux system is either a file or a process
* Defined: A file is anything that can be a source or destination of data.
!!File types: (ls will show the type)
* ''Ordinary files'' (also called regular files) - items such as text files, programs, and images
** {{Command{file}}} command - This command can be used to display a file's type
*** eg: {{Command{file /usr/bin/ls}}}
** {{Command{strings}}} command - This command can be used to extract ascii strings from a binary file
* ''Directory files'' - special files that contain lists of files and other directories. How files are organized on the system.
** Standard conventions: A file written with a trailing slash (eg: {{File{/opt/pub/ncs205/}}} refers to a directory.
** Filesystem Root - Highest level of the file system = {{File{ / }}}
*** The filesystem has a tree-like structure, starting at its root and branching out as you traverse into the sub-directories.
** Navigate directories with the {{Command{ cd }}} command
*** eg: {{Command{ cd /opt/pub/ncs205/ }}}
*** The {{Command{ cd }}} command's default argument will change to your home directory
**** Some commands, like {{Command{ cd}}}, have a default argument. The command will act on the default argument if you do not specify one.
*** {{Command{cd -}}} will change to the last directory you were in. Note the dash after the {{Command{ cd }}} command.
** Display file metadata (eg, permissions and date information) on a directory with {{Command{ls -ld}}}
*** Example: {{Command{ ls -ld /opt/pub/ncs205/ }}}
** Special directories contained within every directory on the system:
*** {{File{.}}} = Refers to the present working directory. (The directory you are currently in)
*** {{File{..}}} = Refers to the parent directory. See warning note below.
** Your home directory. A place for your files on the system. Referred to with the shortcut symbol ({{File{~}}} or the variable $HOME)
*** Either use it alone to refer to your home directory, eg: {{Command{ls ~}}}
*** Or with a username to refer to another user's home directory, eg: {{Command{ls ~//username//}}}
** Relative & absolute path names
*** Absolute path - a path that starts from the root of the filesystem. It will always begin with {{File{ / }}}, eg: {{File{/opt/pub/ncs205/submit/}}}
*** Relative path - a path that starts from your current directory, eg: {{File{ ncs205/submit}}} (notice the lack of a {{Monospaced{ / }}} at the beginning of this path).
** Working directory
*** The //current working directory// is the directory you are currently located in at the shell prompt.
*** The command {{Command{pwd}}} (print working directory) will display the full path of your current working directory to the screen. This helps keep track of where you are on the system.
** Obtain disk usage of a directory with the {{Command{du}}} command, eg: {{Command{ du -sh ~ }}}
* ''Symbolic Links'' - special files that are pointers to other files
* ''Hardware devices'' - storage medium (hard drive, DVD drive, flash drives, etc), display, network cards, etc.
** Character devices - device that reads or writes data one character at a time
** Block device - device that reads or writes data one block at a time
* ''FIFO'' (aka named pipe) - Used for interprocess communication
* ''Socket'' - Used for network communication
{{Warning{''Note:'' The special directory {{File{..}}} refers to the ''parent directory'', not //previous directory//. The word previous is ambiguous and could mean the last directory you were in. The last directory you were in could be anywhere on the filesystem. Referring to the special directory {{File{..}}} as //previous directory// will be considered incorrect. }}}
!! File and directory names
Good file naming practices:
* File names can be any sequence of ASCII characters up to 255 characters
* Start your file names with an alphabetic character
* Try to avoid spaces. Instead use dividers to separate parts of the name, such as {{Monospaced{ _ - : . }}}
* Use an extension that describes the file type
** For example, the file extension {{File{ homebackup_december.tgz }}} suggests this is a gzip-compressed tar archive.
* Files beginning with a dot are hidden from normal view
** These are typically configuration files, like {{File{.bash_profile}}} or {{File{.bashrc}}}
* Avoid special characters (shell metacharacters) in your file names, such as: {{Monospaced{ & * \ | [ ] { } < > ( ) # ? ' " / ; ^ ! ~ % ` }}}
** All of these symbols mean something special to the shell. Keep track of what they mean as they are introduced during the semester. [[Shell Metacharacter Table|handouts/ShellMetacharacterTable.pdf]]
** If special characters were used, escape them with a {{Monospaced{ \ }}}
*** Example: {{Command{vi commands\&examples.txt}}}
** Put quotes around your file name if spaces must be used
*** Example: {{Command{vi "Long File Name.txt"}}}
** We'll discuss escaping, quoting, and metacharacters in more detail later
!! Basic file manipulation
!!! Listing files - {{Command{ls}}}
The {{Command{ls}}} command will list the contents of a directory. Extra options can be used to alter the default behavior of the {{Command{ls}}} command:
* {{Command{-a}}} - This option will include hidden files in the output
* {{Command{-l}}} - This option will display the output in //long listing// format. Additional information about the files will be displayed. Display your files with a long listing is preferred so you a presented with the additional detail.
! Assignment
!! Read Chapter 2 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
- Complete [[Lab 2|labs/lab2.pdf]]
! Version Control with git
!! Online videos
At least watch these two:
* Fast Intro: https://www.youtube.com/watch?v=hwP7WQkmECE
* A little more detailed intro: https://www.youtube.com/watch?v=USjZcfj8yxE
''Note:'' We have git already installed on the class shell server. There is no need to install it on your system.
Feel free to create a scratch space in your home directory if you would like to work along with the video.
----
Optional, if you'd like even more detail
* Part 1: https://www.youtube.com/watch?v=hrTQipWp6co
* Part 2: https://www.youtube.com/watch?v=1ibmWyt8hfw
** When working with our ~GitHub clone, don't mess with the usernames, ~URLs, tokens, or SSH keys like the video is suggesting. The documented steps below will work for you.
* Part 3: https://www.youtube.com/watch?v=Q1kHG842HoI
''Two additional notes about these three videos'':
# You don't have to create a ~GitHub account. We will be working with our own ~GitHub clone that will mirror a lot of its functionality.
## If you'd like to create an online repository and follow along with the video, scroll down to the //Setting up your Git repository// section below for how to access your account on our clone. Just use a different name for your demo repo.
# He uses a GUI IDE tool for code development where we use the command line. These videos are more to explain the concepts. The directions below will explain the methods for how to apply these concepts to the command line.
----
!! Introduction to Version Control
Version control is a system that allows you to manage changes to files and code over time. It's an essential tool for anyone who works with code or digital content, as it provides a way to track changes, collaborate with others, and restore previous versions of your work. With version control, you can work on a project with others without worrying about conflicting changes, easily revert to a previous version of your work, and keep track of who made changes and when.
While it's primary use may be for programming projects, it's also useful for any application that involves tracking the evolution of changes to plain text files, such as:
* My resume and lab manuals for other classes are written in ~LaTeX, a plain text markup language, which can then be tracked in a version control repo
* I use the Markdown language for some technical documentation. This gives basic formatting while keeping the documents plain text
* Our class website is a single HTML file that is checked in to a repo as changes are made. The comment posted to the repo is also sent as a notification to Discord.
* Many of my server configuration files are tracked in repos
* Infrastructure as Code - text files can be used as the basis for creating and configuring server and network infrastructure
* Diagram as code - python can be used for generating diagrams: https://diagrams.mingrammer.com/
!! Introduction to Git
Git is a free and open-source version control system that is widely used in software development. It was created by Linus Torvalds, the creator of the Linux kernel, and designed to handle projects of all sizes, from small personal projects to large-scale enterprise applications.
Git allows developers to track changes made to code and collaborate on projects with other developers. It does this by creating a repository, which is a directory that contains all files and version history of a project. Developers can make changes to the code, save those changes in a commit, and then push those changes to a remote central repository. Git also allows developers to create branches, which are separate lines of development that can be merged back into the main codebase.
Benefits of a central version control system are:
* ''Collaboration'': Git makes it easy for developers to work collaboratively on a project. Multiple developers can work on the same codebase at the same time, making changes to the same files without interfering with each other's work. Git keeps track of all changes made to the codebase, making it easy to merge changes and resolve conflicts.
* @@''Version Control''@@: Git allows developers to keep track of all changes made to the codebase over time. Each commit in Git represents a snapshot of the code at a particular point in time. This makes it easy to roll back to a previous version of the code if necessary.
* @@''Backup''@@: Git provides a backup system for codebases. All changes made to the codebase are saved in a repository, even if they are later undone. This means that developers can always go back to a previous version of the code if necessary and easily compare different versions of code.
* ''Experimentation'': Git makes it easy to experiment with different ideas without affecting the main codebase. Developers can create branches to work on new features or ideas, and merge them back into the main codebase if they are successful.
* ''Flexibility'': Git can be used for any type of project, regardless of its size or complexity, on any operating system. It is a flexible tool that can be adapted to suit the needs of any project.
We will focus on the concepts above highlighted @@in yellow@@
!!! Basic Git terminology
* @@''Repository''@@ - A repository is a directory where Git stores all files and version history of a project.
* @@''Commit''@@ - A commit is a saved snapshot of the changes made to a file or files in the repository.
* ''Branch'' - A branch is a separate line of development that allows multiple developers to work on the same project without interfering with each other's work.
* ''Merge'' - Merging is the process of combining two branches to create a single branch with the changes from both branches.
* @@''Remote''@@ - A remote is a Git repository that is hosted on a remote server, such as ~GitHub or ~GitLab.
* @@''Clone''@@ - Cloning is the process of copying a repository from a remote server to a local machine.
* @@''Push''@@ - Pushing is the process of uploading changes made to a local repository to a remote repository.
* ''Pull'' - Pulling is the process of downloading changes made to a remote repository to a local repository.
* ''Fork'' - Forking is the process of creating a copy of a repository on a remote server, which can be used for experimentation or collaboration.
* ''Checkout'' - Checking out is the process of switching between different branches or commits in a repository.
We will focus on the terms above highlighted @@in yellow@@
!!! Basic Git Commands
* {{Command{git add //file(s)//}}} - Add a file to a repository
* {{Command{git rm //file(s)//}}} - Remove a file from a repository
* {{Command{git mv //oldfile// //newfile//}}} - Rename a file in a repository
* {{Command{git commit //file(s)//}}} - Record any changes to the repository
* {{Command{git status}}} - Display the status and information about the working tree
* {{Command{git log}}} - Display the commit logs
* {{Command{git remote}}} - Set the location of the remote repository
* {{Command{git push}}} - Send committed changes to the remote repository
* {{Command{git diff}}} - Compare changes between commits as well as between the current working version and committed version of a file.
Linux manpages are available for each of these {{Command{git}}} sub-commands. Put a dash between the main command {{Command{git}}} and the sub-command you want to learn more about. For example, {{Command{man git-add}}}
!!! ~GitHub
Git and ~GitHub are related but different tools.
''Git'' is the command-line tool that allows developers to create a repository, make changes to code, and save those changes in commits. Git is designed to work locally on a developer's machine, but it can also be used to collaborate with other developers using remote repositories.
''~GitHub'' is a web-based platform that provides central hosting for Git repositories. It allows developers to create and host remote repositories, collaborate with other developers, and manage projects. ~GitHub provides a user-friendly interface that makes it easy to manage Git repositories.
~GitHub provides developers with additional features, including:
* Cloud storage for your repository
* Web-based interface for managing repositories and projects
* Collaboration features, including pull requests, code review, and issue tracking
* Documentation features, such as a project wiki
* Integration with other tools, including CI/CD pipelines, text editors, and project management tools
* Social features, including profiles, followers, and open-source contributions
We will work with a local, self-hosted clone of ~GitHub called //Gogs//. Everyone will have an account on our local clone where you can create your own repositories.
!! Setting up your Git repository
* Log into our Gogs instance at https://git.ncs205.net:3000/user/login
** Use your campus username
** Your initial password can be found in the file {{File{~///username//.git.txt}}}
*** //username// is your campus username
** You may change your password after logging in. Be sure to record it for future use.
* Create a new repository from within Gogs. Click the ''+'' within the orange box
[img[img/gogs01.png]]
* Call your new repository //''scripts''//, configured as follows. Add a description if you'd like.
[img[img/gogs02.png]]
* You should now have an empty repository in our online Git server. We must now clone it on our class shell server and add our script files.
{{Warning{''Note:'' If this was a real project, we would most likely clone with SSH and utilize an SSH keypair for authentication. This is a bit more work to set up, but is more secure and easier to use going forward. We will instead choose the HTTPS method now and sacrifice some security for ease of use while we're learning new git concepts. Just be aware some of the authentication choices we are making now are only to allow us to focus more on git and not get distracted with more advanced SSH concepts. Our authentication choices here are not an endorsement of these methods for future projects. Use SSH keypairs instead for those.}}}
> Change to your {{File{bin}}} directory: {{Command{ cd ~/bin/ }}}
> A README file is a place to document your repository. Content is recorded in [[Markdown|https://www.markdownguide.org/basic-syntax/]]. Create an empty file: {{Command{ touch README.md }}}
> Initialize the new repository: {{Command{ git init }}}
> Add the README to the repo: {{Command{ git add README.md }}}
> Add your current Lab 35 scripts to the repo: {{Command{ git add ncs205-lab35-q0?.sh }}}
> Display the status of changes awaiting commit. You should see the README.md and your three script files listed as //new file//: {{Command{ git status }}}
> Commit your changes with the message //first commit//: {{Command{ git commit -m "first commit" }}}
> Set the remote location to push your changes to. ''Be sure to change my username to yours.'' {{Command{ git remote add origin https://git.ncs205.net:3000/merantn/scripts.git }}}
> Configure git to cache your credentials in memory:
> {{Command{ git config --global credential.helper cache }}}
> {{Command{ git config --global credential.helper 'cache --timeout=3600' }}}
> ''Optional'' - If you prefer {{Command{nano}}} instead of {{Command{vi}}}, set git to use nano as the default editor: {{Command{ git config --global core.editor nano }}}
> For simplicity while we're learning, we'll do all of our work out of the master branch: {{Command{ git config --global push.default current }}}
> Send your changes to the remote repo: {{Command{ git push }}}
Here is my run through these steps:
{{Commands{
[merantn@lab ~/bin]$ ''git init''
Initialized empty Git repository in /home/merantn/bin/.git/
[merantn@lab ~/bin]$ ''git add README.md''
[merantn@lab ~/bin]$ ''git add ncs205-lab35-q0?.sh''
[merantn@lab ~/bin]$ ''git status''
# On branch master
#
# Initial commit
#
# Changes to be committed:
# (use "git rm --cached <file>..." to unstage)
#
# new file: README.md
# new file: ncs205-lab35-q01.sh
# new file: ncs205-lab35-q02.sh
# new file: ncs205-lab35-q03.sh
#
[merantn@lab ~/bin]$ ''git commit -m "first commit"''
[master (root-commit) 021c95d] first commit
Committer: Nick Merante <redacted>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:
git config --global user.name "Your Name"
git config --global user.email you@example.com
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
4 files changed, 3 insertions(+)
create mode 100644 README.md
create mode 100755 ncs205-lab35-q01.sh
create mode 100755 ncs205-lab35-q02.sh
create mode 100755 ncs205-lab35-q03.sh
[merantn@lab ~/bin]$ ''git remote add origin https://git.ncs205.net:3000/merantn/scripts.git''
[merantn@lab ~/bin]$ ''git config --global credential.helper cache''
[merantn@lab ~/bin]$ ''git config --global credential.helper 'cache --timeout=3600'''
[merantn@lab ~/bin]$ ''git config --global push.default current''
[merantn@lab ~/bin]$ ''git push''
Counting objects: 6, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), 416 bytes | 0 bytes/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To https://git.ncs205.net:3000/merantn/scripts.git
* [new branch] HEAD -> master
}}}
You can now view your repository online and see the files you've just checked in:
[img[img/gogs03.png]]
!!! Committing changes to the repository
New scripts can be added to the repository with the {{Command{git add}}} command above and changes to existing files can be committed with {{Command{git commit}}}. Adding files and checking in changes can serve as a checkpoint and backup to roll back to in case things go awry. It's wise to periodically check in changes at important milestones or before large adjustments are to be made so you have a good checkpoint to roll back to in case things don't work out.
Here is an example of a change being made to an existing file. I forgot to add the shebang to the top of one of my scripts. I will first make that change, and then check the new file into the repository:
1. Update the script. It now contains the missing shebang:
{{Commands{
[merantn@lab ~/bin]$ ''head ncs205-lab35-q03.sh''
#!/bin/sh
# File name: ncs205-lab35-q03.sh
# Author: Nick Merante
# Date Written: Mar 4, 2023
# Assignment: Lab 35
# Purpose: Calculate product of two integers
# Description:
# - Prompt user for input of two integers
# - Calculate and display product
#
}}}
2. We can compare my new version with the version currently checked into the repo:
* Lines beginning with a {{Monospaced{''+''}}} represent additions
* Lines beginning with a {{Monospaced{''-''}}} represent deletions.
Notice the {{Monospaced{''+''}}} before the new shebang I just added:
{{{
[merantn@lab ~/bin]$ git diff ncs205-lab35-q03.sh
diff --git a/ncs205-lab35-q03.sh b/ncs205-lab35-q03.sh
index 806d2a4..bc66030 100755
--- a/ncs205-lab35-q03.sh
+++ b/ncs205-lab35-q03.sh
@@ -1,3 +1,4 @@
+#!/bin/sh
# File name: ncs205-lab35-q03.sh
# Author: Nick Merante
# Date Written: Mar 4, 2023
}}}
3. Commit your change by executing one of these methods:
* Commiting just the changed files you specify: {{Command{ git commit ncs205-lab35-q03.sh }}}
* Committing all changes: {{Command{ git commit -a }}}
4. A text editor will open for you to add a comment to your commit. This is a place to describe the changes you're making. Comments help track what changes were made and why they were made.
* Add your comment
{{{
Adding missing shebang
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Committer: Nick Merante <merantn@lab.ncs205.net>
#
# On branch master
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: ncs205-lab35-q03.sh
#
}}}
* Save and quit the editor to finalize your commit
{{{
[master 7690513] Adding missing shebang
Committer: Nick Merante <merantn@lab.ncs205.net>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:
git config --global user.name "Your Name"
git config --global user.email you@example.com
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
1 file changed, 1 insertion(+)
}}}
5. Push your changes to the online repository with {{Command{ git push }}}
{{Commands{
[merantn@lab ~/bin]$ ''git push''
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 301 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To https://git.ncs205.net:3000/merantn/scripts.git
14f729c..7690513 HEAD -> master
}}}
!!! Reverting a failed change to a file in the working directory
Let's suppose I make a change to one of my files that doesn't work out. I haven't commited that change yet, and I want to revert my working copy to the last checked-in version.
1. Make your change
For simple demonstration purposes, I added a new line to the description (highlighted yellow below). A more common use of this would be to undo several failed changes throughout a piece of code:
{{Commands{
[merantn@lab ~/bin]$ ''head ncs205-lab35-q03.sh''
#!/bin/sh
# File name: ncs205-lab35-q03.sh
# Author: Nick Merante
# Date Written: Mar 4, 2023
# Assignment: Lab 35
# Purpose: Calculate product of two integers
# Description:
# - Prompt user for input of two integers
# - Calculate and display product
# @@- some major goof I want to undo@@
}}}
{{{
[merantn@lab ~/bin]$ git diff ncs205-lab35-q03.sh
diff --git a/ncs205-lab35-q03.sh b/ncs205-lab35-q03.sh
index bc66030..3ffd4a5 100755
--- a/ncs205-lab35-q03.sh
+++ b/ncs205-lab35-q03.sh
@@ -7,4 +7,4 @@
# Description:
# - Prompt user for input of two integers
# - Calculate and display product
-#
+# - some major goof I want to undo
}}}
2. Undo that change, reverting to the last checked-in version. Notice my addition to the Description is now missing and there are no differences between the local and checked-in copies:
{{Commands{
[merantn@lab ~/bin]$ ''git checkout -- ncs205-lab35-q03.sh''
[merantn@lab ~/bin]$ ''head ncs205-lab35-q03.sh''
#!/bin/sh
# File name: ncs205-lab35-q03.sh
# Author: Nick Merante
# Date Written: Mar 4, 2023
# Assignment: Lab 35
# Purpose: Calculate product of two integers
# Description:
# - Prompt user for input of two integers
# - Calculate and display product
#
[merantn@lab ~/bin]$ ''git diff ncs205-lab35-q03.sh''
[merantn@lab ~/bin]$
}}}
!!! Comparing differences between previous commits:
Differences in a file between two previous commits can be compared.
1. First display the commit log for your file to obtain the commit ID, highlighted in yellow below
{{Commands{
[merantn@lab ~/bin]$ ''git log ncs205-lab35-q03.sh''
commit @@7690513d3642de5a83342dd16c85dcf506cdf95e@@
Author: Nick Merante <merantn@lab.ncs205.net>
Date: Sun Mar 5 15:00:23 2023 -0500
Adding missing shebang
commit @@14f729c85b6ea4156db49ccf69583f7822a951c5@@
Author: Nick Merante <merantn@lab.ncs205.net>
Date: Sun Mar 5 14:44:25 2023 -0500
add header
commit @@d7105d4ae4f04807b1fa9f30171677a396a26de8@@
Author: Nick Merante <merantn@lab.ncs205.net>
Date: Sun Mar 5 14:35:53 2023 -0500
first commit
}}}
2. Use the commit messages to identify the points in time you would like to compare, providing the two commit ~IDs and the filename to the next command. Below, I can see the line that was removed and the lines which were then added.
{{{
[merantn@lab ~/bin]$ git diff d7105d4ae4f04807b1fa9f30171677a396a26de8 14f729c85b6ea4156db49ccf69583f7822a951c5 ncs205-lab35-q03.sh
diff --git a/ncs205-lab35-q03.sh b/ncs205-lab35-q03.sh
index 340c263..806d2a4 100755
--- a/ncs205-lab35-q03.sh
+++ b/ncs205-lab35-q03.sh
@@ -1 +1,9 @@
-echo fake demo script 3
+# File name: ncs205-lab35-q03.sh
+# Author: Nick Merante
+# Date Written: Mar 4, 2023
+# Assignment: Lab 35
+# Purpose: Calculate product of two integers
+# Description:
+# - Prompt user for input of two integers
+# - Calculate and display product
+#
}}}
!!! README files
The file {{File{README.md}}} can contain documentation for your project in [[Markdown format|https://www.markdownguide.org/basic-syntax/]]. This content will also be displayed in the git web UI for the files in that directory.
1. Edit the {{File{README.md}}} file and add a synopsis for each of your scripts, similar to my example below. Notice the use of [[Markdown|https://www.markdownguide.org/basic-syntax/]] in the document.
{{{
[merantn@lab ~/bin]$ cat README.md
# NCS205 Scripts
## Lab 35
1. `ncs205-lab35-q01.sh` - Turn up to 9 words into uppercase
2. `ncs205-lab35-q02.sh` - Accept two integers as arguments and compute their product
3. `ncs205-lab35-q03.sh` - Prompt for two integers and compute their product
}}}
2. Commit your new {{File{README.md}}} file
3. Push the changes to the repository
4. Observe the new Readme in the web UI:
[img[img/gogs04.png]]
! Assignment
1. Be sure [[Lab 35|labs/lab35.pdf]] has been completed
2. Get familiar with {{Command{git}}} and our online repository
3. Check in your ''Lab 35'' scripts and a proper {{Monospace{''README''}}} file following the instructions above
4. Experiment with committing and rolling back changes. We will use this more with future material
Also Complete [[Lab 36|labs/lab36.pdf]] after working with your repo.
! Material
!! Lab & VM notebook:
* Start keeping good notes of what you are doing with your ~VMs.
** The software installed today should be included.
** These notes will come in handy later when you need to repeat these steps on future ~VMs
!! Read - //Mastering Ubuntu Server//
* Chapter 3 - Managing Software Packages
** Our systems are running Ubuntu and will be using the {{Command{apt}}} package management commands.
* Chapter 21 - Securing Your Server
** Only the //Installing Security Updates// section starting on page 490 is needed now
* Chapter 7 - Controlling and Managing Processes
** Only the //Managing system processes// section starting on page 148 is needed here
! Notes
{{Note{As technology users, we should know by now how to submit usable problem reports.
If you have a problem, please send a report I can work with. I need details of the problem, what you tried, steps you took to diagnose it, documentation you reviewed, screenshots, logs, etc. If you send me something vague like "//X command doesn't work//" with no supporting details, there may not be much I can do for you and I will wait for you to follow up your message with meaningful information.
The level of assistance I can provide will be proportionate to your effort to troubleshoot and supply details. If you do nothing to troubleshoot and send me no information to work with, you should then expect that much effort put into a response.
}}}
!! Expanding our systems
!!! The {{Command{apt}}} package manager
Software package management is one of the customized components of a Linux distribution and differs between unix operating systems and linux distributions.
* Before software packages, all software was distributed as source code and needed to be compiled. This process could take hours to complete.
* Software packages are pre-compiled for the target OS and architecture. The heavy processing work needed to compile from source is done ahead of time.
The core components of a Linux distribution are:
* Linux kernel
* Base utilities (typically GNU tools)
** Many g* utilities are from the GNU project (eg: gcc)
** Stallman's GNU (GNU's not Unix) project from the early 80's. He wanted to create a totally free OS and started with the utilities.
** He came from the Free Software Foundation and [[a philosophy of freedom|http://audio-video.gnu.org/video/TEDxGE2014_Stallman05_LQ.webm]] (freedom (speech), not price (beer) ).
** Software should be free to run, change, copy and modify so users are the ones in control, free from corporate control so better software develops - GNU license
** GNU tools differ somewhat from early ~FreeBSD tools (sed is a good example)
* Package manager. Different Linux distributions adopt different package management tools.
Extra (optional) components:
* Specialized utilities (Like the tools that come with a distro like Kali)
* X server / Window manager
Each distribution combines these core components in different ways depending on their focus and goals.
Debian-based systems (including Ubuntu) use the {{Monospaced{deb}}} package format and {{Command{dpkg}}} package system with the {{Command{apt}}} package management utility.
Other package management systems exist for other distros
- {{Command{yum}}} (~RedHat Linux and derivatives)
- portage (Gentoo)
- ports (~FreeBSD)
- ~DistroWatch [[Package management cheat sheet|http://distrowatch.com/dwres.php?resource=package-management]]
{{Command{dpkg}}} - this is a very basic utility
* It will mainly just install, update, or remove single packages which were manually downloaded and installed
* You will need to acquire the {{Monospaced{.deb}}} package file yourself or have a direct URL for it
** A {{Monospaced{.deb}}} file is a collection of pre-compiled binaries, configuration files, and support files for an application compiled for the target architecture.
** For example, when I want to install Discord on my Ubuntu Desktop workstation, I have to download the Discord {{Monospaced{.deb}}} file from their website and manually install it with {{Command{dpkg}}}
* Conflicts and dependencies will need to be sorted out manually
* Package updates will not be tracked
{{Command{apt}}} - high-level utility for package management
* A package manager will interact with repositories (collections of package files) to obtain software
* it will take care of any conflicts
* it will install necessary dependencies
* records what is installed and any changes made to the system to facilitate updates, package removal, or audit
* Can check for any updates and update your system with a single command
There are different package repositories (repos)
* Repository configuration files are stored within {{File{/etc/apt/}}}
* A repository is the central distribution point for our linux packages
* Typically, each distro has its own repository on the internet for base packages
* The repository creator determines which applications it contains
* Repos are mirrored for accessibility and speed.
* Other repositories offer additional packages
** Some software is in its own repository and not tracked with the standard OS repos.
!!! Apt package manager commands:
{{Command{apt}}}
<<<
The primary command for the apt package manager. Run this by itself to see all sub-commands
<<<
{{Command{apt update}}}
<<<
Update your local repository index to include the latest information about available packages and software versions. No software updates are performed.
<<<
{{Command{apt install //package_name(s)//}}}
<<<
Install a package and all dependencies
<<<
{{Command{apt remove //package_name(s)//}}}
<<<
Remove a package
<<<
{{Command{apt search //keyword//}}}
<<<
Display a list of packages which match the given keyword
<<<
{{Command{apt-cache show //package_name//}}}
<<<
Display information about a specified package
<<<
{{Command{apt list --upgradable}}}
<<<
Check the repositories for any available updates and display the results, without applying an updates.
<<<
{{Command{apt dist-upgrade}}}
<<<
Check the repositories for any available updates. After reviewing the results, the user will be prompted to apply them.
''A {{Command{reboot}}} will be required if a kernel update is included in the list. Otherwise, only updated services may need to be restarted for the updates to take effect.''
<<<
{{Command{ apt autoremove}}}
<<<
Remove old dependency packages which are no longer required
<<<
{{Command{ apt clean all}}}
<<<
Remove cached packages after install is completed.
<<<
We can install most required software using packages with {{Command{apt}}}:
* Keep a record of what is installed as we go.
* Get started with:
** Install on all systems: {{Monospaced{ man wget telnet rsync openssh-client bzip2 ncat }}}
!!! Other software distribution methods
Some software packages are also distributed with all dependencies self-contained and may run in their own sandbox. This simplifies software distribution and helps ensure there are no conflicts with system packages. These are mostly useful for Desktop-based distributions with a graphical UI.
* The snap system in Ubuntu is an example. Snap also runs its applications in an isolated sandbox to restrict access to other areas of the system.
* The [[AppImage format|https://appimage.org/]] is another type. These software packages are distributed by the author as a single file. Simply download it, make the file executable, and run it. The [[BitWarden Linux client|https://bitwarden.com/download/#downloads-desktop]] is a good example.
!! Install web server software
The following tasks must now be completed to bring your web server online. Refer to the notes above and in previous notes to identify the proper commands to achieve these goals.
Complete these tasks on your web server VM:
# Install the following packages: apache2 php
# Set the ''apache2'' service to start on system startup
# Start the ''apache2'' service now
!! Verify the service with the {{Command{telnet}}} & {{Command{curl}}} commands
The {{Command{telnet}}}, {{Command{curl}}}, and {{Command{nc}}} commands are excellent tools for verifying that you're able to communicate with a host or a service. These are great for troubleshooting and everyone should know how to use all three.
Here I'm using telnet to connect to my web server on localhost. Run the telnet command to make a TCP connection and then begin speaking HTTP to the server. The HTTP command {{Command{GET /}}} will return the website. A lot of HTML will be returned, so I only have the first couple lines in the sample output below.
These connection tests can be run from any system which can communicate with the web service. I'm using my //test// VM for the validation demo.
{{{
root@test:/home/lab# telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
(truncated)
}}}
The {{Command{curl}}} command is another great tool for verifying TCP services and is generally available on every unix system. We'll add the {{Monospaced{-v}}} flag here for additional verbosity that's helpful for troubleshooting. Again, I'm truncating the output.
{{{
root@test:/home/lab# curl -v http://localhost/ | head
* Host localhost:80 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 14 Nov 2024 06:00:34 GMT
< Server: Apache/2.4.58 (Ubuntu)
< Last-Modified: Thu, 14 Nov 2024 05:57:23 GMT
< ETag: "29af-626d91fcb9833"
< Accept-Ranges: bytes
< Content-Length: 10671
(truncated)
* Connection #0 to host localhost left intact
}}}
Once a successful connection has been made, view the apache log files to verify the connection.
- Apache logs are located in {{File{/var/log/apache2/}}}
A log entry for a successful connection will resemble the following. Note the ''200'' HTTP status code:
{{{
192.168.12.10 - - [05/Apr/2024:10:45:42 -0400] "GET / HTTP/1.1" 200 86 "-" "curl/7.29.0"
}}}
When we're starting out, you might instead see a ''403'' status code indicating an error:
{{{
127.0.0.1 - - [05/Apr/2024:10:59:26 -0400] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"
}}}
We can see why in the {{File{error_log}}} file:
{{{
[Wed Apr 05 10:59:26.052681 2023] [autoindex:error] [pid 1659] [client 127.0.0.1:44834] AH01276: Cannot serve directory /var/www/html/: No matching
DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive
}}}
The web server is missing the file for its default website, the {{File{index.html}}}, and cannot proceed. We will fix this error in the //Web Server// lab below.
Now try to connect to your web server from your test VM using {{Command{telnet}}} or {{Command{curl}}}. If you use telnet, don't forget to send the {{Command{GET /}}} command.
{{{
[root@test ~]# telnet 192.168.12.25 80
Trying 192.168.12.25...
Connected to 192.168.12.25.
Escape character is '^]'.
GET /
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>Apache HTTP Server Test Page powered by CentOS</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
( truncated )
}}}
Your web server is now online.
! Additional Material
[[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]] - Chapter 14 (Package Management)
!! Useful commands:
* {{Command{wget}}} - great tool for downloading files from a web or FTP server
* {{Command{tar}}} - Standard linux archive tool. Files are usually distributed or stored as tarballs (the Linux equivalent of Zip). This tool will create or extract them
* {{Command{telnet}}} - Useful tool for testing TCP ports
* {{Command{curl}}} - Useful tool for testing TCP ports or downloading content from the web
* {{Command{apachectl}}} - Tool to manage an Apache server. Good to know if exists, but we likely won't be using it.
! Assignment
<<tiddler [[Lab 52 - VM updates & software installation]]>>
<<tiddler [[Lab 53 - Web Server]]>>
! Material
!! Crypto & Securing Communication:
!!! Read:
* Mastering Ubuntu Server, Chapter 14
** Start with //Securing Apache with TLS// on page 293
** Stop at //Installing and configuring NGINX// on page 299
!!! Watch:
* Brief overview: https://www.youtube.com/watch?v=w0QbnxKRD0w
* Crypto overview: https://www.youtube.com/watch?v=AQDCe585Lnc
** The math behind asymmetric encryption: https://www.youtube.com/watch?v=YEBfamv-_do
* The TLS handshake: https://www.youtube.com/watch?v=cuR05y_2Gxc
!! Scheduled Tasks:
!!! Read:
* Mastering Ubuntu Server, Chapter 14
** Start with //Scheduling tasks with cron// on page 151
{{Warning{''Warning:'' The textbook is light on the details and only discusses cron from a user perspective. There is not much mentioned about the system scheduled tasks saved in the files within {{File{/etc/}}}. The notes below are more thorough.}}}
! Notes
Understanding the core concepts involved with securing network communication is important for a security practitioner. The advent of [[Let's Encrypt|https://letsencrypt.org/]] and the free SSL certificates they offer has made trusted encryption available to the masses. Prior to the Let's Encrypt project, a site operator had to pay a commercial certificate authority to issue an SSL certificate for their site. This added expense limited encryption to those with the time and budget to pay for it.
We're now going to cover some core encryption concepts while implementing secure communication for our web sites.
!! There are different data handling concepts for different purposes
* ''Encoding'' - Data is transformed from one form to another. Usually this is easily reversible and not secure.
** base64 encoding can be used to convert a binary file to text for transmission over email (a text-based medium)
*** The {{Command{base64}}} command will encode text or decode a base64 encoded string
** audio or video encoding, employing a codec (coder-decoder)
* ''Hashing'' - Data is converted into a fixed-size string (a hash) using a non-reversable hash function. The length of the hash is always the same, regardless of the amount of input data.
** Algorithms have evolved over time. Current available algorithms are md5, sha-256, and sha-512.
** Hashes are also used to secure passwords. A hash of your password is stored on the system. When you log in, the password you enter is hashed and compared to the hash stored on the system.
*** Salt: Extra data that is added to a password to ensure the same passwords do not have the same hash. This significantly slows down brute force attacks against a password file containing user hashes.
*** Generate a password hash with perl: {{Command{ perl -e 'print crypt("PlaintextPassword","\$6\$hash-salt\$") . "\n"' }}}
**** {{Monospaced{//PlaintextPassword//}}} is the password you would like to hash
**** {{Monospaced{//hash-salt//}}} is the the salt to use. Salts should be valid random characters and ideally unique to each stored password.
** Hashes are also used for integrity validation. A hash of a file can be saved. If the file changes, its hash will no longer match the stored copy.
*** Commands to generate a hash of a file: {{Command{md5sum}}}, {{Command{sha256sum}}}, and {{Command{sha512sum}}}.
*** Hashes of your labs are stored in {{File{/opt/pub/ncs205/submit/checksums}}} when they are collected to ensure changes are not made after they are graded. If a file is modified, the hash will no longer match.
* ''Encryption'' - Use of an encryption algorithm and a secret key (cipher) to transform a private message (plain-text) into encrypted data (cipher-text). Only those possessing the secret key can view the original message.
** Encryption is reversible only with the encryption key
!! Main Encryption Goals:
* Confidentiality - Prevent disclosure to unauthorized parties
* Integrity - Prevent tampering of transmitted data
* Authenticity - Ensure communication is genuine and only with the intended target
!! Encryption basics
There are two different types of encryption algorithms - symmetric and asymmetric. It is important to understand the differences between them and where each is appropriate.
!!! Symmetric cryptography:
* The same key is used for both encryption and decryption of the message
* This is also known as a shared secret
** {{Command{openssl aes-256-cbc -a -salt -in secretfile.txt}}}
** Secure file transfer with netcat and openssl
*** receiver# {{Command{ nc -l 4444 | openssl aes-256-cfb -salt -d | tar -xvf - }}}
*** sender# {{Command{ tar -cvf - file | openssl aes-256-cfb -salt -e | nc client 4444 }}}
* Pros:
** Fast
** Not resource intensive
** Useful for small and large messages
* Cons:
** Key exchange must occur over a secure channel
** How can you exchange crypto keys over a secure channel that doesn't yet exist because you haven't exchanged keys yet? Another chicken and egg problem.
!!! Asymmetric:
* Public key cryptography
* There are now two keys instead of one shared secret
** public key - available for everyone. This key can be published for others to use.
** private key - kept secret and secure. Typically locked with a passphrase.
* Data encrypted with one key can be viewed or verified by the other
** These keypairs can be used for either encrypting or signing messages
*** Encryption: Data is encrypted with the public key and decrypted with the private key. This secures a message.
*** Message Signing: Data is encrypted (signed) with the private key and decrypted with the public key. This authenticates a message.
* Pros:
** Encryption allows safe key distribution over an insecure channel
* Cons:
** Slow
** More resource intensive
** Only useful for small messages
!!! Symmetric / Asymmetric Hybrid
* Use asymmetric encryption only to transmit a symmetric key.
* Then use symmetric encryption for the actual message.
* The best of both algorithm types:
** Can safely exchange key data
** Fast
** Not resource intensive
** Useful for small and large messages
!! Encryption Uses:
PGP / GPG
* Encrypt or sign files and messages
* Command line tools available: {{Command{gpg}}}
SSH
* use keys for authentication instead of passwords
** Use {{Command{ssh-keygen}}} to generate keypairs
* SSH also uses host keys for encrypting communication between the client and server
!!! SSL Certificates
* A SSL certificate contains information about the owner of the certificate and their public key
* Signed by a Certificate Authority (CA) to establish trust
** The Certificate Authority is //supposed// to take some steps to verify the site owner actually owns the site and they're only issuing certificates to the owners.
** Typically a commercial company like Godaddy, Verisign, or Entrust
** Or nonprofit CA to issue free certs: https://letsencrypt.org/
** ~Self-signed certificates can be created without using an external entity, but they won't be trusted by default
* The CA's signature is added to the certificate to establish trust
** If you view a site's certificate in your web browser, you can see the chain of trust from the Certificate Authority to the site certificate.
* Certificate Verification occurs by matching the host name in the certificate to the host name in the network communication
** Host name in the URL must match the host name in the certificate (single host)
** Or use a wildcard certificate for many sites (*.sunyit.edu)
!!!! SSL Trust
CA certificate stores
* Hardcoded list of trusted root CA certificates in either the application or operating system
** Stored within {{File{/etc/ssl/certs/ca-certificates.crt}}} in Ubuntu or {{File{/etc/pki/tls/certs/ca-bundle.crt}}} on ~RedHat derivatives.
** {{Command{grep Issuer /etc/pki/tls/certs/ca-bundle.crt | less}}} to see who's CA certificates the OS is trusting
* Intermediate CA resellers
** These resellers are trusted by a root CA certificate to issue certs on their behalf
* Transitive trust
** If A trusts B and B trusts C, thus A trusts C
* We now have a web of trust instead of a direct chain
** There are so many trusted certificate authorities, a weak link in any one of them completely destroys the ability to truly trust any of them.
* Certificate Authority weaknesses
** Several Breaches at CA Intermediaries. This sort of thing seems to happen a lot.
***[[DigiNotar (2011)|https://security.googleblog.com/2011/08/update-on-attempted-man-in-middle.html]] - Issued a wildcard certificate for google. About 500 other fake certificates were issued.
** Bad actors
*** Man in the middle proxies
**** [[Gogo Serving Fake SSL Certificates to Block Streaming Sites|http://www.pcmag.com/article2/0,2817,2474664,00.asp]]
**** [[SSL/TLS Interception Proxies and Transitive Trust|http://www.secureworks.com/cyber-threat-intelligence/threats/transitive-trust/]]
*** Malware
**** [[Lenovo's Superfish|http://www.slate.com/articles/technology/bitwise/2015/02/lenovo_superfish_scandal_why_it_s_one_of_the_worst_consumer_computing_screw.html]]
*** [[Microsoft Blacklists Fake Finnish Certificate|http://yro.slashdot.org/story/15/03/18/2048244/microsoft-blacklists-fake-finnish-certificate]]
*** [[Chinese CA issues Google certificates|https://googleonlinesecurity.blogspot.com/2015/03/maintaining-digital-certificate-security.html]]
*** [[Thawte issues certs for domains it doesn't own|http://www.itworld.com/article/2999145/security/google-threatens-action-against-symantec-issued-certificates-following-botched-investigation.html]]
*** [[Researchers find, analyze forged SSL|http://www.net-security.org/secworld.php?id=16843]]
** Certificate revocation problems
*** The mechanisms in place to get the word out that an issued certificate cannot be trusted aren't very robust.
* Leaks or vulnerabilities - Many ways to attack the infrastructure
** [[Heartbleed|https://en.wikipedia.org/wiki/Heartbleed]] - Broke encryption for almost the entire Internet. For two years anyone could obtain a server's encryption key.
** [[Poodle|https://www.openssl.org/~bodo/ssl-poodle.pdf]]
** Weak ciphers and downgrade attacks
** There are too many other examples
!!!! Countermeasures for the weaknesses in the trust system:
* Public key pinning
** Lock a certificate to a specific CA
** http://thenextweb.com/apps/2014/09/02/firefox-32-arrives-new-http-cache-public-key-pinning-support-easy-language-switching-android/
** https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
** https://raymii.org/s/articles/HTTP_Public_Key_Pinning_Extension_HPKP.html
* ~DNS-based Authentication of Named Entities (DANE)
** Remove the ~CAs from the process
** Use DNS to authenticate certificates much like SSH fingerprint records
** Add a ''tlsa'' resource record to your DNS zone
** Move from web of trust to chain of trust like DNSSEC
** Since DNS is totally open, if something is compromised it should be detectable
** Easy to revoke certificates
** https://www.huque.com/bin/gen_tlsa
** [[DerbyCon Video|http://www.irongeek.com/i.php?page=videos/derbycon4/t404-dns-based-authentication-of-named-entities-dane-can-we-fix-our-broken-ca-model-tony-cargile]]
* Online Certificate Status Protocol - Obtain revocation status of a certificate
* Hardcoded blocklists in web browsers
!! Web Encryption
!!! Background
HTTP messages are encapsulated with TLS
- TLS = Transport Layer Security
- Replacement for SSL protocol
This is an encryption layer on top of HTTP
- Used to authenticate the server and encrypt the communication
Use Hybrid encryption:
* Encryption algorithms are decided by the browser and the server
* The most secure method available to both is used
* Symmetric encryption is used for the transaction
* But we need to safely share the key
** Asymmetric crypto is used to send the key
** The server's public key is stored inside the site's certificate
HTTPS Handshake between the server and your browser:
* Browser initiates the connection
* Server responds with its certificate
* Browser advertises its encryption methods and sends a symmetric session key encrypted with the server's public key
* Server decides which cipher to use
* Server and client use this session key for symmetric encryption of the data
** Forward Secrecy - key agreement protocols which ensure a compromise of private keys will not lead to a compromise of past session keys. This helps mitigate future SSL attacks.
Public key crypto is only used here to establish faster symmetric encryption
!!! Implementation
Let's set all of this up to secure communication to our web server.
''Note:'' The examples below contain my username, IP address, and DNS records. Be sure to replace them with your values.
!!!! Key generation
* Save the SSL certificates on your web VM within the directory {{File{/etc/ssl/private/}}}
** This is the standard system directory for keys and certificates
----
!!!!! There are two ways to create SSL keys and certificates:
A) Manually:
- ''Do not do this'' for your web server ~VMs; this is only here for informational purposes. We will be using the automated process below.
* SSL Keys
** Create and secure a SSL key with a passphrases
** Create key: {{Command{ openssl genrsa -aes256 -out www.//username//.key 2048 }}}
** Verify key: {{Command{ openssl rsa -noout -text -in www.//username//.key }}}
* Certificate Signing Request (CSR)
** The CSR will collect the server information. It would be sent to a commercial certificate authority who will then create and sign the certificate.
** Create CSR: {{Command{ openssl req -new -sha256 -key www.//username//.key -out www.//username//.csr }}}
** Verify CSR: {{Command{ openssl req -noout -text -in www.//username//.csr }}}
* Certificate Authority (CA)
** Either commercial or self-signed
** Commercial - send the CSR file to the CA
** ~Self-Signed:
*** Become an untrusted certificate authority yourself:
**** Create CA Key: {{Command{ openssl genrsa -aes256 -out ncs205CA.key 4096 }}}
**** Create CA Cert: {{Command{ openssl req -x509 -new -sha256 -key ncs205CA.key -out ncs205CA.crt -days 18250 }}}
**** Sign a certificate: {{Command{ openssl x509 -req -in www.merantn.csr -CA ncs205CA.crt -~CAkey ncs205CA.key -~CAcreateserial -out www.merantn.crt -days 735 }}}
**** Display the new certificate: {{Command{ openssl x509 -noout -text -in ncs205CA.crt }}}
Whoever control's a systems certificate store controls secure communication to that system! If I create a CA certificate and can add it to your system, any certificates I issue will be trusted. Man in the middle attacks would not display the normal "Untrusted site" warnings in your browser unless the site operator has deployed countermeasures.
'' - or - ''
B) ''Automated:'' Using Let's Encrypt and {{Command{acme.sh}}} to generate SSL certificates
- Use this method to create an SSL certificate for your web servers
A trusted SSL certificate must only be issued to someone who can prove they own the domain.
Let's Encrypt and the {{Command{acme.sh}}} tool can obtain this verification either by checking for a small file placed within the ~DocumentRoot of your website or checking DNS for a particular record. Only someone who actually owns the domain should be able to perform these tasks. Normally, the first method is utilized because it is easy to automate. This is not an option for us because our web servers are on a private network which cannot be accessed directly from the internet. The Let's Encrypt servers will not be able to access our web servers to verify them. Our only option is DNS verification because you control your DNS servers and the DNS records are available outside of our lab environment.
{{Warning{''Warning:'' Let's Encrypt certificates are only valid for 90 days and must be continually renewed. An automated mechanism must be in place to ensure the certificates are renewed prior to expiration. A scheduled task must be created to renew the certificates before they expire. This isn't a concern for us since the class will end before these certificates expire but will need to be done if you're doing this for real. The [[acme.sh documentation|https://acme.sh]] will walk you through how to set up this scheduled task.}}}
You cannot proceed if your DNS server is not yet set up properly. Ensure an external test properly returns an IP address before continuing with SSL configuration. Replace my username with yours everywhere it appears throughout these instructions.
{{{
[merantn@core ~]$ dig www.merantn.ncs205.net @1.1.1.1
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> www.merantn.ncs205.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63979
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.merantn.ncs205.net. IN A
;; ANSWER SECTION:
www.merantn.ncs205.net. 300 IN A 192.168.12.25
;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Tue Nov 26 03:34:59 UTC 2024
;; MSG SIZE rcvd: 67
}}}
Complete the following steps on your web VM to generate an SSL certificate with {{Command{acme.sh}}}:
{{Warning{''Warning'': Getting this all set up is a well choreographed dance. All of the pieces need to fall together precisely in order for it to work. Once it's set up, it'll run flawlessly for years. Go slow, pay attention, and mind the typos. Copy & paste as much as you can. Typing all of this out is asking for pain and punishment.
Replace my username and IP addresses where you see them. Copy the large commands from this page to notepad, make the edits, and then paste the commands into the shell. Be sure to review everything closely before executing.}}}
On your DNS server:
# Add a CAA ([[Certification Authority Authorization|https://letsencrypt.org/docs/caa/]]) record to your zone file. A CAA DNS record indicates which Certificate Authorities are allowed to issue certificates for the domain.
# Add a CNAME record resembling the following to your forward zone:
** {{Monospaced{_acme-challenge.www IN CNAME //username//.acme.ncs205.net. }}}
** Be sure to replace //username// with your username
** See my full zone below for an example.
# Increment your SOA serial number and reload your zone file with {{Command{rndc reload}}}
** Notice the serial number below has changed from the last time I posted my zone file. Do not omit this crucial step.
My full zone file:
{{{
[root@core ~]# cat /etc/bind/master/merantn.ncs205.net.fwd
$TTL 5m
@ IN SOA ns1.merantn.ncs205.net. hostmaster.merantn.ncs205.net. (
2024112500 ; serial number
1d ; refresh
5d ; retry
2w ; expire
30m ; minimum
)
IN NS ns1.merantn.ncs205.net.
IN NS ns5.ncs205.net.
IN CAA 128 issue "letsencrypt.org"
ns1 IN A 192.168.12.26
test IN A 192.168.12.24
www IN A 192.168.12.25
core IN A 192.168.12.26
loghost IN CNAME core
ntp IN CNAME core
directory IN CNAME core
_acme-challenge.www IN CNAME merantn.acme.ncs205.net.
}}}
* Don't forget to replace my username with yours everywhere it appears.
Wait for DNS to propagate and you're able to verify your records exist in local and external DNS before proceeding. These two commands can be used to verify that the DNS records are ready. We see validation here for //my// DNS records both locally and globally.
{{Commands{
''1)'' Local check:
[root@core ~]# ''dig _acme-challenge.www.merantn.ncs205.net CNAME @localhost +noall +answer''
@@_acme-challenge.www.merantn.ncs205.net. 300 IN CNAME merantn.acme.ncs205.net.@@
''2)'' External DNS check against the ~CloudFlare DNS server at 1.1.1.1
[root@core ~]# ''dig _acme-challenge.www.merantn.ncs205.net CNAME @1.1.1.1 +noall +answer''
@@_acme-challenge.www.merantn.ncs205.net. 300 IN CNAME merantn.acme.ncs205.net.@@
}}}
You should see the same CNAME record value returned by both queries. __If you don't see any output in the second command, then you likely either forgot to properly increment your serial number or reload your zone.__
{{Note{''Note:'' I only added the ''+noall'' and ''+answer'' options to ''dig'' to trim the output so it's easier to post here. You don't have to use these options in your checks. Omitting them will yield more details which are often very useful.}}}
Once the CNAME records are fully in place and verified, complete the following steps on your web server VM:
On your web VM:
# Download the {{Command{acme.sh}}} shell script and save it to {{File{/usr/local/sbin/}}} on your web server VM
## {{Command{wget -O /usr/local/sbin/acme.sh https://raw.githubusercontent.com/acmesh-official/acme.sh/master/acme.sh}}}
## Be sure to make the file executable
# Download the ~CloudNS API plugin
## {{Command{mkdir -p /root/.acme.sh/dnsapi/}}}
## {{Command{wget -O /root/.acme.sh/dnsapi/dns_cloudns.sh https://raw.githubusercontent.com/acmesh-official/acme.sh/master/dnsapi/dns_cloudns.sh}}}
# Define the API credentials
** We will be performing DNS verification via API. These two commands will save the API username and password to the shell environment for the acme.sh script to retrieve.
## {{Command{export ~CLOUDNS_SUB_AUTH_ID="//username//"}}}
## {{Command{export ~CLOUDNS_AUTH_PASSWORD="//password//"}}}
## Verify the username and password variables: {{Command{ echo $~CLOUDNS_SUB_AUTH_ID - $~CLOUDNS_AUTH_PASSWORD}}}. You should see them repeated back to you.
** The actual username and password will be posted to the Discord channel for this week's material.
# Begin the authorization process for a certificate for your web server from Let's Encrypt. ''Be sure to replace my username with yours everywhere it appears''
## Perform a test of the SSL certificate issue: {{Command{/usr/local/sbin/acme.sh --server letsencrypt --issue -d www.merantn.ncs205.net --domain-alias merantn.acme.ncs205.net --log --dns dns_cloudns --dnssleep 60 --test}}}
If the certificate is properly issued, you should see it displayed to the screen followed by similar text:
{{Monospaced{
-----END CERTIFICATE-----
Tue Nov 26 03:24:30 UTC 2024] Your cert is in: /root/.acme.sh/www.merantn.ncs205.net_ecc/www.merantn.ncs205.net.cer
[Tue Nov 26 03:24:30 UTC 2024] Your cert key is in: /root/.acme.sh/www.merantn.ncs205.net_ecc/www.merantn.ncs205.net.key
[Tue Nov 26 03:24:30 UTC 2024] The intermediate CA cert is in: /root/.acme.sh/www.merantn.ncs205.net_ecc/ca.cer
[Tue Nov 26 03:24:30 UTC 2024] And the full-chain cert is in: /root/.acme.sh/www.merantn.ncs205.net_ecc/fullchain.cer
}}}
The {{Monospaced{--test}}} option at the end of our last command means this is only a test run of the process. We must ensure everything works before requesting the real certificate. If things aren't set up correctly and you request the real certificate too many times, you'll end up blocked by the validation servers.
If the previous command completed successfully, run it again without the {{Monospaced{--test}}} option at the end. You may also need to add a {{Monospaced{-f}}} option to force it.
# Enable the SSL Apache module with the command {{Command{a2enmod ssl}}}. This module will add encryption.
# Edit the Apache SSL configuration file for the SSL site within the directory {{File{/etc/apache2/sites-available/}}}
## Update the ~DocumentRoot directory to match our new location
## Search for the ''~SSLCertificateFile'' directive and change the path to {{File{/etc/ssl/certs/www.merantn.ncs205.net.cer}}}
*** This file contains our server's public key
## Search for the ''~SSLCertificateKeyFile'' directive and change the path to {{File{/etc/ssl/private/www.merantn.ncs205.net.key}}}
*** This file contains our servers private key
## Search for the ''~SSLCertificateChainFile'' directive and change the path to {{File{/etc/ssl/certs/www.merantn.ncs205.net.fullchain.cer}}}
*** This file contains the intermediate certificates. Our browser contains the Root CA certificate. Including the intermediate certificate in our bundle completes the trust chain.
*** This configuration directive may need to be uncommented.
# Enable the SSL site with the command {{Command{a2ensite default-ssl.conf}}}
# Verify the apache config: {{Command{apachectl configtest}}}
** This will identify any errors that might have been introduced.
** It's always wise to validate your configuration before restarting a service. If you don't and there's a problem, your service will be offline while you sort it out.
** Not a big deal here, but this will be a big deal in the future when downtime costs money and might get you fired.
# Install the certificates to their proper place on the system and restart Apache:
** {{Command{/usr/local/sbin/acme.sh --install-cert -d www.merantn.ncs205.net --log --cert-file /etc/ssl/certs/www.merantn.ncs205.net.cer --key-file /etc/ssl/private/www.merantn.ncs205.net.key --fullchain-file /etc/ssl/certs/www.merantn.ncs205.net.fullchain.cer --reloadcmd "/usr/sbin/apachectl restart" }}}
# Verify Apache is running: {{Command{systemctl status apache2}}}.
----
View the server certificate chain: {{Command{ true | openssl s_client -connect www.merantn.ncs205.net:443 -showcerts}}}
- Replace my username with yours.
- You should see ''Verify return code: 0 (ok)'' near the bottom. This return code means a valid certificate was presented and the connection is fully trusted.
- You can also run that command with my username to see what the expected output should look like
Verify your SSL certificate is fully trusted with {{Command{curl}}}. Notice the certificate information and dates in the connection details. Be sure to use the ''https'' protocol for an encrypted connection:
{{{
[root@www ~]# curl -v https://www.merantn.ncs205.net/
* Host www.merantn.ncs205.net:443 was resolved.
* IPv6: (none)
* IPv4: 127.0.1.1
* Trying 127.0.1.1:443...
* Connected to www.merantn.ncs205.net (127.0.1.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=www.merantn.ncs205.net
* start date: Nov 26 02:25:59 2024 GMT
* expire date: Feb 24 02:25:58 2025 GMT
* subjectAltName: host "www.merantn.ncs205.net" matched cert's "www.merantn.ncs205.net"
* issuer: C=US; O=Let's Encrypt; CN=E5
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET / HTTP/1.1
> Host: www.merantn.ncs205.net
> User-Agent: curl/8.5.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Date: Tue, 26 Nov 2024 04:11:25 GMT
< Server: Apache/2.4.58 (Ubuntu)
< Last-Modified: Thu, 14 Nov 2024 06:32:49 GMT
< ETag: "56-626d99e7fca54"
< Accept-Ranges: bytes
< Content-Length: 86
< Vary: Accept-Encoding
< Content-Type: text/html
<
<HTML>
<BODY>
<BR><BR><BR>
<center><B>Welcome to NCS205!</B></center>
</BODY>
</HTML>
* Connection #0 to host www.merantn.ncs205.net left intact
}}}
When an SSL certificate is deployed, connections must be by hostname because that is the primary mechanism to verify that the SSL certificate is valid. Connections by IP address or localhost will return verification errors.
- Look in the {{Command{curl}}} output above for the certificate ''Subject:'' field, {{Monospaced{subject: CN=www.merantn.ncs205.net}}}. The hostname in the certificate subject field must match the hostname used by curl or your browser to access the site.
We can see the verification errors if something other than the hostname used in the certificate is used to access the website:
{{{
[root@www ~]# curl https://localhost/
curl: (60) SSL: no alternative certificate subject name matches target host name 'localhost'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
}}}
!!! Helpful sites:
* Test server SSL quality: http://www.ssllabs.com/
* https://mozilla.github.io/server-side-tls/ssl-config-generator/
* https://cipherli.st/
!!! Additional Reading
https://www.schneier.com/blog/archives/2010/07/dnssec_root_key.html
Implementation http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/x64.html
SSL/TLS Details: http://en.wikipedia.org/wiki/Transport_Layer_Security
[[Bulletproof SSL & TLS|https://www.feistyduck.com/books/bulletproof-ssl-and-tls/]]
!! Scheduled tasks
Any system will contain jobs which must be run at some sort of periodic interval. They can be regular maintenance tasks common to all Unix systems or unique tasks custom to a specific server.
Log file rotation is an example of a regularly scheduled task common to all systems. Every day, cron executes the {{Command{logrotate}}} command to ensure log files do not accumulate indefinitely. Log files are renamed and sometimes compressed according to the schedule outlined in it's configuration file {{File{/etc/logrotate.conf}}}. Very old log files will be deleted once their retention period has expired.
The collection of your PDF labs is an example of a unique task custom to our class shell server. A script is executed every hour which collects your new labs from the {{File{/opt/pub/ncs205/submit/}}} directory and copies them to my grading queue. Here is the cron job which makes that happen:
{{{
[root@shell ~]# cat /etc/cron.d/lab-collect
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
0 * * * * root /opt/rsl/collect.sh
}}}
Tasks may be scheduled using these two services
* {{Monospaced{cron}}} - run periodically at a specified interval. This service is for tasks which will run regularly.
* {{Monospaced{at}}} - run once at a scheduled date & time. This service is for tasks which are one-offs.
!!! More examples of tasks to schedule at a regular interval:
* SSL certificate renewal
* Daily reports
* Garbage collection (remove old or temporary files to free disk space)
* Vulnerability checks
* System updates
* Source code updates
* {{Command{mysqldump}}} - database backups
** create a read-only mySQL account
** save backup to {{File{/opt/work/backups/}}}
* System backups
* Or, using {{Command{at}}}, scheduling a one-time job for a more convenient time
** ie: something that may be bandwidth or CPU intensive.
!!! cron - run periodically
* Background service: {{Command{crond}}}
** A service that is typically installed with the OS and running by default
*** Our Ubuntu ~VMs are barebones and don't have it. Install the {{Monospaced{cron}}} service with {{Command{apt}}}.
** The service is started automatically on system boot
** Permission for regular users to use this tool is granted or revoked via the {{File{/etc/cron.{deny,allow}}}} files
* System cron configuration files & directories:
** {{File{/etc/crontab}}} file - The main configuration file. Each line in this file is a job to run at the specified schedule
** {{File{/etc/cron.d/}}} directory - Where to put individual files containing single or related jobs. Separate files are easier to maintain in an automated fashion.
** {{File{ /etc/cron.{hourly,daily,weekly,monthly} }}} - Scripts placed in these directories will be executed on that interval
* User cron configuration:
** {{Command{crontab}}} command for accessing scheduled ''user'' jobs
** {{Command{crontab}}} {{Monospaced{[-e|-l]}}} - edit or list the ''user'' cron jobs
** User cron config files are stored within {{File{/var/spool/cron/}}} if you'd like to review them
{{Note{''Note:''
- Scheduled tasks related to the function of the system or its services should be saved in the cron configuration files within {{File{/etc/}}}.
- Scheduled tasks which are only for a specific user (generally a non-root user) should be saved using the {{Command{crontab -e}}} command. }}}
!!!Crontab file format
* See {{File{/etc/crontab}}} for an explanation of the fields.
* Declare variables (if you need to) - Shell variables can be set to define items such as the $PATH to use for executables and the user account to email any results to.
** The defaults are reasonable. You generally only need to set variables to make changes to the defaults or define something not already set.
* Output optionally sent to the owner via email (on by default)
* Command execution fields:
** Time to execute command, (minute, hour, day, month, weekday)
** User to run as (This can only be specified when running from a system crontab configuration file)
** Command string to execute
*** It's a best practice to use the full, absolute path to the command you want to execute.
* Special time nicknames:
** @reboot
** @daily
** Complete list is available in the cron man page
* Special time formatting:
** */2 : Every 2 hours, months, days, etc
** 1-4 : Range, for example from 1 to 4
** 1-6/2 : Every other between 1 and 6
** 2,5 : Multiple, for example 2 and 5
!!! Cron man pages:
* Check man page for the {{Command{cron}}} command
* Notice the ''See Also'' section at the bottom where {{Command{crontab}}} is listed in two different manual sections
* {{Command{man man}}} - will describe how to access different manpage sections
The website https://cron.help/ is also be a good resource for validating your cron scheduling.
!!! Troubleshooting cron
* Logging on Linux does a great job providing useful information to help troubleshoot issues
** Most services write to a log file somewhere below the directory {{File{/var/log/}}}
* The cron log file will be a good resource - {{File{/var/log/cron.log}}}
** On Ubuntu, you need to first enable this log destination in the file {{File{/etc/rsyslog.d/50-default.conf}}} by removing a comment.
* Be sure to include the last few lines of the cron log if you need to reach out for help
!!! at - run once
* System dependent:
** ~FreeBSD - The atrun utility is executed via cron every 5 minutes to check for and run {{Command{at}}} scheduled tasks
** ~CentOS - atd - Daemon running in the background for processing tasks scheduled via the {{Command{at}}} utility
** Ubuntu - Same as ~CentOS, except the package is called {{Monospaced{at}}}.
** This is not installed or running by default on most Linux distributions.
** It must be set to start on boot and be manually started after installation, just like any other new service we add.
* {{Command{at}}} user command to add jobs. Run it with the time the task should execute as an argument
** flexible time formatting
*** {{Command{ at +15 minutes }}}
*** {{Command{ at 4:15 }}}
*** {{Command{ at 4pm tomorrow }}}
*** {{Command{ at 4pm October 15 }}}
* Display scheduled job with {{Command{at -c}}}
** Scheduled jobs stored in {{File{/var/spool/at/}}} files
** {{Command{ atq }}} - display scheduled at jobs
** {{Command{ atrm }}} - remove scheduled at job
** Can use {{File{/etc/at.{allow,deny}}}} files to control access to this utility
!!!! Examples:
For our recent security competition, I wanted to lock access to a system for lunch and re-enable access after the lunch break ended:
{{{
root@vce1:~# at 12pm
warning: commands will be executed using /bin/sh
at> pct stop 1215
at> <EOT>
job 2 at Sat Apr 9 12:00:00 2024
root@vce1:~# at 1pm
warning: commands will be executed using /bin/sh
at> pct start 1215
at> <EOT>
job 3 at Sat Apr 9 13:00:00 2024
root@vce1:~# atq
3 Sat Apr 9 13:00:00 2024 a root
2 Sat Apr 9 12:00:00 2024 a root
}}}
! Assignment
<<tiddler [[Lab 60 - SSL Certificates]]>>
<<tiddler [[Lab 61 - Scheduled Tasks]]>>
! Material
!! Read:
* Mastering Ubuntu Server Chapter 2 - Managing Users & Permissions
* [[sudo tutorial|https://phoenixnap.com/kb/linux-sudo-command]]
* [[sudoedit tutorial|https://www.howtoforge.com/tutorial/how-to-let-users-securely-edit-files-using-sudoedit/]]
!! Watch:
* {{Command{sudo}}} use and configuration: https://www.youtube.com/watch?v=YSSIm0g00m4
/% ** Note: In the video, the sudoers file is edited by executing {{Command{sudo visudo}}}. This may not work on our ~VMs. Instead use {{Command{su}}} to become root and then run {{Command{visudo}}} to edit the sudo configuration. %/
! Notes - Access control & user management
''Authentication'' - Who you are. The process of ascertaining that someone is actually who they claim to be
''Authorization'' - What you are allowed to do. Rules to determine who is allowed to perform certain tasks
!! Access control
!!! Unix maintained a multi-user system from the beginning
* All objects (files & processes) have owners
* A user owns new objects they create
* A superuser (typically root) can act as the owner of any object
** Only the superuser, or someone with superuser privileges, can perform most administrative tasks
!!! Groups
A mechanism to grant permissions to groups of users, such as all students in a particular class.
* The filesystem has a more sophisticated access control system
* Each file has a user owner and a group owner
* Permissions can be set so group members may have their own set of access controls (rwx permissions)
* Groups can be harnessed to control access to the system
For example, the directory {{File{/opt/pub/ncs205/}}} is set so only those in this class can access its files.
!!! The superuser (uid 0)
* The root user is the standard unix superuser account
* There's nothing special about the user name - it's all about the user ID (UID) number
** Unix systems track everything by number: process ~IDs, device ~IDs, IP addresses, uid, and gid
** We prefer names over numbers.
* Check out the {{File{/etc/passwd}}} on your ~VMs. There's a second UID 0 user account named {{Monospaced{toor}}}
** An unknown UID 0 backdoor account would normally be a huge red flag. But this account is so I can get into your systems to help if something breaks.
** It has the same privileges as your {{Monospaced{root}}} account but instead uses a password that I have.
** As far as the system is concerned, {{Monospaced{root}}} & {{Monospaced{toor}}} are the same person because they have the same UID number.
!!! Privilege separation
* superuser (UID 0) - The superuser - ideally only use the system with superuser privileges when necessary.
** {{Command{sudo}}} assists with this. Only execute commands with privileges when you need to.
* normal users - the regular users on the system. This is how we all access and use the class shell server.
* service accounts - These are the accounts our services run as, such as the {{Monospaced{apache}}} and {{Monospaced{mysql}}} users on your web server or {{Monospaced{named}}} user on your core VM.
** {{Monospaced{nobody}}} and {{Monospaced{daemon}}} are generic service accounts
** Service accounts are unprivileged accounts which run services as unprivileged users in case the services are broken into. This way they'll have very limited access to the rest of the system.
** Services ran as {{Monospaced{root}}} in the old days. If a service was exploited and an attacker was able to access files or run commands, they would then have access to the entire system.
** We now apply the principle of least privilege and only grant users (and services) the access they need. If an account or service is broken into, the damage will be limited.
** This is why we don't all access the shell server as {{Monospaced{root}}}. I use an unprivileged user also and only elevate to {{Monospaced{root}}} when necessary.
!!! Privilege escalation
* Limit direct access to the {{Monospaced{root}}} account.
** This improves security because we limit the use of shared passwords.
** And also improves accountability since someone must first log in as themselves.
* Privilege separation - Only obtain superuser privileges when you need them
** Don't always operate as the {{Monospaced{root}}} user
* Instead log in as a regular user and escalate when needed
** {{Command{su}}} command - Substitute user
*** Change the effective userid to another system user
*** Your //real id// is the userid you log in as, the user id associated with the process that created the current process
*** Your //effective id// is one the system uses to determine whether you have access to a resource
*** http://bioinfo2.ugr.es/OReillyReferenceLibrary/networking/puis/ch04_03.htm
*** {{Command{su [username]}}} - Change to another user, inheriting the current shell environment.
*** {{Command{su - [username]}}} - Change to another user, simulating a full login. The current shell environment will not be inherited.
** {{Command{sudo}}} - Allow elevated privileges on a limited scale (per command).
*** {{Command{sudo}}} Allows an administrator to grant root privileges to users without divulging the root password.
*** Or allow a user to just run a few commands as the superuser.
*** Display what you are allowed to access via sudo: {{Command{sudo -l}}}
*** {{Command{sudo //command//}}} - Run a command as another user (defaults to the root user)
*** {{Command{sudoedit //file//}}} - Edit a file as another user (defaults to the root user). Running {{Command{sudoedit //filename//}}} is the same as running {{Command{sudo -e //filename//}}}.
*** sudoers file: {{File{/etc/sudoers}}} - This is where the sudo configuration is saved. Don't edit this file directly. Use {{Command{visudo}}} to edit it.
**** Separate sudo configuration files can also be saved within the directory {{File{/etc/sudoers.d/}}} to keep things better organized.
**** {{Command{visudo}}} will lock the file and perform syntax checks after saving it.
** You can control who can access particular resources with user or group permissions
* Both {{Command{su}}} and {{Command{sudo}}} will log escalation events
** su will log when a unprivileged user switches to another user
** sudo/sudoedit will log each command executed or file modified
* setuid bit
** set ID upon execute
** An extra permission bit that can be set with chmod
** The program will run as the user who owns the file.
** Examples: passwd and crontab commands
** The passwd command needs extra privileges in order to change a user's password, so extra system privileges are granted just to that command.
** Notice the ''s'' in the user execute permission field. This shows that the setuid bit is set and the command will run as the user who owns the file. This is another security red-flag to watch out for.
{{{
[merantn@shell:~]$ ls -l /bin/passwd
-rwsr-xr-x. 1 root root 33560 Apr 18 2022 /bin/passwd
}}}
!!! Finer grained access controls
* ~SELinux and mandatory access controls (MAC)
** Enabled by default in Red Hat Linux and derivatives.
** ~SELinux will cause us problems if we don't either configure or disable it
** Controlled by the {{Command{setenforce}}} command for current boot
** and by the /etc/selinux/config file on boot
** It's something to be aware of, but we're not using it on our Ubuntu ~VMs
* Filesystem access control lists (~ACLs)
** Finer grained per user access to files
** Controlled by {{Command{setfacl}}} and displayed by {{Command{getfacl}}}
** Active ~ACLs noted with a + at the end of the file permissions list
** Another good thing to be aware of, but we're not using for the purposes of this class
!!! Verifying users with PAM
* ''P''luggable ''A''uthentication ''M''odules (mentioned briefly on page 51)
* Configuration resides in /etc/pam.d/
* Originally access was determined by just checking passwords against the password files
* PAM modules are now used for user validation and verification
** They can determine who you are
** And if you have permission to access the resource
** They can also enable additional types of authentication, such as two-factor with hardware or soft tokens.
* Examples:
** {{File{/etc/pam.d/su}}} - limit who can use the {{Command{su}}} command
*** uid 0 users can always run the {{Command{su}}} command
*** We can change this to require wheel group membership as an additional security measure
*** Or we could set it to implicitly trust members of the wheel group (dangerous!)
* Other PAM functions:
** PAM can create home directories on first login with pam_mkhomedir
** Can check password complexity with pam_crack
** Can lock accounts on too many failed attempts with pam_tally or pam_faillock
!! Users and Groups
!!! Password files
* {{File{/etc/passwd}}} - Everyone can read this file
** Contains fields identifying the user
** It used to also contain the hashed password but this was moved elsewhere to hide it from normal users
** Don't leave the old password field (position 2) blank! If blank, no password is required for login. Use the placeholder character {{Monospaced{''x''}}}.
*** An {{Monospaced{''x''}}} in {{File{/etc/passwd}}} column 2 means see {{File{/etc/shadow}}} for the password hash
* {{File{/etc/shadow}}} (Linux) or {{File{/etc/master.passwd}}} (~FreeBSD) - Only root can read this file
** A secure file which contains the password hashes so normal users cannot read them for brute force cracking
** Also contains password and account expiration attributes
** More detail on this file and its fields can be found at https://linuxize.com/post/etc-shadow-file/
* Use {{Command{vipw}}} to edit these files so you have file locking and format verification
** This verification prevents errors from breaking access to the system
** {{Command{vipw}}} will edit the password file
** {{Command{vipw -s}}} will edit the shadow file
* password hashing:
** Sample password hash: {{Monospaced{ $6$hA6IJImd$~TCWDXE6zeHgRYKBNAG2jqHNMyPp9FCW2KdlVFKGWto9BcV9chEjCX3zZAzxx5tqbKn3wve13VWLD8Vb5O214x1 }}}
*** The full hash has three components, separated by the {{Monospaced{$}}} delimiter: Algorithm type, salt, and hashed password
** Different hashing algorithms and their tags from old (weak) to new (strong): DES, ~MD5 ({{Monospaced{$1$}}}), Blowfish ({{Monospaced{$2a$}}}), ~SHA256 ({{Monospaced{$5$}}}), ~SHA512 ({{Monospaced{$6$}}})
*** The tag at the beginning of the password hash identifies the algorithm used.
** salting
*** Randomize hashes by adding a salt to the password before hashing
*** This prevents identical passwords from having the same hash
*** Increases the difficulty for brute force attacks or hash lookup tables (rainbow tables), since now a potential password value has to be tested for each possible hash value.
** Password cracking tools:
*** John the Ripper
*** hashcat
*** GPU processing makes this all much faster now, especially for weak algorithms and passwords
*** Protect your hashes!
/% ** {{Command{authconfig --test | grep hash}}} - See what hashing algorithm is used by default on your system
** {{Command{authconfig --passalgo=md5 --update}}} - Change the default hash type (don't actually run this)
** {{File{/etc/sysconfig/authconfig}}} - Authentication configuration settings
** {{File{/etc/libuser.conf}}} %/
{{Note{''Note:'' On page 41, when the book discusses the {{File{/etc/shadow}}} file, they incorrectly refer to the password as //encrypted//. Passwords are //hashed//, not //encrypted//. There's a difference.
* Hashing is a one-way function. Once plaintext is hashed it cannot be converted back to plaintext. We validate passwords by hashing the password someone provides at login and comparing that hash to the one that's stored.
* Encryption is a two-way function. If plaintext is encrypted, it can later be decrypted and turned into plaintext again if you have the key. We briefly reviewed encryption with the web servers. Your browser negotiates an encrypted channel with the web server. They each send encrypted data that the other decrypts with the proper key.
The subtle difference may not matter for normal Linux users, but it's important for a security practitioner to understand the difference between hashing and encryption.
}}}
Time it takes to brute force these password types:
[img[img/2023_Password_Table.jpg]]
* uid numbers
** Multiple users with same UID number - the system only cares about the number. If multiple users have the same UID number, then they are effectively the same user and can access each other's files
** System accounts (UID < 10)
** Service accounts (~UIDs between 10 and 500)
** Users UID > 1000 (Linux) - Regular users.
!!! Group file
* {{File{/etc/group}}} - Where groups and group memberships are defined.
** {{Monospaced{wheel}}} group - special administrator group. Usually allows extra system access
!!! Shells
* default shell : {{Command{/bin/bash}}} (Linux) or {{Command{/bin/tcsh}}} (BSD)
* lockout shell : {{Command{/sbin/nologin}}}
** Users with this shell are not allowed to log into the system. Service accounts or banned users will be set to this shell.
* Available shells defined in {{File{/etc/shells}}}
!!! Locking accounts
* Replace the hash with a {{Monospaced{*}}} or {{Monospaced{!!}}} to lock the account.
** This is not enough on //some// systems. Users may still be able to log in with SSH keys instead of passwords.
* Also change shell to {{Command{/sbin/nologin}}}
** This is a standard lockout shell. A user must have a valid login shell in order to connect to a system
** The command {{Command{/sbin/nologin}}} just echos //This account is currently not available.// and terminates, thus disconnecting the user from the system.
* {{File{/var/run/nologin}}} or {{File{/etc/nologin}}}
** If this file exists, only root will be allowed to log into the system. The contents of the file will be displayed to the user before they are disconnected.
*** This is helpful if a system needs to be closed for temporary maintenance.
* Check out service accounts in the password file - they should not have passwords or valid shells
** A service account with a password or valid shell is being abused by an attacker.
!!! Adding a new user:
* Use utilities ({{Command{useradd}}}, {{Command{userdel}}}, {{Command{usermod}}}) or edit the password files directly
* Create a home directory for the user
** Be sure to set home directory ownership and permissions so the new user can access it
* Set up the environment (dot files)
** Copy the environment configuration files within {{File{/etc/skel/}}} (Linux) or {{File{/usr/share/skel/}}} (~FreeBSD) to the new user's home directory
*** Note: All environment configuration file names begin with a dot.
*** Don't forget to change ownership on the environment files in the user's home directory too
!!! Remove or lock a user
* Delete or comment lines in the password files
** Commenting the lines in the password files is a non-destructive way to disable access
** Change password hash and change shell
!!! Authentication factors:
Multi-factor authentication (MFA):
* Passwords are not good enough anymore; they are easily stolen.
* We must now increase security by combining multiple authentication factors.
* More sites and organizations are now requiring MFA
** [[Linux Kernel Git Repositories Add 2-Factor Authentication|http://www.linux.com/news/featured-blogs/203-konstantin-ryabitsev/784544-linux-kernel-git-repositories-add-2-factor-authentication]]
** SUNY Poly recently switched from GMail to MS Outlook and added MFA for email account login
!!!! Methods of authentication:
* ''Something you know'': eg, passwords
** Passwords should be of sufficient length and complexity to be hard to crack
** Minimum of 10-12 characters
** {{Monospaced{correct horse battery staple}}} - a good take on how we (incorrectly) handle passwords: http://xkcd.com/936/
** Passwords should be unique across systems
*** It's common to try to use stolen passwords for other services. For example, a username/password combo stolen from a commerce site to be attempted at a bank.
*** [[Russian Hackers Amass Over a Billion Internet Passwords|http://www.nytimes.com/2014/08/06/technology/russian-gang-said-to-amass-more-than-a-billion-stolen-internet-credentials.html?_r=0]]
*** [[Stolen user data used to access account|http://community.namecheap.com/blog/2014/09/01/urgent-security-warning-may-affect-internet-users/]]
*** [[ebay|http://money.cnn.com/2014/05/21/technology/security/ebay-passwords/]]
*** https://haveibeenpwned.com/ - Check if your email address was involved in a data breech
** Password Cards offer an offline way to remember unique passwords: http://www.passwordcard.org/en, etc
** Password vaults
*** [[BitWarden|https://bitwarden.com/]] - Standard online password manager. Can also be self-hosted.
*** [[Password Safe|https://www.pwsafe.org]] - A good offline & cross-platform option
*** [[KeePass|https://keepass.info/]]
** One-time passwords (OTPW) - A list of passwords that can only be used once to prevent reuse if they're intercepted. These are mostly used only as backup passwords now.
* ''Something you have''
** [[yubikey|http://www.yubico.com/]] - A standard hardware key option that's easy to implement
** [[Google Titan Key|https://cloud.google.com/titan-security-key/]]
*** https://www.cnet.com/news/google-made-the-titan-key-to-toughen-up-your-online-security/
** [[DoD CAC card|http://www.cac.mil/common-access-card/]]
** [[Google 2 factor|https://www.google.com/landing/2step/]]
** Most of these are going to soft tokens now, using your phone as the second authentication factor
* ''Something you are''
** biometrics: fingerprint, retina, voice print, facial, vein patterns
* ''Somewhere you are''
** Geofencing - Tie authentication to a particular location
*** Someone may only log in or may not log in from a specific geographic location
*** Useful to flag logins from foreign countries
** ~GeoIP libraries
** pam_geoip
!!! SSH authentication & increasing security
!!!! ssh keys
* Access systems with keys instead of just passwords for added security
* 1.5 factor authentication: Slightly better then just passwords
* Create keypairs with ssh-keygen
** Asymmetric keypairs are used for authentication. You keep the private key secure and locked with a passphrase. The public key is distributed to systems you have permission to access.
* Public keys are stored in {{File{~/.ssh/authorized_keys}}}
* Host public keys are stored in {{File{~/.ssh/known_hosts}}}
* {{Command{ssh-agent}}} & {{Command{ssh-add}}} : add your ssh keys to the agent to be used for connecting to multiple systems
* {{Command{ssh-copy-id}}} : An easy way to transfer your public key to a remote system if you can log in with a password
* {{Command{pssh}}} - parallel ssh for connecting to multiple systems and running command across them at once.
!!!! sshd configuration
* Host keys
** Host keys are used to encrypt the traffic between Unix systems. They're typically found within the {{File{/etc/ssh/}}} directory.
** These keys must be protected. If an attacker obtains them, they can impersonate and access traffic.
** Host key warning - A warning appears on new systems or if they host keys change asking you to verify the host key. This helps ensure you're not the victim to a man-in-the-middle attack
* Requiring user SSH keys to access the system will increase security (disable password authentication)
** This is more secure then just passwords. An attacker cannot only capture a password, they also must capture the SSH private key.
* Deny direct root SSH login - Don't allow users to log in directly as root. They must first log in first as a regular, unprivileged user and then escalate to root with either {{Command{su}}} or {{Command{sudo}}}
** No system should allow direct root login. Turning this off is an excellent security first-step
** Our shell server sees about 50 attempts per day to log in as root. Countermeasures identify and block these attackers.
** {{Command{grep 'sshd-root.*Found' /var/log/fail2ban.log | wc -l}}}
* Require group membership - Users must be in a particular group to log in to the system via ssh
** An option, but not typically done
! Assignment
<<tiddler [[Lab 62 - VM Lockdown - Secure your VMs]]>>
----
<<tiddler [[Lab 63 - sudo]]>>
----
<<tiddler [[Lab 64 - Enable Two-Factor Authentication]]>>
----
/%
<<tiddler [[Lab 65 - SSH Intrusion]]>>
%/
----
! Material
!! Read:
* Chapter 3 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
!! Watch:
* [[Navigating the filesystem|https://www.youtube.com/watch?v=j6vKLJxAKfw]]
! Working on the command line
!! File types
In the Unix world, everything is represented by a file. Regular files and directories are files, but so is hardware. Each piece of hardware connected to the system is associated with a file within the directory {{File{/dev/}}}.
There are three main types of files we will be working with:
* Ordinary files. These are regular files on the system, divided into:
** Plain text - Files you can read if you were to open them in a text editor
** Binary - Machine code, such as programs intended to be executed.
* Directories - The organizational unit for files within the filesystem
* Symbolic Links - These are pointers to another file somewhere on the system. The symbolic link only contains the path to its target.
The [[Week 2, Part 1]] page has a more complete breakdown. Be sure to review the information on that page again.
!! Navigating the filesystem:
!!! Directory paths
* Paths in Unix are very similar to the command line on Windows
* Here, the path separator is a forward slash - {{File{/}}}
* You can change directories with the {{Command{cd}}} command
** eg: {{Command{cd /opt/pub/ncs205/submit}}}
* List the contents of the directory with the {{Command{ls}}} command
* List the contents of the directory in long format with the {{Commandls -l}}} command
** eg: {{Command{ls -l /opt/pub/ncs205/submit}}}
* Some directories of interest:
** {{File{/home/}}} - //Most// user home directories reside somewhere within this directory, but not all.
*** This is only a convention, not a requirement. A user home directory can be anywhere on the system.
** {{File{/opt/pub/ncs205/submit/}}} - Lab/Homework assignments are uploaded to this directory
** {{File{/opt/pub/ncs205/data/}}} - Data files for labs are stored here
** {{File{/opt/pub/ncs205/returned/}}} - Graded and returned Lab/Homework assignments are moved to this directory for your retrieval.
** {{File{/tmp/}}} - Temporary scratch space available to everyone
** {{File{/bin/}}} & {{File{/usr/bin/}}} - Where most program files reside on our systems
!!! Listing files - {{Command{ls}}}
The {{Command{ls}}} command will list the contents of a directory. Extra options can be used to alter the default behavior of the {{Command{ls}}} command:
!!!! Display in long list format: {{Command{ls -l}}}
The {{Command{-l}}} option will display the contents of the directory in long listing format. This displays additional metadata about a file, such as the file type, ownership information, size, and modification timestamp.
The first character on the line indicates the type of file:
* ''d'' for directory
* ''l'' for link
* ''-'' for regular file
{{{
[root@lab ~]# ls -l
total 20
drwx------ 2 root root 173 Nov 23 2020 bin
-rw-------. 1 root root 903 Feb 18 2021 commands.txt
-rw------- 1 root root 57238 Aug 25 2020 packages.txt
-rw------- 1 root root 465 Sep 4 13:16 user2109.txt
}}}
!!!! Display hidden files: {{Command{ls -a}}}
Files beginning with a dot are hidden and not normally displayed with the {{Command{ls}}} command. Adding the {{Command{-a}}} option will allow them to appear:
{{{
[root@lab ~]# ls -al
total 152
dr-xr-x---. 6 root root 4096 Sep 4 13:16 .
dr-xr-xr-x. 17 root root 256 Aug 31 12:42 ..
-rw-r--r--. 1 root root 287 Dec 15 2020 .bash_profile
-rw-r--r--. 1 root root 176 Dec 28 2013 .bashrc
drwx------ 2 root root 173 Nov 23 2020 bin
-rw-------. 1 root root 903 Feb 18 2021 commands.txt
-rw-r--r--. 1 root root 100 Dec 28 2013 .cshrc
-rw------- 1 root root 57238 Aug 25 2020 packages.txt
drwx------. 2 root root 71 Aug 28 17:38 .ssh
-rw------- 1 root root 465 Sep 4 13:16 user2109.txt
}}}
!!!! Sort by modification time: {{Command{ls -t}}}
Adding the {{Command{ls -t}}} option will sort by modification time instead of by file name with the oldest files on the bottom. The {{Command{ls -r}}} reverses the default sort to instead put the newest files on the bottom.
Combining all of these options to see a long listing sorted by reversed time (with the newest files on the bottom) is often handy:
{{{
[root@lab ~]# ls -lrt /opt/pub/ncs205/data
total 496
drwxr-xr-x. 3 root 10000 61 Feb 7 2018 lab7
drwxr-xr-x. 2 root 10000 74 May 27 2019 lab8
drwxr-xr-x. 2 root 10000 16384 May 31 2019 lab9
-rwxr-xr-x. 1 merantn users 471148 Jan 30 2020 whatami
drwxr-xr-x. 2 root 10000 136 Feb 13 2020 lab10
drwxr-xr-x 2 root root 41 Feb 16 2020 lab15
d--------- 2 root root 18 May 4 2020 final
drwxr-xr-x 2 root root 238 Oct 16 2020 filter_examples
drwxr-xr-x 2 root users 207 Mar 13 12:31 lab21
}}}
!! Executing commands
!!! Structure of a command string:
* {{Command{command [options] [arguments]}}}
** options alter the default behavior of a command
** options will begin with either a single dash (''‐'') or a double dash (''‐‐'')
** arguments provide additional information to a command, such as a file or directory to act upon
** options and arguments may be optional or required depending on the command
** Best practice is to enter the three components in that order - First the command, then any options, finally any arguments
** The {{Monospaced{'' [ ] ''}}} in the usage example above indicates that component is optional. The command is always required. Some commands also require arguments. The synopsis in the command's man page will indicate which components are required for different ways to use the command.
!! Viewing files
* Display the contents of a file: {{Command{cat //filename//}}}
* Display the contents of a file one page at a time: {{Command{less //filename//}}}
* Edit a text file: {{Command{nano //filename//}}} -or- {{Command{vi //filename//}}}
!! Working efficiently - some shortcuts
* View your previously executed commands with the {{Command{history}}} command
* Tab completion - Press the tab key to autocomplete commands or file paths
** Enter the first few letters of a command or file, press tab, and the shell will fill in the remaining letters (if it can)
** Example video: https://www.youtube.com/watch?v=5mOHSBFuSy4
* Up / Down arrows - search up and down through your command history to rerun a previously executed command
* Page Up / Page Down - Use these keys to search through your command history for the last commands which begin with a given string
** Type a few letters from the beginning of a previously executed command and press Page Up. The shell will return to the last command you executed which began with those letters. Keep pressing Page Up to cycle through the commands.
!! Other useful commands:
* {{Command{touch}}} - create an empty file
* {{Command{file}}} - examine a file to identify its type
* {{Command{strings}}} - display the plain text strings within a binary file. Often useful for forensics and identifying what a binary is or does.
! Assignment
!! Read Chapter 3 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
* Complete [[Lab 3|labs/lab3.pdf]] & [[Lab 4|labs/lab4.pdf]]
! Material
!! First complete this lab for review:
- Complete [[Lab 5|labs/lab5.pdf]]
!! Read:
* Chapter 4 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
** You can gloss over the parts about wildcards (pp 26-27) for now. We'll come back to them later.
** Focus on becoming familiar with the available commands (also listed below).
!! Watch:
* Creating and Deleting files and directories: https://www.youtube.com/watch?v=91FhiTyEaCU
* Moving and copying files: https://www.youtube.com/watch?v=GKEGNdNIQrw
! Manipulating Files & Directories
Every operating system has a basic set of utilities to manipulate files and directories on the command line. This reading assignment will introduce those commands in the Linux operating system.
!! Basic file manipulation commands
Rename and move files - {{Command{mv}}}
Copy files - {{Command{cp}}}
* Recursive copy: {{Command{cp -R}}}
Create directories - {{Command{mkdir}}}
Delete files and directories - {{Command{rm}}} & {{Command{rmdir}}}
* Difference between unlink & delete - Removing a file doesn't actually delete it from the system, it only marks the space it was occupying as available for reuse.
* This is more appropriately called "unlinking". We're only removing the //link// between the filesystem and the data blocks. The actual data will still reside on the disk and can be forensically recovered.
* The {{Command{srm}}} or {{Command{shred}}} commands will actually destroy a file by overwriting its data blocks before unlinking it.
** These commands are not installed by default on most systems
** Securely wiping a file is more resource intensive than simply unlinking it because random data must first be generated and then written to disk.
View the contents of files with {{Command{cat}}} and {{Command{less}}} (or {{Command{more}}}, an old version of {{Command{less}}})
** [[less quick reference]]
Edit text files:
* The standard Unix text editors are {{Command{nano}}} (Basic and easy to use) and {{Command{vi}}} (more powerful, but harder to get used to)
* {{Command{vi}}} comes installed on every Unix/Linux system. The {{Command{nano}}} editor may need to be installed separately.
* {{Command{vi}}} has a learning curve, but is the professional text editor. If you'll be doing this work in the future, learning {{Command{vi}}} is well worth the time investment.
** Some {{Command{vi}}} handouts: [[Command line summary handout|handouts/UnixCommandSummary.pdf]] (last page) & [[vi diagram handout|handouts/viDiagram.pdf]]
!! Other useful commands:
* {{Command{touch}}} - create an empty file
* {{Command{file}}} - examine a file to identify its type
! Deliverables
!! Review Material:
* Complete [[Lab 5|labs/lab5.pdf]]
!! New Material:
* Complete [[Lab 6|labs/lab6.pdf]] & [[Lab 7|labs/lab7.pdf]]
! Material
!! Read:
* Chapter 28 - Reading Keyboard Input
* Chapter 32 - Positional Parameters
The [[Shell scripting best practices]] page will have useful information as you're writing your scripts. These best practices will be considered when assigning grades. Be sure to note the shell scripting grading rubric outlined at the top of [[Shell script submission requirements]].
! Notes
This portion will cover four main concepts:
- Obtaining input from the user
- Positional Parameters (obtaining input from command-line arguments)
- for loops
- Input Validation (optional)
!! Obtaining input from the user:
Chapter 28 covers the {{Command{read}}} command, along with a lot of advanced detail that is beyond the scope of this class.
The {{Command{read}}} command will accept input from the keyboard and save it to the specified variable: {{Command{read //variablename//}}}
- Use {{Command{read}}} with the ''-p'' option to supply a prompt.
Examples:
{{{
#!/bin/sh
# ./nums.sh
read -p "Enter a number: " number1
echo you entered $number1
echo -n "Enter a number: "
read number2
echo you entered $number2
echo
echo -n "Sum is: "
expr $number1 + $number2
read -p "Enter two numbers: " number1 number2
echo
echo -n "Sum is: "
expr $number1 + $number2
}}}
!! Positional Parameters
The read command will allow us to prompt for input. Positional parameters (variables) will allow our scripts to obtain input from command-line arguments. Chapter 32 will discuss them in more detail.
Special shell variables:
| !Variable | !Description |
| $0 |Name of the script|
| $1 - $9 |Values of command line arguments 1 - 9|
| $# |Total number of command line arguments|
| $* |Value of all command line arguments|
| $@ |Value of all command line arguments; each quoted if specified as "$@"|
| $? |Exit status of most recent command|
| $$ |Process ID of current process|
| $! |PID of most recent background process|
Command line arguments can be passed to a shell script and stored in $1 through $9
{{{
#!/bin/sh
# ./vars.sh
echo $1 - $2 - $3
echo All command line arguments: $*
echo Number of command line arguments: $#
echo Name of the current command: $0
echo Exit status of the previous command: $?
[root@shell ~]$ sh vars.sh first orange third wolf
first - orange - third
All command line arguments: first orange third wolf
Number of command line arguments: 4
Name of the current command: vars.sh
Exit status of the previous command: 0
}}}
! Assignment
!! Complete
* Complete [[Lab 33|labs/lab33.pdf]] & [[Lab 34|labs/lab34.pdf]]
* Complete [[Lab 35|labs/lab35.pdf]] - ''Due Wednesday, Oct 30''
** ''Be sure to follow the [[Shell script submission requirements]] to submit your shell script labs''
{{Note{''Note:'' Lab 34 pro tip - avoid typing out long commands or scripts. Typing out long commands manually is a great way to introduce typos and break things. Use copy/paste instead. If you haven't noticed yet, the [[Linux Shortcuts]] page covers how to copy/paste in putty/Linux.}}}
! Material
* Read Chapter 19 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]].
This material is a little more difficult to understand than most of what we'll be working on. Make use of Discord if you're running into trouble! We can also spin up a Zoom session if anyone would like to chat.
We only have enough time to skim the surface. Regular expressions are very handy tools, especially when combined with our filters or when you need to perform complex text manipulations in vi. I'd like to at least introduce the topic.
If anyone would like to dive deeper into this topic, I recommend [[Mastering Regular Expressions|https://www.oreilly.com/library/view/mastering-regular-expressions/0596528124/]]. It's an excellent technical book.
! Regular Expression Intro
* What is a regular expression?
** Compact way of specifying a pattern of characters
** text is evaluated against the pattern to determine if there is a match.
*** We can then perform actions based on that test
** We can use a regular expression to examine the contents of an entire file, such as with {{Command{grep}}}
** Or we can examine just a string (or variable), such as with input validation
* What are they used for?
** pattern matching - instead of searching for a simple word, search for more complex patterns -
** Similar to how file globbing works. Only here, we're examining the //content// of the files and using the metacharacters a little differently
** Complex substitutions (search & replace)
*** Helpful in {{Command{vi}}} or a Windows text editor such as notepad++
** Input verification
*** Make sure the user is providing an appropriate input in a script or program. For example, if we prompt for an IP address, ensure what's provided is a valid IP address.
* How are they used?
** {{Command{grep}}} - basic regular expressions
** {{Command{egrep}}} or {{Command{grep -E}}} - extended regular expressions
** {{Command{sed}}}
** {{Command{vi}}}
** {{Command{python}}}
** any other programming language
What to match: a combination of ''atoms'' and ''operators''
''atom'' = something we're trying to match
''operator'' = how we're trying to match it
''Atoms'' - Any regular character, or:
| !Symbol | !Meaning | !Example | !Escape | !Not supported by |
| {{Monospaced{'' [ ] ''}}} |Character groups (match any one character listed) | [a-c0-3] | | |
|~|Characters may be specified singly or in ranges|~|~|~|
| {{Monospaced{'' [^ ] ''}}} |Negated character group (match any one character not listed| [^a-c0-3] | | |
| {{Monospaced{'' . ''}}} |Any single character| | | |
''Operators'' - Modify the atoms
These operators act as quantifiers, specifying the number of times an atom is matched.
| !Symbol | !Meaning | !Escape | !Not supported by |
| {{Monospaced{'' ? ''}}} |Optional item. Match 0 or 1. | | sed |
| {{Monospaced{'' * ''}}} |Repetition: 0 or more| | |
| {{Monospaced{'' + ''}}} |Repetition: 1 or more. | | sed |
| {{Monospaced{'' { } ''}}} |Repetition: Defined range of matches {//min//,//max//} or {//min//,} or {//exactly//}| * | |
| !Symbol | !Meaning | !Escape | !Not supported by |
| {{Monospaced{'' ( ) ''}}} |Grouping - next operator applies to whole group| | |
|~|Alternation (match any one of the sub-expressions)| | |
| {{Monospaced{'' {{{|}}} ''}}} |Or. Match either expression it separates. Use with ( )| | |
Grouping with the ''( )'' allow us to apply an operator to a group of atoms. For example, to make the entire string ''abcde'' optional: ''(abcde)?''. The ''?'' will apply to the entire sequence of characters within the ''( )''.
{{Note{''Note:'' We use file globbing to identify groups of files and regular expressions to identify groups of characters. File globbing is for file //names// and regular expressions are used for the file's //contents// or any other strings of characters. }}}
{{Warning{''Warning:'' Regular expressions use many of the same metacharacters as file globbing, but they work differently here. For example, if we want to list files which begin with the letter a, I would use the command {{Command{ls a*}}}. But with regular expressions, if I want to find a string that begins with an a, I need to use the regular expression {{Monospaced{ ''^a.*'' }}}. In file globbing, the {{Monospaced{'' * ''}}} stands alone. In regular expressions it is a modifier and changes the behavior of what comes before it. With regular expressions, the {{Monospaced{'' . ''}}} is used to specify any single character and the {{Monospaced{'' * ''}}} modifies it to match 0 or more occurrences.}}}
''Anchors'' can also be used to specify where our match must occur:
| !Symbol | !Meaning |
| ^ |Start of line|
| $ |End of line|
| \< |Beginning of word|
| \> |End of word|
Typically, when we are searching for a pattern, we are searching for a sub-string (a smaller string within a larger series of characters). For example, if I want to display lines which contain //only// numbers, the command {{Command{egrep '[1-9]+' data.txt}}} will return the output:
12345
a1234
b1234c
But I want lines which contain //only// numbers. I don't want to display the lines which also contain a letter. The solution to prevent other characters from sneaking into your output is to use the anchors to ensure your regular expression is ''//completely matched//''. This command will anchor the numbers to the beginning and end of the line so no other characters can be matched: {{Command{egrep '^[1-9]+$' data.txt}}}
[[Regular expression metacharacters]]
[[ASCII Chart|handouts/ascii-chart.gif]]
Regular expressions and file globbing have common metacharacters that have different meanings in the two contexts. ''Be sure you know how the metacharacters differ.''
!! Examples
!!!! Match:
* grey with either spelling - {{Monospaced{'' gr[ea]y ''}}}
* color with American or British spelling - {{Monospaced{'' colou?r ''}}}
* variations of third street (3rd street or third street) - {{Monospaced{'' (3|thi)rd street''}}}
* The month June either full or abbreviated - {{Monospaced{'' June? ''}}}
* find words with 5 or more vowels in a row - {{Command{'' egrep '[aeiou]{5}' /usr/share/dict/words ''}}}
* Find words in the unix dictionary file that begin with book and end with y - {{Command{ grep '^book.*y$' /usr/share/dict/words }}} - Don't forget the anchors!
* Find words that are 4 characters beginning with a b and ending with a k - {{Command{ grep '^b..k$' /usr/share/dict/words }}}
* Find words in the dictionary file that begin with or end with the string book - {{Command{ egrep '(^book|book$)' /usr/share/dict/words }}}
* Match Kunsela ~C-Wing B&W printer queue names - {{Command{ grep '^c...lpr' /opt/pub/ncs205/data/filter_examples/printcap | cut -d '|' -f 1}}}
* Extract the mail sender and subject for the last 4 emails received - {{Command{egrep "^(From|Subject): " /var/mail/$USER | tail -n 8}}}
!!!! Input verification:
Completely match -
* a valid campus username - {{Monospaced{'' [a-z]{1,8}([0-9]{1,2})? ''}}} - 1 to 8 lowercase letters optionally followed by 1 or 2 numbers.
* a phone number - {{Monospaced{'' '^[0-9]{3}[-. ][0-9]{3}[-. ][0-9]{4}$' ''}}}
* time (12hr clock) - {{Monospaced{'' '^(1-9|1[0-2]):[0-5][0-9] [ap]m$' ''}}}
* a Dollar amount
What is the difference between [ab]* and (a*|b*)
!!! More complex examples:
!!!! Search the ssh configuration files for all configuration directives that are enabled with a yes:
Read this from bottom to top
{{{
egrep '^[^#]+ yes' /etc/ssh/ssh{,d}_config
^ ^ ^^--- ----
| | || ^ ^--- Match both ssh_config and sshd_config
| | || |--------------------- Followed by the string yes
| | ||----------------------- There must be a space between the configuration item and its value
| | |------------------------ The + modifies the [^#] to allow for any length of characters here.
| |-------------------------- Followed by any character which is not a #. We don't want comments.
|----------------------------- The line begins
}}}
This part is the regular expression that searches for the text: {{Monospaced{'' ^[^#]+ yes ''}}}
This part is the file globbing pattern that identifies the files to search: {{Monospaced{''/etc/ssh/ssh{,d}_config''}}}
!!!! Search all networking configuration files on the class shell server for an IP address:
{{Command{egrep '((1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])' /etc/sysconfig/network-scripts/ifcfg-*}}}
It's important to understand how the metacharacters differ between file globbing and regular expressions. In regular expressions, the {{Monospaced{'' ? ''}}} or {{Monospaced{'' * ''}}} modify the atom immediately before them. For example, in the IP address above, the {{Monospaced{'' ? ''}}} in {{Monospaced{'' 1?[0-9]?[0-9] ''}}} means both the {{Monospaced{'' 1 ''}}} and the {{Monospaced{'' [0-9] ''}}} range are optional. In file globbing, the {{Monospaced{'' ? ''}}} or {{Monospaced{'' * ''}}} stand alone and represent either one single character or 0 or more characters by themselves.
The regular expression to identify an IP address is rather complex. We can't just do {{Monospaced{'' [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3} ''}}} if we're only trying to match IP addresses and call it a day. This one would work to identify a proper IP like 192.168.0.1 but would also match an invalid IP address like 555.555.555.555.
To break this regex down piece by piece:
{{{
((1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
^ ^ ^ ^ ----------- -------^ ^ ^ ---------------------------------
| | | | ^ ^ | | | ^-- Repeat the first group again to identify a single octet without the dot at the end.
| | | | | | | | |--- Repeat the previous group within the ( ) three times for the first three octets
|---------------------------------|-------- These ( ) are used for alternation with the |, eg: (This|or|that) will match one of those three words.
| | | | | |------ Escape the dot so it's actually just a dot and cannot represent any single character
| | | | |------------ Match an octet from 250-255
| | | |---------------------- Match an octet from 200-249
| | |------------------------------- Notice the lack of the ?. We must have at least a single digit in each octet
| |---------------------------------- The ? makes the [0-9] optional. This would allow for a two digit IP address
|---------------------------------------- This optional 1 allows for octets in the 100-199 range
}}}
! Assignment
* Complete labs [[Lab B1|labs/labB1.pdf]], [[Lab B2|labs/labB2.pdf]], [[Lab B3|labs/labB3.pdf]], & [[Lab B4|labs/labB4.pdf]]
** These all are extra credit labs.
** Be sure to use an uppercase {{Monospaced{''B''}}} as part of the lab number
** We haven't done much with regular expressions and this is a complex topic. Please make use of the discussion boards if you have any questions or run into any trouble.
! Material
!! Read:
* Chapter 11 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
* Chapter 26 in [[The Linux Command Line|http://www.merantn.net/reference/TLCL-19.01.pdf]]
** The goal from Ch 26 is to get an understanding for functions. The whole thing can be summed up by the example on page 385 and the contents of the //Shell Functions In Your .bashrc File// box on page 391.
!! Watch:
* [[Customizing Your Terminal: .bash_profile and .bashrc files|https://www.youtube.com/watch?v=vDOVEDl2z84]]
* [[Customizing Your Terminal: Adding Color and Information to Your Prompt|https://www.youtube.com/watch?v=LXgXV7YmSiU]]
* [[Creating Aliases for Commands|https://www.youtube.com/watch?v=0liXeoADU6A]]
! Notes
!! Working with the shell
A user automatically executes a shell when they log into a Unix system. A shell is a special type of program that receives and interprets commands. This shell is what users interact with on the command line. You are then logged out of the system when your shell terminates.
The shell you are using is specified as the last column in the {{File{/etc/passwd}}} file. Bash is the standard default but many others exist.
The different shells available on the system are usually listed in the file {{File{/etc/shells}}}.
* {{Command{/sbin/nologin}}} : A special shell used for disabled accounts which should not be able to log in.
** You'll see service accounts in {{File{/etc/passwd}}} with this shell. Users should never be able to log as those service accounts
** A service account with a valid shell is a major red flag and a sign your system has been tampered with.
!!! Bash Customization
RTFM! {{Command{man bash}}}
!!!! Shell initialization and configuration scripts - executed upon login
Your shell environment can be customized by the system administrator and by the user. The sysadmin may have some site-specific changes to make. For example, I change the default umask for everyone on our class shell server. Each user may customize their shell environment either cosmetically, such as by changing the shell prompt, or functionally, such as by changing the PATH or adding command aliases.
The shell environment is customized through a series of script files. They are executed in the following order so the admin can set defaults that the users can override with their own customizations. These scripts work like any of the scripts we've been writing for this class. Any commands entered will be executed when these scripts run at login or logout.
Interactive login shell execution sequence. When you first log in to the system, the following are executed (if they exist):
* {{File{/etc/profile}}} contains general system defaults
* All scripts in the directory {{File{/etc/profile.d/}}}
** Putting individual settings in their own files makes it easier to maintain the changes
** The file {{File{/etc/profile.d/umask.sh}}} sets our default umask
* {{File{~/.bash_profile}}} is controlled by the user for their custom changes
** Put things you want to run during a new login session in this file. Items in this file will not be executed if a new shell instance is executed.
* {{File{~/.bashrc}}} is executed only in interactive shells. This file may contain extra functions and aliases.
** Put settings (like aliases and prompt changes) in this file so they will be activated if a new shell session is run
* {{File{~/.profile}}} may exist instead of {{File{~/.bash_profile}}} on some systems
* //User disconnects//
* {{File{~/.bash_logout}}} will execute when the user logs out.
{{Command{ source //file// }}} (or {{Command{ . //file// }}})
<<<
Read and execute commands from //file// in the current shell environment. Apply changes within an environment script file to the current login shell.
<<<
!!!! Example:
Suppose each time a user logs in, we want to display their last three logins to the screen. The following would be added to either the site-wide {{File{/etc/profile.d/}}} directory or appended to their {{File{~/.bash_profile}}}. We would choose {{File{/etc/profile.d/}}} if we didn't want the users to be able to remove it. We would choose the user's {{File{~/.bash_profile}}} if we wanted users to be able to override it. We would not put it in {{File{~/.bashrc}}} because we only want this information displayed when the users log in, not when they just run a new shell.
{{{
last -3 $USER
}}}
!!!! Default user dotfiles
The directory {{File{/etc/skel}}} contains default copies of {{File{~/.bash_profile}}}, {{File{~/.bashrc}}}, and {{File{~/.bash_logout}}}. These can be copied to the home directories of new users so they have defaults available for their accounts.
!!!! Other shell configuration files:
Readline library - A library for reading a line of input from the terminal
* Configured by {{File{/etc/inputrc}}} and {{File{~/.inputrc}}}
* These files mostly control additional key bindings
* I like to enable the ~PageUp and ~PageDown keys on other systems for fast command recall. They're not enabled by default on Debian.
{{File{/etc/~DIR_COLORS}}}
<<<
Configure directory listing colorization
<<<
Disable ls colorization on the ~VMs. Sometimes color makes it hard to read the text
* Edit {{File{/etc/~DIR_COLORS}}}
* change {{Monospaced{ ''tty'' }}} to {{Monospaced{ ''none'' }}}
{{File{/etc/motd}}} - A ''m''essage ''o''f ''t''he ''d''ay to display to users after they log in
!!! Aliases
Command aliases provide a way to execute long command strings with fewer keystrokes. Additional options and arguments can be added to an alias. For example, running {{Command{ l. -l }}} will display all files which begin with a dot in long-listing format. The {{Command{ l. }}} alias will be translated to {{Command{ ls -d .* }}} and then the {{Monospaced{ -l }}} option will be added.
Display currently defined aliases: {{Command{alias}}}
Set an alias: {{Command{alias name='long_cmd -abcd | next_command -efgh'}}}
Standard aliases
* {{Command{ll}}} - Aliased to {{Command{ ls -l }}} on most systems
* {{Command{l.}}} - Aliased to {{Command{ ls -d .* }}} - Display //only// files which begin with a dot.
Override aliases:
* The {{Command{ rm }}} command is usually aliased to {{Command{ rm -i }}} on most systems so you are prompted before deleting each file.
* Prefix your command with a \ (backslash) to suppress this alias expansion and execute {{Command{ rm }}} normally: {{Command{ \rm foo }}}
Remove an alias for the current login session: {{Command{unalias //alias//}}}
{{Command{which}}} and {{Command{type}}}
* These commands will display how each argument would be interpreted if executed as a command
* Aliases will be translated to their actual commands so you know what is really being executed
!!! Core shell options:
Stored in the {{Monospaced{$SHELLOPTS}}} variable
Manipulated with the set command
Enable a shell option: {{Command{set -o //option//}}}
Disable a shell option: {{Command{set +o //option//}}}
Examples:
Toggle command line input method between vi and emacs:
{{Command{set -o vi}}}
{{Command{set -o emacs}}}
Enable noclobber:
With noclobber enabled, an existing file will not be overwritten by redirecting STDOUT to a file
{{Command{set -o noclobber}}}
{{Command{set +o noclobber}}}
!!! Extra shell options:
{{Command{shopt}}} - Display a list of available options
''-s'' to enable an option
''-u'' to disable an option
Examples:
* {{Command{ shopt -s cdspell }}} - minor errors in the spelling of a directory component in a cd command are corrected.
* {{Command{ shopt -s checkjobs }}} - lists the status of any stopped and running jobs before exiting an interactive shell.
!!! Environment & Shell variables
In bash, variables are defined on the command line with this syntax: {{Command{variable=value}}}
By default all variables are local and will not be inherited by child processes
The {{Command{export}}} command will make a variable global and accessible to any child process
{{Command{export}}} can be used when defining a global variable. eg: {{Command{export foo=bar}}}
Or, can be used to elevate a currently defined variable to global. eg: {{Command{foo=bar ; export foo}}}
{{Command{set}}} will display all currently set variables
{{Command{unset}}} can be used to unset a variable
The shell environment can be manipulated through variables:
For example, the {{Monospaced{$PATH}}} and the prompt variable, {{Monospaced{$~PS1}}}:
The prompt:
* ~PS1 - Primary prompt string is stored in this variable
* Other secondary PS variables exist.
** See https://ss64.com/bash/syntax-prompt.html for more details.
** This site has a handy prompt generator: https://ezprompt.net/
Display your current prompt string: {{Command{ echo $~PS1 }}}
The last character in your prompt - {{Monospaced{ ''#'' }}} vs {{Monospaced{ ''$'' }}}
* {{Monospaced{ ''$'' }}} at the end of the prompt means the user is a regular, unprivileged user.
* {{Monospaced{ ''#'' }}} at the end of the prompt means the user is a superuser.
* This tagging makes it easier to see your privilege level.
Customized prompts I like for this class. This prompt makes it easier to see the full path to the current directory and show long command strings on the projector. The second version adds color.
{{{
PS1='\n[\u@\h \w] :\n\$ '
PS1='\n[\e[1;31m\u\e[m@\e[1;33m\h\e[m \w] :\n\$ '
}}}
Changing the ~PS1 variable by running one of the above commands applies the change immediately to your login session. It will be reset when a new shell executes. Add the change to your {{File{~/.bashrc}}} to make it permanent.
!!! Functions:
Functions can provide a shortcut to more complicated command sequences. They can be used in shell scripts or directly from the command line.
Append to your {{File{~/.bashrc}}}:
{{{
function bak() {
# This function creates a backup in the current working directory of any single file passed as an argument.
# Example: bak test.sh
cp "$@" "$@".`date +%y%m%d:%H%M`.bak
}
}}}
After adding this function to your {{File{~/.bashrc}}}, activate the new version by running {{Command{ . ~/.bashrc}}} or reloading the shell.
!!! History substitution:
* Your command history is saved in a buffer for the current login session
* By default, the buffer is appended to {{File{~/.bash_history}}} upon logout
* You can then display the current session's history buffer with the {{Command{history}}} command.
There are history configuration variables to change this behavior:
- {{Command{set | grep HIST}}}
! Assignment
* An extra credit environment / scripting lab is available - [[Lab C1|labs/labC1.pdf]]
** Be sure to use an uppercase {{Monospaced{''C''}}} as part of the lab number
** There is no firm due date for this lab. Please try to have it in by the end of November
Like most wikis, TiddlyWiki supports a range of simplified character formatting:
| !To get | !Type this |h
| ''Bold'' | {{{''Bold''}}} |
| ==Strikethrough== | {{{==Strikethrough==}}} |
| __Underline__ | {{{__Underline__}}} (that's two underline characters) |
| //Italic// | {{{//Italic//}}} |
| Superscript: 2^^3^^=8 | {{{2^^3^^=8}}} |
| Subscript: a~~ij~~ = -a~~ji~~ | {{{a~~ij~~ = -a~~ji~~}}} |
| @@highlight@@ | {{{@@highlight@@}}} |
| Tiddler Comments | {{{/%}}} text {{{%/}}}. |
| [[Make me a tiddler]] | {{{[[Make me a tiddler]]}}} |
| ~NoTiddler | {{{~NoTiddler}}} |
| {{{This is monotype}}} | {{{{{{This is monotype}}}}}} |
*sample:
|!th1111111111|!th2222222222|
|>| colspan |
| rowspan |left|
|~| right|
|bgcolor(#a0ffa0):colored| center |
|caption|c
For advanced effects, you can control the CSS style of a table by adding a row like this:
{{{
|cssClass|k
}}}
<<<
The highlight can also accept CSS syntax to directly style the text:
@@color:green;green coloured@@
@@background-color:#ff0000;color:#ffffff;red coloured@@
@@text-shadow:black 3px 3px 8px;font-size:18pt;display:block;margin:1em 1em 1em 1em;border:1px solid black;Access any CSS style@@
<<<
!!@@display:block;text-align:center;centered text@@
//For backwards compatibility, the following highlight syntax is also accepted://
{{{
@@bgcolor(#ff0000):color(#ffffff):red coloured@@
}}}
@@bgcolor(#ff0000):color(#ffffff):red coloured@@
/*{{{*/
@@color(yourcolorhere):colored text@@
@@color(fuchsia):colored text@@
@@bgcolor(yourcolorhere):your text here@@
[img[title|filename]]
[img[filename]]
[img[title|filename][link]]
[img[filename][link]]
[[text|url]]
[[Existing Tiddler Name|UglyTiddlerName]]
<<macro>>
<hr> = ----
*Entry One
**Sub-entry A
***Sub-sub-entry i
***Sub-sub-entry ii
**Sub-entry B
*Entry Two
*Entry Three
Use number signs (#'s) instead of asterisks for <OL type=1>
Tables:
|!Headings: add an exclamation point (!) right after the vertical bar.|!Heading2|!Heading3|
|Row 1, Column 1|Row 1, Column 2|Row 1, Column 3|
|>|>|Have one row span multiple columns by using a >|
|Have one column span multiple rows by using a ~|>| Use a space to right-align text in a cell|
|~|>| Enclose text in a cell with spaces to center it |
|>|>|bgcolor(green):Add color to a cell using bgcolor(yourcolorhere):|
|Add a caption by ending the table with a vertical bar followed by a c|c
!Header 1
!!Header 2
!!!Header 3
!!!!Header 4
!!!!!Header 5
Here's the code for a blockquote:
<<<
Here's the quoted text.
<<<
/*}}}*/
!Links
[[Calendar generator|http://zrenard.com/tiddlywiki/cal.php]]
Entities in HTML documents allow characters to be entered that can't easily be typed on an ordinary keyboard. They take the form of an ampersand (&), an identifying string, and a terminating semi-colon (;). There's a complete reference [[here|http://www.htmlhelp.com/reference/html40/entities/]]; some of the more common and useful ones are shown below. Also see [[Paul's Notepad|http://thepettersons.org/PaulsNotepad.html#GreekHtmlEntities%20HtmlEntitiesList%20LatinHtmlEntities%20MathHtmlEntities]] for a more complete list.
|>|>|>|>|>|>| !HTML Entities |
| &nbsp; | | no-break space | | &apos; | ' | single quote, apostrophe |
| &ndash; | – | en dash |~| &quot; | " | quotation mark |
| &mdash; | — | em dash |~| &prime; | ′ | prime; minutes; feet |
| &hellip; | … | horizontal ellipsis |~| &Prime; | ″ | double prime; seconds; inches |
| &copy; | © | Copyright symbol |~| &lsquo; | ‘ | left single quote |
| &reg; | ® | Registered symbol |~| &rsquo; | ’ | right single quote |
| &trade; | ™ | Trademark symbol |~| &ldquo; | “ | left double quote |
| &dagger; | † | dagger |~| &rdquo; | ” | right double quote |
| &Dagger; | ‡ | double dagger |~| &laquo; | « | left angle quote |
| &para; | ¶ | paragraph sign |~| &raquo; | » | right angle quote |
| &sect; | § | section sign |~| &times; | × | multiplication symbol |
| &uarr; | ↑ | up arrow |~| &darr; | ↓ | down arrow |
| &larr; | ← | left arrow |~| &rarr; | → | right arrow |
| &lArr; | ⇐ | double left arrow |~| &rArr; | ⇒ | double right arrow |
| &harr; | ↔ | left right arrow |~| &hArr; | ⇔ | double left right arrow |
The table below shows how accented characters can be built up by subsituting a base character into the various accent entities in place of the underscore ('_'):
|>|>|>|>|>|>|>|>|>|>|>|>|>|>|>|>|>| !Accented Characters |
| grave accent | &_grave; | À | à | È | è | Ì | ì | Ò | ò | Ù | ù | | | | | | |
| acute accent | &_acute; | Á | á | É | é | Í | í | Ó | ó | Ú | ú | | | Ý | ý | | |
| circumflex accent | &_circ; | Â | â | Ê | ê | Î | î | Ô | ô | Û | û | | | | | | |
| umlaut mark | &_uml; | Ä | ä | Ë | ë | Ï | ï | Ö | ö | Ü | ü | | | Ÿ | ÿ | | |
| tilde | &_tilde; | Ã | ã | | | | | Õ | õ | | | Ñ | ñ | | | | |
| ring | &_ring; | Å | å | | | | | | | | | | | | | |
| slash | &_slash; | | | | | | | Ø | ø | | | | | | | |
| cedilla | &_cedil; | | | | | | | | | | | | | | | Ç | ç |
<HTML><a href="http://checkettsweb.com/#%5B%5BCSS-Colors%20and%20Backgrounds%5D%5D%20%5B%5BCSS-Text%20and%20Fonts%5D%5D%20OldStyleSheet%20Rin%20%5B%5BTiddlyWiki%20Structure%5D%5D">CSS Info</a></html>
<<version>>
[[Plugins]]
[[Styles]]
[[TagglyTagging]]
[[systemConfig]]
[[systemTiddler]]
[[excludeSearch]]
[[excludeLists]]
! Material
This page will discuss two topics:
1. Authenticating to Unix systems with SSH keys
2. Terminal multiplexing with GNU {{Command{screen}}}
These are both optional, but good to know and will make working with our lab systems much easier.
!! 1. Authenticating to Unix systems with SSH keys
Two mechanisms exist for SSH authentication:
# normal passwords
# key pairs used in asymmetric encryption
A key pair contains a private key that you keep secure and a public key that is distributed to the systems you have permission to connect to. The private key you have is used to establish your identity. The presence of your public key on a remote system is used to establish your authorization to access it. Private keys should be secured with a passphrase to ensure they cannot be maliciously used if they are captured by an attacker. SSH authentication with passphrase-protected key pairs is much safer than passwords, since now an attacker must also capture the private key file in order to impersonate you. For this reason, it is common to minimally block password authentication to a server when logging in as root or ideally only allow key authentication for all users. More sensitive systems should require key-based authentication as part of general system hardening.
Forcing key-based authentication gives us multi-factor authentication (MFA) when the key is properly secured with a passphrase:
# Something you have (the private key)
# Something you know (the key's passphrase)
We begin by creating a SSH keypair on the class shell server.
{{Command{cd ~/.ssh/}}}
<<<
Change to the ~/.ssh/ directory, the default location for a user's ssh configuration files.
<<<
{{Command{ssh-keygen -t ed25519 -f ncs205 }}}
<<<
Create a SSH key pair using default settings, except for changing the key type to ed25519 and naming the key ncs205. The algorithm and key size can also be adjusted via flags. The remaining defaults are reasonable. You will be prompted to set a passphrase. Choose something secure which you can remember. This [[xkcd cartoon|https://xkcd.com/936/]] may be helpful. The more entropy the better.
<<<
{{Command{ssh-copy-id -i ncs205 root@192.168.12.''//x//''}}}
<<<
Copy your public key to each of your ~VMs. It will be saved to the file {{File{~/.ssh/authorized_keys}}} on the remote system (your VM). The administrator may have to add the key for you on systems you're not able to log into yet.
<<<
{{Command{ssh -l root 192.168.12.''//x//''}}}
<<<
Try to connect to your test VM. You should be prompted for a password since our private key is not in the default location and was not specified on the command line.
<<<
{{Command{ssh -i ncs205 -l root 192.168.12.''//x//''}}}
<<<
You should now be prompted for your SSH passphrase instead of password. If an available and authorized SSH key is found it will be offered for use instead of your password. Authentication will fall back to regular password if key-based fails.
<<<
{{Command{exit}}}
<<<
Disconnect from your VM
<<<
Having to specify the username and key file to use for each login to your ~VMs can be eliminated by using a ssh client configureation. Edit {{File{~/.ssh/config}}} on the shell server and set a default username and ssh key for class ~VMs
Edit the file {{File{~/.ssh/config}}} and add the following:
{{{
Host test
HostName 192.168.12.x
Host www
HostName 192.168.12.x
Host *
IdentityFile ~/.ssh/ncs205
User root
}}}
Be sure to change the x above to your actual IP address. This addition will also eliminate the need for specifying full IP addresses for each connection. You'll be able to then connect with just {{Command{ssh //hostname//}}} and the IP address, user, and key file will be added for you. Add new ~VMs to the config as they are issued to you.
!!! SSH agent - Unlock your key once for multiple connections
The SSH agent is a keyring which your SSH private keys can be attached to. Once set up, future connections will look to that key ring when an authentication request is made instead of prompting you for your SSH passphrase each time. The idea is one authentication event for many remote connections.
{{Command{ssh-agent > ~/.ssh/env}}}
<<<
Create a SSH agent, saving the environment information to the specified file. This environment must be imported in order to make use of the agent.
<<<
{{Command{eval `cat ~/.ssh/env`}}}
<<<
Import the environment settings into the current shell environment
<<<
{{Command{ssh-add ~/.ssh/ncs205}}}
<<<
Add your ncs205 private key to your ssh agent keyring. You should be prompted for its passphrase.
<<<
Once the SSH agent is established you may communicate to your lab systems without being prompted to authenticate each time. Notice the lack of passphrase prompts:
{{Commands{
[merantn@shell ~]$ ''ssh test''
Last login: Mon Oct 19 15:18:26 2020 from 192.168.12.10
[root@test ~]# ''exit''
logout
Connection to 192.168.12.24 closed.
[merantn@shell ~]$ ''ssh www''
Last login: Mon Oct 19 15:19:51 2020 from 192.168.12.10
[root@www ~]# ''exit''
logout
Connection to 192.168.12.25 closed.
}}}
!! 2. Terminal multiplexing with GNU screen
GNU {{Command{screen}}} is a very useful tool for those working with the command line on many systems from different locations on a daily basis. From within {{Command{screen}}}, connections can be made to many systems. The user can detach from the screen session, change physical locations, and reconnect to their screen session continuing work where they left off. GNU {{Command{screen}}} and ssh agents make a great combination for connecting to multiple machines over the course of your work day.
This video might help get you started: https://www.youtube.com/watch?v=Mw6QvsChxo4
{{Command{cp ~merantn/.screenrc ~/}}}
<<<
Copy this default screen configuration file to your home directory. This will establish some baseline settings.
<<<
If you first run the steps in Section 1 to set up ssh-agent and then launch {{Command{screen}}} to start your screen instance, your SSH Agent will be established for all screen windows. You thus will not need to authenticate to your ~VMs as you move between them. You will only need to run the {{Command{screen}}} command without any options once. It will stay active with your tasks running in the background until you either terminate it or the class shell server restarts.
Screen commands:
| !Key Sequence | !Action |
| ~CTRL-a , 0 |Switch to window 0|
| ~CTRL-a , 1 |Switch to window 1|
| ~CTRL-a , 2 |Switch to window 2|
| ~CTRL-a , //n// |Switch to window //n//|
| ~CTRL-a , c |Create a new screen window|
| ~CTRL-a , " |Display available screen windows|
| ~CTRL-a , ' |Switch to a screen window by number|
| ~CTRL-a , A |Title the current screen window|
| ~CTRL-a , ? |Display screen help|
With screen now running, enter these screen commands to get things set up:
* Create a new window: {{Command{~CTRL-a, ~CTRL-c}}}
* Switch to window 1: {{Command{~CTRL-a, 1}}}
** Connect to your test VM with ssh
* Create a new window: {{Command{~CTRL-a, ~CTRL-c}}}
* Switch to window 2: {{Command{~CTRL-a, 2}}}
** Connect to your www VM with ssh
* Switch to window 0: {{Command{~CTRL-a, 0}}}
** Use this window to work on the class shell server
* Detach from screen (as if you're done working for the day): {{Command{~CTRL-a, d}}}
* Reconnect to your screen session (as though you're coming back later to continue work): {{Command{screen -dr}}}
Now, when you disconnect from the shell server, all of your tasks will stay running in the background. Log in again and run {{Command{screen -dr}}} to continue where you left off. Create new windows inside of screen as you need them for new ~VMs or to run additional tasks concurrently.
More screen commands:
| !Key Sequence | !Action |
| ~CTRL-a , | |Split window vertical|
| ~CTRL-a , S |Split window horizontal|
| ~CTRL-a , TAB |Switch between split windows|
| ~CTRL-a , X |Close a split window|
|>|>|
| ~CTRL-a , d |Detach from screen|
| ~CTRL-a , :password |Set a password for your screen session|
{{Note{[[This video|SSH]] may be a helpful demonstration}}}
3. Defeating firewalls with SSH to access protected resources
See the [[Tunnels & Proxies with SSH]] page.
/%
!! 1. Authenticating to Unix systems with SSH keys
Two different sets of keys are used with SSH: one for securing communication between the client and server and, optionally, a set to authenticate remote users.
!!! SSH Host keys
* Public key crypto is used for encrypting communication between client and server
* Server keys are stored in the files {{File{/etc/ssh/ssh_host_*}}}
* Fingerprints for new systems are shown and stored in the user's {{File{~/.ssh/known_hosts}}} file. This keeps a record of trusted systems.
** This file can leak identities of systems you are communicating with
** Hash your current known hosts file if you'd like to mask the systems: {{Command{ ssh-keygen -H }}}
* Fingerprints for known systems are compared on each login to identify MITM attacks
** The user is alerted if a mismatch is found
*** This is the warning you see if you connect to a new system for the first time or there's a server change when connecting to an existing system.
** The user should take steps to verify the host key has legitimately changed. If this change is due to a MITM attack, the attacker could capture your credentials
** Display the fingerprint of a SSH public key: {{Command{ssh-keygen -lf //file//.pub}}}
!!!! Demo:
{{Monospaced{
[merantn@shell ~]$ ''ssh head.ncs205.net''
The authenticity of host 'head.ncs205.net (192.168.12.15)' can't be established.
ECDSA key fingerprint is ~SHA256:bHKouQIItQNr5r1Im3tI0uk2ArpfYU1Yvop0SQhOLVY.
ECDSA key fingerprint is ~MD5:9f:0d:9c:2d:f6:2c:ef:9e:6a:bb:ab:e5:4b:c5:55:e4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'head.ncs205.net' (ECDSA) to the list of known hosts.
# You can't log into this system, so press ~CTRL-C to abort:
merantn@head.ncs205.net's password:
# Here's the fingerprint of this system:
[merantn@shell ~]$ ''grep head ~/.ssh/known_hosts''
head.ncs205.net ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhZIx/NElfvUL0nI/KwOotqk5Fypf01LQpn8YIe7FfXI8xnwEzESmqZTOiC791SrvOaoIxIFu9WW9xO7+BcgSw=
# Hash the hosts in the file:
[merantn@shell ~]$ ''ssh-keygen -H''
/home/merantn/.ssh/known_hosts updated.
Original contents retained as /home/merantn/.ssh/known_hosts.old
WARNING: /home/merantn/.ssh/known_hosts.old contains unhashed entries
Delete this file to ensure privacy of hostnames
Now a grep returns no results:
[merantn@shell ~]$ ''grep head ~/.ssh/known_hosts''
}}}
%/
! Additional Details
This video is a deep dive into SSH and has a lot of great info: https://www.youtube.com/watch?v=fnkG9_jy2qc
! Wrapping up
!! Closing out the semester
!!! VM deletion
Our lab environment for this class will be decommissioned on ''Saturday, Dec 28''. If there is anything you would like to complete or back up, please do so by then. Please let me know if there's anything you need help saving.
!!! Additional Resources
!!!! A lab environment similar to ours can easily be replicated from open source tools:
* [[Proxmox|https://www.proxmox.com/en/]]: The hypervisor our ~VMs are running on. Works great on a spare server or PC that's kicking around.
* [[Naemon|https://www.naemon.org/]]: Infrastructure monitoring
* [[SaltStack|https://www.saltstack.com/]]: Infrastructure Management & Orchestration - I used this to easily run commands on all class ~VMs and maintain baseline configurations
* [[NameCheap|https://www.namecheap.com]]: Domain Registration - A simple, clean interface and free domain privacy.
* [[DigitalOcean|https://www.digitalocean.com/]]: Low cost cloud ~VMs - I use these for my personal infrastructure. Good Linux ~VMs for $6 per month.
* [[Hetzner|https://www.hetzner.com/sb?country=us]]: Low-cost bare metal cloud servers. This is the hosting provider for our current class lab environment.
!!! Class website mirror
The entire class website runs from a single HTML file. A zip file containing the HTML file along with linked images, videos, and lab ~PDFs can be downloaded from https://www.ncs205.net/ncs205.zip
* Last updated 12/17/24 @ 20:45
/% - The link will be live once finals end - %/
/%!!! Feedback
I hope everyone enjoyed this class and got something useful from it. The material I included is the highlights of what you'll need to be exposed to if you'll be using Linux in and beyond the NCS program. If you have any feedback to offer, good or bad, please let me know. I'm always looking for ways to improve the class for the next semester. %/
Type the text for 'excludeLists'
Type the text for 'excludeSearch'
!!!! [[hack6 break-in]]
A VM named ''hack6'' has just been added for each of you. It's presently powered on and assigned to the last IP address of your range via DHCP.
This VM has information which can be found and vulnerabilities which can be exploited to gain access. All required knowledge is based on our class material. Information disclosure will yield the first three flags and a vulnerability can be exploited to ultimately obtain root-level access. There's flags to capture as your intrusion progresses to show you have increasingly gained access and to show how far you were able to get.
Access hack6 for a little CTF (capture the flag). These CTF challenges are a fun way to demonstrate skill, creativity, and understanding of the material.
* This VM is currently running on the last IP address of your range
** Create an A record so host name hack6 points to this IP address
* The ultimate objective is to break into the system and gain full root privileges.
* Flags to capture are in the following files:
** {{File{flag1.txt}}}
** {{File{flag2.txt}}}
** {{File{/home///user///flag3.txt}}}
** {{File{/root/flag4.txt}}}
** //user// is a placeholder for a valid user on the system. Flag 3 is in one of the system user's home directories.
** The "flag" to capture is the string of text located inside each of those four files. It will look something like this: {{Monospaced{ ''flag{//sometext//}'' }}}
What's the highest number flag you can access? Capturing all four shows you have root-level access and have fully taken over the system.
The first two flags can be obtained by interacting with services running on the VM. The second two flags can be obtained after gaining shell access to the VM.
* Don't overthink it; the flags (especially the first three) only require basic interaction with the system
* Everyone seems to first gravitate towards brute-force tactics. This is the path of the unskilled. These are actually rarely successful when targeting a specific system or account and will not help you here.
This challenge will primarily draw from material covered [[Week 15, Part 2]] (Access control and user management), [[Week 15, Part 1]] (Scheduled Tasks), and Section 1 of [[Working more efficiently with GNU screen & SSH keys]] (Authenticating to Unix systems with SSH keys)
* And will require a bit of creativity
* Linux Administration Chapter 23 also contains a lot of good info on SSH
* The {{Command{nmap}}} command can be used to scan a system to identify listening services. The basic syntax is {{Command{nmap //ip_address//}}}.
** You will need to interact with those services to obtain shell access. You will not see this VM in the Proxmox UI and do not have a user account's password to log in directly.
Most system intrusions take advantage of misconfiguration to gain a higher level of access. Developing an understanding of how things work is necessary for a defender to properly configure (and thus secure) their systems. Understanding how things work also makes it easier for an attacker to exploit any misconfigurations.
The first two flags will be found through basic system discovery & information disclosure. Flag 2 will be accompanied by your key into the VM. The final flag and full system compromise will be obtained by exploiting a system misconfiguration.
The path to full system compromise is linear. All flags will need to be obtained in order. There is only one route to obtain the first three flags. There are two different ways to obtain the fourth. Can you find both misconfigurations which will grant root access?
!!!! Grading
The Final Exam will be worth 20 points. The point breakdown for the flags and responses is:
* Flags 1 & 2: 5 points each
* Flag 3: 2 points
* Flag 4: 3 points
* Page 3 question: 5 points
Standard rules for lab assignments apply. The deliverable PDF must contain your name at the top of the first page and must be properly submitted; it must be uploaded to the correct directory on the class shell server and given the correct filename. These skills were part of the material this semester and are thus in scope for the Final exam content. No points will be awarded if these requirements are not completed correctly. The grading queue is visible and can be used as verification that the deliverable PDF was properly collected for evaluation.
!!!! Deliverable:
Complete and submit [[Final Exam|exam/ncs205-final.pdf]] with your steps and flag contents. Upload the PDF to the class shell server to the directory {{File{/opt/pub/ncs205/submit/final/}}}. The file name must be {{File{ncs205-final-//username//.pdf}}}.
@@ ''The final exam write-up will be due by EOD Thursday, May 2'' @@
Consider pacing this out over a longer period of time. With tasks like this, it's sometimes helpful to step away for a bit if you get stuck. That'll give you some time to think about it in the background or come back later with a fresher set of eyes and perspective.
!! less
| !Command | !Action |
| Page Up or ''b'' | Scroll back one page |
| Page Down or space | Scroll forward one page |
| Up arrow | Scroll up one line |
| Down arrow | Scroll down one line |
| ''G'' | Move to the end of the text file |
| ''1G'' or ''g'' | Move to the beginning of the text file |
| /characters | Search forward to the next occurrence of //characters// |
| ''n'' | Search for the next occurrence of the previous search |
| ''h'' | Display help screen |
| ''q'' | Quit less |
config.macros.listTags = { text: "Hello" };
config.macros.listTags.handler = function(place,macroName,params)
{
var tagged = store.getTaggedTiddlers(params[0],params[1]);
//<< Second parameter is field to sort by (eg, title, modified, modifier or text)
var ul = createTiddlyElement(place,"ul",null,null,"");
for(var r=0;r<tagged.length;r++)
{
var li = createTiddlyElement(ul,"li",null,null,"");
createTiddlyLink(li,tagged[r].title,true);
}
}
/***
|''Name:''|Plugin setDefaults|
|''Version:''|1.0.1 (2006-03-16)|
|''Source:''|http://tiddlywikitips.com/#%5B%5BPlugin%20setDefaults%5D%5D|
|''Author:''|Jim Barr (jim [at] barr [dot] net)|
|''Licence:''|[[BSD open source license]]|
|''TiddlyWiki:''|2.0|
|''Browser:''|Firefox 1.0.4+; Firefox 1.5; InternetExplorer 6.0|
!Description
These settings simply set "default" values for several system features and Plugins.
***/
/***
Standard settings:
***/
//{{{
config.options.chkRegExpSearch = false; // default false
config.options.chkCaseSensitiveSearch = false; // default false
config.options.chkAnimate = false; // default true
//config.options.txtUserName = "Nick"; // default "YourName"
config.options.chkSaveBackups = false; // default true
config.options.chkAutoSave = false; // default false
config.options.chkGenerateAnRssFeed = false; // default false
config.options.chkSaveEmptyTemplate = false; // default false
config.options.chkOpenInNewWindow = true; // default true
config.options.chkToggleLinks = false; // default false
config.options.chkHttpReadOnly = true; // default true
config.options.chkForceMinorUpdate = false; // default false
config.options.chkConfirmDelete = true; // default true
config.options.txtBackupFolder = ""; // default ""
config.options.txtMainTab = "tabTimeline"; // default "tabTimeline"
config.options.txtMoreTab = "moreTabAll"; // default "moreTabAll"
config.options.txtMaxEditRows = "30"; // default "30"
config.options.chkInsertTabs = true; // tab inserts a tab when editing a tiddler
//}}}
/***
Custom Plugin settings:
***/
//{{{
config.options.chkSinglePageMode = false; // default "true"
config.options.chkSearchTitlesFirst = true;
config.options.chkSearchList = true; // default "false"
config.messages.messageClose.text = "X"; // default "close"
// config.views.wikified.defaultText = ""; // default "The tiddler '%0' doesn't yet exist. Double-click to create it"
config.options.chkStepWiseNavigationOn = true; // default "false"
config.options.chkDisableAutoSelect =true;
config.options.chkTextAreaExtensions =true;
//}}}
Type the text for 'systemConfig'
Type the text for 'systemTiddler'