Home

bits of code

  • pretify json via commandline
    cat myfile.json | python -m json.tool
  • Start Unetbootin in debian. change sudo to su accordingly.
    sudo export QT_X11_NO_MITSHM=1 unetbootin

    archlinux forum
  • Reset git HEAD
    git reset HEAD~1
  • Include full path in the output of find command for current directory
    find -name "filename" -exec readlink -f {} \;
  • Check progress of dd command while writing to a drive. Very useful while creating bootable pen drives.

    Create bootable drive.
    dd if=[iso path] of=[device] bs=4


    While the above command is running open another terminal and give this command and check output in previous terminal
    watch -n5 'sudo kill -USR1 $(pgrep ^dd)'
  • To encrypt file or folder from cli. make sure you have gpg and tar
    tar zcvf - [file1|folder1] [file2|folder2] | gpg -c > myfiles-backup.tar.gz.gpg

    this will ask for passphrase.

    and to decrypt
    gpg myfiles-backup.tar.gz.gpg

    For more details
  • How to come out of ssh connections that got closed.
    Enter, ~, .
     type these on the terminal whose connection is lost.
  • Play music from a folder and sub folders.
    mplayer -playlist <(find "$PWD" -name "*.mp3" -type f)
  • One command to backup whole system
    rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder

    for more details check this link
    Arch Linux Page
  • Adjust recording time for gnome default recorder which is activated with CTRL+ALT+SHIFT+R and to stop use the same combination before max recording time is reached or recording will stop automatically reaching the max time.
    gsettings set org.gnome.settings-daemon.plugins.media-keys max-screencast-length 360

    360 sec
  • Extract audio from video file
    ffmpeg -i sample.avi -q:a 0 -map a sample.mp3
  • List all public_html folder with long list in home directory
    find . -maxdepth 1 -type d -exec ls -l {} + | grep 'public_html' | grep 'drw'
  • List files based on extension, date and copy them to a directory
    find . -type f -newermt 2018-04-18 ! -newermt 2018-04-19 | grep php | xargs -P 2 -I _FILE_ cp _FILE_ [foldername]/
  • Sort unique values and get word with length between 8 and 63
    cat [textfile] | sort | uniq | pw-inspector -m 8 -M 63 > [textfile]
  • Exclude folders from compression with tar. Make sure to remove trailing ‘/’
    tar -zcv --exclude='[folder]' -f [backup.tgz] [dir_to_backup]
  • find zombie processes
    ps xao pid,ppid,pgid,uname,stat,comm | awk '$5=="Zs"'
  • Find size of directories from the current path and sort them in decending order with highest first.
    du -sch * | sort -h -r
  • While doing a full disk encrypt after partitioning and creating filesystem change the ownership of the mounted folder to make it writeable by normal users.
    How to create encrypted disk
  • Command to get hardware information in the system
    inxi -Fxz

Configuring Thunderbird 91 with remote Calendar AddressBook (CalDEV, CardDEV)

This setup is straight forward with a couple of add-ons to sync Calendars, AddressBooks can be synchronized. After setting up email and syncing mails Follow these steps to sync Calendars and AddressBooks.

Step 1

Install these Addons, TbSync and Provider for CalDAV & CardDAV which is a dependency for TbSync.

Step 2

Go to Account actions > Add new Account > CalDAV & CardDAV it will show this menu.

Step 3

Configure the account.

Step 4

Now Select sync to sync the AddressBook and Calendar. After that select the Calendars and AddressBooks that you want to see in thunderbird.

Xrandr adding custom display resolution

# First we need to get the modeline string for xrandr
# Luckily, the tool "gtf" will help you calculate it.
# All you have to do is to pass the resolution & the-
# refresh-rate as the command parameters:
gtf 1920 1080 60

# In this case, the horizontal resolution is 1920px the
# vertical resolution is 1080px & refresh-rate is 60Hz.
# IMPORTANT: BE SURE THE MONITOR SUPPORTS THE RESOLUTION

# Typically, it outputs a line starting with "Modeline"
# e.g. "1920x1080_60.00"  172.80  1920 2040 2248 2576  1080 1081 1084 1118  -HSync +Vsync
# Copy this entire string (except for the starting "Modeline")

# Now, use "xrandr" to make the system recognize a new
# display mode. Pass the copied string as the parameter
# to the --newmode option:
xrandr --newmode "1920x1080_60.00"  172.80  1920 2040 2248 2576  1080 1081 1084 1118  -HSync +Vsync

# Well, the string within the quotes is the nick/alias
# of the display mode - you can as well pass something
# as "MyAwesomeHDResolution". But, careful! :-|

# Then all you have to do is to add the new mode to the
# display you want to apply, like this:
xrandr --addmode VGA1 "1920x1080_60.00"

# VGA1 is the display name, it might differ for you.
# Run "xrandr" without any parameters to be sure.
# The last parameter is the mode-alias/name which
# you've set in the previous command (--newmode)

# It should add the new mode to the display & apply it.
# Usually unlikely, but if it doesn't apply automatically
# then force it with this command:
xrandr --output VGA1 --mode "1920x1080_60.00"

https://unix.stackexchange.com/questions/227876/how-to-set-custom-resolution-using-xrandr-when-the-resolution-is-not-available-i

Generate self-signed certificate with openssl

To generate self-signed certificate for a quick test or for development. If you are looking for getting certificate for production site then I strongly suggest to go for a proper certificate like LetsEncrypt. You can get it easily with certbot.

To generate a self-signed SSL certificate using the OpenSSL, complete the following steps:

  1. Write down the Common Name (CN) for your SSL Certificate. The CN is the fully qualified name for the system that uses the certificate. If you are using Dynamic DNS, your CN should have a wild-card, for example: *.api.com. Otherwise, use the hostname or IP address set in your Gateway Cluster (for example. 192.16.183.131 or dp1.acme.com).
  2. Run the following OpenSSL command to generate your private key and public certificate. Answer the questions and enter the Common Name when prompted.
    openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
  3. Review the created certificate:
    openssl x509 -text -noout -in certificate.pem
  4. Combine your key and certificate in a PKCS#12 (P12) bundle:
    openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12
  5. Validate your P2 file.
    openssl pkcs12 -in certificate.p12 -noout -info
  6. For further tests you can give these commands. All of these commands are optional
    openssl pkey -in privateKey.key -pubout -outform pem | sha256sum
    openssl x509 -in certificate.crt -pubkey -noout -outform pem | sha256sum
    openssl req -in CSR.csr -pubkey -noout -outform pem | sha256sum

From step 4 all the steps are optional.

Source for these command got from ibm. Not a great source will change it in the future.

Apache reverse proxy to forward requests on a custom port to a local system

This configuration of apache is a little specific to forwarding all requests coming to a custom port on the host system to a local system which maybe running a service on custom port.

Let us say we have 2 systems. System A and B where A can be accessed from the internet on lets say port 8084 and system B which is in the local system is hosting a service on port 8700. Now if someone from outside wants to access B:8700 then they can not do it directly unless we find a way to redirect requests to it from the public facing system A. Apache reverse proxy comes handy to do the job and it is relatively easy to configure.

The following setup is tested on debian bullseye with apache 2.4. The same may work on other systems to as it is fairly simple to setup.

On system A

First make sure apache is installed and running if not install it with apt-get install apache2. Then follow these steps.

  1. Enable these modules for reverse proxy. use the command a2enmod. Some of the modules may not be needed I just enabled them following a guide.
    1. proxy
    2. proxy_http
    3. proxy_ajp
    4. rewrite
    5. deflate
    6. headers
    7. proxy_balancer
    8. proxy_connect
    9. proxy_html

Restart apache systemctl restart apache2

2. Enable custom port in /etc/apache2/ports.conf

Listen 80
Listen 8084 <This is added in the file>

3. Now create a file for virtualhost in /etc/apache2/sites-available/proxy.conf and add these lines. You may change the ports and local ip according to your setup.

<VirtualHost *:8084>
       ProxyPreserveHost On
       ProxyPass / http://192.168.1.3:8700/
       ProxyPassReverse / http://192.168.1.3:8700/

       ErrorLog ${APACHE_LOG_DIR}/error.log
       CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Restart apache and test with public IP:8084. the traffic should be going to system B which is at 192.168.3:8700.

I took help from these sources

https://www.digitalocean.com/community/tutorials/how-to-use-apache-http-server-as-reverse-proxy-using-mod_proxy-extension

https://www.mirantis.com/blog/quick-tip-use-apache-as-a-proxy-server-to-access-internal-ips-from-an-external-machine/

Extract data from postgres in docker and save it to csv

The case is simple postgres database is hosted in a docker image and to access the database use this command. docker image name can be found with this command docker ps -a.

docker exec -it <docker-image> psql -U postgres

docker exec -it <docker-image> psql -U postgres -a <dbname> -c "copy(select u.name user_name,p.name place_name, attempts,p.created from
place p inner join users u on u.id=p.user_id order by attempts desc) to stdout with csv header" > data-list.csv

Breaking down the command u.name user_name is the field in user table with alias of user_name. p.name place_name is the field in place table with alias of place_name, r.created is the field in place table. place p is naming place table with ‘p’ and inner join joins the user id in both the tables.

Moodle migration

Moodle migration from one server to another while upgrading to a new moodle version

Enable maintenance mode before doing anything. This will make sure that there is no data lose and also inform users that the site will not be available. Plan the migration steps before starting the migration which will speed up the process and also reduce the down time. For future reference these are the steps that have to be followed.

Change DNS Nameservers

If you are hosting in a vm and changed the IP update the ip pointing to the server. Later on when configuring certbot it is necessary to have the new IP for the domain.

Preparing the old System

We have to backup these folders and database. Change the paths according to where you have installed moodle. In this case moodle in installed as a default portal for the system in /var/www/html, moodledata can anywhere this is just an example.

  • /var/www/html
  • /var/www/moodledata
  • database.sql Create a folder to copy all of them to one location. In this example we will create a folder called backup and copy all the files and folders to it. ‘~/’ is for home directory of the user in this case it is root.
  mkdir ~/backup/
  rsync -avz --progress --numeric-ids /var/www/html ~/backup
  rsync -avz --progress --numeric-ids /var/www/moodledata ~/backup
  mysqldump -u <dbuser> -p <dbname> > database.sql

for mysqldump it will ask for a password give the password of the database used in config.php file

Preparing the new System

The following setup is for Debian 10 adjust package names and other options accordingly to a different OS. Install all the requirments as suggested by moodle. For convenience use this command which will install all the packages
apt install apache2 mariadb-server php php-mysql php-mbstring php-curl php-tokenizer php-xmlrpc php-soap php-common php-zip php-gd php-xml php-intl php-json

Now transfer the files from old system to the new one

  mkdir ~/imported/
  rsync -avz --progress --numeric-ids user:OLDSERVER-IP:backup imported

Create mariadb database and user then import data to the new db. login to mariadb prompt by given the command ‘mysql’.

  CREATE DATABASE databasename;
  GRANT ALL PRIVILEGES ON databasename.* TO "username"@"hostname" IDENTIFIED BY "password";
  FLUSH PRIVILEGES;

Import database
mysql -u <dbuser> -p <dbname < database.sql

Move moodledata to new location

  cp -r ~/imported/moodledata /var/www/
  chown -R www-data.www-data /var/www/html

Download or clone the latest version from git repo, follow this guide into /var/www/html. At this point you may need to copy some plugins from old installation which you can do once you have started the upgrade. The upgrade page will show what are the missing plugins and you can copy them from back to the new instance.

Update config.php to reflect new paths and in case of new database add those changes.
example config.php

For updating mysql to give 4 byte support follow this link.

  <?php  // Moodle configuration file

  unset($CFG);
  global $CFG;
  $CFG = new stdClass();

  $CFG->dbtype    = 'mariadb';
  //$CFG->dbtype    = 'mysqli';
  $CFG->dblibrary = 'native';
  $CFG->dbhost    = 'localhost';
  $CFG->dbname    = '<dbname>';
  $CFG->dbuser    = '<dbuser>';
  $CFG->dbpass    = '<dbpass>';
  $CFG->prefix    = 'mdl_';
  $CFG->dboptions = array (
    'dbpersist' => 0,
    'dbport' => '',
    'dbsocket' => '',
    'dbcollation' => 'utf8mb4_unicode_ci', #<-- this is optional for storing emoji's in db. to get it check mysql configuration and update it.
  );

  $CFG->wwwroot   = '<url>'; #<-- This url is important that what ever we give here either url or ip is used to access the portal
  $CFG->dataroot  = '/var/www/moodledata'; #<-- Change this if data is in a different location
  $CFG->admin     = 'admin';

  $CFG->directorypermissions = 0777;

  require_once(dirname(__FILE__) . '/lib/setup.php');

  // There is no php closing tag in this file,
  // it is intentional because it prevents trailing whitespace problems!
  ?>

Upgrade the server from web interface

Once all the files are in place open site url. The site will show upgrade page. Once you continue it will show all the plugins that will be upgraded, installed, deleted. The once which are shown ‘missing from disk’ copy them from the back to the new instance and reload. Once the upgrade is done disable maintenance mode.

For reference these where missing files and plugins from the instance that we where upgrading. ~/imported being the root of moodle.

~/imported/learning/mod/attendance/
~/imported/learning/filter/wiris/
~/imported/learning/lib/editor/atto/plugins/wiris/ atto/plugins/
~/imported/learning/lib/editor/tinymce/plugins/tiny_mce_wiris/
~/imported/learning/auth/a2fa/
~/imported/learning/user/profile/field/afaqr/
~/imported/learning/report/benchmark/
chown -R www-data.www-data /var/www/html

Reset password in matrix synapse

To reset the password of matrix user in debian

After login as root. Change the user to postgres

#su postgres

At the postgres prompt connect to synapse database and update users table with new hash. generate the hash with hash_password command

$hash_password
Password: <hidden>
Confirm password: <hidden>
$2b$12$eGgHG99FQ1O9Q8jSnwOJ2e88tND7nznMctJcASV.Is655Hnr7ZFtG <example output>
$psql
postgres=# \c synapsedb;
synapsedb=# select name from users where name = '@<username>:<domain>';
[This will show if the user exists]
synapsedb=# UPDATE users SET password_hash = '$2b$12$eGgHG99FQ1O9Q8jSnwOJ2e88tND7nznMctJcASV.Is655Hnr7ZFtG WHERE name='@<username>:<domain>';'

Setup a VPN tunnel with wireguard in debian

Updated on [09/10/2019]

Wireguard is a new VPN (Rather than calling it a VPN its better to call it tunnel to a different system) and is experimental but much faster and very sleek. It is very straight forward and will not take more than 10 min to setup the whole thing. But if you want to tweak around it will take time. This is way better than setting up openVPN or ipsec.

1. Server setup

Install wireguard in both server and client. The package is currently in unstable, developers are working really hard to include this into mainline kernel. Very soon we will see wireguard included in mainline kernel.

Step 1. Generate keys

Create a directory say something like wg and from terminal go to that directory

mkdir wg
cd wg

Then generate the keys
wg genkey | tee privkey | wg pubkey > pubkey

secure the keys and change the permissions on privkey file to 600.
chmod 600 privkey

Step 2. Create conf file

Create a file in /etc/wireguard/wg0.conf. wg0 will become the interface. any name can be given to the file and that name will become the network device. add the following to the conf file.

[Interface]
PrivateKey = (Output of privatekey file that contains server private key)
Address = 10.0.0.1/24, fd86:ea04:1115::1/64
ListenPort = 60550 (This is not a standard port you can choose any port number above 10000 which is also not mandatory)
SaveConfig = true

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

Step 3. Enable packet forwarding

nano /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

sysctl -p

Step 4. Enable firewall

It is a good practice to enable firewall. In this case we are using ufw. Install ufw with apt install ufw, Enable the port we configured in the conf file.
ufw allow 60550/udp
ufw allow 22/tcp
ufw enable

check the status
ufw status

Step 5. Start the server

wg-quick up wg0
systemctl enable wg-quick@wg0

This is start the interface on reboot

Step 6. Add clients to conf file

Add all the clients (peers) which are allowed to connect to the server by adding their details to wg0.conf file.

[Interface]
PrivateKey = (Output of privatekey file that contains server private key)
Address = 10.0.0.1/24, fd86:ea04:1115::1/64

[Peer]
PublicKey = (Clients public key)
AllowedIPs = 10.0.0.2/32 (clients IP address as configured by client)

Another way of adding clients is from cli and later these settings can be saved to the config file.

wg set <server wireguard network interface> peer <public key of the client> allowed-ips <ip assigned to the client>

2. Client setup

Step 1. Generate the keys

Generate keys and secure them they way we created for server.

Step 2. create config file

Create a config file in /etc/wireguard/wgc.conf. Give any name to the file and that name will become device name.
[Interface]
PrivateKey = (Private key of the client)
Address = 10.0.0.2/24, fd86:ea04:1115::5/64

[Peer]
PublicKey = (Server's public key from *wg show* command on server)
Endpoint = (Linux server's Public IP):60550
AllowedIPs = 0.0.0.0/0, ::/0

Start the interface

wg-quick up wgc

Now you are connected to check the interface. Check the connect on both ends with wg command.

Took some help from.
setup wireguard
Debian Wiki