0%

Delete Useless AAD App With PowerShell Script

A powershell script used to delete useless AAD app under your account.

Prerequsite

Usage

  1. Create a deleteAADAapp.ps1 file.

    1
    2
    3
    4
    5
    6
    7
    8
    az ad app list --filter "startswith(displayName, 'dilin')" --query '[].appId' > rawAppIds.log

    (Get-Content rawAppIds.log | Where-Object { $_ -ne '[' -and $_ -ne ']' -and $_ }) -split '\n' | ForEach-Object -Process {
    $id = $_.Trim() -replace ',','' -replace "``n","";
    $deleteAppCommand = 'az ad app delete --id ' + $id;
    Write-Output "Executing delete command: $deleteAppCommand";
    Invoke-Expression $deleteAppCommand
    }
  2. Change the filter expression in deleteAADAapp.ps1 to suit your need. [REF: OData filter / az ad app list]

  3. In powershell terminal, execute the below command:

    1
    2
    3
    4
    5
    # login account you would like to delete useless AAD app
    az logout && az login
    az account show

    .\deleteAADAapp.ps1

Reference GitHub Repo

delete-useless-aad-app-by-name-tool

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

Vim Toolkit

Graphical Cheat Sheet

vim.png
vi-vim-cheat-sheet
vim.png
vi-vim-cheat-sheet-ChineseVersion

Personal Cheat Sheet

  1. 上下左右

    image.png

  2. dd: 删除一行
    u: undo
    ctrl-r: redo(undo undo)

Git ToolKit

Git config

1
2
3
4
git config -l # list all the config of the repo
git config --global -l # list all global config
git config --global core.whitespace cr-at-eol # (Windows Git): Hide ^M (Carriage Return) in Diff
git config --global core.autocrlf true # 提交时转换为LF(Linux \n),检出时转换为CRLF(Windows \r\n)

Create a new branch

  1. create a new branch and track remote branch

    1
    2
    3
    $ git checkout -b serverfix origin/serverfix
    Branch serverfix set up to track remote branch serverfix from origin.
    Switched to a new branch 'serverfix'
  2. Create a new feature branch in remote.

    1
    2
    3
    4
    5
    6
    7
    # 1. Create a new branch:
    git checkout -b feature_branch_name

    # 2. Edit, add and commit your files.

    # 3. Push your branch to the remote repository:
    git push -u origin feature_branch_name

Undo the most recent commit

1
git reset HEAD~

Git submodule

  1. If you want to git clone a repo alone with submodules. Use the following command:

    1
    git clone --recursive https://github.com/<repo-name>.git
  2. If you have git clone a repo with submodule, you can find submodules’ folders, but the folders are empty.

    1
    2
    3
    git clone https://github.com/<repo-name>.git
    git submodule init # Init your local configuration file
    git submodule update # Will pull submodules you need

    Use the above commands and you can have a complete repo.

Git clone

  1. git clone a private repo

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # Normal case
    git clone https://username:password@github.com/username/repository.git
    # or later input password
    git clone https://username@github.com/username/repository.git
    Password:

    # [!!!] When using two-factor authentication(https://stackoverflow.com/a/52011442/8522166)
    # 2fa(https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)
    $ git clone https://github.com/username/repo.git
    Username: your_username
    Password: your_token
    # or inline
    git clone https://username:token@github.com/username/repository.git
  2. git clone a single branch

    1
    git clone --single-branch --branch <branchname> host:/dir.git

Git pull

git pull & git pull origin master

  • git pull = git fetch + git merge

  • git pull --rebase = git fetch + git rebase

  • git pull <repo> <branch>:
    1st argument by default is the current branch’s remote.
    2nd argument by default is the current branch’s merge.

Say you are working on a local branch dev, whose upstream branch is origin/dev.

1
2
3
4
5
6
7
8
$ git branch -vv
* dev 328c854 [origin/dev] Chapter_5 outline
master 7dedf87 [origin/master] May 9 discussion (#14)

$ git config --get branch.dev.remote
origin
$ git config --get branch.dev.merge
refs/heads/dev

git pull will pull from origin/dev and merge to current dev branch.
git pull origin master can pull origin/master and merge to current dev branch.[!!]

Example:

You are developing on local dev branch, push to the origin/dev at times for a pr. Now the master has some new updates and you would like to sync with master. You can use git pull origin master to pull master’s new update and merge to your branch.

1
2
$ git pull origin master
Merge branch 'master' of github.com:arthma/iot-book into dev

Merge Two GitHub Repo

Say you have two github repo A and B. A is a copy of B and develop some features. Now you want to merge A back to B so the new feature can be added to B.

Go to B’s repo, git pull A’s code.

1
2
# In B repo. Pull A code
git pull git@github.com:A-organization/A-project-name.git

Then solve conflicts, submmit commit, git push, done.

Git rebase

Docker Toolkit

Clean all container

1
2
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

Clean all

1
docker container stop $(docker container ls -a -q) && docker system prune -a -f --volumes

Remove all images except one

1
docker image prune -a --force --filter "label!=image_name"

Check disk usage

1
2
docker system df
docker system df -v

Enable docker CLI in WSL(with Docker for Windows installed)

  1. Tick the checkbox in docker setting:

    Expose docker daemon on tcp://localhost:2357 without TSL

  2. Install docker cli in WSL.

    apt-get install docker.io

  3. Expose tcp port in WSL.

    1
    2
    3
    4
    5
    docker -H tcp://0.0.0.0:2375 images
    echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc

    # If you are using zsh
    echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.zshrc && source ~/.zshrc
  4. Verify docker is enabled in WSL

    1
    docker --version

    Now you are using docker in WSL.

Dockerfile Optimization

Use this website to double check your Dockerfile.

Size of Docker Image

Docker image is compressed for faster download speed, and will be decompressed after being downloaded to your local machine.

The compressed size counts for the download time needed in certain development internet transmission environment, and the decompressed image size matters since it consumes local disk.

Actual Size of Docker Images

To get decompressed size of a docker image (i.e. the actual size), run the following command.

1
2
3
4
docker image ls

# or
docker system df -v

Compressed Size of Docker Images

To get compressed size of a docker image is more complicated.

  1. Get compressed image size information via Docker Hub Registry HTTP API v2.

    1. If the image is host on Docker Hub

      1
      2
      curl -s -H "Authorization: JWT ${TOKEN}" "https://hub.docker.com/v2/repositories/library/<image-name>/tags/?page_size=100" | jq -r '.results[] | select(.name == "<tag-name>") | .images[0].size' | numfmt --to=iec-i
      # "numfmt --to=iec-i" is for human readibility

      Example to get the compressed image size of alpine:latest

      1
      curl -s -H "Authorization: JWT ${TOKEN}" "https://hub.docker.com/v2/repositories/library/alpine/tags/?page_size=100" | jq -r '.results[] | select(.name == "latest") | .images[0].size' | numfmt --to=iec-i
    2. If the image is host on Microsoft Container Registry

      Parse method depends on the registry’s implementation and compability of docker HTTP Registry API.

      1
      docker manifest inspect -v devicedevex.azurecr.io/alpine-azure-sdk | grep size | awk -F ':' '{sum+=$NF} END {print sum}' | numfmt --to=iec-i
  2. Push your image to your domain in docker hub, you can see the compressed size listed on docker hub.

  3. Use docker save to save image to a .tar file and then compress it a .tar.gz file.

    1
    2
    3
    4
    5
    6
    7
    docker save my-image:latest > my-image.tar

    # Compress the .tar file
    gzip my-image.tar

    # Check the size of the compressed image
    ls -lh my-image.tar.gz
  4. Using internet monitor to capture the data size transfered during docker pull might be inconcise due to HTTP’s network congestion control, etc. This method is not recommended.

Bash and Linux ToolKit

工具

标准输入输出

Command Description
echo echo "Hello world!" > hello.sh
printf printf "Test the code %d times, and it still can fail the %d-th time." 100 101
stdout stderr ls 1> /tmp/ls.stdout 2> /tmp/ls.stderr
&& `
$(this_is_a_command) or `` touch `seq 1 3` or touch $(seq 1 3)

文件系统

  1. 工作目录(working directory)

    Command Description
    pwd ls cd
    du du -sh *

    display disk usage statistics
    df df -h /mnt

    display free disk space
  2. 查找文件

    Command Description
    find find . -type f

    找到本目录下所有的文件并输出到屏幕 (包括子目录,下同)
    -name: find . -name "*.jpg"
    -maxdepth: find -maxdepth 1
  3. 文件操作(移动、拷贝、创建文件夹)

    Command Description
    cp mv rm mkdir rmdir 在~/.bashrc文件中,加入下面的命令:
    alias cp='cp -i'
    alias rm='rm -i'
    alias mv='mv -i'

    -i: 如果命令会删除或者覆盖文件,会提示用户confirm。
    chmod chmod a-w somefile

    把somefile的所有写权限去掉
    ln ln -s $src $dest

    创建一个软链接$dest, 它指向的原文件在$src

文件内容

  1. 查看文件

    Command Description
    cat cat <file>
    head tail head -n <file>
    less more lessmore的升级版,推荐使用less
    diff diff file1 file2
    md5sum md5sum <file>

    计算文件的md5码,用来比较本机和远程文件是否相同
  2. 文件内容操作

    Command Description
    cut cut -f1,3 file1

    cut out selected portions of each line of a file
    -b 按字节
    -f显示某几列
    bc echo "1.212*3" | bc

    任意精度的计算器
    wc find . -name "*.jpg" | wc -l

    数出本目录下所有的jpg文件数量
    sort sort a.list > a.sort

    对文件a.list排序并写到a.sort
    -u: 去重
    -k: sort -k2,2 a.list, 按第2列排序
    -n: 按数字排序
    -V:可以理解为加强版的按数字排序
    -r: 倒序
    -R: 随机
    uniq cat list1 list2 | sort | uniq > final.list

    将list1, list2合并,排序后去重,写到final.list
    (其实sort一个命令就够了)
    -c: 用来统计很方便
    -u
    -d
    sed sed "s@origin@corrected@" a.list -i
    将a.list中每行出现的第一个origin替换为corrected
    -i: inplace change

    sed -i "s/old/new/g" `grep old -rl /www`
    /www路径下递归的所有文件中包含old字符串的文件中的old都替换成new
    awk awk -F'_' '{print $1}' a.list

    将a.list中每行按’_’分隔,每行输出分隔后的第一个字符串,例如1_2_3-> 1
    xargs find . -name "*.jpg" | xargs -i rm {}
    find . -name "*.jpg" | xargs rm

    找到本目录下所有的jpg文件并删除
    -P: 多进程
    -n: 一般配合”-P”参数
    comm comm -23 a.sort b.sort

    输出在a中,不在b中的行
    grep cat a.list | grep png

    找到a.list中包含png的行
    -R 递归查找
    -n 列出行号
    -l 列出匹配的文件名
    -a
    -o
    -i不区分大小写

    grep "abc" . -nr 查找当前路径下含有“abc”的文件,列出文件名,行号和匹配内容
    grep "abc" . -lr 查找当前路径下含有“abc”的文件,列出文件名
    rev rev <file>
    find . -type f | rev | cut -d'/' -f1 | rev # revcut'/'

    reverse the file(or reverse the stdin if no file provided)

系统相关

Command Description
htop top 查看各个进程的实时CUP占用率、内存使用,进程ID,命令行,启动时间,用户等信息。以及机器整体的CPU占用率和内存使用情况。
ps ps aux
ps auxf
dstat 系统资源统计。可以查看每秒的CPU,磁盘读写,网络出入等信息
nvidia-smi 查看GPU的状态
ssh
scp scp -r $scr $dest

机器之间复制文件
rsync rsync -avzP $src $dest

和scp接近,文件传输工具。特别适用于大量小文件场合,此时速度会明显快于scp。
-e: rsync -avhP -e "ssh -carcfour"
可以指定ssh参数,如端口/证书等
tmux https://gist.github.com/MohamedAlaa/2961058

tmux is a terminal multiplexer,你会经常在服务器上工作,应该尽快掌握tmux
kill pkill killall kill <process_id>
kill -9 <process_id>
killall python
pkill _unittest

杀死进程
后台运行相关 ./a_time_comsuming_process &> 1.log
jobs
fg (ctrl-z)
bg (ctrl-z)

网络相关

Command Description
netstat netstat -anp|grep <port> 网络详细信息
lsof lsof -i:<port>
ping ping www.baidu.com
ifconfig

常用工具

Command Description
wget wget http://some.server.com/thewantedarticle.pdf -O /tmp/local.pdf
下载工具
curl
jq 用于解析json
$ echo '{"key1":"value1","key2":"value2"}' | jq '.key2'
"value2"
parallel 用于并行执行多个相近的命令
cat 20urls.list | parallel -j 4 wget -q {}
xargs xargs非常强大
find . -type f |grep '2018/03' | xargs -I {} wc -l {}
find . -type f |grep '2018/03' | xargs wc -l

Linux Package Manager

Command Description
sudo apt-get update 更新源列表中的软件列表
sudo apt-get upgrade 把本地已安装的软件,与刚下载的软件列表里对应软件进行对比,如果发现已安装的软件版本太低,就会提示你更新。

Bash脚本语法

“#!”: shebang。使用chmod +x file使sh脚本文件可以运行。

1
#!/bin/bash

变量

  1. 基本用法

    1
    2
    3
    4
    des="awe and cool"
    echo $des
    echo ${des}
    echo "This is ${des}"
  2. 内置变量

    $0, $1, $2…: 脚本参数

  3. 输出赋值

    1
    2
    3
    4
    5
    ret=`cat readme.txt`
    echo $ret

    ret=$(cat readme.txt)
    echo $ret
  4. 运算

    let:

    1
    2
    3
    4
    5
    6
    7
    let "a = 5 + 6"
    echo $a

    x=11
    y=22
    let "z = x * y"
    echo $z

    expr:

    1
    2
    3
    4
    5
    6
    $ expr 5 + 6
    11

    $ foo=$(expr 5 + 6)
    $ echo $foo
    11

    注意:运算符和各个数字之间需要有空格

    双括号

    1
    a=$((5+6))
  5. 判断语句

    1
    2
    3
    4
    5
    6
    if <condition>
    then
    <cmd>
    else
    <cmd>
    fi
    1
    2
    3
    4
    5
    tmp=35
    if [ ${tmp} -ge 30 ]
    then
    echo "It's too hot"
    fi
    Operator Description
    ! EXPRESSION The EXPRESSION is false.
    -n STRING The length of STRING is greater than zero.
    -z STRING The lengh of STRING is zero (ie it is empty).
    STRING1 = STRING2 STRING1 is equal to STRING2
    STRING1 != STRING2 STRING1 is not equal to STRING2
    INTEGER1 -eq INTEGER2 INTEGER1 is numerically equal to INTEGER2
    INTEGER1 -gt INTEGER2 INTEGER1 is numerically greater than INTEGER2
    INTEGER1 -lt INTEGER2 INTEGER1 is numerically less than INTEGER2
    -d FILE FILE exists and is a directory.
    -e FILE FILE exists.
    -r FILE FILE exists and the read permission is granted.
    -s FILE FILE exists and it’s size is greater than zero (ie. it is not empty).
    -w FILE FILE exists and the write permission is granted.
    -x FILE FILE exists and the execute permission is granted.

    test命令:

    1
    2
    3
    $ test 5 -ge 4
    $ echo $?
    0

    test命令的结果,0表示true或命令运行成功;1表示false或运行失败。

  6. 布尔运算

    &&, ||

    1
    2
    3
    4
    if [ $code_review = "pass" ] && [ $regression_test = "pass" ]
    then
    echo "ship it!"
    fi
  7. 循环语句

    1
    2
    3
    4
    for var in <list>
    do
    <cmd>
    done
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    for name in John, Emma, Tom
    do
    echo "My name is $name"
    done

    for num in {1..10}
    do
    echo "count $num"
    done

    for ((i=0; i<10; i++))
    do
    echo "count $i"
    done
  8. until语句

    1
    2
    3
    4
    until [ some_test ]
    do
    <cmd>
    done
    1
    2
    3
    4
    5
    6
    suffix=1
    until [ ! -e "foo${suffix}" ]
    do
    let suffix++
    done
    echo "The file name foo${suffix} is good.
  9. break

  10. continue

  11. 函数

    1
    2
    3
    function_name () {
    <commands>
    }
    1
    2
    3
    4
    5
    6
    7
    8
    9
    discuss_langauges() {
    language1=$1
    language2=$2
    echo "$1 is a good language."
    echo "$2 is also a good language."
    }

    # 调用不需括号
    discuss_langauges c++ python
    1
    2
    3
    4
    5
    6
    7
    8
    9
    find_cpp_files() {
    local folder=$1
    local ret=$(find ${folder} -name '*.cpp' | wc -l)
    return $ret
    }
    folder=/home/hchen/code/ficus/common/machine_learning
    find_cpp_files $folder
    num_files=$?
    echo "There are ${num_files} cpp files in ${folder}"

调试技巧

  • set -e
  • set -x

实用案例

  1. Test if a comand outputs an empty string:

    if [[ $(YOUR_COMMAND)]]

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # ls directory
    if [[ $(ls -A) ]]; then
    echo "there are files"
    else
    echo "no files found"
    fi

    # grep file context
    if [[ $(cat file | grep key_word) ]]; then
    echo "find key_word in the file"
    then
    echo "Not found"
    fi
  2. Check whether the last command complete successfully.

    $?: the exit code of the last command (zero for success, non-zero for failure).

    1
    2
    3
    4
    5
    6
    7
    8
    files=$(ls -A)
    if [[ $? != 0 ]]; then
    echo "Command failed."
    elif [[ $files ]]; then
    echo "Files found."
    else
    echo "No files found."
    fi