Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

January 6, 2026

Setting up Ubuntu 24.04.x LTS desktop and server, RStudio server, and JypyterLab at home

Why? Set up a Linux server computer on a home network if:

  • You want to run code that takes a long time to complete. Let it run on the server computer so you can use your main computer for other things.
  • Your main computer is low on storage space. Upload all data files to the server and you can run analysis on the server remotely, e.g. from a laptop while you sit on your couch.
  • You want a server specifically for Linux (e.g. for bioinformatics or programming purposes), whereas your main computer is Windows or MacOS.

Who? This tutorial is intended for computer-savvy people with enough patience and knowledge to troubleshoot their own computer by searching the internet for error codes and following instructions. Ideally, you should have some programming experience (beginner level in any language is enough) and know about terminals or command prompt windows, but you can learn what you need as you go. This is a great hands-on way to learn to use Linux. Expert Linux knowledge is not a prerequisite. 

What? Ubuntu Desktop 24.04.x LTS "Noble Numbat" (or the latest version), OpenSSH for server capabilities, RStudio server, and Jupyter Lab. Ubuntu is a specific version of Linux.

Cost? All the software is free. For hardware, you need the following:

September 20, 2024

How to get MD5 checksums to detect data corruption (for bioinformatics data curation)

What are MD5 checksums? Checksums are nonsense text strings used to "summarize" a file version. No matter the size of the file (1 kb or 30 GB), the checksum algorithm gives you a conveniently short nonsense string of letters and numbers. The exact same file will give you the exact same checksum every time. If you change a single character or pixel, you will get a different checksum.

MD5 is a specific popular algorithm to get checksums.

Why use checksums? The purpose of checksums is to notice data corruption, especially when downloading files from or uploading files to a server. Every time you transfer files between computers, there is risk of data corruption. For small files, the risk is small and you'll most likely notice, for example if your email attachment download fails due to an internet interruption. 

For large files such as raw sequencing data files, it's a bigger issue and you might not notice right away (or ever) if the last few RNA-seq reads of a >30 million reads file are missing. Therefore, the best practice when downloading new sequencing is to create MD5 checksums yourself and compare them with the MD5 checksum created by the originating computer (the sequencing core's server). They should be the same. If not, something went wrong during file transfer! Try re-downloading the data.

Similarly, when you upload sequencing data to a public repository (e.g. NCBI GEO), you provide MD5 checksums so that the receivers (NCBI's data curators) can confirm the upload was successful.

How to get an MD5 checksum for an individual file? See example below using the Linux terminal. I created a text file containing only the phrase "hello pretend this is sequencing data". The checksum for that file is "b088d8d4d1d831af2d8d16147389aa7d". If I change the first letter to uppercase, the checksum completely changes.

 

September 28, 2021

Updating RStudio server on Ubuntu server 20.04.2 LTS

Problem: R version 4.1.1 crashes when creating plots with ggplot2 and RStudio server version 4.1.1106. 

  • Downgrading to R 4.0.5 apparently works.
  • Adding x11() in front of the plot code may work as well.
  • Updating RStudio server is supposed to work as well.

How to update RStudio server:

sudo apt-get install gdebi-core

wget https://download2.rstudio.org/server/bionic/amd64/rstudio-server-2021.09.0%2B351-amd64.deb 

sudo gdebi rstudio-server-2021.09.0+351-amd64.deb

sudo ufw allow 8787/tcp #optional, skip if you haven't enabled firewall yet


Result: Yes, updating RStudio server worked for me. 

November 25, 2018

Unix/Linux cheatsheet and simple bash

Don't type the #comments (everything after the hashtag is commentary).

Common commands to check your account and environment

date #show date and time on the system (if it's wrong, that could cause problems)

history #shows the history of everything you've typed into the terminal

history > history.txt #saves your history to a text file, overwriting it if it already exists

history >> history.txt #saves your history to a text file, adding to the end (appending) if it already exists instead of overwriting it

ip a #prints your internet IP addresses (ethernet and wifi)

ls #shows you all the filenames in your directory

ls -la #shows you all the information about the files, not just the filenames

pwd #print current working directory; default is /home/username

which #shows you where a command lives (commands are programs so this helps troubleshoot if a command doesn't work)

which ls #example showing you where the ls command lives on your system

whoami #prints your current login username

[control+C] #this cancels or quits whatever is happening, good if you freeze

March 21, 2017

Python: Starting Python for the First Time (Windows & Linux)

Python is a programming language that is popular as a first programming language. This "getting started" tutorial is aimed at Windows users, with suggestions for Linux users.

Install Python and related tools

Install one of these distributions of Python, pick only one:
  • Just pick ONE of these. Installing multiple will create different versions of Python on your computer and that will cause conflicts later.
  • Anaconda
    • If drive space isn't an issue (you'll need 5-6 GB), get this. It is the most common distribution used by beginners and advanced users.
    • Comes with a common packages for life sciences pre-installed (e.g. pandas, numpy, matplotlib)
    • Makes creating different Python environments easy
    • Pick this if you are attending an Intro to Python workshop
  • Miniconda
    • Lighter version of Anaconda. It comes with less pre-installed programs. I like it.
    • Won't have "Anaconda Navigator" window, so you will need to control Python environments through the terminal (might be frustrating for beginners)
    • Pick this if you already know how to program, have other programming software on your computer, and don't want the full Anaconda installation to create duplicates of Jupyter Lab and other tools.
  • Python.org
    • If drive space is very limited, download this basic version
    • Any additional packages will need to be installed separately. Beware, this requires more steps and is less beginner friendly. 
  • During installation, select click the box to add whichever Python version to your PATH environment variable. You want this! It allows you to run Python scripts from the terminal later. 
Install a text editor for programmers (lightweight software), pick one or more: 
  • The default text application (e.g. Notepad.exe for Windows) will work because programs are just text files with special file extensions, but text editors specifically for programmers will make your life easier by color-coding the programming language syntax. This helps you notice when you forget a closing parenthesis or add an extra apostrophe.
  • Notepad++ (Windows)
  • Sublime Text (Windows, Linux, macOS) 
  • Geany (Linux) 
Install more complex IDE software (highly recommended for beginners), pick one or more: 
  • Jupyter Lab is popular and great for multi-script projects. 
    • From Windows terminal (cmd.exe):
      python -m pip install jupyterlab
    • From Python terminal:
      pip install jupyterlab
  • Spyder comes bundled with Anaconda and is my favorite IDE for Python. It is less complicated the Jupyter Lab.
  • RStudio Desktop now supports Python but I don't recommend it if you are learning Python for the first time. It is useful if you plan to use both R and Python languages together in the future. 
Install pip and use it to install Python packages

Open Jupyter Lab and start writing basic Python code

On Windows with Anaconda, open "Anaconda Navigator" 
  • Click "Launch" on the box for "Jypyter Lab" or "Jupyter Notebook".

On Windows with Miniconda, open software "Anaconda Powershell Prompt"
  • Type into the terminal and press [ENTER] button:
    jupyter lab

Jupyter Lab will launch
Whichever method used, the computer will launch Jupyter Lab on a web browser.

Create a new Jupyter notebook file with...
File: New: Notebook

Navigate to the new notebook tab (Untitled.ipynb) and type code into the gray text book. For example:
x = 5
y = 10
x + y 
 
To run the cell (the block of code), either press the play triangle icon or use keyboard shortcut [SHIFT] + [ENTER]

You should see the output of your code and a new cell (gray text box for code):
15

Type and press [SHIFT] + [ENTER]:
z = x*x

Your code will run and you will get a new cell. However, it doesn't look like it ran (no output). Your code assigned a value to variable z, but it did not instruct Python to print the output.

To print the output, try one of these:
z
print(z)

Either of these will print the output (the value) of variable z to your window.

Note that function print() cannot have a space between print and the open parenthesis. The function needs to be attached to the input inside the parentheses.

Try this and output:
print(x+y)
print("Hello world")
z = "Beautiful"
print("Hello",z)


Note that you just replaced the value of variable z so now it has a string value ("Beautiful") instead of a numerical value (x*x or 25). Variable values can be replaced. When writing code, keep track of what assigns values to your variables and when you are rewriting them.

You can also see the data types of your variables, for example:
print(x, type(x))
print(y, type(y))
print(z, type(z))
 
Remember to close every parenthesis that you open, or you will get an error.
'int' = integer
'str' = string (the programming name for "text" data)

There are different type of number datatypes. For example, change y to a number with decimals:
y = y+0.01
print(y, type(y))

'float' = number with decimals

Math expressions with comments after the hashtag (#):
2 + 1 # addition
2 - 1 # subtraction
2*3  # multiplication
18 / 2 # division
2**3 # exponent

Adding comments is very helpful to take notes on your code. Write #comments regularly.

Conditional statements, if else code blocks, and loops

Relational operators:

x = 100

y = 2 

x < y  # less than

x > y  # greater than

x  <= y # less than or equal to

x >= y # greater than or equal to

x == y # double equal sign evaluates for equivalency (is x equal to y?)

x != y # is x NOT EQUAL to y?


Create ifelse statements.

x = 15

y = 10

if x > y: #Note the use of the colon to indicate the end of the if statement to be evaluated

print(str(x)+ " is greater than " +str(y))  #Code to run if statement is true

else: #What to run if the conditional statement is not true

print(str(x)+ "is " +str(y)+ "or less")


Note that Python requires correct indentation of lines for ifelse statements. Not all programming languages enforce this, but Python is strict about it.

The colons after the if conditional statement and after else are important to tell Python what to do. Python will evaluate if the conditional statement is true or not, then decide which code block to run.

Create a while loop. It repeatedly runs a loop until the conditional statement is no longer true.

iteration=0

while iteration < 5:

print("Iteration:", iteration)

iteration += 1 #increases variable 'iteration' by 1 each time the while loop is run


Be careful with while loops. If you don't create a way for it to be interrupted, you can manually stop the code with the square stop icon.

Combine conditional statements and decide if either needs to be true, or if both need to be true.

x=2 

y=15

a=5

b=6

is_true = x>y or a<b

print(is_true)

is_true = x>y and a<b

print(is_true)


Copying and aliasing comparisons:

a=[2,3,4]
b=a #tell computer to use the same value from a for b
print(a is b)
b=[2,3,4]  #write object b with values [2,3,4], same values as a but it's a different object
print(a is b) 
a==b #compare values of the variables

True
False
True

Lists, dictionaries

Ranges:

for i in range(5):

print(i)

0

1

2

3

4

for i in range(25,30):

    print(i)

25
26
27
28
29

Pre-defined lists and using list indexes to pull out values:

my_list = [1, True, 3, 'Vegas', 5]

print(my_list[3])

Vegas

Because Python is zero-indexed (lists begin with zero), the list item 3 is actually the 4th item because Python counts 0,1,2,3,4.

Strings can also be lists of characters:

word = "Python"

for letter in word:

    print(letter)

p

y

t

h

o

n


Use loops to add values to lists:

Long way:

squares = []
for i in range(5):
    squares.append(i**2)
print(squares)

[0, 1, 4, 9, 16]


Concise way:

squares = [i**2 for i in range(5)]
print(squares)


Slice lists (get list subsets):

fruits = ["apple", "banana", "kiwi", "guava", "cherry"]

for fruit in fruits[1:-1]: 
    print(fruit)

banana
kiwi
guava

The code was told where to start (at index 1, which is the second value since Python starts with 0), and where to end (at index -1, which is the second-to-last spot because Python can go backwards).


Dictionaries store key-value pairs:

student = {
    "name": "Jose Soandso",
    "age": 20,
    "major": "Math"
}

## see the dictionary
print(student)

## pull a specific value
student["age"]

## print in loop
for key, value in student.items():
    print(f"{key}: {value}") #f-string formatting


Functions


How to write a simple Python program inside a text file, then run it on the Windows terminal

R programming lesson #2: merging pdf files

Use R package "pdftools" to merge separate pdf into one pdf file. You will never need to use sketchy websites or pay for software ...