DevOps Interview linux Preparation Study Notes

1. Troubleshooting SSH Connection to EC2 Instance

Definition: SSH (Secure Shell) connection failures to EC2 instances can occur due to security group configurations, key permissions, network issues, or instance problems.

Key Points:

• Check security groups first - they’re the most common blocker
• Verify the instance is running and passed status checks
• Ensure your private key has proper permissions (chmod 400)

Step-by-Step Troubleshooting:

1. Verify instance state
   • Check if the instance is running in the EC2 Console
   • Ensure the instance has passed its status checks
2. Check security group rules
   • Verify inbound rules allow SSH traffic (port 22) from your IP
   • Command: aws ec2 describe-security-groups --group-ids sg-xxxxx
3. Validate network settings
   • Confirm the subnet has a route to the internet gateway
   • Ensure no Network ACLs are blocking port 22
4. Check key pair
   • Verify you’re using the correct private key (.pem file)
   • Set proper permissions: chmod 400 your-key.pem
5. Use verbose mode for more details

   ssh -v -i your-key.pem ec2-user@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
   

6. Try the EC2 Instance Connect
   • Use as an alternative to verify the instance is accepting connections
7. Check system logs in EC2 console
   • System log may show SSH service failures

2. Troubleshooting 5xx Errors on Ubuntu Server

Definition: 5xx errors are server-side errors indicating the server failed to fulfill a valid request, commonly seen in web applications.

Key Points:

• 5xx errors almost always indicate server problems, not client issues
• Common causes: resource exhaustion, application crashes, or misconfiguration
• Log files are your best diagnostic tools

Step-by-Step Troubleshooting:

1. Check web server logs
   • Apache: /var/log/apache2/error.log
   • Nginx: /var/log/nginx/error.log
2. Check application logs
   • Check application-specific logs (e.g., /var/log/application.log)
   • For containerized apps, check container logs
3. Monitor system resources

   # Check CPU and memory usage
   top
      
   # Check disk space
   df -h
      
   # Check for running processes
   ps aux | grep apache2  # or nginx, etc.
   

4. Restart web services

   sudo systemctl restart apache2  # or nginx
   

5. Check configuration files
   • Validate config syntax: sudo apache2ctl configtest or sudo nginx -t
6. Check database connections
   • Ensure database services are running
   • Check connection parameters in configuration files
7. Review recent changes
   • Check deployment logs for recent changes
   • Verify configuration changes

3. Adding and Mounting a New Disk on a Server

Definition: Adding a new disk involves attaching the physical/virtual storage, creating a filesystem, and mounting it to make it accessible within the filesystem hierarchy.

Key Points:

• Always check disk identity before partitioning/formatting
• Create a permanent mount entry in /etc/fstab for persistence after reboot
• Consider filesystem type based on your needs (ext4 is common for Linux)

Step-by-Step Guide:

1. Identify the new disk

   lsblk
   # or
   fdisk -l
   

2. Create a partition

   sudo fdisk /dev/xvdf  # Replace with your disk
   # n (new partition)
   # p (primary partition)
   # Accept defaults for partition number and size
   # w (write changes)
   

3. Create a filesystem

   sudo mkfs -t ext4 /dev/xvdf1  # Replace with your partition
   

4. Create a mount point

   sudo mkdir /data  # Or any preferred directory
   

5. Mount the filesystem

   sudo mount /dev/xvdf1 /data
   

6. Configure automatic mounting

   # Get the UUID
   sudo blkid /dev/xvdf1
      
   # Add to /etc/fstab
   echo "UUID=<uuid-from-blkid> /data ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   

7. Verify the mount

   df -h
   

4. Recovering from a Lost PEM File for an AWS Ubuntu Server

Definition: A lost PEM file means you’ve lost SSH access to your instance because the private key is no longer available to authenticate your connection.

Key Points:

• AWS doesn’t store or can’t recover your private key
• Recovery involves creating a new key and applying it to the instance
• For EBS-backed instances, you can use another instance to help recover access

Step-by-Step Recovery:

1. Create a new key pair

   aws ec2 create-key-pair --key-name NewKeyPair --query 'KeyMaterial' --output text > NewKeyPair.pem
   chmod 400 NewKeyPair.pem
   

2. Stop the instance (Note: This will cause downtime)

   aws ec2 stop-instances --instance-ids i-1234567890abcdef0
   

3. Detach the root volume

   aws ec2 detach-volume --volume-id vol-1234567890abcdef0
   

4. Launch a temporary instance
   • Launch in the same AZ as the detached volume
5. Attach the volume to the temporary instance

   aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-abcdef1234567890 --device /dev/sdf
   

6. Mount the volume and modify authorized_keys

   sudo mkdir /mnt/recovery
   sudo mount /dev/xvdf1 /mnt/recovery
   sudo cp ~/.ssh/authorized_keys /mnt/recovery/home/ubuntu/.ssh/
   

7. Detach, reattach to original instance, and restart

   sudo umount /mnt/recovery
   aws ec2 detach-volume --volume-id vol-1234567890abcdef0
   aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-1234567890abcdef0 --device /dev/sda1
   aws ec2 start-instances --instance-ids i-1234567890abcdef0
   

8. Connect with the new key

   ssh -i NewKeyPair.pem ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com