Migrate data from Hadoop to S3 and from S3 to redshift

here are examples of migrating data from Hadoop to S3 and from S3 to Redshift using Python:

Migrating Data from Hadoop to S3:

import boto3
from boto3.s3.transfer import TransferConfig
from hdfs import InsecureClient

# Connect to Hadoop
hdfs = InsecureClient('http://<hadoop-hostname>:50070', user='<hadoop-username>')

# Connect to S3
s3 = boto3.resource('s3', aws_access_key_id='<aws-access-key>', aws_secret_access_key='<aws-secret-key>')

# Configure transfer settings
config = TransferConfig(multipart_threshold=1024 * 25, max_concurrency=10, multipart_chunksize=1024 * 25, use_threads=True)

# Define the Hadoop path and S3 bucket/key to transfer to
hadoop_path = '/path/to/hadoop/files/'
s3_bucket = '<s3-bucket-name>'
s3_key = 'path/to/s3/files/'

# Loop through files in Hadoop directory and copy to S3
for file in hdfs.walk(hadoop_path):
    if file['type'] == 'FILE':
        source_file = hadoop_path + file['pathSuffix']
        dest_file = s3_key + file['pathSuffix']
        s3.Object(s3_bucket, dest_file).upload_file(source_file, Config=config)

Migrating Data from S3 to Redshift:

import boto3
import psycopg2

# Connect to S3
s3 = boto3.resource('s3', aws_access_key_id='<aws-access-key>', aws_secret_access_key='<aws-secret-key>')

# Connect to Redshift
conn = psycopg2.connect(
    host='<redshift-hostname>',
    port=<redshift-port>,
    user='<redshift-username>',
    password='<redshift-password>',
    database='<redshift-database>'
)
cur = conn.cursor()

# Define the S3 bucket and key where the data is stored
s3_bucket = '<s3-bucket-name>'
s3_key = 'path/to/s3/files/'

# Define the Redshift table to load the data into
table_name = '<redshift-table-name>'

# Create the SQL COPY command to load the data from S3 into Redshift
copy_sql = f"""
    COPY {table_name}
    FROM 's3://{s3_bucket}/{s3_key}'
    IAM_ROLE '<aws-iam-role>'
    FORMAT AS JSON 'auto'
"""

# Execute the SQL COPY command
cur.execute(copy_sql)
conn.commit()

# Close the database connection
cur.close()
conn.close()

The code is using the boto3 library, which is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python.

The first section of the code is defining the AWS credentials and the S3 bucket name where the data is stored.

Then, the upload_file function is called to upload the data file from the local system to the specified S3 bucket. The upload_file function takes three parameters: the local file path, the S3 bucket name, and the S3 object key (which is the filename and path within the bucket).

The second section of the code is defining the AWS credentials and the Redshift cluster information where the data will be loaded.

Next, the copy function is called to copy the data from S3 to Redshift. The copy function takes a SQL query that specifies the table and columns where the data will be loaded, as well as the S3 file location and the AWS credentials.