Flattening a collection within collection to get a single List<>

I’m quite new to using Java and was trying to flatten collections within collections using map to try and get a single List. However I don’t seem to be able to get this working. In order to reproduce this I’ve created quite a simple example of what I’m trying to do. Here is what I have so far:

ClassA

import java.util.List;

public class ClassA {
    private List<ClassB> listOfClassB;

    public ClassA(List<ClassB> listOfClassB) {
        this.listOfClassB = listOfClassB;
    }

    public List<ClassB> getListOfClassB() {
        return this.listOfClassB;
    }
}

ClassB

    public class ClassB {
    private String item;

    public ClassB(String item) {
        this.item = item;
    }

    public String getItem() {
        return item;
    }
}

Main

public class Main {
    public static void main(String[] args) {
        ClassB firstClass = new ClassB("A");
        ClassB secondClass = new ClassB("B");
        ClassB thirdClass = new ClassB("C");
        ClassB fourthClass = new ClassB("D");
        ArrayList<ClassB> firstClassList = new ArrayList<>();
        ArrayList<ClassB> secondClassList = new ArrayList<>();

        firstClassList.add(firstClass);
        firstClassList.add(secondClass);
        secondClassList.add(thirdClass);
        secondClassList.add(fourthClass);

        ArrayList<ClassA> classes = new ArrayList<>();
        classes.add(new ClassA(firstClassList));
        classes.add(new ClassA(secondClassList));

        List<List<String>> collect = classes.stream().map(c -> c.getListOfClassB().stream().map(ClassB::getItem).collect(Collectors.toList())).collect(Collectors.toList());
    }
}

As you can see on the bottom I am able to get List<List<String>> but what I’m looking to get is a List<String> of the items within ClassB. I’ve tried using a flatmap for this but I couldn’t get it working and was looking for some guidance.

Thanks in advance.

Solution:

Here is the flatmap example which works fine:

classes.stream().flatMap(aclass -> aclass.getListOfClassB().stream())
    .forEach(b -> System.out.println("Class B Item Name : "+b.getItem()));

It gives the following output:

Class B Item Name : A
Class B Item Name : B
Class B Item Name : C
Class B Item Name : D

and to get the exact answer:

List<String> collect2 = classes.stream().flatMap(aclass -> aclass.getListOfClassB().stream())
    .map(b -> b.getItem())
    .collect(Collectors.toList());

it gives me a list as follows:

collect2 : [A, B, C, D]

Else clause in lambda expression

I use the following lambda expression to iterate over PDF files.

public static void run(String arg) {

        Path rootDir = Paths.get(arg);
        PathMatcher matcher = FileSystems.getDefault().getPathMatcher("glob:**.pdf");
        Files.walk(rootDir)
                .filter(matcher::matches)
                .forEach(Start::modify);
    }

    private static void modify(Path p) {
        System.out.println(p.toString());
    }

This part .forEach(Start::modify); executes the static method modify from the same class where the lambda expression is located. Is there a possibility to add something like else clause when no PDF file is found?

Solution:

You could collect the result after the filter operation into a list instance and then check the size before operating on it.

List<Path> resultSet = Files.walk(rootDir)
                            .filter(matcher::matches)
                            .collect(Collectors.toList());
if(resultSet.size() > 0){
    resultSet.forEach(Start::modify);
}else {
    // do something else   
}

Alternatively, you could do something like this:

if(Files.walk(rootDir).anyMatch(matcher::matches)) {
         Files.walk(rootDir)
              .filter(matcher::matches)
              .forEach(Start::modify);
}else {
        // do something else    
}

python "binom" with fewer dependencies?

My Python 3.6 script works great with

from scipy.special import binom

Running the code in AWS lambda, however, does not work. Attempting to load the zipped deployment package from S3 gives the error:

Unzipped size must be smaller than 262144000 bytes

Surely somewhere there is a Python package which can do what “binom” does without needing all of “scipy” which seems to require “numpy” ?

Solution:

The binomial coefficient is a known and fairly trivial calculation; binom(n, k) is just n! / (k! * (n - k)!). Python built-ins can do this in a perfectly straightforward (if theoretically sub-optimal, since it can produce excessively large intermediates that more tuned approaches could avoid, but it hardly matters most of the time) way:

from math import factorial

def binom(n, k):
    return factorial(n) // (factorial(k) * factorial(n - k))

If you need it somewhat faster, gmpy2 offers a bincoef function, and gmpy2 is a bit more standalone than scipy/numpy (it needs GMP/MPFR/MPC, but it’s on the order of a few MB of binaries all told, not a few hundred MB). It returns a gmpy2.mpz type, which is largely interoperable with int, or you can just force conversion back to int by wrapping:

from gmpy2 import bincoef

def binom(n, k):
    return int(bincoef(n, k))

AWS ServerLess Web App Example

I was looking into AWS services for a serverless web app development.i was able to host a static webpage in S3.
But I can’t find any sample projects with java as lambda functions. I don’t have experience with web services and I am confused in the following scenarios

1) Creating a lambda java web API which takes some parameters and generates some results
2) How to call a lambda web API from HTML using javascript, passing some form data as parameters
3) Accessing the output from the web API within the HTML and showing it as a content in the web page

I really appreciate some basic sample codes for the above requirements, i tried a lot online for solutions but can’t find any understandable one.

Thank you

Solution:

Hi think you are at wrong place, this is not place to find sample code. Yet I will refer you few code links.

  1. Creating a lambda java web API which takes some parameters and generates some results

https://github.com/serverless/examples/tree/master/aws-java-simple-http-endpoint

2) How to call a lambda web API from HTML using javascript, passing
some form data as parameters & 3) Accessing the output from the web
API within the HTML and showing it as a content in the web page

https://api.jquery.com/jQuery.post/

In my opinion, you should not go through this code and develop straight up your app. You first should clear few basic workflow of Client Server and how website works, also try to sharpen your documentation reading skill.

Go Through every technolog’s document such as in your case serverless, javascript, html. You will find plenty of getting started code and in detail understanding, which will help you build your product better.

How to get the thumbnail of base64 encoded video file in nodejs

I am developing a web application using Nodejs. I am using Amazon S3 bucket to store files. What I am doing now is that when I upload a video file (mp4) to the S3 bucket, I will get the thumbnail photo of the video file from the lambda function. For fetching the thumbnail photo of the video file, I am using this package – https://www.npmjs.com/package/ffmpeg. I tested the package locally on my laptop and it is working.

Here is my code tested on my laptop

var ffmpeg = require('ffmpeg');

module.exports.createVideoThumbnail = function(req, res)
{
    try {
        var process = new ffmpeg('public/lalaland.mp4');
        process.then(function (video) {

            video.fnExtractFrameToJPG('public', {
                frame_rate : 1,
                number : 5,
                file_name : 'my_frame_%t_%s'
            }, function (error, files) {
                if (!error)
                    console.log('Frames: ' + files);
                else
                    console.log(error)
            });

        }, function (err) {
            console.log('Error: ' + err);
        });
    } catch (e) {
        console.log(e.code);
        console.log(e.msg);
    }
    res.json({ status : true , message: "Video thumbnail created." });
}

The above code works well. It gave me the thumbnail photos of the video file (mp4). Now, I am trying to use that code in the AWS lambda function. The issue is the above code is using video file path as the parameter to fetch the thumbnails. In the lambda function, I can only fetch the base 64 encoded format of the file. I can get id (s3 path) of the file, but I cannot use it as the parameter (file path) to fetch the thumbnails as my s3 bucket does not allow public access. So, what I tried to do was that I tried to save the base 64 encoded video file locally in the lambda function project itself and then passed the file path as the parameter for fetching the thumbnails. But the issue was that aws lamda function file system is read-only. So I cannot write any file to the file system. So what I am trying to do right now is to retrieve the thumbnails directly from the base 64 encoded video file. How can I possibly do it, please?

Solution:

Looks like you are using a wrong file location,

/tmp/* is your writable location for temporary files and limited to 512MB

Checkout the tutorial that does the same as you like to do.

https://concrete5.co.jp/blog/creating-video-thumbnails-aws-lambda-your-s3-bucket

Lambda Docs:

https://docs.aws.amazon.com/lambda/latest/dg/limits.html

Ephemeral disk capacity (“/tmp” space) 512 MB

Hope it helps.

context.awsRequestId from lambda

Is the uuid from context.awsRequestId really unique? I want to use it in association with the creation of a resource, so I can now when the resource was created:

const uuid = require('uuid');
const AWS = require('aws-sdk');

const dynamoDb = new AWS.DynamoDB.DocumentClient();

module.exports.create = (event, context, callback) => {
  const timestamp = new Date().getTime();
  const data = JSON.parse(event.body);
  if (typeof data.text !== 'string') {
    console.error('Validation Failed');
    callback(null, {
      statusCode: 400,
      headers: { 'Content-Type': 'text/plain' },
      body: 'Couldn\'t create the todo item.',
    });
    return;
  }

  const params = {
    TableName: process.env.DYNAMODB_TABLE,
    Item: {
      id: context.awsRequestId,
      text: data.text,
      checked: false,
      createdAt: timestamp,
      updatedAt: timestamp,
    },
  };

  // write the todo to the database
  dynamoDb.put(params, (error) => {
    // handle potential errors
    if (error) {
      console.error(error);
      callback(null, {
        statusCode: error.statusCode || 501,
        headers: { 'Content-Type': 'text/plain' },
        body: 'Couldn\'t create the todo item.',
      });
      return;
    }

    // create a response
    const response = {
      statusCode: 200,
      body: JSON.stringify(params.Item),
    };
    callback(null, response);
  });
};

Thank you.

Solution:

I don’t think this is clearly documented, but based on observations, these UUIDs do not appear to be randomly generated (which is good for uniqueness). Instead, they look like they are a variant of Type 1 UUIDs, where most of the bytes actually represent a timestamp, so it would appear to be a safe assumption that they are both spatially and temporally unique.

When the digit M in xxxxxxxx-xxxx-Mxxx-xxxx-xxxxxxxxxxxx is set to 1, the UUID should be a Type 1 and should represent a high resolution timestamp and “node” identifier, although in this case the node component seems to carry no meaningful information… but the timestamps seem close to real time (though not precisely so).

How to limit in groupBy java stream

This is my course model in class named Course:

public class Course{
    private int courseId;
    private String courseName;
    private Teacher teacher;
}

This is my teacher model in class named Teacher:

public class Teacher{
    private int teacherId;
    private String name;
}

I want to get a Map<String, List<Course>> but if the teacherId is repeated just add that Course into list of map.

I am using groupBy for it

Map<Integer, List<Course>>  result = courses.stream()
        .collect(Collectors.groupingBy(c -> c.getTeacher().getTeacherId(), Collectors.toList()));

and it’s giving result as expected.

But I want to limit here that As soon as 5 teachers are found stop to process and returned the result.

How can it be done??

Solution:

There’s no direct support for this, as stopping and living with potentially incomplete data is rather unusual.

A straight-forward solution collecting the first five groups completely, would be

Set<Integer> firstFive = courses.stream()
    .map(c -> c.getTeacher().getTeacherId())
    .distinct().limit(5)
    .collect(Collectors.toSet());
Map<Integer, List<Course>> result = courses.stream()
    .filter(c -> firstFive.contains(c.getTeacher().getTeacherId()))
    .collect(Collectors.groupingBy(c -> c.getTeacher().getTeacherId()));

Here, the Course lists of these first five teacher ids are complete.


A solution that truly stops after encountering the 5th teacher id, would be simpler with a loop:

Map<Integer, List<Course>> result = new HashMap<>();
for(Course c: courses) {
    result.computeIfAbsent(c.getTeacher().getTeacherId(), x -> new ArrayList<>()).add(c);
    if(result.size() == 5) break;
}

But there is not much sense in collecting lists of Courses, when you can’t trust these lists afterwards. Keep in mind, that even the source list’s very last element could belong to the first encountered teacher ID, so you need to process the entire list even if you are interested in only one teacher’s complete list of courses.

Python: pass "not" as a lambda function

I need to pass a function as a parameter that works as the boolean “not”. I tried something like this but it didn’t work because not isn’t a function.

theFunction(callback=not) # Doesn't work :(

I need to do the following, but I wonder if there exists any predefined function that does this simple job, so that I don’t have to redefine it like this:

theFunction(callback=lambda b: not b, anotherCallback=lambda b: not b)

Note: I can’t change the fact that I have to pass a function like this because it’s an API call.

Solution:

Yes, there is the operator module: https://docs.python.org/3.6/library/operator.html

import operator
theFunction(callback=operator.not_)

AWS API Gateway & Lambda Function for SOAP / REST Conversion

I have a legacy application which currently utilises a SOAP interface to invoke functions against.

I wish to serve this interface to a website, with it being preferential to be done using REST with a JSON scheme.

I wish to get feedback and validate my thinking for the solution to this problem as described below:

In order to do the transformation from SOAP (XML) <–> REST (JSON) I am planning on using the following components:

  • Amazon API Gateway: Receives a POST request from my website containing basic payload data.

  • Lambda Function: API Gateway invokes an AWS lambda function which is contains a soap client package. The function handles any necessary transformations and mappings and it is the Lambda function which issues the request to my legacy SOAP service.

  • SOAP service: Receives request, executed required function and returns XML response

  • Lambda Function: Receives response, parses any required data / handles any errors and returns the response call back to the API Gateway.

  • API Gateway: Returns result of Lamdba function invocation.

A few questions:

  1. I am unsure whether it is the Lambda function which carries out the request directly to my SOAP service, I am assuming so as this is just an encapsulated function. Is this correct?

  2. Does the Lambda function return the response back to the gateway request? I believe this is correct as the function can execute synchronously so will still have the context to return the response back.

  3. Do you generally handle any authentication checks within the API Gateway for the initial inbound request or does the request in its entirety get passed down to the Lambda function where I would then conduct and execute any auth policies/checks?

Solution:

For Question 1 & 2:
Yes, it is correct to do the SOAP request and transformation in the Lambda function.

For Question 3: To handle authentication at API Gateway, you can use a separate Lambda function called Custom Authorizer. You can also do authorization for API Gateway endpoints in Custom Authorizer Lambda function.

Write to /tmp directory in aws lambda with python

Goal

I’m trying to write a zip file to the /tmp folder in a python aws lambda, so I can extract manipulate before zipping, and placing it in s3 bucket.

Problem

Os Errno30 Read Only FileSystem

This code was tested locally on my computer to make sure the file would write to my working directory before I uploaded it to aws. This is the code i’m trying to use.

file = downloadFile() #This is api call that returns binary zip object
newFile = open('/tmp/myZip.zip','wb')
newFile.write(file)
extractAll('/tmp/myZip.zip')

here is the code that is trying to extract the zip file

def extractAll(self,source):
        with zipfile.ZipFile(source, 'r') as archive:
            archive.extractall()

here is the trace

[Errno 30] Read-only file system: '/var/task/test-deploy': OSError
Traceback (most recent call last):
File "/var/task/web.py", line 31, in web
performAction(bb, eventBody)
File "/var/task/src/api/web.py", line 42, in performAction
zipHelper.extractAll('/tmp/myZip.zip')
File "/var/task/src/shared/utils/zipfilehelper.py", line 24, in extractAll
archive.extractall()
File "/var/lang/lib/python3.6/zipfile.py", line 1491, in extractall
self.extract(zipinfo, path, pwd)
File "/var/lang/lib/python3.6/zipfile.py", line 1479, in extract
return self._extract_member(member, path, pwd)
File "/var/lang/lib/python3.6/zipfile.py", line 1538, in _extract_member
os.mkdir(targetpath)
OSError: [Errno 30] Read-only file system: '/var/task/test-deploy'

Solution:

extractAll() will extract files in the current directory, which is /var/task/test-deploy in your case.

You need to use os.chdir() to change the current directory:

import os, zipfile

os.chdir('/tmp')
with zipfile.ZipFile(source, 'r') as archive:
    archive.extractall()

Even better, you can create a temporary directory and extract the files there, to avoid polluting /tmp:

import os, tempfile, zipfile

with tempfile.TemporaryDirectory() as tmpdir:
    os.chdir(tmpdir)
    with zipfile.ZipFile(source, 'r') as archive:
        archive.extractall()

If you want to restore the current working directory after extracting the file, consider using this decorator:

import os, tempfile, zipfile, contextlib

@contextlib.context_manager
def temporary_work_dir():
    old_work_dir = os.getcwd()
    with tempfile.TemporaryDirectory() as tmp_dir:
        os.chdir(tmp_dir)
        try:
            yield
        finally:
            os.chdir(old_work_dir)

with temporary_work_dir():
    with zipfile.ZipFile(source, 'r') as archive:
        archive.extractall()