607 questions
1
vote
0
answers
52
views
Unable to upload file using spark Java
I'm using Spark java with below dependency, and I'm trying to make one API endpoint to receive an file using multipart file, Below is the code attached, I've tried multiple ways to do that, but ...
1
vote
0
answers
40
views
Spark - ReadStream Kafka-Events and delete entries in my S3 bucket based on attributes in Kafka-Event
I Want to listen to a Kafka-Topic. Every event on that topic represents that an entry of my S3 Bucket can be deleted.
I've got the following Code to read the Events from my Topic:
private static ...
- 35
2
votes
0
answers
109
views
java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.spark.unsafe.types.VariantVal
I evaluate Spark 4 try_variant_get method handling variant type data. First I make sql statements examples.
CREATE TABLE family (
id INT,
data VARIANT
);
INSERT INTO family (id, data)
VALUES
(1, ...
- 1,433
0
votes
1
answer
202
views
Apache Spark log4j2.properties file does not generate log files
I try to generate user-defined log file on spark 4.0.
OS : Windows 11
Spark : spark-4.0.0-preview2-bin-hadoop3
First I make log4j2.properties file on %SPARK_HOME%\conf folder.
rootLogger.level = info
...
- 1,433
0
votes
0
answers
408
views
java.lang.NoSuchFieldError: LZ4 : spark.sql()
I am trying to run one of my methods which has -
spark.sql("DROP TABLE IF EXISTS " + dbNameString + "." + tableNameString)
When I am running the method, the code breaks on the ...
- 1
-2
votes
1
answer
58
views
How run SparkJava from Java
I am trying to bundle a spark java app in maven, any idea?
I've used the maven-compiler plugin but no luck, the dependencies are not loaded.
<plugin>
<groupId>...
- 476
0
votes
1
answer
89
views
how to handle exceptions in SparkJava
I'm trying to handle all exception (exception handler) for SparkJava
I am currently using try/catch blocks in each endpoint but that approach doesn't scale properly.
Something that can handle ...
0
votes
1
answer
627
views
Grouping and sorting on spark structured streaming
I have a usecase where I have streaming dataset like mobile number ,starttime and call duration.
I need to do group by on mobile number and sort the group based on starttime and filter out the calls ...
- 15
0
votes
1
answer
118
views
Error when working with geoscan on databricks
I used geoscan libraries on Azure Databricks. However, when I configured for my task, I got this error.
`TypeError: 'JavaPackage' object is not callable
------------------------------------------------...
1
vote
1
answer
216
views
Configuring Apache Spark's MemoryStream to simulate Kafka stream
I was requested to look into using Apache Spark's MemoryStream to simulate a Kafka stream in a Java Spring Boot service. The documentation / online community is a bit small on this topic so I am ...
- 464
0
votes
1
answer
109
views
Is @Bind needed?
I discovered by accident that in a spring boot project I didn't have to bind argument in a query like the one below.
@SqlQuery("""
select id, name
from organisations
...
- 4,383
0
votes
1
answer
422
views
OkHttp new session
I have an http server, and it uses sessions, but every time I send a request from okhttp the server creates a new session.
The server uses jetty with sparkjava. In a browser it works normally, and the ...
- 39
-1
votes
1
answer
128
views
Spark Java sum is giving incorrect value
Spark Java sum is giving incorrect value
Java sample code is as below
List<Double> points = Arrays.asList(-6221.4, 6380.46);
Dataset<Row> dt = spark.createDataset(points, Encoders.DOUBLE()...
- 9
1
vote
1
answer
511
views
How to set schema into spark.sql.function.from_csv?
I use spark-3.4.1-hadoop3 on windows 11. And I try to generate the schema to pass into from_csv function parameter.
Belows are my codes.
import org.apache.spark.sql.Column;
import org.apache.spark.sql....
- 1,433
0
votes
1
answer
191
views
repartition not working with xml file in Spark
I have dataframe which I want to save as multiple xml files. This is my code:
employees
.repartition(col("first_name"))
.write()
.option(&...
- 3,769