Hi,
I have been trying to connect to locally deployed arcade db with pyspark using postgres JDBC driver and faced an interesting problem. While executing simple queries (e.g. SELECT id FROM table on a non-empty table with valid column id), I would get empty results. However, when checking the count of the dataframe loaded by pyspark, it returns a correct number of rows in the table. Showing the dataframe content displays N empty rows.
I have researched this for a while, and figured out that spark queries the database by adding WHERE 1=0 to the query to retrieve data schema. On regular postgres database, this does not return any rows, but the schema is returned. For example:
In arcade db, nothing is returned:
Hi,
I have been trying to connect to locally deployed arcade db with pyspark using postgres JDBC driver and faced an interesting problem. While executing simple queries (e.g.
SELECT id FROM tableon a non-emptytablewith valid columnid), I would get empty results. However, when checking the count of the dataframe loaded by pyspark, it returns a correct number of rows in the table. Showing the dataframe content displays N empty rows.I have researched this for a while, and figured out that spark queries the database by adding
WHERE 1=0to the query to retrieve data schema. On regular postgres database, this does not return any rows, but the schema is returned. For example:In arcade db, nothing is returned: