Skip to content

Implement returning for postgresql#39968

Closed
00dav00 wants to merge 11 commits intorails:mainfrom
00dav00:postgres_returning_values
Closed

Implement returning for postgresql#39968
00dav00 wants to merge 11 commits intorails:mainfrom
00dav00:postgres_returning_values

Conversation

@00dav00
Copy link

@00dav00 00dav00 commented Aug 1, 2020

Summary

This is an attempt to implement #34237

It adds a class method that will receive the attributes that should be returned on record creation or update because they are calculated inside postgresql.

Steps to reproduce

1. Create a model

class Order < ApplicationRecord
end

2. Create a migration with a DB function as default.

class CreateOrders < ActiveRecord::Migration[5.2]
  def change
    create_table :orders do |t|
      t.string :uuid, default: -> { "gen_random_uuid()" }
      t.references :user
      t.references :product
      t.text :comment
 
      t.timestamps
    end
  end
end

rails db:migrate

3. Create a new order

rails console

order = Order.create(user_id: 1, product_id: 1)

Desired behavior

I would like create to set uuid (by the postgres default) and for it to set the new uuid value into the order AR instance, as it does with the id.

> order
# =>
# +----+--------------------------------------+---------+------------+---------+----------------+----------------+
# | id | uuid                                 | user_id | product_id | comment | created_at     | updated_at     |
# +----+--------------------------------------+---------+------------+---------+----------------+----------------+
# | 1  | bec94fe0-f7c3-4ede-8b25-c4bce702ec00 | 1       | 1          |         | 2018-10-17 ... | 2018-10-17 ... |
# +----+--------------------------------------+---------+------------+---------+----------------+----------------+

Actual behavior

The uuid does get set in the db, but it is not returned into the order instance.

@natematykiewicz
Copy link
Contributor

natematykiewicz commented Aug 15, 2020

My company's Rails app (using Postgres) has quite a few columns that are managed by database triggers. A RETURNING statement to populate them on insert/update sounds amazing! Right now, we have to reload the record after save, to get the new values. It's tedious and inefficient.

@davegson
Copy link

just a gentle fyi - the failing checks should probably be tackled

@natematykiewicz
Copy link
Contributor

I just "requested changes" to make Rubocop happy.

@00dav00
Copy link
Author

00dav00 commented Aug 21, 2020

Will take a look tomorrow @natematykiewicz :)

@gmaliar
Copy link

gmaliar commented Oct 2, 2020

@00dav00 this is amazing! My company would appreciate this PR a lot, is there anyway I can help? Is it okay if I try to fix the breaking tests?

@00dav00
Copy link
Author

00dav00 commented Oct 3, 2020

Hi @gmaliar , the biggest problem this PR has is that the feature is not officially approved, that's why I haven't continue working on it. Here is the rails forum issue, if you know someone that can get admins attention on this it would be greatly appreciated.

@jrochkind
Copy link
Contributor

@matthewd suggested in #34237 that they thought some functionality in this area did make sense as part of AR core, perhaps they could provide some feedback or advice on this PR to try to move it along.

Specifically they said:

To my eye, the ideal behaviour would be @tbuehlmann's explicit self.returning_attributes = (for manual additions reflecting things like triggers), plus automatic detection of unparseable default expressions [to always be returned on insert] and computed columns [to always be returned on insert or update].

@jorge-precich
Copy link

Loving the idea and the PR. My company is also having issues with gen_random_uuid() from PostgreSQL. You should really follow @jrochkind suggestion. I'd like to help with any you may need 👍

@00dav00
Copy link
Author

00dav00 commented Nov 13, 2020

Hi @jorge-precich , I do started this PR based on that issue that @jrochkind mentioned and would definitely love any feedback or advice from @matthewd . I guess that maybe if the rails forrum issue gets more interaction the rails core team may be open to consider the addition.

@00dav00 00dav00 force-pushed the postgres_returning_values branch from 543627f to 9afee65 Compare December 20, 2020 14:46
Base automatically changed from master to main January 14, 2021 17:01
@rails-bot
Copy link

rails-bot bot commented Apr 17, 2021

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Thank you for your contributions.

@rails-bot rails-bot bot added the stale label Apr 17, 2021
@natematykiewicz
Copy link
Contributor

I still would really like this to get merged

@rails-bot rails-bot bot removed the stale label Apr 17, 2021
@jrochkind
Copy link
Contributor

I am also very interested in this functionality. We have a PR. Is there anything we can do to help? Is there any way to get Rails committer attention?

@00dav00
Copy link
Author

00dav00 commented Apr 17, 2021

@natematykiewicz @jrochkind @deepj I will rebase against the last changes in master to prevent it from being closed. I'm not sure what we can do get the attention of a committer. Do you have any suggestions?

@zzak
Copy link
Member

zzak commented Jun 2, 2021

Trying to take a quick stab at summarizing this PR:

  • 👀 Need a review on this implementation (similar to insert_all)
  • 📈 Current implementation could maybe be optimized further
  • It seems that test coverage could improve

That said, I think the first point is by far the most important.

We can also add more tests and refactor after that.

@00dav00
Copy link
Author

00dav00 commented Jun 2, 2021

Hi @zzak thanks so much for checking this!
Is there something I have to/can do about the Need a review on this implementation part or is this an rails team task? Please let me know :)

@zzak
Copy link
Member

zzak commented Jun 3, 2021

@00dav00 For review, it helps to have someone with additional context and in this particular case my best guess is @matthewd -- so we may just have to wait for them on this.

@00dav00
Copy link
Author

00dav00 commented Jun 3, 2021

Understood @zzak , thanks for clearing this up. I guess nothing to do but wait for this to reach @matthewd 's radar.

@natematykiewicz
Copy link
Contributor

Kasper is doing some PR review this week. I suggested reviewing this PR.

He has some feedback:
https://twitter.com/natematykiewicz/status/1421342172796882944

@jrochkind
Copy link
Contributor

Thanks @natematykiewicz . The feedback "try closing it and then reopening a new PR doing it in a different way" -- without having actually reviewed the code here, and thus with absolutely no tips on what sort of different way might be good -- seems to me profoundly discouraging and unhelpful. I don't know why anyone would put their time into another PR based on that, I sure wouldn't.

My conclusion is basically just that nobody on Rails core team is interested in this, and there is no way to get a PR reviewed in that case, so it's unlikely to be a good use of time to try an alternate PR.

I wonder if there's any way to get this feature with a relatively light monkey-patch that seems possibly likely to survive future releases? That would seem to be the only hope.

@natematykiewicz
Copy link
Contributor

Is there a way to support more than just Postgres in this? A PR that's just Postgres-specific might not gain as much traction as something that's more widely supported.

@00dav00
Copy link
Author

00dav00 commented Aug 3, 2021

@natematykiewicz Thanks for checking with Kasper, I think I will just let this be closed by GH bots cause it looks like there is no way of getting this reviewed.

@jrochkind I'm not sure about monkey patching for this case may because it may be to fragile given that the changes sit at the core of active record.

@natematykiewicz
Copy link
Contributor

natematykiewicz commented Aug 3, 2021

Monkey patching it is a really bad idea. The Composite Primary Keys gem does a similar monkey patch to return your composite primary key columns back, and the gem is very unstable between patch versions of ActiveRecord. Having something like this in Rails would be a huge help to a lot of things. I see 2 things that I'm guessing would help this PR along:

  1. It's currently Postgres only
  2. It uses the word returning, which is a Postgres term (I think)

I guess a big question is, does MySQL have a similar feature available that this could also support? Basecamp uses MySQL, so making this be a feature for that would probably help this gain traction.

@jrochkind
Copy link
Contributor

Googling, it's unfortuantely looking to me like MySQL offers no equivalent. :(

@deepj
Copy link
Contributor

deepj commented Aug 21, 2021

This feature is supported by PostgreSQL and SQLite (more limited than in PostgreSQL). MySQL doesn't support it yet.

@jrochkind
Copy link
Contributor

jrochkind commented Sep 27, 2021

Here's a PR that just get merged for something supported by Postgres but not MySQL. It may provide a model of how to structure code in Rails for features only supported by some adapters and not others (as well as evidence that such features can be merged sometimes!).

#41487

However, without interest from Rails committers, it may be that this PR isn't going to go anywhere no matter how much time is put into it. :(

@jrochkind
Copy link
Contributor

jrochkind commented Oct 18, 2021

Here's another PR for returning on SQLite. #42955

Maybe if they can be reconciled to use similar code, and the attention of any rails committers can be obtained (that's the hard part), there'd be a chance of merge?

And here's yet another merged PR for an unrelated feature that supports only postgres and SQLite, demonstrating that Rails is in some cases willing to merge AR features only supported on postgres and sqlite, not mysql. #40491

@davegson
Copy link

Just to chime in on how to get attention: I had a read of the CONTRIBUTING guide. As this PR is clearly labeled as a feature request this section applies:

Do you intend to add a new feature or change an existing one?

  • Suggest your change in the rubyonrails-core mailing list and start writing code.

  • Do not open an issue on GitHub until you have collected positive feedback about the change. GitHub issues are primarily intended for bug reports and fixes.

So I guess it is best to drop by in their forums and reference this. I do not have the time to do so myself but am grateful if anybody else would take a chance. In my opinion, the discussions on here and the 17 👍 as of today should surely count as plenty of positive feedback as required by the guide.

@00dav00
Copy link
Author

00dav00 commented Oct 19, 2021

@davegson this is the rails forum issue

@davegson
Copy link

my bad - bummer to see it overlooked there too...

@OuYangJinTing
Copy link
Contributor

OuYangJinTing commented Feb 23, 2022

Here's another PR for returning on SQLite. #42955

Maybe if they can be reconciled to use similar code, and the attention of any rails committers can be obtained (that's the hard part), there'd be a chance of merge?

And here's yet another merged PR for an unrelated feature that supports only postgres and SQLite, demonstrating that Rails is in some cases willing to merge AR features only supported on postgres and sqlite, not mysql. #40491

Actually. MariaDB support returning clause too. MariaDB and MySql use same adapter. So I think returing function should be more widely used, not just limited to postgresql. I have also proposed a feature in rails forum: https://discuss.rubyonrails.org/t/feature-proposal-activerecord-update-all-delete-all-support-use-sqls-returning-clause-statements/78729

@00dav00
Copy link
Author

00dav00 commented Mar 15, 2022

Hi @OuYangJinTing , I think this has more to do with getting the right person to review and agree on the changes than been able to make the feature broader, as @zzak commented before.

@fatkodima
Copy link
Member

I think, this PR should be also extended to automatically support virtual columns, because currently there is a need to do a reload to get the value.

# id and return that value.
#
# If "returning" param is provided in Postgresql, the method will return
# a hash with the values including the id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing the return type is, imho, problematic

@winnieoursbrun
Copy link

Would be great too for updates, specially with increment_counter method, with that we can have atomic counter for unique name generation based on a count, for example.

@JuPlutonic
Copy link

@winnieoursbrun FYI I used this psql code in Rails production for the same reason:

  # foobars_id is a field used in unique name generation, e.g. with concatinated version of foobars
  def generate_number
    self.version_number = 1
    query = "select setval('foobars_id_seq'::regclass, nextval('foobars_id_seq'::regclass), false)"
    doc_num = ActiveRecord::Base.connection.select_values(query)[0]
    self.document.id = doc_num
    self.number = "#{doc_num.to_s.rjust(5, '0')}-#{self.version_number}"
end

@quentindemetz
Copy link

My company is using https://github.com/pennylane-hq/active_record-returning_attributes/tree/qdm/rails7 for the moment

@jrochkind
Copy link
Contributor

jrochkind commented Jun 5, 2023

It looks like this has finally been handled, separately by separate development, here? #48241

this one can probably be closed, with that one (hooray!) merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.