Import proposals from a spreadsheet by lahdeero · Pull Request #7084 · decidim/decidim

lahdeero · 2021-01-07T09:56:12Z

🎩 What? Why?

Non-digital natives and people who don't want their personal information on the internet, tell their ideas (proposals) more preferably to a trusted person. Therefore, admin users want to bulk import paper collected ideas to Decidim. This PR has implementation for importing proposals from files (csv, xls and json). Implementation is similar to export feature and should be easily expandable to other components as well.

📌 Related Issues

https://meta.decidim.org/processes/roadmap/f/122/proposals/14939

Testing

Create a csv or excel spreadsheet with column names title/en and body/en (or another locale which you prefer eg. title/ca and body/ca). Also scope/id and category/id columns are supported. For each row enter valid title and body, then save. After saving the spreadsheet go to -> Admin dashboard -> Processes -> pick process -> Proposals -> Import -> Import from a file -> Browse -> -> Start import -> check out new proposals

📋 Checklist

🚨 Please review the guidelines for contributing to this repository.

📷 Screenshots

Please add screenshots of the changes you're proposing

♥️ Thank you!

carolromero · 2021-01-12T11:47:19Z

Hi @lahdeero! Is there any staging where we can test this? Thanks!

lahdeero · 2021-01-14T16:06:17Z

Hello @carolromero! Unfortunately not currently, but we have plans to make one.

oriolgual · 2021-01-15T10:53:28Z

Hello @carolromero! Unfortunately not currently, but we have plans to make one.

I'll create one

oriolgual · 2021-01-15T10:54:20Z

decidim-admin/app/commands/decidim/admin/create_import.rb

+
+      def call
+        return broadcast(:invalid) if form.invalid?
+        return broadcast(:invalid) unless form.creator


Could this be moved to a form validation?

Yep. I removed creator check from the command.

mrcasals · 2021-01-15T11:56:16Z

@decidim/product this can be tested in https://decidim-staging-pr-264.herokuapp.com/

carolromero · 2021-01-18T10:14:06Z

Hi @lahdeero the import works very well, good job!
A couple of comments/doubts:

I see that there is a drop-down list but with only one value -proposal creator- what is the reason? Are other "creators" expected?

We should add some guidance on the expected file format. Also for consistency with other existing import options in the admin panel, can we change the button name only to "Import"?

Thanks!

lahdeero · 2021-01-19T14:52:43Z

* I see that there is a drop-down list but with only one value -`proposal creator`- what is the reason? Are other "creators" expected?

Yes, other creators are expected. For proposals component probably not (maybe for comments), but for budgets for example, we could have separate creators for budgets and projects.

* We should add some guidance on the expected file format. Also for consistency with other existing import options in the admin panel, can we change the button name only to "Import"?

Yep. Added guidance and updated button label.

Thanks for the feedback!

mrcasals · 2021-01-19T19:40:44Z

@lahdeero some tests seem to be failing after the latest changes, can you check them please? 😄

lahdeero · 2021-01-21T07:12:01Z

@mrcasals Done 😅

mrcasals · 2021-01-21T10:25:29Z

@lahdeero can you redeploy this so @decidim/product can re-review it? 😄

lahdeero · 2021-01-21T14:21:18Z

@mrcasals I haven't created the instance. @oriolgual Can you redeploy please?

mrcasals · 2021-01-21T14:47:58Z

Whoops, sorry about that @lahdeero.

@decidim/product I've redeployed the app with the latest changes! Give it 15min from now and you should be able to review them 😄

carolromero · 2021-01-21T16:01:38Z

@mrcasals can't access to the admin panel of the review app. It throws a system error 🤔

mrcasals · 2021-01-22T07:49:30Z

@carolromero solved now, sorry about that!

virgile-dev · 2021-01-22T08:45:33Z

Hey did a little testing on my end on the link you shared.

When clicking on "edit" for existing processes / assemblies I get a server error
I created a new one and when clicking on "import from file" I get a server error too (I tested here https://decidim-staging-pr-264.herokuapp.com/admin/assemblies/ooo/components/93/manage/ )

carolromero · 2021-01-22T08:49:25Z

@mrcasals I'm getting the server error as well

mrcasals · 2021-01-22T14:19:37Z

@virgile-dev @carolromero sorry, there was an issue with the review app. Should be fixed now!

carolromero · 2021-01-26T09:18:13Z

@mrcasals I'm still getting the server error when accessing the import from file option 😢

mrcasals · 2021-01-26T09:45:56Z

@lahdeero the error we're getting is this:

ActionView::Template::Error (The asset "decidim/admin/import_guidance.js" is not present in the asset pipeline.

I think you need to either require the file in decidim-admin/app/assets/config/decidim_admin_manifest.js or add it in decidim-admin/lib/decidim/admin/engine.rb:32 😄

mrcasals · 2021-01-26T09:46:17Z

Also, some conflicts appeared @lahdeero!

lahdeero · 2021-01-27T12:04:48Z

I rebased and added import_guidance.js.es6 into decidim-admin/app/assets/config/decidim_admin_manifest.js. @mrcasals

mrcasals · 2021-01-27T13:34:52Z

@carolromero can you re-check the review app, please? https://decidim-staging-pr-264.herokuapp.com/

carolromero · 2021-01-28T13:29:04Z

@mrcasals now it's working great, thanks!

Yep. Added guidance and updated button label.

@lahdeero thanks for adding the help on the file format!

Yes, other creators are expected. For proposals component probably not (maybe for comments), but for budgets for example, we could have separate creators for budgets and projects.

Regarding the creator field, we have discussed it with the rest of @decidim/product and we prefer to remove it for now. If in the future there is a need to add it because there are another roles we can discuss it.

lahdeero · 2021-01-28T14:33:25Z

Regarding the creator field, we have discussed it with the rest of @decidim/product and we prefer to remove it for now. If in the future there is a need to add it because there are another roles we can discuss it.

I made it so that if there is less than 2 creators, it doesn't show creator select. Is this ok? @carolromero

carolromero · 2021-01-28T16:15:36Z

@lahdeero sorry to bother you with this but I just don't understand the logic 😢 .
Let's recap:

There is a Creator field with the value "proposal creator". But this only determines the role of who imports (creates) the proposals, right?
And then there is a field Create imports as (maybe we could better call it "Create proposals as") where a drop-down list appears with the user and the groups to which he/she belongs.

So, for better understanding, could you give me an example of another possible value that could appear in the Creator field (bearing in mind that this option is for the proposal component only)? Thanks and sorry!

lahdeero · 2021-01-29T09:08:40Z

Not bothering at all 😊 Clearly there is room for making this more understandable for users.

Little bit technical background: I have tried to making this similar to export feature that we can use in all components. In every resource needs to be created differently (purpose of creator class), but the file upload view for example would look like same in every component. Now that I think more about this, maybe this is what it should look like:

* There is a `Creator` field with the value "proposal creator". But this only determines the role of who imports (creates) the proposals, right?

It determines (under the hood) which class is responsible for the creating actual resource from parsed data.

* And then there is a field `Create imports as` (maybe we could better call it "Create proposals as") where a drop-down list appears with the user and the groups to which he/she belongs.

We could call it by resource, but if there is more resources (creators before) in same component we would have to change text (dynamically) when we select another resource and that might be confusing for the user as well.

So, for better understanding, could you give me an example of another possible value that could appear in the Creator field (bearing in mind that this option is for the proposal component only)? Thanks and sorry!

Another value could be Comment creator, if we want to import comments (as we can already export them). That creator we could use with other commentable resources too.

Bottom line is that before importing we need to have following information:

Resource
User or group
File

Hopefully this clears this at least a bit. @carolromero

carolromero · 2021-02-01T12:45:41Z

Hi @lahdeero, thank you very much for the explanation ❤️!

It determines (under the hood) which class is responsible for the creating actual resource from parsed data.

Understood!

We could call it by resource, but if there is more resources (creators before) in same component we would have to change text (dynamically) when we select another resource and that might be confusing for the user as well.

yes, better to leave a generic text then and not the name of the specific resource. Although I am anticipating that the help text in the upload file will have to be dynamic. But we can discuss that when that option is available.

Ok, so to move forward with this PR, let's do what you proposed, if there is only one creator it is not shown to the user. It's already working like that, isn't it?

lahdeero · 2021-02-02T11:50:56Z

yes, better to leave a generic text then and not the name of the specific resource. Although I am anticipating that the help text in the upload file will have to be dynamic. But we can discuss that when that option is available.

Yes, help text is dynamic already (not that it really matters yet with just one creator/resource).

Ok, so to move forward with this PR, let's do what you proposed, if there is only one creator it is not shown to the user. It's already working like that, isn't it?

Yes, selection is hidden if component has just one creator.

Thanks!

oriolgual · 2021-02-02T14:37:11Z

decidim-admin/app/commands/decidim/admin/create_import.rb

+      def call
+        return broadcast(:invalid) if form.invalid?
+
+        form.context[:user_group] = user_group


I think this should be inside the form, with a def user_group rather than modifying the form from the command

Yep. Moved user_group assign to the form.

oriolgual · 2021-02-02T14:37:56Z

decidim-admin/app/commands/decidim/admin/create_import.rb

+        importer = importer_for(form.file_path, form.mime_type)
+        imported_data = importer.prepare
+
+        return broadcast(:invalid_lines, importer.invalid_lines) unless importer.invalid_lines.empty?
+


Maybe consider moving this logic also to the form and adding a validation there?

Yep. Form validates invalid_lines now.

oriolgual · 2021-02-02T14:38:35Z

decidim-admin/app/controllers/decidim/admin/imports_controller.rb

+      include Decidim::ComponentPathHelper
+      def new


Suggested change

include Decidim::ComponentPathHelper

def new

include Decidim::ComponentPathHelper

def new

oriolgual · 2021-02-02T14:39:18Z

decidim-admin/app/controllers/decidim/admin/imports_controller.rb

+      include Decidim::ComponentPathHelper
+      def new
+        enforce_permission_to :import, :component_data, component: current_component
+        @import = form(Admin::ImportForm).from_params(


I'd rename this to @form so it's easier to udnerstand

Suggested change

@import = form(Admin::ImportForm).from_params(

@form = form(Admin::ImportForm).from_params(

Yep. Renamed import -> form

oriolgual · 2021-02-02T14:39:26Z

decidim-admin/app/controllers/decidim/admin/imports_controller.rb

+      def create
+        enforce_permission_to :import, :component_data, component: current_component
+
+        @import = form(Admin::ImportForm).from_params(


oriolgual · 2021-02-02T14:41:32Z

decidim-dev/lib/decidim/dev/assets/.~lock.import_proposals.xls#

@@ -0,0 +1 @@
+,elahdenpera,THINK7,07.01.2021 11:38,file:///home/elahdenpera/.config/libreoffice/4;


Should this file be here?

mrcasals

Looks good to me, thanks @lahdeero!

Feedback addressed

lahdeero changed the title ~~Feature/import proposals spreadsheet~~ Feature/import proposals from spreadsheet Jan 7, 2021

lahdeero changed the title ~~Feature/import proposals from spreadsheet~~ Feature/import proposals from a spreadsheet Jan 7, 2021

andreslucena changed the title ~~Feature/import proposals from a spreadsheet~~ Import proposals from a spreadsheet Jan 11, 2021

oriolgual reviewed Jan 15, 2021

View reviewed changes

github-actions bot mentioned this pull request Jan 15, 2021

Review app for feature/import_proposals_spreadsheet by mrcasals codegram/decidim-staging#264

Closed

lahdeero added 6 commits January 27, 2021 10:41

Base architecture and dropdown in ui

2d8f042

Trying to create proper routes

a514ca2

Trying to create createimport call

901e942

Trying to parse/read from spreadsheet

b92ac5c

Can import proposals from csv

65c7c65

Component can have many importers

a1c3e25

lahdeero added 3 commits January 27, 2021 10:41

Render new instead of redirect if form error

02a262f

Add import helps to ignore_unused

bc67886

Add import_guidance.js to admin_manifest.js

4ae54e2

lahdeero force-pushed the feature/import_proposals_spreadsheet branch from e349dca to 4ae54e2 Compare January 27, 2021 08:43

Hide creator select if less than 2 creators

dc56083

oriolgual previously requested changes Feb 2, 2021

View reviewed changes

lahdeero added 4 commits February 3, 2021 12:54

Move validations from command to form

5fa70d2

Add support for conferences

7cb58a3

Restart tests

30e71ae

Remove invalid lines broadcast from import controller

df12379

mrcasals approved these changes Feb 10, 2021

View reviewed changes

mrcasals added module: admin module: proposals type: feature labels Feb 10, 2021

mrcasals merged commit 6c5841d into decidim:develop Feb 10, 2021

ahukkanen deleted the feature/import_proposals_spreadsheet branch February 12, 2021 11:39

martanducas mentioned this pull request Mar 1, 2021

Import proposals from CSV PEMB2030/decidim-pemb#5

Closed

oliverbarnes mentioned this pull request Mar 16, 2021

Bump the module and the demo to Decidim 0.23 liquidvotingio/decidim-module-liquidvoting#33

Closed

ahukkanen mentioned this pull request Sep 3, 2021

Private space user CSV import fails when using an incorrect column separator #8298

Closed

lahdeero mentioned this pull request Mar 11, 2022

Create admin log records when proposals are imported from a file #9006

Merged

12 tasks

ahukkanen mentioned this pull request Jul 29, 2022

Standardize CSV import formats and fix private users CSV import with invalid file #9627

Merged

	@import = form(Admin::ImportForm).from_params(
	@form = form(Admin::ImportForm).from_params(

		@@ -0,0 +1 @@
		,elahdenpera,THINK7,07.01.2021 11:38,file:///home/elahdenpera/.config/libreoffice/4; No newline at end of file

Uh oh!

Conversation

lahdeero commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎩 What? Why?

📌 Related Issues

Testing

📋 Checklist

📷 Screenshots

Uh oh!

carolromero commented Jan 12, 2021

Uh oh!

lahdeero commented Jan 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oriolgual commented Jan 15, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrcasals commented Jan 15, 2021

Uh oh!

carolromero commented Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lahdeero commented Jan 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrcasals commented Jan 19, 2021

Uh oh!

lahdeero commented Jan 21, 2021

Uh oh!

mrcasals commented Jan 21, 2021

Uh oh!

lahdeero commented Jan 21, 2021

Uh oh!

mrcasals commented Jan 21, 2021

Uh oh!

carolromero commented Jan 21, 2021

Uh oh!

mrcasals commented Jan 22, 2021

Uh oh!

virgile-dev commented Jan 22, 2021

Uh oh!

carolromero commented Jan 22, 2021

Uh oh!

mrcasals commented Jan 22, 2021

Uh oh!

carolromero commented Jan 26, 2021

Uh oh!

mrcasals commented Jan 26, 2021

Uh oh!

mrcasals commented Jan 26, 2021

Uh oh!

lahdeero commented Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrcasals commented Jan 27, 2021

Uh oh!

carolromero commented Jan 28, 2021

Uh oh!

lahdeero commented Jan 28, 2021

Uh oh!

carolromero commented Jan 28, 2021

Uh oh!

lahdeero commented Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carolromero commented Feb 1, 2021

Uh oh!

lahdeero commented Feb 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lahdeero Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

lahdeero commented Jan 7, 2021 •

edited

Loading

lahdeero commented Jan 14, 2021 •

edited

Loading

carolromero commented Jan 18, 2021 •

edited

Loading

lahdeero commented Jan 19, 2021 •

edited

Loading

lahdeero commented Jan 27, 2021 •

edited

Loading

lahdeero commented Jan 29, 2021 •

edited

Loading

lahdeero Feb 3, 2021 •

edited

Loading