Skip to content

[Feature:TAGrading] Add Image Annotation#11921

Open
martig7 wants to merge 49 commits intomainfrom
add-image-annotation
Open

[Feature:TAGrading] Add Image Annotation#11921
martig7 wants to merge 49 commits intomainfrom
add-image-annotation

Conversation

@martig7
Copy link
Copy Markdown
Contributor

@martig7 martig7 commented Jul 28, 2025

Why is this Change Important & Necessary?

PDF Annotation was recently removed, and so as a replacement we agreed to instead support image annotation.

What is the New Behavior?

What steps should a reviewer take to reproduce or test the bug or new feature?

Bulk upload to a gradable on the new branch, and assign the result to a student. Go to their submisson, and open up an image. This page should now show, and editing the image should render the changes. These changes should persist across instances. Downloading should download the image with the annotations built on top.
image

The student can now see their annotated image from the gradable at the bottom of the page, which is the same as with pdf annotation:

image image

Automated Testing & Documentation

This feature will likely need submitty.org documentation.

Other information

A future PR/issue should store this annotated image somewhere without having to build to see the annotated image.

@github-project-automation github-project-automation bot moved this to Seeking Reviewer in Submitty Development Jul 28, 2025
@martig7 martig7 requested a review from lavalleeale July 28, 2025 19:24
@automateprojectmangement automateprojectmangement bot moved this from Seeking Reviewer to In Review in Submitty Development Jul 28, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Jul 28, 2025

Codecov Report

❌ Patch coverage is 0.41322% with 482 lines in your changes missing coverage. Please review.
✅ Project coverage is 21.45%. Comparing base (d5914cd) to head (deac387).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##               main   #11921      +/-   ##
============================================
- Coverage     21.67%   21.45%   -0.23%     
- Complexity     9491     9615     +124     
============================================
  Files           267      269       +2     
  Lines         36273    36995     +722     
  Branches        474      547      +73     
============================================
+ Hits           7863     7937      +74     
- Misses        27940    28515     +575     
- Partials        470      543      +73     
Flag Coverage Δ
autograder 21.31% <ø> (ø)
js 1.83% <0.00%> (-0.24%) ⬇️
migrator 100.00% <ø> (ø)
php 20.66% <1.48%> (-0.02%) ⬇️
python_submitty_utils 80.08% <ø> (ø)
submitty_daemon_jobs 90.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@Chriun Chriun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearing annotations does not clear the rendered view but it is cleared in the view when you click edit annotations. Downloading the image also contains the annotations that were supposed to be cleared.

@martig7 martig7 requested a review from Chriun August 1, 2025 15:35
@martig7 martig7 requested a review from Chriun August 6, 2025 20:18
Copy link
Copy Markdown
Contributor

@Chriun Chriun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from what I tested. Code may need a refactor and additional features (such as viewing annotations from other graders) in the future.

As of now, though, image annotation and student viewing of the graded annotated images works. Others should most likely test as well.

Copy link
Copy Markdown
Contributor

@jeffrey-cordero jeffrey-cordero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the image annotation features look very robust. I could only find an issue with being unable to view/download annotations on the student-view, which I'll look into after a VM reset, but otherwise, the annotation components worked well. Having a native save button within the modal itself, and clicking on submissions_processed automatically opening up the modal would be nice, but these can be future enhancements.

Copy link
Copy Markdown
Contributor

@lavalleeale lavalleeale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added code comments but functionality works well for me

@github-project-automation github-project-automation bot moved this from In Review to Work in Progress in Submitty Development Aug 8, 2025
@williamjallen williamjallen self-requested a review August 8, 2025 18:24
@automateprojectmangement automateprojectmangement bot moved this from Work in Progress to In Review in Submitty Development Aug 8, 2025
Copy link
Copy Markdown
Contributor

@jeffrey-cordero jeffrey-cordero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The work so far to implement image annotation looks solid, laying great groundwork for further optimizations and new features.

Copy link
Copy Markdown
Contributor

@williamschen23 williamschen23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks decent and functionality works as intended


$annotation_path = FileUtils::joinPaths($this->core->getConfig()->getCoursePath(), 'annotations', $gradeable_id, $id, $active_version);
$annotation_jsons = [];
$md5_path = md5($path);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

md5 probably provides enough entropy here given that the input data is not easily controlled by users, but we could consider a better hash function if desired.

}
}
else {
$anon_path = $this->getPath($file_path, $gradeable_id, false);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getPath() doesn't do any form of path sanitization when the third argument is false. I believe that means that this code allows an arbitrary read if a carefully constructed $_POST['file_path'] argument is provided. It would be better to use FileUtils::joinPaths() in getPath() at a minimum, but it would be even better to just get rid of any sort of logic where we pass paths from the client side to the server side.

Even if these security vulnerabilities are fixed now, passing paths around unnecessarily opens the door to developers unintentionally creating new security vulnerabilities in the future.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to understand exactly how to prevent this pass from client side to server side. Should I just be passing a file name to the post request and building the rest of the filepath? Should I not be not passing full filepaths to the client from the viewi at all, and if so does that mean I should also just be just passing a filename there? My understanding of the filepath specifics isn't the greatest because I tried to use the same functions/logic as the previous pdf annotator.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous PDF annotator is unfortunately something we've had security issues with in the past, so it's best to not rely on it too much. In general, I think passing the most minimal part needed to uniquely identify a file is the best way to approach that. Is that just the filename, or a part of the filename?

$display_file_url = $this->core->buildCourseUrl(['display_file']);

$localcss = [];
$localcss[] = $this->core->getOutput()->timestampResource(FileUtils::joinPaths('image', 'image_annotation.css'), 'css');
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use addInternalCss() here instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't figure out how to make addInternalCss work here, is there a performance reason for switching them out?

$localjs = [];

if ($jquery) {
$localjs[] = $this->core->getOutput()->timestampResource(FileUtils::joinPaths('jquery', 'jquery.min.js'), 'vendor');
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use addVendorJs() here.

import * as luxon from 'luxon';
import * as Twig from 'twig';
import * as markerjs3 from '@markerjs/markerjs3';
import * as markerjsUI from '@markerjs/markerjs-ui';
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How big are these libraries?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image Unless I'm checking this wrong they should be 23 and 27 kBs respectively.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these libraries loaded for every page, or only the pages where annotation is needed?

@github-project-automation github-project-automation bot moved this from In Review to Work in Progress in Submitty Development Aug 16, 2025
@github-actions github-actions bot added the Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete label Sep 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete

Projects

Status: Work in Progress

Development

Successfully merging this pull request may close these issues.

6 participants