Skip to content

Is test split = p1-test + p2-test + p3-test ?? #1

@anirbanl

Description

@anirbanl

Hi,

I went through the code and found that p1-test, p2-test, p3-test splits are created from the test split of ReaSCAN-v1.0/ReaSCAN-compositional and they when added up should give test split (the numbers also hint that 921+2120+2712 = 5753).

However, when I check the dataset splits (my code snippet below executed in root dir ReaSCAN-v1.0 after extracting ReaSCAN-v1.0.zip), I find that that p1-test+p2-test+p3-test NOT EQ test. Maybe I am doing something wrong here and I needed help in figuring out:

import json
import collections
p1=json.load(open('ReaSCAN-compositional-p1-test/data-compositional-splits.txt', 'r'))["examples"]["test"]
p2=json.load(open('ReaSCAN-compositional-p2-test/data-compositional-splits.txt', 'r'))["examples"]["test"]
p3=json.load(open('ReaSCAN-compositional-p3-test/data-compositional-splits.txt', 'r'))["examples"]["test"]
p123=p1+p2+p3
ptest=json.load(open('ReaSCAN-compositional/data-compositional-splits.txt', 'r'))["examples"]["test"]
p123_command=[x['command'] for x in p123]
ptest_command=[x['command'] for x in ptest]
p123_command_sorted = sorted(p123_command)
ptest_command_sorted = sorted(ptest_command)
if collections.Counter(p123_command_sorted) == collections.Counter(ptest_command_sorted):
    print ("The lists are identical")
else :
    print ("The lists are not identical")

I get the output: The lists are not identical

Further checking produces:

>p123_command_sorted[:10]
['pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle,cautiously',
 'pull,the,big,blue,circle,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,big,blue,square,and,inside,of,a,red,box,while,spinning',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,big,yellow,cylinder,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,big,yellow,cylinder,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,small,blue,square']

>ptest_command_sorted[:10]
['pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle',
 'pull,the,big,blue,circle,cautiously',
 'pull,the,big,blue,circle,cautiously',
 'pull,the,big,blue,circle,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,big,yellow,cylinder,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,small,blue,square,hesitantly',
 'pull,the,big,blue,circle,that,is,in,the,same,column,as,a,small,blue,square,hesitantly']

Thanks in advance for your help! :)

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdocumentationImprovements or additions to documentation

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions